Re: [Libreoffice] [Proof-of-concept] Drastically speedup the offapi (and oovbaapi) build
Hi Norbert, On 2011-09-01 at 23:43 -0500, Norbert Thiebaud wrote: Ah right, I just copied the way the response file was created in the old implementation; Tempfile.mk sounds better. wouldn't it be nicer and more efficient to adapt iclc to have a mode where stdin is the list of file to compile(1) ? the ouput follow usually a pattern based on the input file that way you avoid the whole tempfile dance... Oh indeed, and that would allow to do all of them at once; though again, I wonder how would it compare to the per-directory approach with -j 1... Hard to fine-tune it :-) Thank you, Kendy ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] [Proof-of-concept] Drastically speedup the offapi (and oovbaapi) build
Hi all, On 2 September 2011 08:31, Jan Holesovsky ke...@suse.cz wrote: Hi Norbert, On 2011-09-01 at 23:43 -0500, Norbert Thiebaud wrote: Ah right, I just copied the way the response file was created in the old implementation; Tempfile.mk sounds better. wouldn't it be nicer and more efficient to adapt iclc to have a mode where stdin is the list of file to compile(1) ? the ouput follow usually a pattern based on the input file that way you avoid the whole tempfile dance... Oh indeed, and that would allow to do all of them at once; though again, I wonder how would it compare to the per-directory approach with -j 1... Hard to fine-tune it :-) If it's possible to process more at once it would be better I think. The thing is that the directories are too small to make that really faster. Anyway, I'm going to push this. May be the gbuild part could be written nicer but I have done that this way, feel free to change anybody. Matúš ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] [Proof-of-concept] Drastically speedup the offapi (and oovbaapi) build
Hi Kendy, all, On Thu, 01 Sep 2011 10:25:20 +0200 Jan Holesovsky ke...@suse.cz wrote: With the attached patch, the offapi/UnoApi_offapi.mk (and oovbaapi) has to change to something like the following; this is of course only part of that, the other rules have to be adapted the same way, but that should be ~trivial. Second part of the problem is that idlc have to be patched to bail out with != 0 exit value, according to the comments it is not the case right now. Just a word of warning after a short skimming of the patch: It seems like you are using $? with huge amounts of files to create a commandline. This is quite risky on Linux, where you easily hit the command length limits of the shell on certain installs (depending on ulimits). There is solenv/gbuild/Tempfile.mk to work around that (to create response files). Tempfile.mk btw is the easiest way to trigger that good old make 3.81 bug. Best, Bjoern -- https://launchpad.net/~bjoern-michaelsen ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] [Proof-of-concept] Drastically speedup the offapi (and oovbaapi) build
Hi Bjoern, On 2011-09-01 at 11:21 +0200, Bjoern Michaelsen wrote: Just a word of warning after a short skimming of the patch: It seems like you are using $? with huge amounts of files to create a commandline. This is quite risky on Linux, where you easily hit the command length limits of the shell on certain installs (depending on ulimits). There is solenv/gbuild/Tempfile.mk to work around that (to create response files). Tempfile.mk btw is the easiest way to trigger that good old make 3.81 bug. Ah right, I just copied the way the response file was created in the old implementation; Tempfile.mk sounds better. Regards, Kendy ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] [Proof-of-concept] Drastically speedup the offapi (and oovbaapi) build
Hi Kendy, Bjoern On 1 September 2011 10:25, Jan Holesovsky ke...@suse.cz wrote: Hi Matúš, Bjoern, I've found a way how to speedup the build of offapi (which is a major problem on windows - on my box it takes about 10% of the entire build time). The trick is simple - instead of generating the files one by one, several .idl's are treated during one idlc call; something that has been there before gbuildization of the module. Of course, it also handles the case when everything is built, and just 1 .idl changes - in that case, only that one is rebuilt. That's nice. Unfortunately I am heading for vacation, and cannot finish it before I go - do you think anybody of you could have a look, please? So, I had a look, it works that way but is not so much faster. With the attached patch, the offapi/UnoApi_offapi.mk (and oovbaapi) has to change to something like the following; this is of course only part of that, the other rules have to be adapted the same way, but that should be ~trivial. Second part of the problem is that idlc have to be patched to bail out with != 0 exit value, according to the comments it is not the case right now. I was not working on the second part yet and I do not intend to do so. Could someone else do it? I have no idea what idlc is anyway. Enjoy :-) Well, it took a while, or more. Not so much fun but not really problem. Bjoern, could you please approve patches? And how to push them ? First the Kendy's one and then change it? I have attached just one patch for better reviewing but created two. (for UnoApiTarget.mk) Now results.. in offapi: before: make -srj 4 = 75s make -sr = 206s after: make -srj 4 = 71s make -sr = 183s It's little more interesting for oovbaapi: (make -sr) 49s - 34s But that does not take much. May be for Windows it will be bigger difference but also this is something. Or is it possible to process even more files at once? Also from different directories? And there were side-effects. I have no idea why but I had to add include directives for a few idl files (also attached). Thank you, You are welcome, thanks for bringing this up. Matúš diff --git a/solenv/gbuild/UnoApiTarget.mk b/solenv/gbuild/UnoApiTarget.mk index 935c5ea..061c2b7 100644 --- a/solenv/gbuild/UnoApiTarget.mk +++ b/solenv/gbuild/UnoApiTarget.mk @@ -60,46 +60,55 @@ gb_UnoApiTarget_IDLFILES_$(1) := endef define gb_UnoApiTarget_add_idlfiles -$(foreach idl,$(2),$(call gb_UnoApiTarget_add_idlfile,$(1),$(idl))) +$(foreach idl,$(3),$(call gb_UnoApiTarget_add_idlfile,$(1),$(2),$(idl))) + +$(call gb_UnoApiPartTarget_get_target,$(2)/idl.done) : $(foreach repo,$(gb_REPOS),$(foreach idl,$(3),$(realpath $(repo)/$(2)/$(idl).idl))) + $(gb_UnoApiPartTarget__command) endef define gb_UnoApiTarget_add_idlfile -$(call gb_UnoApiTarget_get_target,$(1)) : $(call gb_UnoApiPartTarget_get_target,$(2).urd) -$(call gb_UnoApiPartTarget_get_target,$(2).urd) : $(foreach repo,$(gb_REPOS),$(realpath $(repo)/$(2).idl)) -gb_UnoApiTarget_HPPFILES_$(1) += $(2).hdl -gb_UnoApiTarget_HPPFILES_$(1) += $(2).hpp -gb_UnoApiTarget_IDLFILES_$(1) += $(2).idl +$(call gb_UnoApiTarget_get_target,$(1)) : $(call gb_UnoApiPartTarget_get_target,$(2)/$(3).urd) +$(call gb_UnoApiPartTarget_get_target,$(2)/$(3).urd) : $(call gb_UnoApiPartTarget_get_target,$(2)/idl.done) +gb_UnoApiTarget_HPPFILES_$(1) += $(2)/$(3).hdl +gb_UnoApiTarget_HPPFILES_$(1) += $(2)/$(3).hpp +gb_UnoApiTarget_IDLFILES_$(1) += $(2)/$(3).idl -$(call gb_UnoApiTarget_get_header_target,)$(2).hpp :| $(call gb_UnoApiTarget_get_target,$(1)) -$(call gb_UnoApiTarget_get_header_target,)$(2).hdl :| $(call gb_UnoApiTarget_get_target,$(1)) +$(call gb_UnoApiTarget_get_header_target,)$(2)/$(3).hpp :| $(call gb_UnoApiTarget_get_target,$(1)) +$(call gb_UnoApiTarget_get_header_target,)$(2)/$(3).hdl :| $(call gb_UnoApiTarget_get_target,$(1)) endef define gb_UnoApiTarget_add_idlfiles_noheader -$(foreach idl,$(2),$(call gb_UnoApiTarget_add_idlfile_noheader,$(1),$(idl))) +$(foreach idl,$(3),$(call gb_UnoApiTarget_add_idlfile_noheader,$(1),$(2),$(idl))) + +$(call gb_UnoApiPartTarget_get_target,$(2)/idl_noheader.done) : $(foreach repo,$(gb_REPOS),$(foreach idl,$(3),$(realpath $(repo)/$(2)/$(idl).idl))) + $(gb_UnoApiPartTarget__command) endef define gb_UnoApiTarget_add_idlfile_noheader -$(call gb_UnoApiTarget_get_target,$(1)) : $(call gb_UnoApiPartTarget_get_target,$(2).urd) -$(call gb_UnoApiPartTarget_get_target,$(2).urd) : $(foreach repo,$(gb_REPOS),$(realpath $(repo)/$(2).idl)) -gb_UnoApiTarget_IDLFILES_$(1) += $(2).idl +$(call gb_UnoApiTarget_get_target,$(1)) : $(call gb_UnoApiPartTarget_get_target,$(2)/$(3).urd) +$(call gb_UnoApiPartTarget_get_target,$(2)/$(3).urd) : $(call gb_UnoApiPartTarget_get_target,$(2)/idl_noheader.done) +gb_UnoApiTarget_IDLFILES_$(1) += $(2)/$(3).idl endef define gb_UnoApiTarget_add_idlfiles_nohdl -$(foreach idl,$(2),$(call gb_UnoApiTarget_add_idlfile_nohdl,$(1),$(idl))) +$(foreach idl,$(3),$(call
Re: [Libreoffice] [Proof-of-concept] Drastically speedup the offapi (and oovbaapi) build
On Thu, Sep 1, 2011 at 4:35 AM, Jan Holesovsky ke...@suse.cz wrote: Hi Bjoern, On 2011-09-01 at 11:21 +0200, Bjoern Michaelsen wrote: Just a word of warning after a short skimming of the patch: It seems like you are using $? with huge amounts of files to create a commandline. This is quite risky on Linux, where you easily hit the command length limits of the shell on certain installs (depending on ulimits). There is solenv/gbuild/Tempfile.mk to work around that (to create response files). Tempfile.mk btw is the easiest way to trigger that good old make 3.81 bug. Ah right, I just copied the way the response file was created in the old implementation; Tempfile.mk sounds better. wouldn't it be nicer and more efficient to adapt iclc to have a mode where stdin is the list of file to compile(1) ? the ouput follow usually a pattern based on the input file that way you avoid the whole tempfile dance... Norbert (1) IIRC idlc has a stdin mode, but it is to compile 1 file from stdin ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
