reverse dependency order?
I've run into situations where given: foo: a b c and "b" was missing a dependency on "a". The above did not fail with parallel make for years because "a" finished fast, before "b" actually needed to use it's result. It might be interesting to have a make flag that would reverse the order in which dependencies are considered, this will catch really fast missing dependencies even when building with "make -j1". Is something like that feasible? Would it be easy to implement? Thanks! --Dan ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Parallel Build with GNU make
Martin Dorey wrote: -Original Message- From: bug-make-bounces+martin.dorey=hds@gnu.org [mailto:bug-make-bounces+martin.dorey=hds@gnu.org] On Behalf Of Roland Schwingel Sent: Monday, February 01, 2016 23:40 To: Eli Zaretskii; bug-make@gnu.org Subject: Re: Parallel Build with GNU make Sorry for following up on my own post,but there has been a "bug" in the last sentence... See for correction below Am 02.02.2016 um 08:35 schrieb Roland Schwingel: Hi... Eli Zaretskii wrote on 01.02.2016 20:14:17: > Add "--debug=j" to the make command-line switches and see if it > launches more than one command at a time. > > Anyway, your makefile.mak file can be written in such a way as to > effectively prohibit parallelism, due to how it lays out dependencies. This is something I am struggeling on. I tried this a couple of times but could not make it working in a way that is satisfying for me, as I could not find the correct way to control parallelism. I am aware of .NOTPARALLEL but this might not be enough for me. I have this examplarily layout: subfolder1 - subfolder4 - sub4_1.c sub1_1.c sub4_2.c sub1_2.c ... subfolder2 - sub2_1.c sub2_2.c subfolder3 - sub3_1.c sub3_2.c main1.c main2.c main3.c All subfolders shall be visited recursively first. No parallelism is allowed in visiting folders. But when in a folder and it comes to concrete compiling parallelism should be allowed. This should result in this compileorder for the above example Visit subfolder1 (not in parallel) Visit subfolder4 (not in parallel) Compile sub4_1.c and sub4_2.c in parallel Return to subfolder1 Compile sub1_2.c and sub1_2.c in parallel Return to main Visit subfolder2 (not in parallel) Compile sub2_1.c and sub2_2.c in parallel Return to main Vist subfolder3 (not in parallel) Compile sub3_1.c and sub3_2.c in parallel Return to main Compile main1.c,main2.c and main3.c in parallel When doing a make -j 4 for instance make is visiting all subfolders in parallel which is not desired. When I am using .NOTPARALLEL the whole subfolder is not compiled in parallel which is also not desired. Each folder visit is a concrete submake in my case. Each folder has its own makefile Any clue on how to achieve the flow I want to have? Visiting folders non recursive while compiling itself inside of the folders is recursive? Should read: Visiting folders non PARALLEL while compiling itself inside of the folders is IN PARALLEL. Use a for-loop for anything that must be explicitly serial. Generally, to me this is a sign of a broken Makefiles - when you define your dependencies correctly, everything should naturally run in the correct order, regardless of degree of parallelism. The fact that you want subfolder 2 to be built after subfolder 1 means sub1 *is* a dependency of sub2 and if you haven't stated that in your Makefile then you've made an error in your Makefile. Read this: http://highlandsun.com/hyc/#Make in particular this http://highlandsun.com/hyc/GNUYou.htm -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
RE: Parallel Build with GNU make
I'm not sure that I'll be adding anything that isn't a statement of the obvious but I can explain how we solve this in our environment. We have the subfolders specified in a variable, SUBDIRS. Another variable, BUILD_RECURSION_PREREQUISITES is synthesized from SUBDIRS to contain the like of recurse.%.subfolder1 recurse.%.subfolder2 etc. BUILD_TARGET_STEMS contains phony target names that a user might pass to make, like "release" and "debug". These stems eventually form the % in those prerequisites, with something like: $(BUILD_TARGET_STEMS): %: $(BUILD_RECURSION_PREREQUISITES) Then I serialize the recursion, where appropriate, with this gruesome fragment: JOIN_ADJACENT_PAIRS = $(filter-out @@:% %:@@,$(join $(addsuffix :,$(1) @@),@@ $(1))) CONSTRUCT_DEPENDENCY_SERIALIZATION_RULES = $(foreach STEM,$(1),$(subst %,$(STEM),$(call JOIN_ADJACENT_PAIRS,$(2 SERIALIZE_DEPENDENCIES = $(foreach RULE,$(call CONSTRUCT_DEPENDENCY_SERIALIZATION_RULES,$(1),$(2)),$(eval $(RULE))) $(call SERIALIZE_DEPENDENCIES,$(BUILD_TARGET_STEMS),$(BUILD_RECURSION_PREREQUISITES)) -Original Message- From: bug-make-bounces+martin.dorey=hds@gnu.org [mailto:bug-make-bounces+martin.dorey=hds@gnu.org] On Behalf Of Roland Schwingel Sent: Monday, February 01, 2016 23:40 To: Eli Zaretskii; bug-make@gnu.org Subject: Re: Parallel Build with GNU make Sorry for following up on my own post,but there has been a "bug" in the last sentence... See for correction below Am 02.02.2016 um 08:35 schrieb Roland Schwingel: > Hi... > > Eli Zaretskii wrote on 01.02.2016 20:14:17: > > > Add "--debug=j" to the make command-line switches and see if it > > launches more than one command at a time. > > > > Anyway, your makefile.mak file can be written in such a way as to > > effectively prohibit parallelism, due to how it lays out dependencies. > This is something I am struggeling on. I tried this a couple of times > but could not make it working in a way that is satisfying for me, as I > could not find the correct way to control parallelism. I am aware of > .NOTPARALLEL but this might not be enough for me. > > I have this examplarily layout: > > subfolder1 - subfolder4 - sub4_1.c > sub1_1.c sub4_2.c > sub1_2.c > ... > subfolder2 - sub2_1.c > sub2_2.c > subfolder3 - sub3_1.c > sub3_2.c > main1.c > main2.c > main3.c > > All subfolders shall be visited recursively first. No parallelism is > allowed in visiting folders. But when in a folder and it comes to > concrete compiling parallelism should be allowed. > > This should result in this compileorder for the above example > Visit subfolder1 (not in parallel) > Visit subfolder4 (not in parallel) > Compile sub4_1.c and sub4_2.c in parallel > Return to subfolder1 > Compile sub1_2.c and sub1_2.c in parallel > Return to main > Visit subfolder2 (not in parallel) > Compile sub2_1.c and sub2_2.c in parallel > Return to main > Vist subfolder3 (not in parallel) > Compile sub3_1.c and sub3_2.c in parallel > Return to main > Compile main1.c,main2.c and main3.c in parallel > > When doing a make -j 4 for instance make is visiting all subfolders in > parallel which is not desired. When I am using .NOTPARALLEL the whole > subfolder is not compiled in parallel which is also not desired. Each > folder visit is a concrete submake in my case. Each folder has its own > makefile > > Any clue on how to achieve the flow I want to have? Visiting folders non > recursive while compiling itself inside of the folders is recursive? Should read: Visiting folders non PARALLEL while compiling itself inside of the folders is IN PARALLEL. Sorry and thanks again for your help, Roland ___ Bug-make mailing list Bug-make@gnu.org https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_mailman_listinfo_bug-2Dmake&d=CwICAg&c=DZ-EF4pZfxGSU6MfABwx0g&r=oBMzc8Omr1YTgjig4n4076T3IKL7TuNH9HpVbojD-ms&m=MfFKkv-mCbvJA9yNJWbqRoqitBf5hjsRRy1xBUq8peA&s=Rf15u8uDSCGNGENESJhWzqBOa086GoGTrfTFimBzepg&e= ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Fine-tuning for parallel building with GNU make
> Visiting folders non PARALLEL while compiling itself inside of the folders is > IN PARALLEL. How do you think about to traverse the affected directory hierarchy by a dedicated make recipe? * Have you got any expectations about the amount of parallel work which should be triggered by scripts in subdirectories? * Did you look at any possibilities around a variable like "MAKEFLAGS"? https://www.gnu.org/software/make/manual/html_node/Options_002fRecursion.html Regards, Markus ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Parallel Build with GNU make
Sorry for following up on my own post,but there has been a "bug" in the last sentence... See for correction below Am 02.02.2016 um 08:35 schrieb Roland Schwingel: Hi... Eli Zaretskii wrote on 01.02.2016 20:14:17: > Add "--debug=j" to the make command-line switches and see if it > launches more than one command at a time. > > Anyway, your makefile.mak file can be written in such a way as to > effectively prohibit parallelism, due to how it lays out dependencies. This is something I am struggeling on. I tried this a couple of times but could not make it working in a way that is satisfying for me, as I could not find the correct way to control parallelism. I am aware of .NOTPARALLEL but this might not be enough for me. I have this examplarily layout: subfolder1 - subfolder4 - sub4_1.c sub1_1.c sub4_2.c sub1_2.c ... subfolder2 - sub2_1.c sub2_2.c subfolder3 - sub3_1.c sub3_2.c main1.c main2.c main3.c All subfolders shall be visited recursively first. No parallelism is allowed in visiting folders. But when in a folder and it comes to concrete compiling parallelism should be allowed. This should result in this compileorder for the above example Visit subfolder1 (not in parallel) Visit subfolder4 (not in parallel) Compile sub4_1.c and sub4_2.c in parallel Return to subfolder1 Compile sub1_2.c and sub1_2.c in parallel Return to main Visit subfolder2 (not in parallel) Compile sub2_1.c and sub2_2.c in parallel Return to main Vist subfolder3 (not in parallel) Compile sub3_1.c and sub3_2.c in parallel Return to main Compile main1.c,main2.c and main3.c in parallel When doing a make -j 4 for instance make is visiting all subfolders in parallel which is not desired. When I am using .NOTPARALLEL the whole subfolder is not compiled in parallel which is also not desired. Each folder visit is a concrete submake in my case. Each folder has its own makefile Any clue on how to achieve the flow I want to have? Visiting folders non recursive while compiling itself inside of the folders is recursive? Should read: Visiting folders non PARALLEL while compiling itself inside of the folders is IN PARALLEL. Sorry and thanks again for your help, Roland ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
