Re: Parallel Build with GNU make
Martin Dorey wrote: -Original Message- From: bug-make-bounces+martin.dorey=hds@gnu.org [mailto:bug-make-bounces+martin.dorey=hds@gnu.org] On Behalf Of Roland Schwingel Sent: Monday, February 01, 2016 23:40 To: Eli Zaretskii; bug-make@gnu.org Subject: Re: Parallel Build with GNU make Sorry for following up on my own post,but there has been a "bug" in the last sentence... See for correction below Am 02.02.2016 um 08:35 schrieb Roland Schwingel: Hi... Eli Zaretskii wrote on 01.02.2016 20:14:17: > Add "--debug=j" to the make command-line switches and see if it > launches more than one command at a time. > > Anyway, your makefile.mak file can be written in such a way as to > effectively prohibit parallelism, due to how it lays out dependencies. This is something I am struggeling on. I tried this a couple of times but could not make it working in a way that is satisfying for me, as I could not find the correct way to control parallelism. I am aware of .NOTPARALLEL but this might not be enough for me. I have this examplarily layout: subfolder1 - subfolder4 - sub4_1.c sub1_1.c sub4_2.c sub1_2.c ... subfolder2 - sub2_1.c sub2_2.c subfolder3 - sub3_1.c sub3_2.c main1.c main2.c main3.c All subfolders shall be visited recursively first. No parallelism is allowed in visiting folders. But when in a folder and it comes to concrete compiling parallelism should be allowed. This should result in this compileorder for the above example Visit subfolder1 (not in parallel) Visit subfolder4 (not in parallel) Compile sub4_1.c and sub4_2.c in parallel Return to subfolder1 Compile sub1_2.c and sub1_2.c in parallel Return to main Visit subfolder2 (not in parallel) Compile sub2_1.c and sub2_2.c in parallel Return to main Vist subfolder3 (not in parallel) Compile sub3_1.c and sub3_2.c in parallel Return to main Compile main1.c,main2.c and main3.c in parallel When doing a make -j 4 for instance make is visiting all subfolders in parallel which is not desired. When I am using .NOTPARALLEL the whole subfolder is not compiled in parallel which is also not desired. Each folder visit is a concrete submake in my case. Each folder has its own makefile Any clue on how to achieve the flow I want to have? Visiting folders non recursive while compiling itself inside of the folders is recursive? Should read: Visiting folders non PARALLEL while compiling itself inside of the folders is IN PARALLEL. Use a for-loop for anything that must be explicitly serial. Generally, to me this is a sign of a broken Makefiles - when you define your dependencies correctly, everything should naturally run in the correct order, regardless of degree of parallelism. The fact that you want subfolder 2 to be built after subfolder 1 means sub1 *is* a dependency of sub2 and if you haven't stated that in your Makefile then you've made an error in your Makefile. Read this: http://highlandsun.com/hyc/#Make in particular this http://highlandsun.com/hyc/GNUYou.htm -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
RE: Parallel Build with GNU make
I'm not sure that I'll be adding anything that isn't a statement of the obvious but I can explain how we solve this in our environment. We have the subfolders specified in a variable, SUBDIRS. Another variable, BUILD_RECURSION_PREREQUISITES is synthesized from SUBDIRS to contain the like of recurse.%.subfolder1 recurse.%.subfolder2 etc. BUILD_TARGET_STEMS contains phony target names that a user might pass to make, like "release" and "debug". These stems eventually form the % in those prerequisites, with something like: $(BUILD_TARGET_STEMS): %: $(BUILD_RECURSION_PREREQUISITES) Then I serialize the recursion, where appropriate, with this gruesome fragment: JOIN_ADJACENT_PAIRS = $(filter-out @@:% %:@@,$(join $(addsuffix :,$(1) @@),@@ $(1))) CONSTRUCT_DEPENDENCY_SERIALIZATION_RULES = $(foreach STEM,$(1),$(subst %,$(STEM),$(call JOIN_ADJACENT_PAIRS,$(2 SERIALIZE_DEPENDENCIES = $(foreach RULE,$(call CONSTRUCT_DEPENDENCY_SERIALIZATION_RULES,$(1),$(2)),$(eval $(RULE))) $(call SERIALIZE_DEPENDENCIES,$(BUILD_TARGET_STEMS),$(BUILD_RECURSION_PREREQUISITES)) -Original Message- From: bug-make-bounces+martin.dorey=hds@gnu.org [mailto:bug-make-bounces+martin.dorey=hds@gnu.org] On Behalf Of Roland Schwingel Sent: Monday, February 01, 2016 23:40 To: Eli Zaretskii; bug-make@gnu.org Subject: Re: Parallel Build with GNU make Sorry for following up on my own post,but there has been a "bug" in the last sentence... See for correction below Am 02.02.2016 um 08:35 schrieb Roland Schwingel: > Hi... > > Eli Zaretskii wrote on 01.02.2016 20:14:17: > > > Add "--debug=j" to the make command-line switches and see if it > > launches more than one command at a time. > > > > Anyway, your makefile.mak file can be written in such a way as to > > effectively prohibit parallelism, due to how it lays out dependencies. > This is something I am struggeling on. I tried this a couple of times > but could not make it working in a way that is satisfying for me, as I > could not find the correct way to control parallelism. I am aware of > .NOTPARALLEL but this might not be enough for me. > > I have this examplarily layout: > > subfolder1 - subfolder4 - sub4_1.c > sub1_1.c sub4_2.c > sub1_2.c > ... > subfolder2 - sub2_1.c > sub2_2.c > subfolder3 - sub3_1.c > sub3_2.c > main1.c > main2.c > main3.c > > All subfolders shall be visited recursively first. No parallelism is > allowed in visiting folders. But when in a folder and it comes to > concrete compiling parallelism should be allowed. > > This should result in this compileorder for the above example > Visit subfolder1 (not in parallel) > Visit subfolder4 (not in parallel) > Compile sub4_1.c and sub4_2.c in parallel > Return to subfolder1 > Compile sub1_2.c and sub1_2.c in parallel > Return to main > Visit subfolder2 (not in parallel) > Compile sub2_1.c and sub2_2.c in parallel > Return to main > Vist subfolder3 (not in parallel) > Compile sub3_1.c and sub3_2.c in parallel > Return to main > Compile main1.c,main2.c and main3.c in parallel > > When doing a make -j 4 for instance make is visiting all subfolders in > parallel which is not desired. When I am using .NOTPARALLEL the whole > subfolder is not compiled in parallel which is also not desired. Each > folder visit is a concrete submake in my case. Each folder has its own > makefile > > Any clue on how to achieve the flow I want to have? Visiting folders non > recursive while compiling itself inside of the folders is recursive? Should read: Visiting folders non PARALLEL while compiling itself inside of the folders is IN PARALLEL. Sorry and thanks again for your help, Roland ___ Bug-make mailing list Bug-make@gnu.org https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_mailman_listinfo_bug-2Dmake&d=CwICAg&c=DZ-EF4pZfxGSU6MfABwx0g&r=oBMzc8Omr1YTgjig4n4076T3IKL7TuNH9HpVbojD-ms&m=MfFKkv-mCbvJA9yNJWbqRoqitBf5hjsRRy1xBUq8peA&s=Rf15u8uDSCGNGENESJhWzqBOa086GoGTrfTFimBzepg&e= ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Parallel Build with GNU make
Sorry for following up on my own post,but there has been a "bug" in the last sentence... See for correction below Am 02.02.2016 um 08:35 schrieb Roland Schwingel: Hi... Eli Zaretskii wrote on 01.02.2016 20:14:17: > Add "--debug=j" to the make command-line switches and see if it > launches more than one command at a time. > > Anyway, your makefile.mak file can be written in such a way as to > effectively prohibit parallelism, due to how it lays out dependencies. This is something I am struggeling on. I tried this a couple of times but could not make it working in a way that is satisfying for me, as I could not find the correct way to control parallelism. I am aware of .NOTPARALLEL but this might not be enough for me. I have this examplarily layout: subfolder1 - subfolder4 - sub4_1.c sub1_1.c sub4_2.c sub1_2.c ... subfolder2 - sub2_1.c sub2_2.c subfolder3 - sub3_1.c sub3_2.c main1.c main2.c main3.c All subfolders shall be visited recursively first. No parallelism is allowed in visiting folders. But when in a folder and it comes to concrete compiling parallelism should be allowed. This should result in this compileorder for the above example Visit subfolder1 (not in parallel) Visit subfolder4 (not in parallel) Compile sub4_1.c and sub4_2.c in parallel Return to subfolder1 Compile sub1_2.c and sub1_2.c in parallel Return to main Visit subfolder2 (not in parallel) Compile sub2_1.c and sub2_2.c in parallel Return to main Vist subfolder3 (not in parallel) Compile sub3_1.c and sub3_2.c in parallel Return to main Compile main1.c,main2.c and main3.c in parallel When doing a make -j 4 for instance make is visiting all subfolders in parallel which is not desired. When I am using .NOTPARALLEL the whole subfolder is not compiled in parallel which is also not desired. Each folder visit is a concrete submake in my case. Each folder has its own makefile Any clue on how to achieve the flow I want to have? Visiting folders non recursive while compiling itself inside of the folders is recursive? Should read: Visiting folders non PARALLEL while compiling itself inside of the folders is IN PARALLEL. Sorry and thanks again for your help, Roland ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Parallel Build with GNU make
Hi... Eli Zaretskii wrote on 01.02.2016 20:14:17: > Add "--debug=j" to the make command-line switches and see if it > launches more than one command at a time. > > Anyway, your makefile.mak file can be written in such a way as to > effectively prohibit parallelism, due to how it lays out dependencies. This is something I am struggeling on. I tried this a couple of times but could not make it working in a way that is satisfying for me, as I could not find the correct way to control parallelism. I am aware of .NOTPARALLEL but this might not be enough for me. I have this examplarily layout: subfolder1 - subfolder4 - sub4_1.c sub1_1.c sub4_2.c sub1_2.c ... subfolder2 - sub2_1.c sub2_2.c subfolder3 - sub3_1.c sub3_2.c main1.c main2.c main3.c All subfolders shall be visited recursively first. No parallelism is allowed in visiting folders. But when in a folder and it comes to concrete compiling parallelism should be allowed. This should result in this compileorder for the above example Visit subfolder1 (not in parallel) Visit subfolder4 (not in parallel) Compile sub4_1.c and sub4_2.c in parallel Return to subfolder1 Compile sub1_2.c and sub1_2.c in parallel Return to main Visit subfolder2 (not in parallel) Compile sub2_1.c and sub2_2.c in parallel Return to main Vist subfolder3 (not in parallel) Compile sub3_1.c and sub3_2.c in parallel Return to main Compile main1.c,main2.c and main3.c in parallel When doing a make -j 4 for instance make is visiting all subfolders in parallel which is not desired. When I am using .NOTPARALLEL the whole subfolder is not compiled in parallel which is also not desired. Each folder visit is a concrete submake in my case. Each folder has its own makefile Any clue on how to achieve the flow I want to have? Visiting folders non recursive while compiling itself inside of the folders is recursive? Thanks, Roland ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Parallel Build with GNU make
> From: "Nagarjuna Badigunchala (RBEI/EAP4)" > > Date: Mon, 1 Feb 2016 11:18:38 + > > Can somebody tell me how to use the parallelism feature of GNU make? > I tried to use the following command for parallel build but it is not working. > Make –j 7 –f makefile.mak build > When I execute the above command, I can see that the GNU make is utilizing > 40%-60% of CPU. When I > execute the same command using clearmake(clearmake –j 7 makefile.mak build) > process, I can see that > it is always utilizing 80-100% of CPU. Which informs that the GNU make > process is not using the parallelism. > Note: Versions of GNU make tested (3.81 and 4.1) Add "--debug=j" to the make command-line switches and see if it launches more than one command at a time. Anyway, your makefile.mak file can be written in such a way as to effectively prohibit parallelism, due to how it lays out dependencies. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Parallel Build with GNU make
Hello, Can somebody tell me how to use the parallelism feature of GNU make? I tried to use the following command for parallel build but it is not working. Make -j 7 -f makefile.mak build When I execute the above command, I can see that the GNU make is utilizing 40%-60% of CPU. When I execute the same command using clearmake(clearmake -j 7 makefile.mak build) process, I can see that it is always utilizing 80-100% of CPU. Which informs that the GNU make process is not using the parallelism. Note: Versions of GNU make tested (3.81 and 4.1) Thanks in advance:) Mit freundlichen Grüßen / Best regards Nagarjuna Badigunchala RBEI/EAP Tel. +91(80)6738-1518 ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
