Re: Parallel Build with GNU make
Sorry for following up on my own post,but there has been a "bug" in the last sentence... See for correction below Am 02.02.2016 um 08:35 schrieb Roland Schwingel: Hi... Eli Zaretskii wrote on 01.02.2016 20:14:17: > Add "--debug=j" to the make command-line switches and see if it > launches more than one command at a time. > > Anyway, your makefile.mak file can be written in such a way as to > effectively prohibit parallelism, due to how it lays out dependencies. This is something I am struggeling on. I tried this a couple of times but could not make it working in a way that is satisfying for me, as I could not find the correct way to control parallelism. I am aware of .NOTPARALLEL but this might not be enough for me. I have this examplarily layout: subfolder1 - subfolder4 - sub4_1.c sub1_1.c sub4_2.c sub1_2.c ... subfolder2 - sub2_1.c sub2_2.c subfolder3 - sub3_1.c sub3_2.c main1.c main2.c main3.c All subfolders shall be visited recursively first. No parallelism is allowed in visiting folders. But when in a folder and it comes to concrete compiling parallelism should be allowed. This should result in this compileorder for the above example Visit subfolder1 (not in parallel) Visit subfolder4 (not in parallel) Compile sub4_1.c and sub4_2.c in parallel Return to subfolder1 Compile sub1_2.c and sub1_2.c in parallel Return to main Visit subfolder2 (not in parallel) Compile sub2_1.c and sub2_2.c in parallel Return to main Vist subfolder3 (not in parallel) Compile sub3_1.c and sub3_2.c in parallel Return to main Compile main1.c,main2.c and main3.c in parallel When doing a make -j 4 for instance make is visiting all subfolders in parallel which is not desired. When I am using .NOTPARALLEL the whole subfolder is not compiled in parallel which is also not desired. Each folder visit is a concrete submake in my case. Each folder has its own makefile Any clue on how to achieve the flow I want to have? Visiting folders non recursive while compiling itself inside of the folders is recursive? Should read: Visiting folders non PARALLEL while compiling itself inside of the folders is IN PARALLEL. Sorry and thanks again for your help, Roland ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Parallel Build with GNU make
Hi... Eli Zaretskii wrote on 01.02.2016 20:14:17: > Add "--debug=j" to the make command-line switches and see if it > launches more than one command at a time. > > Anyway, your makefile.mak file can be written in such a way as to > effectively prohibit parallelism, due to how it lays out dependencies. This is something I am struggeling on. I tried this a couple of times but could not make it working in a way that is satisfying for me, as I could not find the correct way to control parallelism. I am aware of .NOTPARALLEL but this might not be enough for me. I have this examplarily layout: subfolder1 - subfolder4 - sub4_1.c sub1_1.c sub4_2.c sub1_2.c ... subfolder2 - sub2_1.c sub2_2.c subfolder3 - sub3_1.c sub3_2.c main1.c main2.c main3.c All subfolders shall be visited recursively first. No parallelism is allowed in visiting folders. But when in a folder and it comes to concrete compiling parallelism should be allowed. This should result in this compileorder for the above example Visit subfolder1 (not in parallel) Visit subfolder4 (not in parallel) Compile sub4_1.c and sub4_2.c in parallel Return to subfolder1 Compile sub1_2.c and sub1_2.c in parallel Return to main Visit subfolder2 (not in parallel) Compile sub2_1.c and sub2_2.c in parallel Return to main Vist subfolder3 (not in parallel) Compile sub3_1.c and sub3_2.c in parallel Return to main Compile main1.c,main2.c and main3.c in parallel When doing a make -j 4 for instance make is visiting all subfolders in parallel which is not desired. When I am using .NOTPARALLEL the whole subfolder is not compiled in parallel which is also not desired. Each folder visit is a concrete submake in my case. Each folder has its own makefile Any clue on how to achieve the flow I want to have? Visiting folders non recursive while compiling itself inside of the folders is recursive? Thanks, Roland ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH1/2] Use spawn() on Cygwin
Hi... bug-make-bounces+roland.schwingel=onevision@gnu.org wrote on 30.07.2013 18:37:35: > On Tue, 2013-07-30 at 18:39 +0300, Eli Zaretskii wrote: > > In general, I feel it's wrong to do this: Cygwin is a Posix platform, > > so it should be using the Posix code, to be as compatible with other > > Posix platforms as possible. EMX is not a Posix platform, so using > > its code will likely make the Cygwin Make deviate from Posix behavior > > at times. > > If we decide to take this change I think we should reduce it to a single > #define, such as HAVE_W32_SPAWN or similar, as we did with the > HAVE_DOS_PATHS, set it in one place based on __EMX__ etc., then use that > single macro where we want to check for spawn() in the code. This sounds IMHO reasonable. Roland ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH1/2] Use spawn() on Cygwin
Hi Eli... Eli Zaretskii wrote on 30.07.2013 18:29:53: > From: Eli Zaretskii > To: Roland Schwingel > Cc: bug-make@gnu.org, p.fe...@samsung.com > Date: 30.07.2013 18:32 > Subject: Re: [PATCH1/2] Use spawn() on Cygwin > > > Cc: bug-make@gnu.org, Pavel Fedin > > From: Roland Schwingel > > Date: Tue, 30 Jul 2013 18:12:55 +0200 > > > > I am using Pavels patch for some months now in my private version of > > gnumake on cygwin heavily and I could not find any regression with it up > > to now and gnumake is in my use cases clearly faster. > > Thanks for the info. > > What version of Make are you using? If it is not the version from the > git repo, please try that, and in particular the new --output-sync > feature. My version is from git, but already some months old (I checked it out on 20130114). So it is not fresh bananas. Currently I am enjailed by a very tight schedule. Maybe I can update my version of gnumake beginning of next week. Sorry, Roland ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH 2/2] Do not use DOS paths on Cygwin
Hi... bug-make-bounces+roland.schwingel=onevision@gnu.org wrote on 30.07.2013 17:43:10: > > Currently make's configure suggests that it should use DOS-style paths on > > Cygwin. This is not true, and this assumption makes path-related mechanisms > > to work incorrectly. Currently Cygwin package supplies manual hint in > > config.cache in order to work around this. > > Sorry, I don't understand the problem you are trying to solve. Could > you please explain it in more detail? > > This feature was added in response to an explicit request of some > Cygwin users. I don't want to remove it without a very good reason. > After all, if you don't need DOS-style file names, you can simply > refrain from using them in your Makefiles, and then you will not see > any effect of this support. > > > I think we should also ask MinGW team about __MSYS__ definition. MSys is > > close to Cygwin, so perhaps this should also be removed. > > MSYS supports DOS-style file names, so removing that would seem wrong. > However, I don't speak for MSYS developers, so I'll defer to their > judgment. I want to throw in my $0.02 here, too. But this time in the other direction. Sorry Pavel. I clearly think the DOS paths mode should remain in even for cygwin. I know that there are objections in cygwins top level management against this. Cygwin wants to have DOS path mode out as - as far as I understand it. Therefore I am maintaining my private version of gnumake for many years now with enabled DOS path mode (and some little other things). Also I could not see any problems building packages on cygwin for cywin with my version of gnumake, but I have to confess that is not my regular use case. I believe I am using a DOS paths enabled gnumake on cygwin now for over 10 years. Having enabled DOS pathes in gnumake makes life with non cygwin tools (eg. javac and other things) in makefiles a lot easier. Again this is my personal opinion. For MSYS I believe it could not be ripped out. It is IMHO vital there and would break it. Roland ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH1/2] Use spawn() on Cygwin
Hi... bug-make-bounces+roland.schwingel=onevision@gnu.org wrote on 30.07.2013 17:39:10: > > From: Pavel Fedin > > Date: Tue, 30 Jul 2013 14:42:23 +0400 > > > > Please take this patch, Cygwin team told that they would like to integrate > > with upstream. I have already posted it some time ago but got no reply. > > The patch significantly improves performance of Make under Cygwin. > > Thanks. > > Is there any discussion we could read about that with the details of > the problem and how/why does the proposed patch solves it? > > In general, I feel it's wrong to do this: Cygwin is a Posix platform, > so it should be using the Posix code, to be as compatible with other > Posix platforms as possible. EMX is not a Posix platform, so using > its code will likely make the Cygwin Make deviate from Posix behavior > at times. I am using Pavels patch for some months now in my private version of gnumake on cygwin heavily and I could not find any regression with it up to now and gnumake is in my use cases clearly faster. It is still not a stallion badged red italian car, but it is cruising its curves noticably snappier on cygwin now. Using EMX is a shortcut to achieve the usage of spawn() over fork(). It might not be the cleanest choice but it works. Maybe Pavel finds the time to make the patch distinct to cygwin itself. Just my $0.02, Roland___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
