Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-16 Thread Peter Eisentraut
On mån, 2010-11-15 at 23:34 -0500, Tom Lane wrote: It's clear to me that we are very far from having a handle on what it'll really take to run parallel builds safely, and I am therefore now of the opinion that we ought to revert the patch. Hypothetical gains in parallelism are useless if we

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-16 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: On mån, 2010-11-15 at 23:34 -0500, Tom Lane wrote: It's clear to me that we are very far from having a handle on what it'll really take to run parallel builds safely, and I am therefore now of the opinion that we ought to revert the patch. We don't

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-15 Thread Bernd Helmle
--On 14. November 2010 11:08:13 -0500 Robert Haas robertmh...@gmail.com wrote: +1. The current master branch fails to build on my (rather new) Mac with make -j2. I could upgrade my toolchain but it seems like more trouble than it's worth, not to mention a possible obstacle to new users

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-15 Thread Peter Eisentraut
On mån, 2010-11-15 at 11:13 +0100, Bernd Helmle wrote: --On 14. November 2010 11:08:13 -0500 Robert Haas robertmh...@gmail.com wrote: +1. The current master branch fails to build on my (rather new) Mac with make -j2. I could upgrade my toolchain but it seems like more trouble than

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-15 Thread Robert Haas
On Mon, Nov 15, 2010 at 4:10 PM, Peter Eisentraut pete...@gmx.net wrote: ld: file not found: ../../../../../../src/backend/postgres collect2: ld returned 1 exit status make[3]: *** [ascii_and_mic.so] Error 1 make[2]: *** [all-ascii_and_mic-recurse] Error 2 make[1]: ***

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-15 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: Very odd, but this completely blew up the first time I tried it. In file included from path.c:34: pg_config_paths.h:2:11: error: missing terminating character FWIW, I didn't replicate that, but I did get this during one attempt with -j4:

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-15 Thread Tom Lane
I tried another experiment, which was make -j100 all on my relatively new Linux box (2 dual-core CPUs). It blew up real good, as per attached stderr output, which shows evidence of more missing dependencies as well as some additional cases of concurrent attempts to build the same target. It's

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-14 Thread Dave Page
On Sat, Nov 13, 2010 at 8:13 PM, Peter Eisentraut pete...@gmx.net wrote: On lör, 2010-11-13 at 20:07 +0200, Peter Eisentraut wrote: On lör, 2010-11-13 at 12:20 -0500, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: On lör, 2010-11-13 at 11:12 -0500, Tom Lane wrote: It looks like

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-14 Thread Tom Lane
Dave Page dp...@pgadmin.org writes: On Sat, Nov 13, 2010 at 8:13 PM, Peter Eisentraut pete...@gmx.net wrote: Well, it looks like $(eval) is pretty broken in 3.80, so either we require 3.81 or we abandon this line of thought. 3.81 might be a problem for Solaris - unless I pay for a support

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-14 Thread Andrew Dunstan
On 11/14/2010 10:44 AM, Tom Lane wrote: And on the fourth hand, what we're buying here is pretty marginal for developers and of no interest whatever for users. I still think it's worth looking into whether the bug can be dodged by shortening the eval calls. But if not, I think I'd vote for

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-14 Thread Robert Haas
On Nov 14, 2010, at 10:44 AM, Tom Lane t...@sss.pgh.pa.us wrote: I still think it's worth looking into whether the bug can be dodged by shortening the eval calls. But if not, I think I'd vote for reverting. Maybe we could revisit this in a couple of years. +1. The current master branch

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-14 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Nov 14, 2010, at 10:44 AM, Tom Lane t...@sss.pgh.pa.us wrote: I still think it's worth looking into whether the bug can be dodged by shortening the eval calls. But if not, I think I'd vote for reverting. Maybe we could revisit this in a couple of

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-14 Thread Tom Lane
I wrote: I still think it's worth looking into whether the bug can be dodged by shortening the eval calls. In fact, that does seem to work; I'll commit a patch after testing a bit more. We still need someone to add the missing build dependencies so that make -j is trustworthy again.

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-14 Thread Robert Haas
On Sun, Nov 14, 2010 at 12:13 PM, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: I still think it's worth looking into whether the bug can be dodged by shortening the eval calls. In fact, that does seem to work; I'll commit a patch after testing a bit more. We still need someone to add the

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Andrew Dunstan
On 11/12/2010 11:25 PM, Tom Lane wrote: Andrew Dunstanand...@dunslane.net writes: On 11/12/2010 03:16 PM, Peter Eisentraut wrote: Improved parallel make support Looks like this patch has pretty comprehensively broken the MSVC build system. I'll see what I can recover from the wreckage.

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: I'm curious to know how much all this buys us. It *would* be nice if make -k worked better. I frequently run into the fact that (with the pre-existing setup) a compile error in the backend prevented make from proceeding with builds of interfaces/,

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Tom Lane
BTW, there's another problem here: make -j2 on my Mac blows up with this on stderr: ld: file not found: ../../../../../../src/backend/postgres collect2: ld returned 1 exit status make[3]: *** [ascii_and_mic.so] Error 1 make[2]: *** [all-ascii_and_mic-recursive] Error 2 make[1]: ***

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Andrew Dunstan
On 11/13/2010 11:12 AM, Tom Lane wrote: It looks like all the unhappy critters are getting the same virtual memory exhausted error. I wonder whether they are all using make 3.80 ... Maybe we need to put back make version logging. Interestingly, narwhal, the mingw machine that has reported,

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Dave Page
On Sat, Nov 13, 2010 at 4:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: It looks like all the unhappy critters are getting the same virtual memory exhausted error.  I wonder whether they are all using make 3.80 Both my Sparc and Intel Solaris critters have 3.80. -- Dave Page Blog:

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Peter Eisentraut
On lör, 2010-11-13 at 11:06 -0500, Andrew Dunstan wrote: But I don't feel we need to squeeze every last pip out of the build system. Probably not on the buildfarm, but when you are developing, saving 20 seconds or 2 minutes per cycle can lead to hours saved. -- Sent via pgsql-hackers

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Peter Eisentraut
On lör, 2010-11-13 at 11:12 -0500, Tom Lane wrote: It looks like all the unhappy critters are getting the same virtual memory exhausted error. I wonder whether they are all using make 3.80 ... It turns out that there is an unrelated bug in 3.80 that some Linux distributions have patched

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Peter Eisentraut
On lör, 2010-11-13 at 11:23 -0500, Tom Lane wrote: Consulting stdout shows that indeed it's launched this series of jobs: make -C backend/utils/mb/conversion_procs all make -C ascii_and_mic all gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: On lör, 2010-11-13 at 11:12 -0500, Tom Lane wrote: It looks like all the unhappy critters are getting the same virtual memory exhausted error. I wonder whether they are all using make 3.80 ... It turns out that there is an unrelated bug in 3.80 that

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Erik Rijkers
On Sat, November 13, 2010 18:15, Peter Eisentraut wrote: On lör, 2010-11-13 at 11:12 -0500, Tom Lane wrote: It looks like all the unhappy critters are getting the same virtual memory exhausted error. I wonder whether they are all using make 3.80 ... It turns out that there is an unrelated

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Peter Eisentraut
On lör, 2010-11-13 at 12:20 -0500, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: On lör, 2010-11-13 at 11:12 -0500, Tom Lane wrote: It looks like all the unhappy critters are getting the same virtual memory exhausted error. I wonder whether they are all using make 3.80 ...

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Peter Eisentraut
On lör, 2010-11-13 at 20:07 +0200, Peter Eisentraut wrote: On lör, 2010-11-13 at 12:20 -0500, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: On lör, 2010-11-13 at 11:12 -0500, Tom Lane wrote: It looks like all the unhappy critters are getting the same virtual memory exhausted

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-13 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: Well, it looks like $(eval) is pretty broken in 3.80, so either we require 3.81 or we abandon this line of thought. [ emerges from some grubbing about in the gmake sources... ] It looks to me like the bug in 3.80 is only triggered when eval expands to a

Re: [HACKERS] [COMMITTERS] pgsql: Improved parallel make support

2010-11-12 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: On 11/12/2010 03:16 PM, Peter Eisentraut wrote: Improved parallel make support Looks like this patch has pretty comprehensively broken the MSVC build system. I'll see what I can recover from the wreckage. There are also at least three non-Windows