Re: [HACKERS] improved parallel make support

[Andrew Dunstan]

On 11/11/2010 03:19 PM, Dave Page wrote:



My hope is that one day CMake will enable us to come up with a universal
solution, but we're some way from that yet.

We used CMake for a couple of projects, but ended up abandoning it for
new stuff. It just didn't work as nicely as we wanted.



Yes, it's been discussed before here too and didn't really go anywhere :-(

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Dave Page]

On Thu, Nov 11, 2010 at 5:19 PM, Mark Cave-Ayland
 wrote:
> Dave Page wrote:
>
>> On Thu, Nov 11, 2010 at 4:51 PM, Andrew Dunstan 
>> wrote:
>>>
>>> Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and
>>> isn't
>>> its Windows version build with MSVC?
>>
>> Yes - it's a PITA as we have to have a dummy build of the server in
>> mingw/msys to compile PostGIS and Slony. We're probably going to be
>> looking at that in the not-to-distant future as we want 64bit builds
>> of both and will be using VC++.
>
> Just for the record, a lot of work was done in the 1.4 release series to
> make MSVC builds possible, and indeed several people have reported success:
>
> http://postgis.refractions.net/pipermail/postgis-devel/2009-March/005102.html
> http://postgis.refractions.net/pipermail/postgis-devel/2010-September/010299.html

Cool - that will help.

> The two main outstanding issues as I see it are:
>
> 1) The GTK-based GUI for shp2pgsql (although if someone wanted to sponsor
> work to convert to wxWidgets to bring us in line with pgAdmin, that would be
> strongly considered).

:-)

> 2) Maintenance of the MSVC build system. So far we have had some complaints
> about not using MSVC, but then no-one has stepped up to maintain the build
> system for it. Forcing all existing developers to suddenly start maintaining
> the Windows build is a total non-starter.

Unless you're making major architectural changes, it shouldn't take
any real effort to add/remove the occasional source file. I'm sure
there are folks that could be persuaded to do that occasionally.

> My hope is that one day CMake will enable us to come up with a universal
> solution, but we're some way from that yet.

We used CMake for a couple of projects, but ended up abandoning it for
new stuff. It just didn't work as nicely as we wanted.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Mark Cave-Ayland]

Dave Page wrote:


On Thu, Nov 11, 2010 at 4:51 PM, Andrew Dunstan  wrote:

Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and isn't
its Windows version build with MSVC?


Yes - it's a PITA as we have to have a dummy build of the server in
mingw/msys to compile PostGIS and Slony. We're probably going to be
looking at that in the not-to-distant future as we want 64bit builds
of both and will be using VC++.


Just for the record, a lot of work was done in the 1.4 release series to 
make MSVC builds possible, and indeed several people have reported success:


http://postgis.refractions.net/pipermail/postgis-devel/2009-March/005102.html
http://postgis.refractions.net/pipermail/postgis-devel/2010-September/010299.html

The two main outstanding issues as I see it are:

1) The GTK-based GUI for shp2pgsql (although if someone wanted to 
sponsor work to convert to wxWidgets to bring us in line with pgAdmin, 
that would be strongly considered).


2) Maintenance of the MSVC build system. So far we have had some 
complaints about not using MSVC, but then no-one has stepped up to 
maintain the build system for it. Forcing all existing developers to 
suddenly start maintaining the Windows build is a total non-starter.


My hope is that one day CMake will enable us to come up with a universal 
solution, but we're some way from that yet.



ATB,

Mark.

--
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Dave Page]

On Thu, Nov 11, 2010 at 4:51 PM, Andrew Dunstan  wrote:
>
> Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and isn't
> its Windows version build with MSVC?

Yes - it's a PITA as we have to have a dummy build of the server in
mingw/msys to compile PostGIS and Slony. We're probably going to be
looking at that in the not-to-distant future as we want 64bit builds
of both and will be using VC++.


-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Andrew Dunstan]

On 11/11/2010 11:43 AM, Mark Cave-Ayland wrote:

Dave Page wrote:


Thanks - installed.

As a matter of policy, I do not want to drop support for a FOSS 
build tool

chain on Windows if at all avoidable.


Nor I, however I only have limited time to dedicate to that goal.


One thing to think about is that since PostGIS uses MingW/PGXS on 
Windows, we use MingW builds in order to generate the Makefiles we 
need (there is no native MSVC build for Windows). Not being able to do 
this would cause us great inconvenience :(






Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and 
isn't its Windows version build with MSVC?


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Mark Cave-Ayland]

Dave Page wrote:


Thanks - installed.


As a matter of policy, I do not want to drop support for a FOSS build tool
chain on Windows if at all avoidable.


Nor I, however I only have limited time to dedicate to that goal.


One thing to think about is that since PostGIS uses MingW/PGXS on 
Windows, we use MingW builds in order to generate the Makefiles we need 
(there is no native MSVC build for Windows). Not being able to do this 
would cause us great inconvenience :(



ATB,

Mark.

--
Mark Cave-Ayland - Senior Technical Architect
PostgreSQL - PostGIS
Sirius Corporation plc - control through freedom
http://www.siriusit.co.uk
t: +44 870 608 0063

Sirius Labs: http://www.siriusit.co.uk/labs

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Dave Page]

On Thu, Nov 11, 2010 at 1:04 PM, Andrew Dunstan  wrote:
>
> No, all you need to unpack those is the basic-bsdtar package.

Ahh, OK. That seems to be in the MinGW (compiler) section of the
downloads for some reason.

> But to save
> you the pain of all this, I have copied the three objects I installed to get
> this working on my likewise pretty old Msys to where you can get them. Just
> grab
> 

Thanks - installed.

> As a matter of policy, I do not want to drop support for a FOSS build tool
> chain on Windows if at all avoidable.

Nor I, however I only have limited time to dedicate to that goal.


-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Andrew Dunstan]

On 11/11/2010 06:58 AM, Dave Page wrote:

On Wed, Nov 10, 2010 at 6:13 PM, Andrew Dunstan  wrote:

Yeah, it's complaining about not finding bison, but configure managed to
find bison just fine. Are you sure the right make was installed? It looks
suspicious because it's not talking about msys virtual maths like the old
make did. It needs to be make-3.81-3-msys-1.0.13

You'll need another couple of libraries as well (libiconv and libintl) if
they are not already installed. Making this change took me a while to get
right on dawn_bat.

I installed the latest make from gnu.org (which I've now uninstalled).
The Msys installation on this box is old, and doesn't support the lzma
packages used by the latest releases - and from what I can tell, it
would take a major upgrade of the installation to get that support.
I'm not sure thats a path I want to go down, as I have no idea how
much will break if I do that, and I don't exactly have much in the way
of spare time to fix it if that happens.

I'm currently leaning towards removing the 9.1 build from the machine;
on a purely selfish note, I have no interest in mingw/msys builds
anymore anyway. However, I'm open to suggestions if anyone knows a
relatively safe way to resolve this.



No, all you need to unpack those is the basic-bsdtar package. But to 
save you the pain of all this, I have copied the three objects I 
installed to get this working on my likewise pretty old Msys to where 
you can get them. Just grab



As a matter of policy, I do not want to drop support for a FOSS build 
tool chain on Windows if at all avoidable.



cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Dave Page]

On Wed, Nov 10, 2010 at 6:13 PM, Andrew Dunstan  wrote:
>
> Yeah, it's complaining about not finding bison, but configure managed to
> find bison just fine. Are you sure the right make was installed? It looks
> suspicious because it's not talking about msys virtual maths like the old
> make did. It needs to be make-3.81-3-msys-1.0.13
> 
> You'll need another couple of libraries as well (libiconv and libintl) if
> they are not already installed. Making this change took me a while to get
> right on dawn_bat.

I installed the latest make from gnu.org (which I've now uninstalled).
The Msys installation on this box is old, and doesn't support the lzma
packages used by the latest releases - and from what I can tell, it
would take a major upgrade of the installation to get that support.
I'm not sure thats a path I want to go down, as I have no idea how
much will break if I do that, and I don't exactly have much in the way
of spare time to fix it if that happens.

I'm currently leaning towards removing the 9.1 build from the machine;
on a purely selfish note, I have no interest in mingw/msys builds
anymore anyway. However, I'm open to suggestions if anyone knows a
relatively safe way to resolve this.

/D

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Andrew Dunstan]

On 11/10/2010 10:32 AM, Peter Eisentraut wrote:

On tis, 2010-11-09 at 03:54 -0800, Dave Page wrote:

Narwhal should be OK now.

The build has issues now, possibly related to the make upgrade.




Yeah, it's complaining about not finding bison, but configure managed to 
find bison just fine. Are you sure the right make was installed? It 
looks suspicious because it's not talking about msys virtual maths like 
the old make did. It needs to be make-3.81-3-msys-1.0.13 
 
You'll need another couple of libraries as well (libiconv and libintl) 
if they are not already installed. Making this change took me a while to 
get right on dawn_bat.


cheers

andrew

Re: [HACKERS] improved parallel make support

[Peter Eisentraut]

On tis, 2010-11-09 at 03:54 -0800, Dave Page wrote:
> Narwhal should be OK now.

The build has issues now, possibly related to the make upgrade.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Dave Page]

On Sat, Nov 6, 2010 at 4:35 AM, Peter Eisentraut  wrote:
> On ons, 2010-11-03 at 16:34 +0200, Peter Eisentraut wrote:
>> On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote:
>> > Do we have a handle on how many buildfarm members this will break?
>>
>> I suppose we don't.  One way to find out would be to commit just this
>> bit
>>
>> +# We need the $(eval) function, which is available in GNU make 3.80.
>> +# That also happens to be the version where the .VARIABLES variable
>> +# was introduced, so this is a simple check.
>> +ifndef .VARIABLES
>> +$(error GNU make 3.80 or newer is required)
>> +endif
>>
>> with a $(warning) instead, and let it run for a bit.
>
> So far, two machines have reported an older make version:
>
> dawn_bat
> narwhal
>
> both of the mingw type.  Andrew, Dave, could you see about upgrading the
> GNU make installation there?

Narwhal should be OK now.

/D

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Andrew Dunstan]

On 11/06/2010 07:35 AM, Peter Eisentraut wrote:

So far, two machines have reported an older make version:

dawn_bat
narwhal

both of the mingw type.  Andrew, Dave, could you see about upgrading the
GNU make installation there?



dawn_bat is done.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Peter Eisentraut]

On ons, 2010-11-03 at 16:34 +0200, Peter Eisentraut wrote:
> On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote:
> > Do we have a handle on how many buildfarm members this will break?
> 
> I suppose we don't.  One way to find out would be to commit just this
> bit
> 
> +# We need the $(eval) function, which is available in GNU make 3.80.
> +# That also happens to be the version where the .VARIABLES variable
> +# was introduced, so this is a simple check.
> +ifndef .VARIABLES
> +$(error GNU make 3.80 or newer is required)
> +endif
> 
> with a $(warning) instead, and let it run for a bit.

So far, two machines have reported an older make version:

dawn_bat
narwhal

both of the mingw type.  Andrew, Dave, could you see about upgrading the
GNU make installation there?

There are a few machines that haven't build in five days or more, but
based on their operating system version, it is fairly safe to assume
that they have an up-to-date version.



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Tom Lane]

Peter Eisentraut  writes:
> On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote:
>> Do we have a handle on how many buildfarm members this will break?

> I suppose we don't.  One way to find out would be to commit just this
> bit

> +# We need the $(eval) function, which is available in GNU make 3.80.
> +# That also happens to be the version where the .VARIABLES variable
> +# was introduced, so this is a simple check.
> +ifndef .VARIABLES
> +$(error GNU make 3.80 or newer is required)
> +endif

> with a $(warning) instead, and let it run for a bit.

+1

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Peter Eisentraut]

On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote:
> Peter Eisentraut  writes:
> > This patch requires GNU make 3.80, because of the above "|" feature and
> > the $(eval) function.  Version 3.80 is dated October 2002, so it should
> > be no problem, but I do occasionally read of make 3.79 around here;
> > maybe it's time to get rid of that.  I did put in a check that makes the
> > build fail right away if a wrong version of make is used.
> 
> Do we have a handle on how many buildfarm members this will break?

I suppose we don't.  One way to find out would be to commit just this
bit

+# We need the $(eval) function, which is available in GNU make 3.80.
+# That also happens to be the version where the .VARIABLES variable
+# was introduced, so this is a simple check.
+ifndef .VARIABLES
+$(error GNU make 3.80 or newer is required)
+endif

with a $(warning) instead, and let it run for a bit.



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improved parallel make support

[Tom Lane]

Peter Eisentraut  writes:
> This patch requires GNU make 3.80, because of the above "|" feature and
> the $(eval) function.  Version 3.80 is dated October 2002, so it should
> be no problem, but I do occasionally read of make 3.79 around here;
> maybe it's time to get rid of that.  I did put in a check that makes the
> build fail right away if a wrong version of make is used.

Do we have a handle on how many buildfarm members this will break?

(fwiw, my hpux box is running 3.79.1)

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] improved parallel make support

[Peter Eisentraut]

I have worked on some improvements on how we handle recursive make in
our makefiles.  Most places uses for loops, which has some
disadvantages: parallel make doesn't work across directories, make -k
doesn't work, and make -q doesn't work.  Instead, I went with the
approach that we already use in the src/backend directory, where we call
the subordinate makes as target prerequisites.

Note that because with this, parallel make really works, the rule
dependencies must be correct.  This has always been the case, but now it
really shows up.  A frequent issue is that this sort of thing no longer
works:

all: submake-libpgport zic

zic: $(ZICOBJS)

because this relies on the "all" target to execute its prerequisites in
order.  Instead, you need to write it like this:

all: zic

zic: $(ZICOBJS) | submake-libpgport

(The bar is necessary so that zic isn't considered constantly out of
date because it depends on a phony target.)

This patch requires GNU make 3.80, because of the above "|" feature and
the $(eval) function.  Version 3.80 is dated October 2002, so it should
be no problem, but I do occasionally read of make 3.79 around here;
maybe it's time to get rid of that.  I did put in a check that makes the
build fail right away if a wrong version of make is used.

diff --git a/GNUmakefile.in b/GNUmakefile.in
index 57f5813..9d4c4ce 100644
--- a/GNUmakefile.in
+++ b/GNUmakefile.in
@@ -8,57 +8,39 @@ subdir =
 top_builddir = .
 include $(top_builddir)/src/Makefile.global
 
+$(call recurse,all install,src config)
+
 all:
-	$(MAKE) -C src all
-	$(MAKE) -C config all
-	@echo "All of PostgreSQL successfully made. Ready to install."
+	+...@echo "All of PostgreSQL successfully made. Ready to install."
 
 docs:
 	$(MAKE) -C doc all
 
-world:
-	$(MAKE) -C doc all
-	$(MAKE) -C src all
-	$(MAKE) -C config all
-	$(MAKE) -C contrib all
-	@echo "PostgreSQL, contrib, and documentation successfully made. Ready to install."
+$(call recurse,world,contrib,all)
+world: all docs
+	+...@echo "PostgreSQL, contrib, and documentation successfully made. Ready to install."
 
 html man:
 	$(MAKE) -C doc $@
 
 install:
-	$(MAKE) -C src $@
-	$(MAKE) -C config $@
-	@echo "PostgreSQL installation complete."
+	+...@echo "PostgreSQL installation complete."
 
 install-docs:
 	$(MAKE) -C doc install
 
-install-world:
-	$(MAKE) -C doc install
-	$(MAKE) -C src install
-	$(MAKE) -C config install
-	$(MAKE) -C contrib install
-	@echo "PostgreSQL, contrib, and documentation installation complete."
+$(call recurse,install-world,contrib,install)
+install-world: install install-docs
+	+...@echo "PostgreSQL, contrib, and documentation installation complete."
 
-installdirs uninstall coverage:
-	$(MAKE) -C doc $@
-	$(MAKE) -C src $@
-	$(MAKE) -C config $@
+$(call recurse,installdirs uninstall coverage,doc src config)
 
-distprep:
-	$(MAKE) -C doc $@
-	$(MAKE) -C src $@
-	$(MAKE) -C config $@
-	$(MAKE) -C contrib $@
+$(call recurse,distprep,doc src config contrib)
 
 # clean, distclean, etc should apply to contrib too, even though
 # it's not built by default
+$(call recurse,clean,doc contrib src config)
 clean:
-	$(MAKE) -C doc $@
-	$(MAKE) -C contrib $@
-	$(MAKE) -C src $@
-	$(MAKE) -C config $@
 # Garbage from autoconf:
 	@rm -rf autom4te.cache/
 
@@ -78,11 +60,7 @@ check: all
 check installcheck installcheck-parallel:
 	$(MAKE) -C src/test $@
 
-installcheck-world:
-	$(MAKE) -C src/test installcheck
-	$(MAKE) -C src/pl installcheck
-	$(MAKE) -C src/interfaces/ecpg installcheck
-	$(MAKE) -C contrib installcheck
+$(call recurse,installcheck-world,src/test src/interfaces/ecpg contrib,installcheck)
 
 GNUmakefile: GNUmakefile.in $(top_builddir)/config.status
 	./config.status $@
@@ -143,4 +121,4 @@ distcheck: dist
 	rm -rf $(distdir) $(dummy)
 	@echo "Distribution integrity checks out."
 
-.PHONY: dist distdir distcheck docs install-docs
+.PHONY: dist distdir distcheck docs install-docs world install-world installcheck-world
diff --git a/contrib/Makefile b/contrib/Makefile
index b777325..e1f2a84 100644
--- a/contrib/Makefile
+++ b/contrib/Makefile
@@ -63,14 +63,4 @@ endif
 #		start-scripts	\ (does not have a makefile)
 
 
-all install installdirs uninstall distprep clean distclean maintainer-clean:
-	@for dir in $(SUBDIRS); do \
-		$(MAKE) -C $$dir $@ || exit; \
-	done
-
-# We'd like check operations to run all the subtests before failing.
-check installcheck:
-	@CHECKERR=0; for dir in $(SUBDIRS); do \
-		$(MAKE) -C $$dir $@ || CHECKERR=$$?; \
-	done; \
-	exit $$CHECKERR
+$(recurse)
diff --git a/contrib/dblink/Makefile b/contrib/dblink/Makefile
index 148961e..cc59128 100644
--- a/contrib/dblink/Makefile
+++ b/contrib/dblink/Makefile
@@ -4,6 +4,7 @@ MODULE_big = dblink
 PG_CPPFLAGS = -I$(libpq_srcdir)
 OBJS	= dblink.o
 SHLIB_LINK = $(libpq)
+SHLIB_PREREQS = submake-libpq
 
 DATA_built = dblink.sql 
 DATA = uninstall_dblink.sql 
diff --git a/src/Makefile b/src/Makefile
index 0e1e431..0d4a6ee 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -12,20 +12,21 @@ subdir =