Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Hi Andrew, On Mon, Sep 23, 2013 at 6:43 PM, Andrew Dunstan and...@dunslane.net wrote: I'm working on it. It appears to have a slight problem or two I want to fix at the same time, rather than backpatch something broken. Any progress on this? I notice that the fixes didn't make it into 9.3.1. Regards, Marti -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On 09/23/2013 12:15 PM, Cédric Villemain wrote: I'm working on it. It appears to have a slight problem or two I want to fix at the same time, rather than backpatch something broken. Would you mind sharing the problems you are facing ? You've noticed the problem about installdirs, I suppose. The attached patch is the fix currently applyed at debian. I will when I'm sure it's not a case of pilot error on my part. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On Fri, Sep 13, 2013 at 8:17 PM, Marti Raudsepp ma...@juffo.org wrote: On Fri, Sep 13, 2013 at 6:42 PM, Cédric Villemain ced...@2ndquadrant.com wrote: Andrew is about to commit (well...I hope) a doc patch about that and also a little fix. Imho this is a bugfix so I hope it will be applyed in older branches. Oh I see, indeed commit 6697aa2bc25c83b88d6165340348a31328c35de6 Improve support for building PGXS modules with VPATH fixes the problem and I see it's not present in REL9_3_0. Andrew and others, does this seem safe enough to backport to 9.3.1? Ping? Will this be backported to 9.3 or should I report to extension authors to fix their Makefiles? I understand that the 9.3.1 release might still be weeks away, I'd just like to get a vague confirmation about what committers think. Note that this patch is already applied to Debian/Ubuntu packages (including those on apt.postgresql.org). Regards, Marti -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Marti Raudsepp wrote: On Fri, Sep 13, 2013 at 8:17 PM, Marti Raudsepp ma...@juffo.org wrote: Oh I see, indeed commit 6697aa2bc25c83b88d6165340348a31328c35de6 Improve support for building PGXS modules with VPATH fixes the problem and I see it's not present in REL9_3_0. Andrew and others, does this seem safe enough to backport to 9.3.1? Ping? Will this be backported to 9.3 or should I report to extension authors to fix their Makefiles? I think this should be backpatched. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On 09/23/2013 11:31 AM, Alvaro Herrera wrote: Marti Raudsepp wrote: On Fri, Sep 13, 2013 at 8:17 PM, Marti Raudsepp ma...@juffo.org wrote: Oh I see, indeed commit 6697aa2bc25c83b88d6165340348a31328c35de6 Improve support for building PGXS modules with VPATH fixes the problem and I see it's not present in REL9_3_0. Andrew and others, does this seem safe enough to backport to 9.3.1? Ping? Will this be backported to 9.3 or should I report to extension authors to fix their Makefiles? I think this should be backpatched. I'm working on it. It appears to have a slight problem or two I want to fix at the same time, rather than backpatch something broken. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Le lundi 23 septembre 2013 11:43:13 Andrew Dunstan a écrit : On 09/23/2013 11:31 AM, Alvaro Herrera wrote: Marti Raudsepp wrote: On Fri, Sep 13, 2013 at 8:17 PM, Marti Raudsepp ma...@juffo.org wrote: Oh I see, indeed commit 6697aa2bc25c83b88d6165340348a31328c35de6 Improve support for building PGXS modules with VPATH fixes the problem and I see it's not present in REL9_3_0. Andrew and others, does this seem safe enough to backport to 9.3.1? Ping? Will this be backported to 9.3 or should I report to extension authors to fix their Makefiles? I think this should be backpatched. I'm working on it. It appears to have a slight problem or two I want to fix at the same time, rather than backpatch something broken. Would you mind sharing the problems you are facing ? You've noticed the problem about installdirs, I suppose. The attached patch is the fix currently applyed at debian. -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et FormationMake the install targets depend on installdirs (not yet upstream, tbd) --- a/src/makefiles/pgxs.mk +++ b/src/makefiles/pgxs.mk @@ -132,29 +132,29 @@ $(INSTALL_PROGRAM) $(PROGRAM)$(X) '$(DESTDIR)$(bindir)' endif # PROGRAM -installcontrol: $(addsuffix .control, $(EXTENSION)) +installcontrol: $(addsuffix .control, $(EXTENSION)) | installdirs ifneq (,$(EXTENSION)) $(INSTALL_DATA) $^ '$(DESTDIR)$(datadir)/extension/' endif -installdata: $(DATA) $(DATA_built) +installdata: $(DATA) $(DATA_built) | installdirs ifneq (,$(DATA)$(DATA_built)) $(INSTALL_DATA) $^ '$(DESTDIR)$(datadir)/$(datamoduledir)/' endif -installdatatsearch: $(DATA_TSEARCH) +installdatatsearch: $(DATA_TSEARCH) | installdirs ifneq (,$(DATA_TSEARCH)) $(INSTALL_DATA) $^ '$(DESTDIR)$(datadir)/tsearch_data/' endif -installdocs: $(DOCS) +installdocs: $(DOCS) | installdirs ifdef DOCS ifdef docdir $(INSTALL_DATA) $^ '$(DESTDIR)$(docdir)/$(docmoduledir)/' endif # docdir endif # DOCS -installscripts: $(SCRIPTS) $(SCRIPTS_built) +installscripts: $(SCRIPTS) $(SCRIPTS_built) | installdirs ifdef SCRIPTS $(INSTALL_SCRIPT) $^ '$(DESTDIR)$(bindir)/' endif # SCRIPTS signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On Tue, Sep 17, 2013 at 10:37 AM, Cédric Villemain ced...@2ndquadrant.com wrote: Erm, isn't apt.postgresql.org supposed to ship the *official* PostgreSQL versions? Given that this issue affects all distros, I don't see why Ubuntu/Debian need to be patched separately. Well, the patches are applyed on the debian packages (not only in apt.pgdg.org). The packages provided by apt.postgresql.org are based on the 'official packages' from debian. (if you allow me this circle) Oh I see, that makes sense. PS: Where are the sources used to build packages on 9.3: http://anonscm.debian.org/loggerhead/pkg-postgresql/postgresql-9.3/trunk/changes Thanks! Regards, Marti -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Apt.pgdg got the patch present in postgresql head applyed. Erm, isn't apt.postgresql.org supposed to ship the *official* PostgreSQL versions? Given that this issue affects all distros, I don't see why Ubuntu/Debian need to be patched separately. Well, the patches are applyed on the debian packages (not only in apt.pgdg.org). The packages provided by apt.postgresql.org are based on the 'official packages' from debian. (if you allow me this circle) Does anyone else think this is problematic? By slipping patches into distro-specific packages, you're confusing users (like me) and bypassing the PostgreSQL QA process. The QA is about distribution of packages here. Debian applies 14 patches on 9.3 builds, not only the things about vpath we're talking about. PS: Where are the sources used to build packages on apt.postgresql.org? 9.3: http://anonscm.debian.org/loggerhead/pkg-postgresql/postgresql-9.3/trunk/changes -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Marti Raudsepp ma...@juffo.org a écrit : On Tue, May 14, 2013 at 4:12 AM, Marti Raudsepp ma...@juffo.org wrote: While testing out PostgreSQL 9.3beta1, I stumbled upon a problem % make DESTDIR=/tmp/foo install /usr/bin/install: will not overwrite just-created ‘/tmp/foo/usr/share/postgresql/extension/semver--0.3.0.sql’ with ‘./sql/semver--0.3.0.sql’ make: *** [install] Error 1 On Wed, May 15, 2013 at 4:49 PM, Peter Eisentraut pete...@gmx.net wrote: That said, I'm obviously outnumbered here. What about the following compromise: Use the configure-selected install program inside PostgreSQL (which we can test easily), and use install-sh under USE_PGXS? Admittedly, the make install time of extensions is probably not an issue. Did we ever do anything about this? It looks like the thread got distracted with VPATH builds and now I'm seeing this problem in 9.3.0. :( This occurs in Arch Linux, but for some odd reason not on Ubuntu when using apt.postgresql.org. Somehow the pgxs.mk supplied by apt.postgresql.org differs from the one shipped in PostgreSQL. Is there a chance of getting this resolved in PostgreSQL or should we get extension writers to fix their makefiles instead? Apt.pgdg got the patch present in postgresql head applyed. Andrew is about to commit (well...I hope) a doc patch about that and also a little fix. Imho this is a bugfix so I hope it will be applyed in older branches. -- Envoyé de mon téléphone. Excusez la brièveté. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On Fri, Sep 13, 2013 at 6:42 PM, Cédric Villemain ced...@2ndquadrant.com wrote: Marti Raudsepp ma...@juffo.org a écrit : Did we ever do anything about this? It looks like the thread got distracted with VPATH builds and now I'm seeing this problem in 9.3.0. Andrew is about to commit (well...I hope) a doc patch about that and also a little fix. Imho this is a bugfix so I hope it will be applyed in older branches. Oh I see, indeed commit 6697aa2bc25c83b88d6165340348a31328c35de6 Improve support for building PGXS modules with VPATH fixes the problem and I see it's not present in REL9_3_0. Andrew and others, does this seem safe enough to backport to 9.3.1? Apt.pgdg got the patch present in postgresql head applyed. Erm, isn't apt.postgresql.org supposed to ship the *official* PostgreSQL versions? Given that this issue affects all distros, I don't see why Ubuntu/Debian need to be patched separately. Does anyone else think this is problematic? By slipping patches into distro-specific packages, you're confusing users (like me) and bypassing the PostgreSQL QA process. PS: Where are the sources used to build packages on apt.postgresql.org? Regards, Marti -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On Tue, May 14, 2013 at 4:12 AM, Marti Raudsepp ma...@juffo.org wrote: While testing out PostgreSQL 9.3beta1, I stumbled upon a problem % make DESTDIR=/tmp/foo install /usr/bin/install: will not overwrite just-created ‘/tmp/foo/usr/share/postgresql/extension/semver--0.3.0.sql’ with ‘./sql/semver--0.3.0.sql’ make: *** [install] Error 1 On Wed, May 15, 2013 at 4:49 PM, Peter Eisentraut pete...@gmx.net wrote: That said, I'm obviously outnumbered here. What about the following compromise: Use the configure-selected install program inside PostgreSQL (which we can test easily), and use install-sh under USE_PGXS? Admittedly, the make install time of extensions is probably not an issue. Did we ever do anything about this? It looks like the thread got distracted with VPATH builds and now I'm seeing this problem in 9.3.0. :( This occurs in Arch Linux, but for some odd reason not on Ubuntu when using apt.postgresql.org. Somehow the pgxs.mk supplied by apt.postgresql.org differs from the one shipped in PostgreSQL. Is there a chance of getting this resolved in PostgreSQL or should we get extension writers to fix their makefiles instead? Regards, Marti -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Le mardi 28 mai 2013 14:10:14, Cédric Villemain a écrit : I just took time to inspect our contribs, USE_PGXS is not supported by all of them atm because of SHLIB_PREREQS (it used submake) I have a patch pending here to fix that. Attached patch fix SHLIB_PREREQS when building with USE_PGXS commit 19e231b introduced SHLIB_PREREQS but failed to port that to PGXS build. The issue is that submake-* can not be built with PGXS, in this case they must check that expected files are present (and installed). Maybe it is better to only check if they have been built ? This fix the build of dblink and postgres_fdw (make USE_PGXS=1) This patch is a bugfix and IMO should be applied in 9.1 and 9.2. Else we should remove the PGXS support from the Makefile of the contribs dblink and postgres_fdw because it is not working currently. -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Le mardi 28 mai 2013 15:15:55, Cédric Villemain a écrit : Le samedi 25 mai 2013 16:41:24, Cédric Villemain a écrit : If it seems to be on the right way, I'll keep fixing EXTENSION building with VPATH. I haven't tried the patch, but let me just say that Debian (and apt.postgresql.org) would very much like the VPATH situation getting improved. At the moment we seem to have to invent a new build recipe for every extension around. Attached patch adds support for VPATH with USE_PGXS It just change recipe for install: in pgxs.mk. I am unsure automatic variables can be used this way with all UNIX variation of make... I also didn't touch MODULE and PROGRAM (yet) This patch can also be seen as a bugfix. The problem is that in case like this one: === FOO=bar.control installcontrol: $(FOO) $(INSTALL_DATA) $(FOO) \ '$(DESTDIR)$(datadir)/extension/' === INSTALL_DATA will install the file defined by FOO (=bar.control). But in the next case (the fix): === FOO=bar.control installcontrol: $(FOO) $(INSTALL_DATA) $ '$(DESTDIR)$(datadir)/extension/' === the $ contains the *filepath* where 'make' found the FOO file. I believe it is because recipes are read once target and prerequisite are set, so $ contains the full path to FOO, but FOO will still contain exactly what has been assigned to FOO. I choose to add targets for the variables that can be set when using PGXS. And say 'install:' target to depends on each. Maybe there is a smarter way to do it... my skills in Makefile are limited. So far the feedback is good for the set of patches: pg_buildext (the debian postgresql builder tool) is still working as expected and can be simplified. I can build each contrib provided with PostgreSQL and some others have been tried (like plr). IMHO this is mostly bugfixing and it just outline that we can support VPATH and PGXS build at the same time. -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On 05/29/2013 06:08 PM, Cédric Villemain wrote: I just took time to inspect our contribs, USE_PGXS is not supported by all of them atm because of SHLIB_PREREQS (it used submake) I have a patch pending here to fix that. Once all our contribs can build with USE_PGXS I fix the VPATH. I've added 'most' of the patches to the commitfest... (I am not sure it is required, as this is more bugfix than anything else IMHO) See https://commitfest.postgresql.org/action/patch_view?id=1122 https://commitfest.postgresql.org/action/patch_view?id=1123 https://commitfest.postgresql.org/action/patch_view?id=1124 I stopped trying to add new item after too many failures from https://commitfest.postgresql.org/action/patch_form So one patch is not in the commitfest yet (fix_install_ext_vpath.patch) failures? what kind of issues did you experience? Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Le jeudi 30 mai 2013 14:32:46, Stefan Kaltenbrunner a écrit : On 05/29/2013 06:08 PM, Cédric Villemain wrote: I just took time to inspect our contribs, USE_PGXS is not supported by all of them atm because of SHLIB_PREREQS (it used submake) I have a patch pending here to fix that. Once all our contribs can build with USE_PGXS I fix the VPATH. I've added 'most' of the patches to the commitfest... (I am not sure it is required, as this is more bugfix than anything else IMHO) See https://commitfest.postgresql.org/action/patch_view?id=1122 https://commitfest.postgresql.org/action/patch_view?id=1123 https://commitfest.postgresql.org/action/patch_view?id=1124 I stopped trying to add new item after too many failures from https://commitfest.postgresql.org/action/patch_form So one patch is not in the commitfest yet (fix_install_ext_vpath.patch) failures? what kind of issues did you experience? I didn't sent too much details as I am not sure if it was my setup which breaks things or not. [...] Just tested with Stephen, looks like a problem with something on my side, sorry for the noise. (rekonq 0 - chromium 1) -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
I just took time to inspect our contribs, USE_PGXS is not supported by all of them atm because of SHLIB_PREREQS (it used submake) I have a patch pending here to fix that. Once all our contribs can build with USE_PGXS I fix the VPATH. I've added 'most' of the patches to the commitfest... (I am not sure it is required, as this is more bugfix than anything else IMHO) See https://commitfest.postgresql.org/action/patch_view?id=1122 https://commitfest.postgresql.org/action/patch_view?id=1123 https://commitfest.postgresql.org/action/patch_view?id=1124 I stopped trying to add new item after too many failures from https://commitfest.postgresql.org/action/patch_form So one patch is not in the commitfest yet (fix_install_ext_vpath.patch) -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
I just took time to inspect our contribs, USE_PGXS is not supported by all of them atm because of SHLIB_PREREQS (it used submake) I have a patch pending here to fix that. Attached patch fix SHLIB_PREREQS when building with USE_PGXS commit 19e231b introduced SHLIB_PREREQS but failed to port that to PGXS build. The issue is that submake-* can not be built with PGXS, in this case they must check that expected files are present (and installed). Maybe it is better to only check if they have been built ? This fix the build of dblink and postgres_fdw (make USE_PGXS=1) -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation diff --git a/src/Makefile.global.in b/src/Makefile.global.in index 89e39d2..1b13f85 100644 --- a/src/Makefile.global.in +++ b/src/Makefile.global.in @@ -415,13 +415,24 @@ libpq_pgport = -L$(top_builddir)/src/port -lpgport \ -L$(top_builddir)/src/common -lpgcommon $(libpq) endif - +# If PGXS is not defined, builds as usual: +# build dependancies required by SHLIB_PREREQS +# If the build is with PGXS, then any requirement is supposed to be already +# build and we just take care that the expected files exist +ifndef PGXS submake-libpq: $(MAKE) -C $(libpq_builddir) all +else +submake-libpq: $(libdir)/libpq.so ; +endif +ifndef PGXS submake-libpgport: $(MAKE) -C $(top_builddir)/src/port all $(MAKE) -C $(top_builddir)/src/common all +else +submake-libpgport: $(libdir)/libpgport.a $(libdir)/libpgcommon.a ; +endif .PHONY: submake-libpq submake-libpgport signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Once all our contribs can build with USE_PGXS I fix the VPATH. The last step is interesting: installcheck/REGRESS. For this one, if I can know exactly what's required (for debian build for example), then I can also fix this target. There is a hack to link the regression data files from the srcdir to the builddir when doing 'make VPATH'. but it failed when used in conjunction with USE_PGXS and out-of-tree build of an extension. Issue is the absence of the data/ directory in the builddir. Attached patch fix that. -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation diff --git a/src/makefiles/pgxs.mk b/src/makefiles/pgxs.mk index bbcfe04..e8ff584 100644 --- a/src/makefiles/pgxs.mk +++ b/src/makefiles/pgxs.mk @@ -263,6 +263,7 @@ test_files_build := $(patsubst $(srcdir)/%, $(abs_builddir)/%, $(test_files_src) all: $(test_files_build) $(test_files_build): $(abs_builddir)/%: $(srcdir)/% + mkdir -p $(dir $@) ln -s $ $@ endif # VPATH signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Once all our contribs can build with USE_PGXS I fix the VPATH. Attached patch set VPATH for out-of-tree extension builds If the makefile is not in the current directory (where we launch 'make') then assume we are building out-of-src tree and set the VPATH to the directory of the *first* makefile... Thus it fixes: mkdir /tmp/a cd /tmp/a make -f extension_src/Makefile USE_PGXS=1 Note that the patch fix things. Still I am not really happy with the rule to get the srcdir. -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation diff --git a/src/makefiles/pgxs.mk b/src/makefiles/pgxs.mk index e8ff584..64732ff 100644 --- a/src/makefiles/pgxs.mk +++ b/src/makefiles/pgxs.mk @@ -61,9 +61,18 @@ ifdef PGXS top_builddir := $(dir $(PGXS))../.. include $(top_builddir)/src/Makefile.global +# If Makefile is not in current directory we are building the extension with +# VPATH so we set the variable here +# XXX what about top_srcdir ? +ifeq ($(CURDIR),$(dir $(firstword $(MAKEFILE_LIST top_srcdir = $(top_builddir) srcdir = . VPATH = +else +top_srcdir = $(top_builddir) +srcdir = $(dir $(firstword $(MAKEFILE_LIST))) +VPATH = $(dir $(firstword $(MAKEFILE_LIST))) +endif # These might be set in Makefile.global, but if they were not found # during the build of PostgreSQL, supply default values so that users signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Le mardi 28 mai 2013 14:16:38, Cédric Villemain a écrit : Once all our contribs can build with USE_PGXS I fix the VPATH. The last step is interesting: installcheck/REGRESS. For this one, if I can know exactly what's required (for debian build for example), then I can also fix this target. There is a hack to link the regression data files from the srcdir to the builddir when doing 'make VPATH'. but it failed when used in conjunction with USE_PGXS and out-of-tree build of an extension. Issue is the absence of the data/ directory in the builddir. Attached patch fix that. use $(MKDIR_P) instead of mkdir -p -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation diff --git a/src/makefiles/pgxs.mk b/src/makefiles/pgxs.mk index bbcfe04..e8ff584 100644 --- a/src/makefiles/pgxs.mk +++ b/src/makefiles/pgxs.mk @@ -263,6 +263,7 @@ test_files_build := $(patsubst $(srcdir)/%, $(abs_builddir)/%, $(test_files_src) all: $(test_files_build) $(test_files_build): $(abs_builddir)/%: $(srcdir)/% + $(MKDIR_P) '$(dir $@)' ln -s $ $@ endif # VPATH signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Le samedi 25 mai 2013 16:41:24, Cédric Villemain a écrit : If it seems to be on the right way, I'll keep fixing EXTENSION building with VPATH. I haven't tried the patch, but let me just say that Debian (and apt.postgresql.org) would very much like the VPATH situation getting improved. At the moment we seem to have to invent a new build recipe for every extension around. Attached patch adds support for VPATH with USE_PGXS It just change recipe for install: in pgxs.mk. I am unsure automatic variables can be used this way with all UNIX variation of make... I also didn't touch MODULE and PROGRAM (yet) -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation diff --git a/src/makefiles/pgxs.mk b/src/makefiles/pgxs.mk index 31746f3..2575855 100644 --- a/src/makefiles/pgxs.mk +++ b/src/makefiles/pgxs.mk @@ -121,33 +121,40 @@ all: all-lib endif # MODULE_big -install: all installdirs -ifneq (,$(EXTENSION)) - $(INSTALL_DATA) $(addprefix $(srcdir)/, $(addsuffix .control, $(EXTENSION))) '$(DESTDIR)$(datadir)/extension/' -endif # EXTENSION -ifneq (,$(DATA)$(DATA_built)) - $(INSTALL_DATA) $(addprefix $(srcdir)/, $(DATA)) $(DATA_built) '$(DESTDIR)$(datadir)/$(datamoduledir)/' -endif # DATA -ifneq (,$(DATA_TSEARCH)) - $(INSTALL_DATA) $(addprefix $(srcdir)/, $(DATA_TSEARCH)) '$(DESTDIR)$(datadir)/tsearch_data/' -endif # DATA_TSEARCH +install: all installdirs installcontrol installdata installdatatsearch installdocs installscripts ifdef MODULES $(INSTALL_SHLIB) $(addsuffix $(DLSUFFIX), $(MODULES)) '$(DESTDIR)$(pkglibdir)/' endif # MODULES +ifdef PROGRAM + $(INSTALL_PROGRAM) $(PROGRAM)$(X) '$(DESTDIR)$(bindir)' +endif # PROGRAM + +installcontrol: $(addsuffix .control, $(EXTENSION)) +ifneq (,$(EXTENSION)) + $(INSTALL_DATA) $ '$(DESTDIR)$(datadir)/extension/' +endif + +installdata: $(DATA) $(DATA_built) +ifneq (,$(DATA)$(DATA_built)) + $(INSTALL_DATA) $^ '$(DESTDIR)$(datadir)/$(datamoduledir)/' +endif + +installdatatsearch: $(DATA_TSEARCH) +ifneq (,$(DATA_TSEARCH)) + $(INSTALL_DATA) $^ '$(DESTDIR)$(datadir)/tsearch_data/' +endif + +installdocs: $(DOCS) ifdef DOCS ifdef docdir - $(INSTALL_DATA) $(addprefix $(srcdir)/, $(DOCS)) '$(DESTDIR)$(docdir)/$(docmoduledir)/' + $(INSTALL_DATA) $^ '$(DESTDIR)$(docdir)/$(docmoduledir)/' endif # docdir endif # DOCS -ifdef PROGRAM - $(INSTALL_PROGRAM) $(PROGRAM)$(X) '$(DESTDIR)$(bindir)' -endif # PROGRAM + +installscripts: $(SCRIPTS) $(SCRIPTS_built) ifdef SCRIPTS - $(INSTALL_SCRIPT) $(addprefix $(srcdir)/, $(SCRIPTS)) '$(DESTDIR)$(bindir)/' + $(INSTALL_SCRIPT) $^ '$(DESTDIR)$(bindir)/' endif # SCRIPTS -ifdef SCRIPTS_built - $(INSTALL_SCRIPT) $(SCRIPTS_built) '$(DESTDIR)$(bindir)/' -endif # SCRIPTS_built ifdef MODULE_big install: install-lib signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
If it seems to be on the right way, I'll keep fixing EXTENSION building with VPATH. I haven't tried the patch, but let me just say that Debian (and apt.postgresql.org) would very much like the VPATH situation getting improved. At the moment we seem to have to invent a new build recipe for every extension around. I have been busy the last week. I just took time to inspect our contribs, USE_PGXS is not supported by all of them atm because of SHLIB_PREREQS (it used submake) I have a patch pending here to fix that. Once all our contribs can build with USE_PGXS I fix the VPATH. The last step is interesting: installcheck/REGRESS. For this one, if I can know exactly what's required (for debian build for example), then I can also fix this target. -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Re: Cédric Villemain 2013-05-25 201305251641.28401.ced...@2ndquadrant.com I just took time to inspect our contribs, USE_PGXS is not supported by all of them atm because of SHLIB_PREREQS (it used submake) I have a patch pending here to fix that. Once all our contribs can build with USE_PGXS I fix the VPATH. The evil part of the problem is that you'd need to fix it all the way back to 8.4 (or 9.0 once 8.4 is EOL) if we want it to build extensions on apt.pg.org. (Or find a way that is compatible with as many majors as possible.) The last step is interesting: installcheck/REGRESS. For this one, if I can know exactly what's required (for debian build for example), then I can also fix this target. Debian builds don't have root access, so make installcheck would need to install into a temp dir (probably DESTDIR). Currently this fails because the server will still insist on reading the extension control files from PGSHAREDIR. (See the thread starting at http://www.postgresql.org/message-id/20120221101903.GA15647@gnash for details.) Otherwise than that, and my pending pg_regress --host /tmp patch, things should probably ok for buildds. Christoph -- c...@df7cb.de | http://www.df7cb.de/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Re: Cédric Villemain 2013-05-17 201305171642.59241.ced...@2ndquadrant.com If it seems to be on the right way, I'll keep fixing EXTENSION building with VPATH. I haven't tried the patch, but let me just say that Debian (and apt.postgresql.org) would very much like the VPATH situation getting improved. At the moment we seem to have to invent a new build recipe for every extension around. Thanks for working on this. Christoph -- c...@df7cb.de | http://www.df7cb.de/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Le jeudi 16 mai 2013 18:53:19, Alvaro Herrera a écrit : Andrew Dunstan wrote: On 05/16/2013 10:39 AM, Cédric Villemain wrote: Le jeudi 16 mai 2013 15:51:48, Tom Lane a écrit : Andrew Dunstan and...@dunslane.net writes: On 05/16/2013 05:41 AM, Dimitri Fontaine wrote: And VPATH building of extension is crucially important for me, as the easiest way I've found to build and package a given extension against all currently supported version of PostgreSQL. Is there documented support for VPATH builds? The core code certainly builds okay in VPATH mode, and I would agree with Dimitri that pgxs builds should as well. But this is more of an autoconf+make feature than ours, so I'm not sure why you'd expect us to document it. Extension does not support VPATH at 100% (well, pgxs.mk). There is a minor hack for some REGRESS files but that's all. Right. My impression is that pgxs.mk actively works against vpath builds. That's my experience, yes. It would be great to get it fixed. OK. I've reduce that to the attached (WIP) patch . Problem is usage of $(srcdir) in the PGXS case. I try to remove it in favor of recipes for each case (DATA, DOCS, ...) It's a quick hack... It does not handle some cases yet (for example it does not find .o when come the time to build .so if the .o was already in VPATH), this later issue is a separate one I think. If it seems to be on the right way, I'll keep fixing EXTENSION building with VPATH. Comments ? -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation diff --git a/src/makefiles/pgxs.mk b/src/makefiles/pgxs.mk index bbcfe04..bc01e0c 100644 --- a/src/makefiles/pgxs.mk +++ b/src/makefiles/pgxs.mk @@ -112,32 +112,44 @@ all: all-lib endif # MODULE_big -install: all installdirs -ifneq (,$(EXTENSION)) - $(INSTALL_DATA) $(addprefix $(srcdir)/, $(addsuffix .control, $(EXTENSION))) '$(DESTDIR)$(datadir)/extension/' -endif # EXTENSION -ifneq (,$(DATA)$(DATA_built)) - $(INSTALL_DATA) $(addprefix $(srcdir)/, $(DATA)) $(DATA_built) '$(DESTDIR)$(datadir)/$(datamoduledir)/' -endif # DATA -ifneq (,$(DATA_TSEARCH)) - $(INSTALL_DATA) $(addprefix $(srcdir)/, $(DATA_TSEARCH)) '$(DESTDIR)$(datadir)/tsearch_data/' -endif # DATA_TSEARCH +install: all installdirs installcontrol installdata installdatatsearch installdocs installscript installscriptbuilt ifdef MODULES $(INSTALL_SHLIB) $(addsuffix $(DLSUFFIX), $(MODULES)) '$(DESTDIR)$(pkglibdir)/' endif # MODULES +ifdef PROGRAM + $(INSTALL_PROGRAM) $(PROGRAM)$(X) '$(DESTDIR)$(bindir)' +endif # PROGRAM + +installcontrol: $(addsuffix .control, $(EXTENSION)) +ifneq (,$(EXTENSION)) + $(INSTALL_DATA) $ '$(DESTDIR)$(datadir)/extension/' +endif + +installdata: $(DATA) $(DATA_built) +ifneq (,$(DATA)$(DATA_built)) + $(INSTALL_DATA) $^ '$(DESTDIR)$(datadir)/$(datamoduledir)/' +endif + +installdatatsearch: $(DATA_TSEARCH) +ifneq (,$(DATA_TSEARCH)) + $(INSTALL_DATA) $^ '$(DESTDIR)$(datadir)/tsearch_data/' +endif + +installdocs: $(DOCS) ifdef DOCS ifdef docdir - $(INSTALL_DATA) $(addprefix $(srcdir)/, $(DOCS)) '$(DESTDIR)$(docdir)/$(docmoduledir)/' + $(INSTALL_DATA) $^ '$(DESTDIR)$(docdir)/$(docmoduledir)/' endif # docdir endif # DOCS -ifdef PROGRAM - $(INSTALL_PROGRAM) $(PROGRAM)$(X) '$(DESTDIR)$(bindir)' -endif # PROGRAM + +installscript: $(SCRIPTS) ifdef SCRIPTS - $(INSTALL_SCRIPT) $(addprefix $(srcdir)/, $(SCRIPTS)) '$(DESTDIR)$(bindir)/' + $(INSTALL_SCRIPT) $^ '$(DESTDIR)$(bindir)/' endif # SCRIPTS -ifdef SCRIPTS_built - $(INSTALL_SCRIPT) $(SCRIPTS_built) '$(DESTDIR)$(bindir)/' + +installscriptbuilt: $(SCRIPTS_built) +ifdef SCRIPTS + $(INSTALL_SCRIPT) $^ '$(DESTDIR)$(bindir)/' endif # SCRIPTS_built ifdef MODULE_big signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Andrew Dunstan and...@dunslane.net writes: DATA_built = sql/$(EXTENSION)--$(EXTVERSION).sql DATA = $(filter-out sql/$(EXTENSION)--$(EXTVERSION).sql, $(wildcard sql/*--*.sql)) Is that right? I think that's still breaking VPATH builds because the widlcard call happens in the current tree, not in the VPATH (source) tree. And VPATH building of extension is crucially important for me, as the easiest way I've found to build and package a given extension against all currently supported version of PostgreSQL. I couldn't figure out how to allow for wildcard and whatnot in the extensions Makefile and have it work in all cases, including VPATH, so my current advice is to get rid of any dynamic code here and just plain copy and paste the file names you're interested into. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On 05/16/2013 05:41 AM, Dimitri Fontaine wrote: Andrew Dunstan and...@dunslane.net writes: DATA_built = sql/$(EXTENSION)--$(EXTVERSION).sql DATA = $(filter-out sql/$(EXTENSION)--$(EXTVERSION).sql, $(wildcard sql/*--*.sql)) Is that right? I think that's still breaking VPATH builds because the widlcard call happens in the current tree, not in the VPATH (source) tree. And VPATH building of extension is crucially important for me, as the easiest way I've found to build and package a given extension against all currently supported version of PostgreSQL. I couldn't figure out how to allow for wildcard and whatnot in the extensions Makefile and have it work in all cases, including VPATH, so my current advice is to get rid of any dynamic code here and just plain copy and paste the file names you're interested into. Is there documented support for VPATH builds? I know of an external script that promises such support, but AFAIK there is nothing in our code that supports it. If we're going to support this I suggest we do it properly. I don't consider myself on the hook to support some external build tool. As for supporting multiple versions, I have given up supporting multiple Postgres versions from a single set of sources, for FDWs at least. For those I use a git branch that mirrors the corresponding Postgres branch. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Andrew Dunstan and...@dunslane.net writes: On 05/16/2013 05:41 AM, Dimitri Fontaine wrote: And VPATH building of extension is crucially important for me, as the easiest way I've found to build and package a given extension against all currently supported version of PostgreSQL. Is there documented support for VPATH builds? The core code certainly builds okay in VPATH mode, and I would agree with Dimitri that pgxs builds should as well. But this is more of an autoconf+make feature than ours, so I'm not sure why you'd expect us to document it. Having said that, if a particular extension's makefile contains features that cause it to not build VPATH, then presumably that extension's author doesn't care. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Le jeudi 16 mai 2013 15:51:48, Tom Lane a écrit : Andrew Dunstan and...@dunslane.net writes: On 05/16/2013 05:41 AM, Dimitri Fontaine wrote: And VPATH building of extension is crucially important for me, as the easiest way I've found to build and package a given extension against all currently supported version of PostgreSQL. Is there documented support for VPATH builds? The core code certainly builds okay in VPATH mode, and I would agree with Dimitri that pgxs builds should as well. But this is more of an autoconf+make feature than ours, so I'm not sure why you'd expect us to document it. Extension does not support VPATH at 100% (well, pgxs.mk). There is a minor hack for some REGRESS files but that's all. I think at least DOCS, DATA and REGRESS needs some addition on pgxs.mk to help extension author allows build out-of-tree (postgresql been built out or not). I'll work on a patch for that. -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On 05/16/2013 10:39 AM, Cédric Villemain wrote: Le jeudi 16 mai 2013 15:51:48, Tom Lane a écrit : Andrew Dunstan and...@dunslane.net writes: On 05/16/2013 05:41 AM, Dimitri Fontaine wrote: And VPATH building of extension is crucially important for me, as the easiest way I've found to build and package a given extension against all currently supported version of PostgreSQL. Is there documented support for VPATH builds? The core code certainly builds okay in VPATH mode, and I would agree with Dimitri that pgxs builds should as well. But this is more of an autoconf+make feature than ours, so I'm not sure why you'd expect us to document it. Extension does not support VPATH at 100% (well, pgxs.mk). There is a minor hack for some REGRESS files but that's all. Right. My impression is that pgxs.mk actively works against vpath builds. I think at least DOCS, DATA and REGRESS needs some addition on pgxs.mk to help extension author allows build out-of-tree (postgresql been built out or not). I'll work on a patch for that. OK, great. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Andrew Dunstan wrote: On 05/16/2013 10:39 AM, Cédric Villemain wrote: Le jeudi 16 mai 2013 15:51:48, Tom Lane a écrit : Andrew Dunstan and...@dunslane.net writes: On 05/16/2013 05:41 AM, Dimitri Fontaine wrote: And VPATH building of extension is crucially important for me, as the easiest way I've found to build and package a given extension against all currently supported version of PostgreSQL. Is there documented support for VPATH builds? The core code certainly builds okay in VPATH mode, and I would agree with Dimitri that pgxs builds should as well. But this is more of an autoconf+make feature than ours, so I'm not sure why you'd expect us to document it. Extension does not support VPATH at 100% (well, pgxs.mk). There is a minor hack for some REGRESS files but that's all. Right. My impression is that pgxs.mk actively works against vpath builds. That's my experience, yes. It would be great to get it fixed. Then there's the outright weird stuff using ancient makefiles .. *grumble* pg_filedump *grumble* -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Alvaro Herrera alvhe...@2ndquadrant.com writes: Then there's the outright weird stuff using ancient makefiles .. *grumble* pg_filedump *grumble* I've never made any effort to improve the original makefile for that. Want to send a patch? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Tom Lane wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: Then there's the outright weird stuff using ancient makefiles .. *grumble* pg_filedump *grumble* I've never made any effort to improve the original makefile for that. Want to send a patch? Not right away, but I will get to it eventually, thanks. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On Tue, May 14, 2013 at 10:30 PM, Stephen Frost sfr...@snowman.net wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: I still think we should revert 9db7ccae2000524b72a4052352cbb5407fb53b02. The argument that the system-provided program might be faster carries very little weight for me --- make install is fast enough already. It's not worth making a bunch of extension authors jump through hoops, whether their style was bad or not. +1. +1. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On 5/14/13 5:45 PM, Tom Lane wrote: We changed to using install-sh unconditionally back in 2001 because we had too many problems with system-provided scripts that didn't do what we expected. I see very little reason to believe that the compatibility problems have disappeared since then, and in fact this complaint seems to me to be sufficient to refute that thesis. The compatibility issues in 2001 were completely different ones and were explicitly resolved with a new configure check (which is used worldwide by thousands of packages, note). Let's not confuse the issue. The argument that the system-provided program might be faster carries very little weight for me --- make install is fast enough already. It is not for me. Note also that make install is part of test runs. It's not worth making a bunch of extension authors jump through hoops, whether their style was bad or not. Well, I consider that this is not a style issue. It's an issue of wide-spread bugginess caused by uninformed copy-and-paste, and I'm glad we found it. Considering the widespread crappiness in PostgreSQL extension makefiles, I don't consider it a problem that a few things need to be fixed. That said, I'm obviously outnumbered here. What about the following compromise: Use the configure-selected install program inside PostgreSQL (which we can test easily), and use install-sh under USE_PGXS? Admittedly, the make install time of extensions is probably not an issue. (The affected extensions will still be buggy because users can still substitute their own install programs. We're just hiding the issue for a while.) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On 5/14/13 10:38 AM, Cédric Villemain wrote: If I follow your example, then I would rewrite http://manager.pgxn.org/howto Oh that's where that is coming from? Well that example has all kinds of problems. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Peter Eisentraut pete...@gmx.net writes: That said, I'm obviously outnumbered here. What about the following compromise: Use the configure-selected install program inside PostgreSQL (which we can test easily), and use install-sh under USE_PGXS? Admittedly, the make install time of extensions is probably not an issue. That works for me, since as you say we can easily fix any such bugs in the core code. The scary thing about this for extension authors is that they may very well see no bug in their own testing, only to have their packages fall over in the wild. We shouldn't make each author who's copied that code rediscover the problem for themselves that expensively. (The affected extensions will still be buggy because users can still substitute their own install programs. We're just hiding the issue for a while.) I'm not following this argument. The old makefile coding explicitly set INSTALL = $(SHELL) $(top_srcdir)/config/install-sh -c The only way I can see that that would be overridden is an explicit make INSTALL=/my/random/script install and surely we cannot guarantee that things don't break when you do something like that. There are hundreds of ways you can break the build if you mess with makefile variables. So I'm not prepared to call it a bug if that doesn't work. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Le mardi 14 mai 2013 15:08:42, Andrew Dunstan a écrit : On 05/14/2013 07:59 AM, Peter Eisentraut wrote: On 5/14/13 4:17 AM, Marti Raudsepp wrote: On Tue, May 14, 2013 at 5:27 AM, Peter Eisentraut pete...@gmx.net wrote: On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote: It's caused by this common pattern in extension makefiles: DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql What is the point of this? Why have the wildcard and then the non-wildcard term? Because the non-wildcard file is built by the same Makefile (it's copied from the sql/$(EXTENSION).sql file). If it wasn't there, a make install from a clean checkout would miss this file. If it's built, then it should be listed in DATA_built. So, AIUI, leaving aside stylistic arguments about use of wildcards, one solution could be: DATA_built = sql/$(EXTENSION)--$(EXTVERSION).sql DATA = $(filter-out sql/$(EXTENSION)--$(EXTVERSION).sql, $(wildcard sql/*--*.sql)) Is that right? Yes. -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On 05/15/2013 10:05 AM, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: That said, I'm obviously outnumbered here. What about the following compromise: Use the configure-selected install program inside PostgreSQL (which we can test easily), and use install-sh under USE_PGXS? Admittedly, the make install time of extensions is probably not an issue. That works for me, since as you say we can easily fix any such bugs in the core code. The scary thing about this for extension authors is that they may very well see no bug in their own testing, only to have their packages fall over in the wild. We shouldn't make each author who's copied that code rediscover the problem for themselves that expensively. +1, although I will be renovating the Makefiles for all my extensions along the lines of my previous email. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On Wed, May 15, 2013 at 09:51:15AM -0400, Peter Eisentraut wrote: On 5/14/13 10:38 AM, Cédric Villemain wrote: If I follow your example, then I would rewrite http://manager.pgxn.org/howto Oh that's where that is coming from? Well that example has all kinds of problems. Would you be so kind as to point out same, or better still, to write up what you think of as a better example intended for the same audience? I'm sure the PGXN people would be delighted to put something better up there. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Le mercredi 15 mai 2013 16:43:17, Andrew Dunstan a écrit : On 05/15/2013 10:05 AM, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: That said, I'm obviously outnumbered here. What about the following compromise: Use the configure-selected install program inside PostgreSQL (which we can test easily), and use install-sh under USE_PGXS? Admittedly, the make install time of extensions is probably not an issue. That works for me, since as you say we can easily fix any such bugs in the core code. The scary thing about this for extension authors is that they may very well see no bug in their own testing, only to have their packages fall over in the wild. We shouldn't make each author who's copied that code rediscover the problem for themselves that expensively. +1, although I will be renovating the Makefiles for all my extensions along the lines of my previous email. pgxs.mk has some old rules to replace $libdir (and some few other maybe). Maybe we can try to find what make sense for most of the extension authors and add rules/target to pgxs.mk to reduce the useless copy/paste in the Makefile of extensions. So what's desirable ? -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On Tue, May 14, 2013 at 5:27 AM, Peter Eisentraut pete...@gmx.net wrote: On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote: It's caused by this common pattern in extension makefiles: DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql What is the point of this? Why have the wildcard and then the non-wildcard term? Because the non-wildcard file is built by the same Makefile (it's copied from the sql/$(EXTENSION).sql file). If it wasn't there, a make install from a clean checkout would miss this file. all: sql/$(EXTENSION)--$(EXTVERSION).sql sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql cp $ $@ I think using wildcard is bad style and you should fix it. Perhaps, but fixing the extensions is not a solution at this point. A large number of extensions use this exact code (it comes from David Wheeler's template AFAIK). We might stand a chance in fixing the public extensions on PGXN, but this would presumably still break non-public extensions that people have written. For example: David Wheeler's pgTAP https://github.com/theory/pgTAP/blob/master/Makefile#L91 Jan Urbański's first_last_agg https://github.com/wulczer/first_last_agg/blob/master/Makefile Andrew Dunstan's json_build https://github.com/pgexperts/json_build/blob/master/Makefile Theo Schlossnagle's pg_amqp http://api.pgxn.org/src/pg_amqp/pg_amqp-0.3.0/Makefile OmniTI's pg_jobmon https://github.com/omniti-labs/pg_jobmon/blob/master/Makefile Regards, Marti -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Le mardi 14 mai 2013 10:17:13, Marti Raudsepp a écrit : On Tue, May 14, 2013 at 5:27 AM, Peter Eisentraut pete...@gmx.net wrote: On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote: It's caused by this common pattern in extension makefiles: DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql What is the point of this? Why have the wildcard and then the non-wildcard term? Because the non-wildcard file is built by the same Makefile (it's copied from the sql/$(EXTENSION).sql file). If it wasn't there, a make install from a clean checkout would miss this file. all: sql/$(EXTENSION)--$(EXTVERSION).sql sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql cp $ $@ I think using wildcard is bad style and you should fix it. Perhaps, but fixing the extensions is not a solution at this point. A large number of extensions use this exact code (it comes from David Wheeler's template AFAIK). We might stand a chance in fixing the public extensions on PGXN, but this would presumably still break non-public extensions that people have written. My understanding is that the offending commit reveals possible bad written instruction in some Makefile. What's wrong ? -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Marti Raudsepp ma...@juffo.org writes: all: sql/$(EXTENSION)--$(EXTVERSION).sql sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql cp $ $@ That's a recipe for problems. Each time I meet with such a construct in an extension's Makefile I propose an hard coded alternative. We're speaking of maintaining less than half a dozen file names here, with one or two additions per year, in most cases. Really, that trick is a recipe for problems. Use at your own risk. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support PS: yes I've been saying that all along, ever since David first cooked that, and I've blogged about it, and I've worked on alternatives and tools, in particular pg_buildext, part of the postgresql-common debian package. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On 5/13/13 10:58 PM, Andrew Dunstan wrote: I'm not sure why the wildcard is a bad idea - don't we want any present update sql files to be installed? Generally, wildcards in makefiles are a bad idea because you will then not discover any broken tarballs and checkouts. The users will just install nothing or a subset of the files you had intended. It might also break vpath builds. That can probably be worked around, though. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On 5/14/13 4:17 AM, Marti Raudsepp wrote: On Tue, May 14, 2013 at 5:27 AM, Peter Eisentraut pete...@gmx.net wrote: On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote: It's caused by this common pattern in extension makefiles: DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql What is the point of this? Why have the wildcard and then the non-wildcard term? Because the non-wildcard file is built by the same Makefile (it's copied from the sql/$(EXTENSION).sql file). If it wasn't there, a make install from a clean checkout would miss this file. If it's built, then it should be listed in DATA_built. Perhaps, but fixing the extensions is not a solution at this point. A large number of extensions use this exact code (it comes from David Wheeler's template AFAIK). So far, the number is still less than the number of extensions broken by the htup header refactoring, so I'm not worried about it. Also, substituting a custom install program has always been supported, so this was already broken anyway, now we just know about it earlier. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On 5/14/13 7:50 AM, Dimitri Fontaine wrote: Marti Raudsepp ma...@juffo.org writes: all: sql/$(EXTENSION)--$(EXTVERSION).sql sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql cp $ $@ That's a recipe for problems. Each time I meet with such a construct in an extension's Makefile I propose an hard coded alternative. We're speaking of maintaining less than half a dozen file names here, with one or two additions per year, in most cases. I don't think this in particular is a problem. I use something like that myself. But you should list all the older extension upgrade files explicitly (to which your point applies). -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On 05/14/2013 07:59 AM, Peter Eisentraut wrote: On 5/14/13 4:17 AM, Marti Raudsepp wrote: On Tue, May 14, 2013 at 5:27 AM, Peter Eisentraut pete...@gmx.net wrote: On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote: It's caused by this common pattern in extension makefiles: DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql What is the point of this? Why have the wildcard and then the non-wildcard term? Because the non-wildcard file is built by the same Makefile (it's copied from the sql/$(EXTENSION).sql file). If it wasn't there, a make install from a clean checkout would miss this file. If it's built, then it should be listed in DATA_built. So, AIUI, leaving aside stylistic arguments about use of wildcards, one solution could be: DATA_built = sql/$(EXTENSION)--$(EXTVERSION).sql DATA = $(filter-out sql/$(EXTENSION)--$(EXTVERSION).sql, $(wildcard sql/*--*.sql)) Is that right? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Le mardi 14 mai 2013 10:17:13, Marti Raudsepp a écrit : On Tue, May 14, 2013 at 5:27 AM, Peter Eisentraut pete...@gmx.net wrote: On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote: It's caused by this common pattern in extension makefiles: DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql What is the point of this? Why have the wildcard and then the non-wildcard term? Because the non-wildcard file is built by the same Makefile (it's copied from the sql/$(EXTENSION).sql file). If it wasn't there, a make install from a clean checkout would miss this file. all: sql/$(EXTENSION)--$(EXTVERSION).sql sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql cp $ $@ There is also something else here, the addition of the all: target. I believe it should be removed and the added target(s) be written after the include of pgxs makefile. If I follow your example, then I would rewrite http://manager.pgxn.org/howto From = PG91 = $(shell $(PG_CONFIG) --version | grep -qE 8\.| 9\.0 echo no || echo yes) ifeq ($(PG91),yes) all: sql/$(EXTENSION)--$(EXTVERSION).sql sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql cp $ $@ DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql EXTRA_CLEAN = sql/$(EXTENSION)--$(EXTVERSION).sql endif PGXS := $(shell $(PG_CONFIG) --pgxs) include $(PGXS) To PG91 = $(shell $(PG_CONFIG) --version | grep -qE 8\.| 9\.0 echo no || echo yes) ifeq ($(PG91),yes) DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql EXTRA_CLEAN = sql/$(EXTENSION)--$(EXTVERSION).sql endif PGXS := $(shell $(PG_CONFIG) --pgxs) include $(PGXS) ifeq ($(PG91),yes) sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql cp $ $@ endif because the default target from the PostgreSQL Makefile is «all:» and the addition of a target before inclusion of PostgreSQL makefile change the default (see DEFAULT_GOAL variable) Thought ? -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On Tue, May 14, 2013 at 2:59 PM, Peter Eisentraut pete...@gmx.net wrote: Perhaps, but fixing the extensions is not a solution at this point. A large number of extensions use this exact code (it comes from David Wheeler's template AFAIK). So far, the number is still less than the number of extensions broken by the htup header refactoring, so I'm not worried about it. I think that's fair. The C API has always been in flux and there are no stability guarantees. We expect C extension writers to know what they're doing. But I think we should be more forgiving to extensions written in stable languages like PL/pgSQL, especially if they break *because* the writer was following PGXN packaging best practices (http://manager.pgxn.org/howto) and not a fault of their own. I did a quick and dirty survey of extensions on PGXN and found that the install change causes problems for (at least) 22% of extensions there. I think it's well worth the time to implement a workaround, rather than hassle extension writers. --- I downloaded and tried to build the latest version of each package on PGXN (103 total). Now obviously I can't build everything because I don't have all the necessary dependencies -- which means it's probably skewed against C extensions. So don't take these results too seriously. But a it's data point regardless: 9.2 9.3beta1 70 61 -- make succeeds 65 33 -- make install succeeds 0 23 -- outputs install: will not overwrite just-created 07 -- outputs implicit declaration of function 'heap_ Note that the htup change causes just a compile-time warning (it will fail at extension load time), so some of these are counted as successful builds. Script and some instructions to replicate this test are attached. Regards, Marti build.sh Description: Bourne shell script -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Marti Raudsepp ma...@juffo.org writes: I did a quick and dirty survey of extensions on PGXN and found that the install change causes problems for (at least) 22% of extensions there. I think it's well worth the time to implement a workaround, rather than hassle extension writers. What's really worrying me about this is that beta1 has been out for less than 48 hours and we've already found one way in which a lot of system-provided install scripts vary from the one we provide. How many more compatibility problems will there be? We changed to using install-sh unconditionally back in 2001 because we had too many problems with system-provided scripts that didn't do what we expected. I see very little reason to believe that the compatibility problems have disappeared since then, and in fact this complaint seems to me to be sufficient to refute that thesis. I still think we should revert 9db7ccae2000524b72a4052352cbb5407fb53b02. The argument that the system-provided program might be faster carries very little weight for me --- make install is fast enough already. It's not worth making a bunch of extension authors jump through hoops, whether their style was bad or not. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
* Tom Lane (t...@sss.pgh.pa.us) wrote: I still think we should revert 9db7ccae2000524b72a4052352cbb5407fb53b02. The argument that the system-provided program might be faster carries very little weight for me --- make install is fast enough already. It's not worth making a bunch of extension authors jump through hoops, whether their style was bad or not. +1. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
Marti Raudsepp ma...@juffo.org writes: While testing out PostgreSQL 9.3beta1, I stumbled upon a problem installing some extensions (pgTAP and semver among others): ... I traced the problem down to commit 9db7ccae2000524b72a4052352cbb5407fb53b02 Use system install program when available and usable. It turns out that 'install' from coreutils 8.21 complains when it's told to install the same source file twice. TBH, I thought that was a dangerous idea from the get-go. My vote for fixing this would be to revert that change, not try to hack all the makefiles to work around it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote: It's caused by this common pattern in extension makefiles: DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql What is the point of this? Why have the wildcard and then the non-wildcard term? I think using wildcard is bad style and you should fix it. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 9.3 beta breaks some extensions make install
On 05/13/2013 10:27 PM, Peter Eisentraut wrote: On Tue, 2013-05-14 at 04:12 +0300, Marti Raudsepp wrote: It's caused by this common pattern in extension makefiles: DATA = $(wildcard sql/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql What is the point of this? Why have the wildcard and then the non-wildcard term? I think using wildcard is bad style and you should fix it. I've been caught by it. I'm not sure why the wildcard is a bad idea - don't we want any present update sql files to be installed? . I don't see any reason to have the redundant addition of the current version's sql file, though. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
