Re: faster testing with symlink installs

[Michael Paquier]

On Thu, Mar 22, 2018 at 11:07:48AM -0400, Robert Haas wrote:
> I suppose we could provide a build-time option to change this behavior.

it seems to me that it is going to be hard to reach a consensus by the
end of the commit fest, so I am marking this patch as returned with
feedback.  Not sure if that's worth it either..
--
Michael


signature.asc
Description: PGP signature

Re: faster testing with symlink installs

[Robert Haas]

On Wed, Mar 21, 2018 at 11:43 PM, Michael Paquier  wrote:
> On Wed, Mar 07, 2018 at 05:06:59PM -0500, Robert Haas wrote:
>> TBH I find that Homebrew example pretty odd.  I would understand
>> installing each major release in a version directory, but putting
>> every point release in a different versioned directory seems like a
>> bad plan.
>
> That's a project policy on their side.  The version of all components is
> kept around this way to give users the easier possibility to link or use
> back a previous version if a component update does wrong.  There are
> advantages to such models as well for projects which care a lot about
> some precise compatibility that some upstream maintainer overlooked,
> while not having to patch an existing instance to filter out only a set
> of components.

Well, under that definition, the behavior Peter is complaining about
upthread is a feature, not a bug.  If you want a separate install of
PostgreSQL for every minor release, you should have a separate version
of each other piece of software you build against it.  Or so it seems
to me, anyway.

I suppose we could provide a build-time option to change this behavior.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: faster testing with symlink installs

[Tom Lane]

Michael Paquier  writes:
> The last complain on the matter I can find actually involves the same
> people as this thread :)
> https://www.postgresql.org/message-id/54DE457F.2090206%40gmx.net

> So the patch should be marked as rejected or at least returned with
> feedback?

IMO, changing this behavior is more likely to draw complaints
than praise.

regards, tom lane

Re: faster testing with symlink installs

[Michael Paquier]

On Wed, Mar 07, 2018 at 05:06:59PM -0500, Robert Haas wrote:
> TBH I find that Homebrew example pretty odd.  I would understand
> installing each major release in a version directory, but putting
> every point release in a different versioned directory seems like a
> bad plan. 

That's a project policy on their side.  The version of all components is
kept around this way to give users the easier possibility to link or use
back a previous version if a component update does wrong.  There are
advantages to such models as well for projects which care a lot about
some precise compatibility that some upstream maintainer overlooked,
while not having to patch an existing instance to filter out only a set
of components.
--
Michael


signature.asc
Description: PGP signature

Re: faster testing with symlink installs

[Robert Haas]

On Wed, Feb 28, 2018 at 9:34 PM, Peter Eisentraut
 wrote:
> Except ... this doesn't actually work.  find_my_exec() resolves symlinks
> to find the actual program installation, and so for example the
> installed initdb will look for postgres in src/bin/initdb/.  I would
> like to revert this behavior, because it seems to do more harm than
> good.  The original commit message indicates that this would allow
> symlinking a program to somewhere outside of the installation tree and
> still allow it to work and find its friends.  But that could just as
> well be done with a shell script.
>
> Reverting this behavior would also allow Homebrew-like installations to
> work better.  The actual installation is in a versioned directory like
> /usr/local/Cellar/postgresql/10.1/... but symlinked to
> /usr/local/opt/postgresql/.  But because of the symlink resolution,
> calling /usr/local/opt/postgresql/bin/pg_config returns paths under
> Cellar, which then causes breakage of software built against that path
> on postgresql upgrades the change the version number.

My initial gut feeling is that changing this would hurt more people
than it would help.  It seems entirely reasonable to install
PostgreSQL in, say, /opt/local/postgresql, and then symlink psql and
pg_ctl into /opt/local/bin or /usr/bin or wherever you like to find
your binaries.  I don't think we want to break that.  It's true that
it will work if you symlink everything, as in your Homebrew example,
but you might not.  TBH I find that Homebrew example pretty odd.  I
would understand installing each major release in a version directory,
but putting every point release in a different versioned directory
seems like a bad plan.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

faster testing with symlink installs

[Peter Eisentraut]

I'm proposing a way to make test runs a bit faster.

A fair amount of time is taken by creating the test installation.  Not
huge compared to the whole test run, but still long enough to get
boring.  The idea (that I stole from somewhere else) is that we make the
installation as symlinks pointing back to the source tree.  So after the
first run, the installation is automatically up to date after each
build.  Then we can skip the whole temp install run if we find that we
already have one (unless you change the set of installed files in a
major way, which is very rate).  This also helps if you use make
installcheck: You just need to rebuild and restart the server, skipping
the make install run.

So that's what the patch does: make install INSTALL_SYMLINKS=1 installs
files by making relative symlinks.  make check TEMP_INSTALL_SYMLINKS=1
creates the temporary installation using symlinks and skips the install
step if a tmp_install already exists.  It requires the "ln" program from
GNU coreutils.  So it would be optional.

Except ... this doesn't actually work.  find_my_exec() resolves symlinks
to find the actual program installation, and so for example the
installed initdb will look for postgres in src/bin/initdb/.  I would
like to revert this behavior, because it seems to do more harm than
good.  The original commit message indicates that this would allow
symlinking a program to somewhere outside of the installation tree and
still allow it to work and find its friends.  But that could just as
well be done with a shell script.

Reverting this behavior would also allow Homebrew-like installations to
work better.  The actual installation is in a versioned directory like
/usr/local/Cellar/postgresql/10.1/... but symlinked to
/usr/local/opt/postgresql/.  But because of the symlink resolution,
calling /usr/local/opt/postgresql/bin/pg_config returns paths under
Cellar, which then causes breakage of software built against that path
on postgresql upgrades the change the version number.

Thoughts?

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
From f101d14195d5bc26f09a84e823f06b6c422fe444 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut 
Date: Sun, 28 Jan 2018 12:24:14 -0500
Subject: [PATCH 1/3] Remove duplicate installation of some header files

In non-vpath builds, some header files such as pg_config.h were actually
installed twice (to the same target location).  Rearrange the make rules
a bit to avoid that.
---
 src/include/Makefile | 11 +++
 1 file changed, 3 insertions(+), 8 deletions(-)

diff --git a/src/include/Makefile b/src/include/Makefile
index a689d352b6..2d1baca04a 100644
--- a/src/include/Makefile
+++ b/src/include/Makefile
@@ -39,13 +39,7 @@ install: all installdirs
$(INSTALL_DATA) $(srcdir)/port.h 
'$(DESTDIR)$(includedir_internal)'
$(INSTALL_DATA) $(srcdir)/postgres_fe.h  
'$(DESTDIR)$(includedir_internal)'
$(INSTALL_DATA) $(srcdir)/libpq/pqcomm.h 
'$(DESTDIR)$(includedir_internal)/libpq'
-# These headers are needed for server-side development
-   $(INSTALL_DATA) pg_config.h '$(DESTDIR)$(includedir_server)'
-   $(INSTALL_DATA) pg_config_ext.h '$(DESTDIR)$(includedir_server)'
-   $(INSTALL_DATA) pg_config_os.h  '$(DESTDIR)$(includedir_server)'
-   $(INSTALL_DATA) utils/errcodes.h '$(DESTDIR)$(includedir_server)/utils'
-   $(INSTALL_DATA) utils/fmgroids.h '$(DESTDIR)$(includedir_server)/utils'
-   $(INSTALL_DATA) utils/fmgrprotos.h 
'$(DESTDIR)$(includedir_server)/utils'
+# Headers for server-side development
 # We don't use INSTALL_DATA for performance reasons --- there are a lot of 
files
cp $(srcdir)/*.h '$(DESTDIR)$(includedir_server)'/ || exit; \
chmod $(INSTALL_DATA_MODE) '$(DESTDIR)$(includedir_server)'/*.h  || 
exit; \
@@ -54,7 +48,8 @@ install: all installdirs
  chmod $(INSTALL_DATA_MODE) '$(DESTDIR)$(includedir_server)'/$$dir/*.h 
 || exit; \
done
 ifeq ($(vpath_build),yes)
-   for file in dynloader.h catalog/schemapg.h parser/gram.h 
storage/lwlocknames.h utils/probes.h; do \
+   for file in pg_config.h pg_config_ext.h pg_config_os.h utils/errcodes.h 
utils/fmgroids.h utils/fmgrprotos.h \
+ dynloader.h catalog/schemapg.h parser/gram.h storage/lwlocknames.h 
utils/probes.h; do \
  cp $$file '$(DESTDIR)$(includedir_server)'/$$file || exit; \
  chmod $(INSTALL_DATA_MODE) '$(DESTDIR)$(includedir_server)'/$$file || 
exit; \
done

base-commit: 51057feaa6bd24b51e6a4715c2090491ef037534
-- 
2.16.2

From afc5738edc445f577d3b7da5503ff61d1debe953 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut 
Date: Sun, 28 Jan 2018 12:24:37 -0500
Subject: [PATCH 2/3] Don't resolve symlinks in find_my_exec()

This reverts 336969e490d71c316a42fabeccda87f798e562dd.  That feature
allowed symlinking a binary to a location outside of the tree and still
be able to find the