Re: [PATCHES] Cleaning up multiply-defined-symbol warnings on OS X
On Tue, May 09, 2006 at 09:18:17AM -0400, Tom Lane wrote:
> Martijn van Oosterhout writes:
> > Eh? It stops a program expecting libpq4 being linked to libpq3 for any
> > reason, so the above situation can't happen. You don't need to version
> > any structs, only the functions using them.
>
> If we have an existing app built against an unversioned libpq, what
> happens if we roll in a versioned libpq? Does it break completely?
> That is, is the introduction of versioning by itself an incompatible
> ABI break?
This is a special rules:
- If a program (like psql) requires a symbol but specifies no symbol
version, it will match against that symbol without a version, or if not
found, the symbol with the lowest version number.
So if you slide a versioned library under a program that doesn't know
about it, it will load fine because unversioned symbols match anything.
Here is a quick example of how it works:
$ cat prog.c
extern int func();
int main()
{
return func();
}
$ cat lib.c
int func()
{
return 1;
}
$ cat ver_script
VERSION_1 { global: *; };
### Create the various versions ###
$ gcc -shared lib.c -Wl,--version-script=ver_script -Wl,-soname=liblib.so -o
liblib_ver.so
$ gcc -shared lib.c -Wl,-soname=liblib.so -o liblib_nover.so
$ gcc -o prog_ver prog.c -L$PWD -llib_ver
$ gcc -o prog_nover prog.c -L$PWD -llib_nover
### Test ###
$ mkdir test
$ cp liblib_nover.so test/liblib.so
$ cp prog_nover test/prog
$ LD_LIBRARY_PATH=$PWD/test test/prog # Works (both no version)
$ cp liblib_nover.so test/liblib.so
$ cp prog_ver test/prog
$ LD_LIBRARY_PATH=$PWD/test test/prog # Doesn't work (prog with version,
lib without)
test/prog: /tmp/t/test/liblib.so: no version information available (required by
test/prog)
$ cp liblib_ver.so test/liblib.so
$ cp prog_ver test/prog
$ LD_LIBRARY_PATH=$PWD/test test/prog # Works (both with version)
$ cp liblib_ver.so test/liblib.so
$ cp prog_nover test/prog
$ LD_LIBRARY_PATH=$PWD/test test/prog # Works (prog without version, lib
with)
You can make that broken variation work too, if the version is marked
as weak, but I couldn't quickly find out how.
http://people.redhat.com/drepper/symbol-versioning
Hope this helps,
--
Martijn van Oosterhout http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to
> litigate.
signature.asc
Description: Digital signature
Re: [PATCHES] Cleaning up multiply-defined-symbol warnings on OS X
Martijn van Oosterhout writes: > Eh? It stops a program expecting libpq4 being linked to libpq3 for any > reason, so the above situation can't happen. You don't need to version > any structs, only the functions using them. If we have an existing app built against an unversioned libpq, what happens if we roll in a versioned libpq? Does it break completely? That is, is the introduction of versioning by itself an incompatible ABI break? regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Cleaning up multiply-defined-symbol warnings on OS X
On Tue, May 09, 2006 at 01:50:38PM +0200, Peter Eisentraut wrote: > Am Dienstag, 9. Mai 2006 10:41 schrieb Martijn van Oosterhout: > > Depends what you mean by signature. The structures of PGconn and > > PGresult have changed over time, so if you you pass a PGresult > > allocated by libpq4 to a function in libpq3, it'll crash. > > Symbol versioning only affects functions (and maybe variables?) but not > structs. Nothing helps in that case. Eh? It stops a program expecting libpq4 being linked to libpq3 for any reason, so the above situation can't happen. You don't need to version any structs, only the functions using them. Have a nice day, -- Martijn van Oosterhout http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to > litigate. signature.asc Description: Digital signature
Re: [PATCHES] Cleaning up multiply-defined-symbol warnings on OS X
Am Dienstag, 9. Mai 2006 10:41 schrieb Martijn van Oosterhout: > Depends what you mean by signature. The structures of PGconn and > PGresult have changed over time, so if you you pass a PGresult > allocated by libpq4 to a function in libpq3, it'll crash. Symbol versioning only affects functions (and maybe variables?) but not structs. Nothing helps in that case. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Cleaning up multiply-defined-symbol warnings on OS X
On Tue, May 09, 2006 at 10:19:56AM +0200, Peter Eisentraut wrote: > Am Samstag, 29. April 2006 21:27 schrieb Martijn van Oosterhout: > > What it does is remove the restriction that any one program can only > > use (directly or indirectly) one version of libpq at any moment. > > Programs can use indirectly postgres via PAM or NSS or other such > > pluggable interfaces. Currently PAM and NSS must be using the same > > version of libpq as any program that might call them or everything > > blows up. > > Symbol versioning only helps if a function's signature has changed, doesn't > it? Have we ever done that? Depends what you mean by signature. The structures of PGconn and PGresult have changed over time, so if you you pass a PGresult allocated by libpq4 to a function in libpq3, it'll crash. Symbol versioning can serve a few purposes: 1. You can have one library supporting several ABIs simultaneously. glibc does this but that's not what we're interested in. 2. When loading a library, the linker checks both the symbol name and the symbol version. Thus, it allows two versions of the same library to co-exist in the same address space simultaneously. This is what we want. Basically, if you symbol version, a program compiled against a particular version with only call functions in that version and won't be accedently misdirected to another version that happened to be loaded first... In the example I posted earlier, say psql starts, compiled against libpq5. It does a gethostbyname() that is redirected to an NSS module originally compiled against libpq4. The linker will load libpq4 but link the actual symbols of the NSS module to libpq5. Any it can't find there it will find in libpq4. It's this splitting up that will get you. Loading this way round is unlikely to cause many problems since we don't tend to remove functions from the API so it may not link any symbols to libpq4. The more dangerous situation is if the old library gets loaded first and something uses symbols that didn't exist in the old version but do in the new version. Have a nice day, -- Martijn van Oosterhout http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to > litigate. signature.asc Description: Digital signature
Re: [PATCHES] Cleaning up multiply-defined-symbol warnings on OS X
Am Samstag, 29. April 2006 21:27 schrieb Martijn van Oosterhout: > What it does is remove the restriction that any one program can only > use (directly or indirectly) one version of libpq at any moment. > Programs can use indirectly postgres via PAM or NSS or other such > pluggable interfaces. Currently PAM and NSS must be using the same > version of libpq as any program that might call them or everything > blows up. Symbol versioning only helps if a function's signature has changed, doesn't it? Have we ever done that? -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PATCHES] Cleaning up multiply-defined-symbol warnings on OS X
On Sat, Apr 29, 2006 at 03:07:30PM -0400, Tom Lane wrote: > Martijn van Oosterhout writes: > > Now that 95% of this patch [1] (currently in the hold queue) has been > > implemented, perhaps we should go the final 5% and version the symbols > > to fix the ambiguous symbol problem forever (on linux at least, though > > anywhere with the GCC toolchain will work). > > I was thinking about that, but came away unconvinced it's worth doing > considering we are not a Linux-only system. Is there any advantage > in having symbol versioning on Linux when we'll still have the same > API-change constraints everywhere else? What it does is remove the restriction that any one program can only use (directly or indirectly) one version of libpq at any moment. Programs can use indirectly postgres via PAM or NSS or other such pluggable interfaces. Currently PAM and NSS must be using the same version of libpq as any program that might call them or everything blows up. Versioning is basically (amongst other things) a way of ensuring that a program runs against the same version of the library it was compiled against. Just renaming a library won't confuse it for example. It's more a preventative measure than anything else. As postgres gets integrated into more things the chance of symbols getting mislinked increases. > I had actually forgotten about your earlier patch :-( I figured as much :) Have a nice day, -- Martijn van Oosterhout http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to > litigate. signature.asc Description: Digital signature
Re: [PATCHES] Cleaning up multiply-defined-symbol warnings on OS X
Martijn van Oosterhout writes: > Now that 95% of this patch [1] (currently in the hold queue) has been > implemented, perhaps we should go the final 5% and version the symbols > to fix the ambiguous symbol problem forever (on linux at least, though > anywhere with the GCC toolchain will work). I was thinking about that, but came away unconvinced it's worth doing considering we are not a Linux-only system. Is there any advantage in having symbol versioning on Linux when we'll still have the same API-change constraints everywhere else? I had actually forgotten about your earlier patch :-( regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Cleaning up multiply-defined-symbol warnings on OS X
Tom Lane wrote: > I just came across some documentation claiming that it's possible to do > similar exported-symbol filtering in Linux: > http://people.redhat.com/%7edrepper/dsohowto.pdf > I'm going to see if I can make that work along the same lines as my > Darwin patch. If we can get it going on both Linux and Darwin then > I think we'll have critical mass to make the change. Now that 95% of this patch [1] (currently in the hold queue) has been implemented, perhaps we should go the final 5% and version the symbols to fix the ambiguous symbol problem forever (on linux at least, though anywhere with the GCC toolchain will work). Have a nice day, [1] http://archives.postgresql.org/pgsql-patches/2005-10/msg00166.php -- Martijn van Oosterhout http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to > litigate. signature.asc Description: Digital signature
Re: [PATCHES] Cleaning up multiply-defined-symbol warnings on OS X
Bruce Momjian writes: > The problem is that the call to thread.c::pqGetpwuid() happens in a > #ifndef WIN32 block, so the function is not seen by Win32, so we don't > get a compile error. > I am thinking your patch is a good idea because it will give us a > non-Win32 platform that has the _check_ behavior. We do have to bump > the major for ecpg libs for this, but not libpq. I just came across some documentation claiming that it's possible to do similar exported-symbol filtering in Linux: http://people.redhat.com/%7edrepper/dsohowto.pdf I'm going to see if I can make that work along the same lines as my Darwin patch. If we can get it going on both Linux and Darwin then I think we'll have critical mass to make the change. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] Cleaning up multiply-defined-symbol warnings on OS X
Tom Lane wrote: > Bruce Momjian writes: > > I am thinking your patch is a good idea because it will give us a > > non-Win32 platform that has the _check_ behavior. We do have to bump > > the major for ecpg libs for this, but not libpq. > > No, because if 8.2 ships with a libpq.so.4 that doesn't expose > pqGetpwuid, it will look like pre-8.2 versions of libecpg will work > with that libpq. Which they won't. You can't remove exported > functions (whether the export was intentional or no) without a > major version bump. OK, right. -- Bruce Momjian http://candle.pha.pa.us EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] Cleaning up multiply-defined-symbol warnings on OS X
Bruce Momjian writes: > I am thinking your patch is a good idea because it will give us a > non-Win32 platform that has the _check_ behavior. We do have to bump > the major for ecpg libs for this, but not libpq. No, because if 8.2 ships with a libpq.so.4 that doesn't expose pqGetpwuid, it will look like pre-8.2 versions of libecpg will work with that libpq. Which they won't. You can't remove exported functions (whether the export was intentional or no) without a major version bump. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] Cleaning up multiply-defined-symbol warnings on OS X
We have two approaches for dealing with pgport leakage. For libraries, we remove libpgport from the link line and recompile our own object files: # Need to recompile any libpgport object files LIBS := $(filter-out -lpgport, $(LIBS)) OBJS= execute.o typename.o descriptor.o data.o error.o prepare.o memory.o \ connect.o misc.o path.o exec.o \ $(filter snprintf.o, $(LIBOBJS)) The problem is that there is no testing of which object files need to be added, but fortunately Win32 has the dllexport which makes libpq export _only_ functions we define, so if we forget something, it shows up as a Win32 link error. For applications, we have them pull symbols from libpgport first, and are not bound to specific libpgport functions being in libpq. Makefile.global has: # Force clients to pull symbols from the non-shared library libpgport # rather than pulling some libpgport symbols from libpq just because # libpq uses those functions too. This makes applications less # dependent on changes in libpq's usage of pgport. To do this we link to # pgport before libpq. This does cause duplicate -lpgport's to appear # on client link lines. ifdef PGXS libpq_pgport = -L$(libdir) -lpgport $(libpq) else libpq_pgport = -L$(top_builddir)/src/port -lpgport $(libpq) endif That is our _portable_ way to do it. So, technically we have everything covered, at least since 8.0.2. The problem is that the call to thread.c::pqGetpwuid() happens in a #ifndef WIN32 block, so the function is not seen by Win32, so we don't get a compile error. I am thinking your patch is a good idea because it will give us a non-Win32 platform that has the _check_ behavior. We do have to bump the major for ecpg libs for this, but not libpq. --- Tom Lane wrote: > Recent versions of Darwin spew a lot of multiply-defined-symbol warnings > while building Postgres, mostly because the programs in src/bin import > both libpq and libpgport, and libpq includes copies of many of the > libpgport files. Since the libpgport routines aren't officially part of > the libpq API, it'd be better if those symbols weren't visible from > outside libpq. The attached patch makes this so, using the already > existing exports.txt list as the definition of libpq's official API. > > I found while testing the patch that we have one API leak in current > sources: ecpglib is depending on the libpq-exported pqGetpwuid() > because src/port/path.c requires it but src/port/thread.c isn't included > into ecpglib. The patch corrects this oversight. This BTW is > sufficient reason for a libpq major version bump if we apply this patch; > we learned that lesson last time we fixed an API leak. (Did we already > bump libpq's major version for 8.2? I don't recall.) > > The reason I didn't go ahead and apply this is that I'm wondering if > there's some more-portable solution that would work for other platforms > besides Darwin. I seem to recall that we've discussed the point before, > but no patch has been forthcoming. > > Comments? > > regards, tom lane > Content-Description: darwin-symbols.patch > Index: src/Makefile.shlib > === > RCS file: /cvsroot/pgsql/src/Makefile.shlib,v > retrieving revision 1.103 > diff -c -r1.103 Makefile.shlib > *** src/Makefile.shlib19 Apr 2006 16:32:08 - 1.103 > --- src/Makefile.shlib20 Apr 2006 00:56:38 - > *** > *** 107,117 > ifeq ($(DLTYPE), library) > # linkable library > DLSUFFIX:= .dylib > ! LINK.shared = $(COMPILER) -dynamiclib -install_name > $(libdir)/lib$(NAME).$(SO_MAJOR_VERSION)$(DLSUFFIX) $(version_link) > -multiply_defined suppress > else > # loadable module (default case) > DLSUFFIX:= .so > ! LINK.shared = $(COMPILER) -bundle > endif > shlib = > lib$(NAME).$(SO_MAJOR_VERSION).$(SO_MINOR_VERSION)$(DLSUFFIX) > shlib_major = lib$(NAME).$(SO_MAJOR_VERSION)$(DLSUFFIX) > --- 107,117 > ifeq ($(DLTYPE), library) > # linkable library > DLSUFFIX:= .dylib > ! LINK.shared = $(COMPILER) -dynamiclib -install_name > $(libdir)/lib$(NAME).$(SO_MAJOR_VERSION)$(DLSUFFIX) $(version_link) > $(exported_symbols_list) -multiply_defined suppress > else > # loadable module (default case) > DLSUFFIX:= .so > ! LINK.shared = $(COMPILER) -bundle -multiply_defined suppress > endif > shlib = > lib$(NAME).$(SO_MAJOR_VERSION).$(SO_MINOR_VERSION)$(DLSUFFIX) > shlib_major = lib$(NAME).$(SO_MAJOR_VERSION)$(DLSUFFIX) > Index: src/interfaces/li
[PATCHES] Cleaning up multiply-defined-symbol warnings on OS X
Recent versions of Darwin spew a lot of multiply-defined-symbol warnings
while building Postgres, mostly because the programs in src/bin import
both libpq and libpgport, and libpq includes copies of many of the
libpgport files. Since the libpgport routines aren't officially part of
the libpq API, it'd be better if those symbols weren't visible from
outside libpq. The attached patch makes this so, using the already
existing exports.txt list as the definition of libpq's official API.
I found while testing the patch that we have one API leak in current
sources: ecpglib is depending on the libpq-exported pqGetpwuid()
because src/port/path.c requires it but src/port/thread.c isn't included
into ecpglib. The patch corrects this oversight. This BTW is
sufficient reason for a libpq major version bump if we apply this patch;
we learned that lesson last time we fixed an API leak. (Did we already
bump libpq's major version for 8.2? I don't recall.)
The reason I didn't go ahead and apply this is that I'm wondering if
there's some more-portable solution that would work for other platforms
besides Darwin. I seem to recall that we've discussed the point before,
but no patch has been forthcoming.
Comments?
regards, tom lane
Index: src/Makefile.shlib
===
RCS file: /cvsroot/pgsql/src/Makefile.shlib,v
retrieving revision 1.103
diff -c -r1.103 Makefile.shlib
*** src/Makefile.shlib 19 Apr 2006 16:32:08 - 1.103
--- src/Makefile.shlib 20 Apr 2006 00:56:38 -
***
*** 107,117
ifeq ($(DLTYPE), library)
# linkable library
DLSUFFIX := .dylib
! LINK.shared = $(COMPILER) -dynamiclib -install_name
$(libdir)/lib$(NAME).$(SO_MAJOR_VERSION)$(DLSUFFIX) $(version_link)
-multiply_defined suppress
else
# loadable module (default case)
DLSUFFIX := .so
! LINK.shared = $(COMPILER) -bundle
endif
shlib =
lib$(NAME).$(SO_MAJOR_VERSION).$(SO_MINOR_VERSION)$(DLSUFFIX)
shlib_major = lib$(NAME).$(SO_MAJOR_VERSION)$(DLSUFFIX)
--- 107,117
ifeq ($(DLTYPE), library)
# linkable library
DLSUFFIX := .dylib
! LINK.shared = $(COMPILER) -dynamiclib -install_name
$(libdir)/lib$(NAME).$(SO_MAJOR_VERSION)$(DLSUFFIX) $(version_link)
$(exported_symbols_list) -multiply_defined suppress
else
# loadable module (default case)
DLSUFFIX := .so
! LINK.shared = $(COMPILER) -bundle -multiply_defined suppress
endif
shlib =
lib$(NAME).$(SO_MAJOR_VERSION).$(SO_MINOR_VERSION)$(DLSUFFIX)
shlib_major = lib$(NAME).$(SO_MAJOR_VERSION)$(DLSUFFIX)
Index: src/interfaces/libpq/Makefile
===
RCS file: /cvsroot/pgsql/src/interfaces/libpq/Makefile,v
retrieving revision 1.143
diff -c -r1.143 Makefile
*** src/interfaces/libpq/Makefile 11 Apr 2006 20:26:40 - 1.143
--- src/interfaces/libpq/Makefile 20 Apr 2006 00:56:39 -
***
*** 125,130
--- 125,143
echo '; Aliases for MS compatible names' >> $@
sed -e '/^#/d' -e 's/^\(.* \)\([0-9][0-9]*\)/\1= _\1/' < $< | sed
's/ *$$//' >> $@
+ # On Darwin, restrict the symbols exported by the library to just the
+ # official list, so as to avoid multiply-defined-symbol warnings in
+ # programs that use libpgport along with libpq.
+
+ ifeq ($(PORTNAME), darwin)
+ $(shlib): exports.darwin
+
+ exports.darwin: exports.txt
+ $(AWK) '/^[^#]/ {printf "_%s\n",$$1}' $< >$@
+
+ exported_symbols_list = -exported_symbols_list exports.darwin
+ endif
+
# depend on Makefile.global to force rebuild on re-run of configure
$(srcdir)/libpq.rc: libpq.rc.in $(top_builddir)/src/Makefile.global
sed -e 's/\(VERSION.*\),0 *$$/\1,'`date '+%y%j' | sed 's/^0*//'`'/' <
$< > $@
***
*** 147,153
rm -f '$(DESTDIR)$(includedir)/libpq-fe.h'
'$(DESTDIR)$(includedir_internal)/libpq-int.h'
'$(DESTDIR)$(includedir_internal)/pqexpbuffer.h'
'$(DESTDIR)$(datadir)/pg_service.conf.sample'
clean distclean: clean-lib
! rm -f $(OBJS) pg_config_paths.h crypt.c getaddrinfo.c inet_aton.c
noblock.c pgstrcasecmp.c snprintf.c strerror.c open.c thread.c md5.c ip.c
encnames.c wchar.c pthread.h
rm -f pg_config_paths.h # Might be left over from a Win32 client-only
build
maintainer-clean: distclean
--- 160,166
rm -f '$(DESTDIR)$(includedir)/libpq-fe.h'
'$(DESTDIR)$(includedir_internal)/libpq-int.h'
'$(DESTDIR)$(includedir_internal)/pqexpbuffer.h'
'$(DESTDIR)$(datadir)/pg_service.conf.sample'
clean distclean: clean-lib
! rm -f $(OBJS) pg_config_paths.h crypt.c getaddrinfo.c inet_aton.c
noblock.c pgstrcasecmp.c snprintf.c strerror.c open.c thread.c md5.c ip.c
encnames.c wchar.c pthread.h expor
