Re: AIX shared libraries (was Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries)
Tom Lane wrote: Is it possible that the rules have changed across AIX versions, and that the code in there now is needful for older versions? I don't think that this behaviour has changed. I remember it from AIX 4.3.2. AIX 4.3 is the first version to support the -brtl. The current code is in place to mimic the behaviour of dlopen, etc, on the older platforms. I think we are at a point that we can stop maintaining AIX older than 4.3 if we want. -rocco ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
The patch did not work for me :-( My buildfarm members failed in local testing to execute the install-check, because initdb failed to find libpq.so. Make check did succeed, so I think there is a possibility of getting it working, but it won't be as simple as adding -brtl to the template. I was also getting duplicate symbols in the link, that I wasn't before either. I don't have time right now to work through all the issues, but wanted to give feedback that the patch isn't quite this simple. Thanks, -rocco -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Albe Laurenz Sent: Tuesday, September 12, 2006 9:01 AM To: Tom Lane *EXTERN* Cc: pgsql-hackers@postgresql.org Subject: Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries Tom Lane wrote: In our case, we have libpq.a and libpq.so in the same directory, so unless you link with -brtl you will get a static link (because libpq.a is a static library). I wonder whether we ought to suppress building (or at least installing) our .a libraries at all on AIX. Adding -btrl to LDFLAGS would help within the context of our own build, but external clients that link to libpq without saying that are going to get undesirable results. I think there's a reasonable argument that by installing a .a file that isn't a shared library, we are violating the platform's conventions. The natural way in AIX would be: - Create libpq.so - Create libpq.a by 'rm -f libpq.a; ar -rc libpq.a libpq.so' - Install only libpq.a For a static build on AIX, you have to specify all the libraries and give the linker -bstatic and -bI:/lib/syscalls.exp Should -brtl be added to src/template/aix? Sounds that way, but that'll only help for psql and other stuff built within our build. Could you try this against CVS tip: * add -brtl to LDFLAGS in the template * Remove the AIX-specific hack on $(libpq) at lines 349-354 of src/Makefile.global.in * see if it configures and builds I have done that (see the attached patch) and it works fine. I don't have the native AIX C compiler, so I could only test it with gcc. I have taken the liberty to modify the static link line in Makefile.global.in to contain the LDAP libraries, I hope that's appropriate. Yours, Laurenz Albe ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
Kevin Brown wrote: Let me expand a little on some of the peculiarities of shared libraries on AIX: - A normal AIX shared library is called libXX.a It is an 'ar' archive that contains the shared object(s). Ah, so the problem really boils down to funny naming conventions. If they use .a for both shared and static libraries, how does anyone tell the difference? It sounds to me like there is no difference. Notice how his example ldd output shows dependencies on specific .o entries within the various .a files that reside on the system, rather than on the .a files as a whole. If those entries had been statically linked then they wouldn't have shown up in the ldd output at all. That is not entirely true. The difference between a static and a shared library on AIX is that the *.o files in a dynamic library are dynamic objects, produced by the linker (what is called *.so in Linux), and the *.o files in a static library are the output of the compiler (what is called *.o in Linux). What IS true is that you can do a static build against a dynamic library. Against a static library you can only do static builds. Yours, Laurenz Albe ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
Tom Lane wrote: In our case, we have libpq.a and libpq.so in the same directory, so unless you link with -brtl you will get a static link (because libpq.a is a static library). I wonder whether we ought to suppress building (or at least installing) our .a libraries at all on AIX. Adding -btrl to LDFLAGS would help within the context of our own build, but external clients that link to libpq without saying that are going to get undesirable results. I think there's a reasonable argument that by installing a .a file that isn't a shared library, we are violating the platform's conventions. The natural way in AIX would be: - Create libpq.so - Create libpq.a by 'rm -f libpq.a; ar -rc libpq.a libpq.so' - Install only libpq.a For a static build on AIX, you have to specify all the libraries and give the linker -bstatic and -bI:/lib/syscalls.exp Should -brtl be added to src/template/aix? Sounds that way, but that'll only help for psql and other stuff built within our build. Could you try this against CVS tip: * add -brtl to LDFLAGS in the template * Remove the AIX-specific hack on $(libpq) at lines 349-354 of src/Makefile.global.in * see if it configures and builds I have done that (see the attached patch) and it works fine. I don't have the native AIX C compiler, so I could only test it with gcc. I have taken the liberty to modify the static link line in Makefile.global.in to contain the LDAP libraries, I hope that's appropriate. Yours, Laurenz Albe aix.link.patch Description: aix.link.patch ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
AIX shared libraries (was Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries)
Albe Laurenz [EMAIL PROTECTED] writes: Tom Lane wrote: I think there's a reasonable argument that by installing a .a file that isn't a shared library, we are violating the platform's conventions. The natural way in AIX would be: - Create libpq.so - Create libpq.a by 'rm -f libpq.a; ar -rc libpq.a libpq.so' - Install only libpq.a Hm. This seems possible with some moderate hacking on Makefile.shlib (certainly it'd be no more invasive than the existing Windows-specific platform variants). However, looking at what's already in Makefile.shlib for AIX makes me doubt the above claim a bit, because AFAICS libpq.so is produced from libpq.a on that platform. Is it possible that the rules have changed across AIX versions, and that the code in there now is needful for older versions? Another issue with installing only .a is that there's no provision for versioning in .a library names ... what happens to someone who needs two generations of libpq on his machine? regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: AIX shared libraries (was Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries)
Tom Lane wrote: The natural way in AIX would be: - Create libpq.so - Create libpq.a by 'rm -f libpq.a; ar -rc libpq.a libpq.so' - Install only libpq.a Hm. This seems possible with some moderate hacking on Makefile.shlib (certainly it'd be no more invasive than the existing Windows-specific platform variants). However, looking at what's already in Makefile.shlib for AIX makes me doubt the above claim a bit, because AFAICS libpq.so is produced from libpq.a on that platform. Is it possible that the rules have changed across AIX versions, and that the code in there now is needful for older versions? I don't think that this behaviour has changed. I remember it from AIX 4.3.2. Of course libpq.so is created from (the static) libpq.a. But once you have the dynamic library, you can link statically against it. Another issue with installing only .a is that there's no provision for versioning in .a library names ... what happens to someone who needs two generations of libpq on his machine? Use different directories and set LIBPATH? I don't know if there is a canonical way to do that. I'll investigate. Yours, Laurenz Albe ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
Albe Laurenz [EMAIL PROTECTED] writes: Tom Lane wrote: Applied, but without that last part. It builds OK for me on Darwin, which is moderately picky about that sort of thing, but someone should try AIX. It builds fine on AIX 5.3 as long as you tell it to link with libpq.so. Static builds against libpq.a will fail. Hm. We have been assuming that AIX's problem is that dynamic libraries don't remember their dependencies properly, but maybe the real issue is that it prefers static over dynamic libraries? If so, what we ought to be doing is adding the prefer-dynamic-libraries switch to the default LDFLAGS on that platform. Should -lldap or -lldap_r be added to pg_config --libs? You have a mistaken idea of the purpose of pg_config --libs. It exists to record what LIBS was at build time, not more, not less. It is certainly not intended as a guide to how to link libpq. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
Martijn van Oosterhout kleptog@svana.org writes: Static links are going to require it on every platform, not just AIX. The question do we want to ask is how easy do we want to make static linking, because the same treatment will have to apply to -lssl, -lcrypto, -lkrb5, -lk5crypto and quite possibly others. Do we really want to go there? Well, we already have a solution for static linking within the PG build environment, the question is do we wish to export it. Given the lack of complaints to date, I think not. Do we really want to encourage anyone to statically link libraries that don't belong to their project? It's not only the build-time dependency hell, it's the difficulty of installing library updates. There's a reason why dynamic link is the default on all modern platforms. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
Tom Lane wrote: It builds fine on AIX 5.3 as long as you tell it to link with libpq.so. Static builds against libpq.a will fail. Hm. We have been assuming that AIX's problem is that dynamic libraries don't remember their dependencies properly, but maybe the real issue is that it prefers static over dynamic libraries? If so, what we ought to be doing is adding the prefer-dynamic-libraries switch to the default LDFLAGS on that platform. AIX shared libraries know their dependencies well, and you don't have to specify them again when linking against the library. Let me expand a little on some of the peculiarities of shared libraries on AIX: - A normal AIX shared library is called libXX.a It is an 'ar' archive that contains the shared object(s). - In static linking mode, the shared object will be included in the executable. In dynamic linking mode (the default) only a reference to the shared object is included. - AIX can also link against shared objects called libXX.so (like other UNIXen). - When it resolves a -lXX flag, the linker searches the -L list for libXX.a and libXX.so. It prefers libXX.a over libXX.so unless invoked with the flag -brtl. So the linker does not prefer static over dynamic libraries, it just perfers libXX.a over libXX.so. In our case, we have libpq.a and libpq.so in the same directory, so unless you link with -brtl you will get a static link (because libpq.a is a static library). To illustrate, let me include the output of 'ldd psql' for a dynamically linked PostgreSQL: 8.2/bin/psql needs: /postgres/8.2/lib/libpq.so /usr/lib/libpthread.a(shr_xpg5_64.o) /usr/lib/libc.a(shr_64.o) /usr/local/lib/libldap_r.a(libldap_r-2.3.so.0) /unix /usr/lib/libcrypt.a(shr_64.o) /usr/lib/libs.a(shr_64.o) See http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/c om.ibm.aix.cmds/doc/aixcmds3/ld.htm Should -brtl be added to src/template/aix? Should -lldap or -lldap_r be added to pg_config --libs? You have a mistaken idea of the purpose of pg_config --libs. It exists to record what LIBS was at build time, not more, not less. It is certainly not intended as a guide to how to link libpq. *nods* Thanks for the clarification. Yours, Laurenz Albe ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
Albe Laurenz [EMAIL PROTECTED] writes: Let me expand a little on some of the peculiarities of shared libraries on AIX: - A normal AIX shared library is called libXX.a It is an 'ar' archive that contains the shared object(s). Ah, so the problem really boils down to funny naming conventions. If they use .a for both shared and static libraries, how does anyone tell the difference? So the linker does not prefer static over dynamic libraries, it just perfers libXX.a over libXX.so. In our case, we have libpq.a and libpq.so in the same directory, so unless you link with -brtl you will get a static link (because libpq.a is a static library). I wonder whether we ought to suppress building (or at least installing) our .a libraries at all on AIX. Adding -btrl to LDFLAGS would help within the context of our own build, but external clients that link to libpq without saying that are going to get undesirable results. I think there's a reasonable argument that by installing a .a file that isn't a shared library, we are violating the platform's conventions. Should -brtl be added to src/template/aix? Sounds that way, but that'll only help for psql and other stuff built within our build. Could you try this against CVS tip: * add -brtl to LDFLAGS in the template * Remove the AIX-specific hack on $(libpq) at lines 349-354 of src/Makefile.global.in * see if it configures and builds regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
Tom Lane wrote: Here is a new patch that replaces the previous one; it adds two macros LDAP_LIBS_FE and LDAP_LIBS_BE for frontend and backend, respectively. I did not only add them to the Makefile for interfaces/libpq, but also everywhere something is linked against libpq in case somebody links static. Applied, but without that last part. It builds OK for me on Darwin, which is moderately picky about that sort of thing, but someone should try AIX. It builds fine on AIX 5.3 as long as you tell it to link with libpq.so. Static builds against libpq.a will fail. Should -lldap or -lldap_r be added to pg_config --libs? If yes, which of them? Yours, Laurenz Albe ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
On Mon, Sep 11, 2006 at 12:13:29PM +0200, Albe Laurenz wrote: Applied, but without that last part. It builds OK for me on Darwin, which is moderately picky about that sort of thing, but someone should try AIX. It builds fine on AIX 5.3 as long as you tell it to link with libpq.so. Static builds against libpq.a will fail. Should -lldap or -lldap_r be added to pg_config --libs? If yes, which of them? Static links are going to require it on every platform, not just AIX. The question do we want to ask is how easy do we want to make static linking, because the same treatment will have to apply to -lssl, -lcrypto, -lkrb5, -lk5crypto and quite possibly others. Do we really want to go there? Have a nice day, -- Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ From each according to his ability. To each according to his ability to litigate. signature.asc Description: Digital signature
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
Here is a new patch that replaces the previous one; it adds two macros LDAP_LIBS_FE and LDAP_LIBS_BE for frontend and backend, respectively. I did not only add them to the Makefile for interfaces/libpq, but also everywhere something is linked against libpq in case somebody links static. Applied, but without that last part. It builds OK for me on Darwin, which is moderately picky about that sort of thing, but someone should try AIX. I'll do that today. Laurenz Albe ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
Tom Lane wrote: Albe Laurenz [EMAIL PROTECTED] writes: Let me expand a little on some of the peculiarities of shared libraries on AIX: - A normal AIX shared library is called libXX.a It is an 'ar' archive that contains the shared object(s). Ah, so the problem really boils down to funny naming conventions. If they use .a for both shared and static libraries, how does anyone tell the difference? It sounds to me like there is no difference. Notice how his example ldd output shows dependencies on specific .o entries within the various .a files that reside on the system, rather than on the .a files as a whole. If those entries had been statically linked then they wouldn't have shown up in the ldd output at all. So the difference is no longer between static libraries and shared libraries -- they're now just libraries. The only difference is how you link to them. What IBM has done here is very sensible, and is really what the other Unixes should have done from the beginning: a library is just a library, and what differs is how it's used. -- Kevin Brown [EMAIL PROTECTED] ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
Albe Laurenz [EMAIL PROTECTED] writes: I did not only add them to the Makefile for interfaces/libpq, but also everywhere something is linked against libpq in case somebody links static. I intensely dislike that part of the patch, but will work on applying the rest. If we do need to start mentioning all of libpq's dependencies everywhere it's linked, I think it's time for a generic solution to that, instead of hacking each such place over again every time a new dependency pops up. But at the moment I'm unconvinced that we need to do it. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
On Fri, Sep 08, 2006 at 03:20:00PM -0400, Tom Lane wrote: Albe Laurenz [EMAIL PROTECTED] writes: I did not only add them to the Makefile for interfaces/libpq, but also everywhere something is linked against libpq in case somebody links static. If we do need to start mentioning all of libpq's dependencies everywhere it's linked, I think it's time for a generic solution to that, instead of hacking each such place over again every time a new dependency pops up. The business of having to include every single dependancy when linking static is quite irritating. It has almost reached the point where people are just giving up static linking because it's too much of a pain. However, if we do want to support it, the way you do it is by extending pg_config to do something like: pg_config --dynamic-lick = returns -lpq pg_config --static-link= returns -lpq plus other libs That way only people who actually want static linking need be bothered. Have a ncie day, -- Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ From each according to his ability. To each according to his ability to litigate. signature.asc Description: Digital signature
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
Albe Laurenz [EMAIL PROTECTED] writes: Here is a new patch that replaces the previous one; it adds two macros LDAP_LIBS_FE and LDAP_LIBS_BE for frontend and backend, respectively. I did not only add them to the Makefile for interfaces/libpq, but also everywhere something is linked against libpq in case somebody links static. Applied, but without that last part. It builds OK for me on Darwin, which is moderately picky about that sort of thing, but someone should try AIX. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] [HACKERS] Fix linking of OpenLDAP libraries
Tom Lane wrote: I have realized that my modifications in configure.in and src/interfaces/libpq/Makefile to link libpq against OpenLDAP are buggy. Here is a proposed patch to fix it. [...] # The backend doesn't need everything that's in LIBS, however ! LIBS := $(filter-out -lz -lreadline -ledit -ltermcap -lncurses -lcurses -lldap_r $(PTHREAD_LIBS), $(LIBS)) This seems pretty risky. What if PTHREAD_LIBS contains -L switches? They'd get removed even if needed for other libraries. It would probably be safer not to put LDAP into LIBS at all, but invent two new macros for configure to set, say LDAP_LIBS and LDAP_LIBS_R, and add these to the link lines in the backend and libpq respectively. Here is a new patch that replaces the previous one; it adds two macros LDAP_LIBS_FE and LDAP_LIBS_BE for frontend and backend, respectively. I did not only add them to the Makefile for interfaces/libpq, but also everywhere something is linked against libpq in case somebody links static. I don't know if that's necessary, or if static builds are supported - if not, the changes to those Makefiles should perhaps not be applied. Tested on Linux, AIX and Windows. Yours, Laurenz Albe ldaplink2.patch Description: ldaplink2.patch ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Fix linking of OpenLDAP libraries
Tom Lane wrote on September 04, 2006: Albe Laurenz [EMAIL PROTECTED] writes: # The backend doesn't need everything that's in LIBS, however ! LIBS := $(filter-out -lz -lreadline -ledit -ltermcap -lncurses -lcurses -lldap_r $(PTHREAD_LIBS), $(LIBS)) This seems pretty risky. What if PTHREAD_LIBS contains -L switches? They'd get removed even if needed for other libraries. It would probably be safer not to put LDAP into LIBS at all, but invent two new macros for configure to set, say LDAP_LIBS and LDAP_LIBS_R, and add these to the link lines in the backend and libpq respectively. That seems like a good idea. I'll try to work it out and resubmit the fix. Yours, Laurenz Albe ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Fix linking of OpenLDAP libraries
Albe Laurenz [EMAIL PROTECTED] writes: # The backend doesn't need everything that's in LIBS, however ! LIBS := $(filter-out -lz -lreadline -ledit -ltermcap -lncurses -lcurses -lldap_r $(PTHREAD_LIBS), $(LIBS)) This seems pretty risky. What if PTHREAD_LIBS contains -L switches? They'd get removed even if needed for other libraries. It would probably be safer not to put LDAP into LIBS at all, but invent two new macros for configure to set, say LDAP_LIBS and LDAP_LIBS_R, and add these to the link lines in the backend and libpq respectively. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] Fix linking of OpenLDAP libraries
I have realized that my modifications in configure.in and src/interfaces/libpq/Makefile to link libpq against OpenLDAP are buggy. Here is a proposed patch to fix it. I write this to pgsql-hackers too because I want to share the difficulty I'm facing - maybe somebody has a better idea. To handle thread safety, OpenLDAP comes with a second library libldap_r. The thread safe API is identical to the normal API, the difference is that you must link against libldap_r instead of libldap to get thead safety. These are my problems: - While libpq should be thread safe when ./configured with --enable_thread_safety, the backend should be linked against the normal libldap. - At least on RedHat Linux, you need to link against the pthread library too if you want to link against libldap_r, because the latter has unresolved dependencies. My solution: - If thread safety is not desired, I link against libldap. No problem. - If thread safety is desired, I first try to link against libldap_r without the thread libraries, and only if that fails add the thread libraries to LIBS. - I tweak src/backend/Makefile so that it strips libldap_r and the thread libs from LIBS and replace it with libldap if necessary. That means that if --enable_thread_safety and --with-ldap is both specified, all executables except 'postgres' will be linked against libldap_r (and the thread libs, if necessary). I tested my patch on RedHat Enterprise Linux 3 and AIX 5.3. The behaviour for Windows (use the native WLDAP32.DLL) is unchanged. Yours, Laurenz Albe ldaplink.patch Description: ldaplink.patch ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
