Re: preparing for 1.5.24
Hi Charles, * Charles Wilson wrote on Wed, Apr 25, 2007 at 05:38:49AM CEST: Ralf Wildenhues wrote: Thanks also for the documentation suggestion. Slightly rewritten suggestion to come up. Ping? [antecedent reposted below] Sigh. I had hoped to have enough time to work on this. Alas, I better just post my thoughts. Apologies for being the eye of the needle here (can you use the phrase this way in English?). First, I think this is a bit ad hoc here: the dllexport/dllimport issue, or more generally, w32-specific issues (although as I understand it's not /strictly/ w32-specific), don't start with using libltdl. As such, I think they deserves its own chapter in the manual, and then the libltdl handling can refer to it, in shortened form. Ideally, all the gory details of how you can have several deplibs, some static some shared, and currently need to define a macro for each based on that (but maybe libtool can help there in the future to automatize that?). AC_LIBTOOL_WIN32_DLL could refer to this section then. I think some of this stuff has been discussed at length on the libtool lists; I assume you were one of the people involved. Even cooler would be to take into account the differences for when we add support for MSVC, wgcc, ... Around line 3546 [probably 4476, now] in libtool.texi, something like: --% @emph{Win32 specific} FWIW, I don't like this way of putting keywords much. It should be a heading, or part of the surrounding sentence. When building a DLL on windows (MinGW,Cygwin) that Let's not speak of a library as a DLL, except where we explain what a DLL is, and except the strictly w32-specific parts of the manual. We should strive to use coherent names for things, here: shared library. Just like you use convenience archive within the same sentence. uses @code{libltdl} as a convenience archive, there are some issues with symbol decoration that must be carefully managed. You must do one of the following, within the code and/or build machinery of your library: @enumerate 1 @item explicitly mark as @code{declspec(dllexport)} all symbols in your library that should be exported when building your library's source code. However, these symbols should @emph{not} be marked @code{declspec(dllexport)} when building @emph{clients} of your library -- instead, the symbols should be marked @code{declspec(dllimport)}. This procedure will require careful macro design within your library's public header files. An example of this may be found in @file{zlib.h} from the zlib distribution @url{http://www.zlib.net/}. Let's rather not refer to an external source here: they could change, the link is more than two links away from what the user really needs to see when doing this, and zlib isn't GNU software (it's free, so there's no real issue here). Rather, a small, self-contained example would be great. Ideally, that example would be identical to a test in the new testsuite, so that we're kept honest in that it actually works as expected. @item use the (win32-specific) linker option @code{-Xlinker --export-all-symbols}. This will restore the expected auto-export behavior of the GNU linker on windows. However, the @code{--export-all-symbols} option is specific to the GNU linker's i386 PE target -- @code{ld} for other targets will not recognize the option. Therefore, using this option will require careful coding in your library's @file{Makefile.am} and @file{configure.ac}. For example: configure.ac: @example ... case $host_os in cygwin* | mingw* | pw32*) This doesn't account for specifity to GNU binutils ld. $with_gnu_ld should help here, bugs in its setting notwithstanding. RESTORE_AUTOEXPORT=-Xlinker --export-all-symbols [...] Cheers, and thanks for all your work, Ralf
Re: preparing for 1.5.24
Ralf Wildenhues wrote: Thanks also for the documentation suggestion. Slightly rewritten suggestion to come up. Ping? [antecedent reposted below] -- Chuck Around line 3546 [probably 4476, now] in libtool.texi, something like: --% @emph{Win32 specific} When building a DLL on windows (MinGW,Cygwin) that uses @code{libltdl} as a convenience archive, there are some issues with symbol decoration that must be carefully managed. You must do one of the following, within the code and/or build machinery of your library: @enumerate 1 @item explicitly mark as @code{declspec(dllexport)} all symbols in your library that should be exported when building your library's source code. However, these symbols should @emph{not} be marked @code{declspec(dllexport)} when building @emph{clients} of your library -- instead, the symbols should be marked @code{declspec(dllimport)}. This procedure will require careful macro design within your library's public header files. An example of this may be found in @file{zlib.h} from the zlib distribution @url{http://www.zlib.net/}. @item use the (win32-specific) linker option @code{-Xlinker --export-all-symbols}. This will restore the expected auto-export behavior of the GNU linker on windows. However, the @code{--export-all-symbols} option is specific to the GNU linker's i386 PE target -- @code{ld} for other targets will not recognize the option. Therefore, using this option will require careful coding in your library's @file{Makefile.am} and @file{configure.ac}. For example: configure.ac: @example ... case $host_os in cygwin* | mingw* | pw32*) RESTORE_AUTOEXPORT=-Xlinker --export-all-symbols ;; *) RESTORE_AUTOEXPORT= ;; esac AC_SUBST(RESTORE_AUTOEXPORT) ... @end example Makefile.am @example ... libmylibrary_la_LDFLAGS = -no-undefined $(RESTORE_AUTOEXPORT) ... @end example @item Use libtool facilities to mimic @code{--export-all-symbols}. Unlike the solution above, this solution has the advantage of requiring no platform-specific workarounds. Simply append @code{-export-symbols-regex .*} to the @code{_LDFLAGS} for your library: Makefile.am @example ... libmylibrary_la_LDFLAGS = -no-undefined -export-symbols-regex .* ... @end example @end enumerate --%
Re: preparing for 1.5.24
Hello Charles, Charles Wilson writes: - It would help me greatly if someone could look into the Cygwin and MinGW mdemo* failures; and documentation updates if needed. Solution in this case is to turn auto-export back on Or to mark all symbols as to be exported, no? (Or to mark all other symbols explicitly. But let's not go that way.) It'd be nice to be able to say whenever libltdl is used as a convenience library to build a DLL, on windows libtool should automatically turn on --export-all-symbols. But that isn't true: if the target DLL is already doing explicit declspec(dllexport), then (1) the target library would work, today, without any changes, and (2) we don't want to override that target DLL's explicit export settings by saying, no, I'm going to export everything. ACK. So, I'm not sure how to solve this, in a way that is platform-agnostic. I think the patch below should work, and be platform-agnostic. WDYT? OK for branch-1-5 and HEAD alike? It fixes the similar failure there. Do you think we should add some documentation bits a la if you have the same problem, consider this solution as well? (FWIW, on HEAD there are other mdemo test failures which I haven't looked at yet.) FWIW, it doesn't appear that libtool's -export-dynamic will do the right thing here. I thought about setting export_dynamic_flag_spec to '${wl}--export-all-symbols' for mingw/cygwin (currently, they inherit the generic '${wl}--export-dynamic' setting [3]), and then using '-export-dynamic' in libmlib_la_LDFLAGS for all platforms. I don't think -export-dynamic is intended to have the semantics of --export-all-symbols. But I'm nervous about changing the definition of export_dynamic_flag_spec -- because that would have wide-ranging effects. Esepcially in a release candidate, and especially especially in the RC for the last-ever 1.5.x libtool. I guess I should have written the last 1.5.x release, unless serious issues show up. When writing, the intention was more of a after this, we work towards 2.0 then anything else. [1] mdemo README: Note that on Windows, for the purposes of illustrating ltdl, the libraries *are* built as dll's, but do not have the __declspec machinery to make them suitable for loading at link time. This is only for clarity inside this example, look at the example in ../demo to see how the __declspec macros should be set up. [2] BTW, ../demo does not set up any __declspec macros. I believe the readme snippet above is really old, and pre-dates auto-export/auto-import functionality. At best, this snippet is now specific only to non-gcc on Windows, which is at present only barely supported in libtool-1.5.x anyway. Notwithstanding my reference to this snippet in the context of libltdl-as-convenience-lib, I think this snippet should be deleted entirely. Indeed. I guess for branch-1-5 it should be removed. For HEAD and eventual MSVC support, those would need to be added again. [3] I'm not sure, but it looks like cygwin's ld simply ignores --export-dynamic. From a brief look at the bfd/ld source code, this ld option seems to be ELF-specific. I guess. I don't know what its effect on Cygwin should be though. Not --export-all-symbols, that's not the right thing. Cheers, and thank you, Ralf * mdemo/Makefile.am (libmlib_la_LDFLAGS): Add -export-symbols-regex .* because the convenience libltdl uses export markers and thus turns off autoexport on w32. Fixes test failures for Cygwin and MinGW. Bug analysis by Charles Wilson. Index: mdemo/Makefile.am === RCS file: /sources/libtool/libtool/mdemo/Attic/Makefile.am,v retrieving revision 1.46.2.2 diff -u -r1.46.2.2 Makefile.am --- mdemo/Makefile.am 3 Feb 2006 09:37:34 - 1.46.2.2 +++ mdemo/Makefile.am 18 Feb 2007 12:07:21 - @@ -19,9 +19,12 @@ libsub_la_SOURCES = sub.c libsub_la_LDFLAGS = -no-undefined +## Use -export-symbols-regex here explicitly because libltdl marks +## its exported symbols, and we use libltdl as a convenience archive. +## Thus, on w32, auto-exporting is turned off. libmlib_la_SOURCES = mlib.c libmlib_la_LIBADD = @LIBLTDL@ -dlopen foo1.la -dlopen libfoo2.la -libmlib_la_LDFLAGS = -no-undefined +libmlib_la_LDFLAGS = -no-undefined -export-symbols-regex .* libmlib_la_DEPENDENCIES = @LIBLTDL@ libsub.la foo1.la libfoo2.la noinst_HEADERS = foo.h
Re: preparing for 1.5.24
Ralf Wildenhues wrote: Hello Charles, Charles Wilson writes: - It would help me greatly if someone could look into the Cygwin and MinGW mdemo* failures; and documentation updates if needed. Solution in this case is to turn auto-export back on Or to mark all symbols as to be exported, no? (Or to mark all other symbols explicitly. But let's not go that way.) Yes, that should work, too. I think the patch below should work, and be platform-agnostic. WDYT? OK for branch-1-5 and HEAD alike? It fixes the similar failure there. Hmm -- seems ok in principle. I don't know if the end behavior is exactly the same, between ld's --export-all-symbols and libtool's -export-symbols-regex .*. I've attached the exports list from my cygmlib-0.dll (taken from the output of objdump -p). How does yours compare? Do you think we should add some documentation bits a la if you have the same problem, consider this solution as well? Yes, I do. Around line 3546 in libtool.texi, something like: --% @emph{Win32 specific} When building a DLL on windows (MinGW,Cygwin) that uses @code{libltdl} as a convenience archive, there are some issues with symbol decoration that must be carefully managed. You must do one of the following, within the code and/or build machinery of your library: @enumerate 1 @item explicitly mark as @code{declspec(dllexport)} all symbols in your library that should be exported when building your library's source code. However, these symbols should @emph{not} be marked @code{declspec(dllexport)} when building @emph{clients} of your library -- instead, the symbols should be marked @code{declspec(dllimport)}. This procedure will require careful macro design within your library's public header files. An example of this may be found in @file{zlib.h} from the zlib distribution @url{http://www.zlib.net/}. @item use the (win32-specific) linker option @code{-Xlinker --export-all-symbols}. This will restore the expected auto-export behavior of the GNU linker on windows. However, the @code{--export-all-symbols} option is specific to the GNU linker's i386 PE target -- @code{ld} for other targets will not recognize the option. Therefore, using this option will require careful coding in your library's @file{Makefile.am} and @file{configure.ac}. For example: configure.ac: @example ... case $host_os in cygwin* | mingw* | pw32*) RESTORE_AUTOEXPORT=-Xlinker --export-all-symbols ;; *) RESTORE_AUTOEXPORT= ;; esac AC_SUBST(RESTORE_AUTOEXPORT) ... @end example Makefile.am @example ... libmylibrary_la_LDFLAGS = -no-undefined $(RESTORE_AUTOEXPORT) ... @end example @item Use libtool facilities to mimic @code{--export-all-symbols}. Unlike the solution above, this solution has the advantage of requiring no platform-specific workarounds. Simply append @code{-export-symbols-regex .*} to the @code{_LDFLAGS} for your library: Makefile.am @example ... libmylibrary_la_LDFLAGS = -no-undefined -export-symbols-regex .* ... @end example @end enumerate --% (FWIW, on HEAD there are other mdemo test failures which I haven't looked at yet.) FWIW, it doesn't appear that libtool's -export-dynamic will do the right thing here. I thought about setting export_dynamic_flag_spec to '${wl}--export-all-symbols' for mingw/cygwin (currently, they inherit the generic '${wl}--export-dynamic' setting [3]), and then using '-export-dynamic' in libmlib_la_LDFLAGS for all platforms. I don't think -export-dynamic is intended to have the semantics of --export-all-symbols. It seems to just delegate to ld's --export-dynamic, which from my reading of 'info ld' seems concerned mainly with the export of symbols from an executable. (kinda like --export-all-symbols, but .exe-specific, and doesn't really export ALL symbols, just most of them). You're right in that we should not overload its behavior on win32 -- that way lies madness. I guess I should have written the last 1.5.x release, unless serious issues show up. When writing, the intention was more of a after this, we work towards 2.0 then anything else. Heck yeah. I don't want to slow THAT down! Indeed. I guess for branch-1-5 [ the blurb in mdemo/README concerning win32 ] should be removed. For HEAD and eventual MSVC support, those would need to be added again. Right. [3] I'm not sure, but it looks like cygwin's ld simply ignores --export-dynamic. From a brief look at the bfd/ld source code, this ld option seems to be ELF-specific. I guess. I don't know what its effect on Cygwin should be though. Not --export-all-symbols, that's not the right thing. Ack. I think it should force a kind of --export-all-symbols, but only for .exe's. Current ld behavior for .exe's is to NOT export any symbols (e.g. no auto-export) -- unless (a) there are symbols in the exe which were explicitly marked declspec(dllexport), or
Re: preparing for 1.5.24
On Feb 17, 2007, at 6:54 PM, Ralf Wildenhues wrote: Hi Peter, Tim, all, * Peter O'Gorman wrote on Sun, Feb 11, 2007 at 01:20:24PM CET: On Feb 11, 2007, at 7:24 PM, Ralf Wildenhues wrote: Should we rather do a real prerelease, so it's archived on alpha.gnu.org,[...] This is good news, and great work. I'm pretty sure finch can handle the load and bandwidth, so I think it is up to you, but I would not bother with a formal alpha. I've looked at the snapshot. It uses Automake 1.9.6 and Autoconf 2.60. Since I'd rather use 1.10 and 2.61 for 1.5.24, I preferred a formal alpha. I've now adjusted the URL for the branch-1-5 snapshot to contain the string 1.5.23c in the name. I assume the switch will happen automatically with the next build? Looks like I was on the ball the day I wrote that script, yeah, it handles the version change without modification. Peter
Re: preparing for 1.5.24
Ralf Wildenhues wrote: I would like to release 1.5.24 as soon as possible. I've flushed my queue now, except for the patch I just posted and whatever w32 changes are still needed. FWIW, I've done a test run of branch-1-5, here's some results: PASS SKIP FAIL config.guess compiler used notes 110 1 1 i686-pc-cygwingcc, g++, g77 fail: mdemo2-make confirmed (1.5.23c, about two hours ago). I've diagnosed the reason and have a patch -- but I'm not sure if it should be applied. I don't consider this a showstopper for 1.5.24, regardless of whether the attached patch is accepted or not. - It would help me greatly if someone could look into the Cygwin and MinGW mdemo* failures; and documentation updates if needed. I believe the mdemo2 failure on cygwin is another ripple from the ltdl declspec() change. mlib uses libltdlc as convenience library, so when linking cygmlib-0.dll there exist symbols (lt_*) that are declared declspec(dllexport) -- so autoexport is turned off. However, the symbols in mlib.c are NOT marked in that way (see mdemo/README snippet, [1]), so without autoexport, the output dll contains only lt_* symbols -- and mlib_func is missing. Solution in this case is to turn auto-export back on (earlier suggestions conerning define LT_SCOPE before #inluding ltdl.h won't work, because we're linking in the ltdl.o object that was already built as PIC (==-DDLL_EXPORT), and LT_SCOPE has already ended up as declspec(dllexport) in that ltdl.o. It'd be nice to be able to say whenever libltdl is used as a convenience library to build a DLL, on windows libtool should automatically turn on --export-all-symbols. But that isn't true: if the target DLL is already doing explicit declspec(dllexport), then (1) the target library would work, today, without any changes, and (2) we don't want to override that target DLL's explicit export settings by saying, no, I'm going to export everything. So, I'm not sure how to solve this, in a way that is platform-agnostic. (short of undoing the -DDLL_EXPORT change, like Ralf wants to. But I won't rehash that argument, except to say that this problem would persist on mingw even if we fixed cygwin in that way) A non-platform-agnostic mechanism could be described as: When building a DLL on Windows (==MinGW,Cygwin) that uses libltdl as a convenience archive, you must do one of two things: (1) explicitly mark as declspec(dllexport) all symbols in _your_ library that should be exported when building your library's source code -- but do NOT mark them dllexport when building clients; this procedure will require much macro magic in your library's public header files, or (2) use the (win32-specific) linker option -Xlinker --export-all-symbols to restore the expected auto-export behavior on windows. This will require some configure magic, as other platforms' linkers will not recognize this option. See mdemo/configure.ac and mdemo/Makefile.am for an example. I used #2 in the attached patch, for the mdemo/mdemo2 issue. FWIW, it doesn't appear that libtool's -export-dynamic will do the right thing here. I thought about setting export_dynamic_flag_spec to '${wl}--export-all-symbols' for mingw/cygwin (currently, they inherit the generic '${wl}--export-dynamic' setting [3]), and then using '-export-dynamic' in libmlib_la_LDFLAGS for all platforms. But I'm nervous about changing the definition of export_dynamic_flag_spec -- because that would have wide-ranging effects. Esepcially in a release candidate, and especially especially in the RC for the last-ever 1.5.x libtool. Anyway, with the attached patch, all tests pass on cygwin. -- Chuck [1] mdemo README: Note that on Windows, for the purposes of illustrating ltdl, the libraries *are* built as dll's, but do not have the __declspec machinery to make them suitable for loading at link time. This is only for clarity inside this example, look at the example in ../demo to see how the __declspec macros should be set up. [2] BTW, ../demo does not set up any __declspec macros. I believe the readme snippet above is really old, and pre-dates auto-export/auto-import functionality. At best, this snippet is now specific only to non-gcc on Windows, which is at present only barely supported in libtool-1.5.x anyway. Notwithstanding my reference to this snippet in the context of libltdl-as-convenience-lib, I think this snippet should be deleted entirely. [3] I'm not sure, but it looks like cygwin's ld simply ignores --export-dynamic. From a brief look at the bfd/ld source code, this ld option seems to be ELF-specific. 2007-02-17 Charles Wilson ... * mdemo/configure.ac: add platform-specific