Re: rtld + static linking
Marcel Moolenaar wrote: > On Thu, Nov 27, 2003 at 03:41:14AM -0800, Terry Lambert wrote: > > > If you can get gcc and binutils to add the necessary support, then > > > we can talk further. Until then it's academic. > > > > I think there are political reasons for not doing this. The > > number one reason would be that you could load GPL'ed code as > > a module into a non-GPL'ed program, and then use it without > > neding to change your own license to GPL. > > The use of dlopen() does not circumvent that you're creating a > modified version. The GPL-incompatible code and the GPL code do > not communicate at arms length. Hence, the GPL-incompatible code > is to be relicensed if the modified work is to be distributed. No. IF you separately distibute your code and the GPL'ed module code, AND you have an alternate user of the interface, THEN you can distribute the code at will without GPL'ing it, even if the intent is to link against the GPL'ed code at runtime. Be, Inc. did this with GPL'd Linux device drivers, and the one lawsuit the FSF began to bring on the basis of a GPL'ed library was resolved when the authors of the original code provided their own non-GPL'ed library, thus putting the GPL'ed code at arms length, legally, with regard to whether or not the combined code was a derivative work of the GPL'ed code. Like it or not, being able to dlopen a GPL'ed module means that you can circumvent the intent of the GPL. In fact, there are a number of PAM modules that are GPL'ed (go ahead and check). > Alternatively, ask the license holder for an exception or see if > the library can be relicensed under the LGPL. Why ask, when you can simply circumvent? The only thing it would buy you is the ability to not redistribute "modularized source", which is of questionable benefit. > From a GPL licensing point of view dlopen() is no different than > static linking. Incorrect. The linking occurs at the behest of the person running the program, rather than at the behest of the person distributing the program,. As distributed, the program and the module together constitute a "mere agregation", and therefore the program itself is not subject to the module's GPL nature. > > Well, it specifically mentions being able to do something like > > linking libc to libresolv, so that you implicitly get the resolver > > library when you explicitly link in the C library. That doesn't > > work with static linking of ELF binaries, so I think we can either > > say static linking is completely broken and needs to be fixed before > > it is allowed to be used, or static linking is undefined by the ELF > > standard. > > Recording dependencies in ELF files is specific to dynamic linking. No, it's specific to all relocatable ELF objects, at least according to the specification. > In fact, there's nothing that you need to add to ELF headers or > sections to make static linking work. It's irrelevant to an ELF > specification. That's why it's not mentioned. Dynamic linking needs > special headers and/or sections in the ELF file to make it work. > > I don't think you can infer from the ELF specification that static > linking is undefined. I don't think that you can infer that it's defined. 8-). Anything that's not defined is, by definition, undefined. -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: rtld + static linking
On Thu, Nov 27, 2003 at 03:41:14AM -0800, Terry Lambert wrote: > > If you can get gcc and binutils to add the necessary support, then > > we can talk further. Until then it's academic. > > I think there are political reasons for not doing this. The > number one reason would be that you could load GPL'ed code as > a module into a non-GPL'ed program, and then use it without > neding to change your own license to GPL. The use of dlopen() does not circumvent that you're creating a modified version. The GPL-incompatible code and the GPL code do not communicate at arms length. Hence, the GPL-incompatible code is to be relicensed if the modified work is to be distributed.i Alternatively, ask the license holder for an exception or see if the library can be relicensed under the LGPL. >From a GPL licensing point of view dlopen() is no different than static linking. > > > I'll also point out that the ELF specification does not define static > > > linking *at all*. > > > > I think that's because it doesn't need any special mention. > > Well, it specifically mentions being able to do something like > linking libc to libresolv, so that you implicitly get the resolver > library when you explicitly link in the C library. That doesn't > work with static linking of ELF binaries, so I think we can either > say static linking is completely broken and needs to be fixed before > it is allowed to be used, or static linking is undefined by the ELF > standard. Recording dependencies in ELF files is specific to dynamic linking. In fact, there's nothing that you need to add to ELF headers or sections to make static linking work. It's irrelevant to an ELF specification. That's why it's not mentioned. Dynamic linking needs special headers and/or sections in the ELF file to make it work. I don't think you can infer from the ELF specification that static linking is undefined. -- Marcel Moolenaar USPA: A-39004 [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: rtld + static linking
Terry Lambert writes: > > I've looked without much success. Could you give a timeframe, a subject > > and/or something? > > Note that the part you snipped indicated that the patches were > posted by a third party, and that my own patches had been offered, > but were not posted in their entirety to the mailing list. In > actuality, I only ever posted portions of my own patches, since > they also required compiler and linker changes. Could you please post your patches in their entirety? M -- Mark Murray iumop ap!sdn w,I idlaH ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: rtld + static linking
Marcel Moolenaar wrote:
> > Ether way, you still need to deal with the linker changes necessary
> > to export the symbol set for all statically linked objects, and to
> > force the inclusion of all archive members when statically linking,
> > if one of the linked libraries is libdl, if you wanted a full
> > implementation.
>
> I think the cure is worse than the decease in this case. You don't
> want the full libc linked into a static sh, simply because we need
> to be able to load a shared library at runtime.
The alternative is to create a version association between the
libc.a and libc.so, to allow it to be recovered at runtime. I
don't know how satisfying this would be, but it would resolve
99.9% of the cases, I think. Arguing against inclusion of
everything for all static libraris without a shared counterpart,
and for the main program, is harder. For my money, it's a lot
easier to dynamically link everything, and only deal with one
type of executable image thereafter.
> If you can get gcc and binutils to add the necessary support, then
> we can talk further. Until then it's academic.
I think there are political reasons for not doing this. The
number one reason would be that you could load GPL'ed code as
a module into a non-GPL'ed program, and then use it without
neding to change your own license to GPL. Editorially, the
FSF compiler folks have some incentive for making dlopen()
inconvenient/difficult to use. If you link dynamic, at
least they get the LGPL behaviour for all your other libraries
as a consolation prize, so making it work is probably on their
list of "Things To Not Do".
If the Linux people want to be SUSv3/POSIX/IEEE 1003.1-2003
compliant, they'll do the work and FreeBSD will get it for
free. If thy don't want to do the work, FreeBSD will have to
decide whether they want to carry around local patches, or if
the feature (killing off static vs. dynamic bikesheds) is
worth the effort.
I'm just offering a potential option that's not one of the two
hotly contested ones, as a less controversial approach to a
solution to the original problem. It's the solution that was
chosen for the original problem that has everyone up in arms,
so a different solution seemed to be called for here...
> > BTW: IEEE 1003.1-2003 requires a full implementation of dlopen, and
> > does not permit an exception for statically linked binaries:
> >
> > http://www.opengroup.org/onlinepubs/007904975/functions/dlopen.html
>
> Yes it does:
>
> \begin{quote}
> ... Implementations may also impose specific constraints on the
> construction of programs that can employ dlopen() and its related
> services.
> \end{quote}
The phrase "constraints on the construction of programs" sounds
to me like you're allowed to require compile options, not you're
allowed to make it not work as it's documented to work otherwise,
or we might as well say that calling dlopen() in a given system's
implementation is allowed to invoke nethack, like the original
GCC implementation of "#pragma" did. 8-).
> > I'll also point out that the ELF specification does not define static
> > linking *at all*.
>
> I think that's because it doesn't need any special mention.
Well, it specifically mentions being able to do something like
linking libc to libresolv, so that you implicitly get the resolver
library when you explicitly link in the C library. That doesn't
work with static linking of ELF binaries, so I think we can either
say static linking is completely broken and needs to be fixed before
it is allowed to be used, or static linking is undefined by the ELF
standard.
> Note that staticly linked executables can be in violation of platform
> runtime specifications.
A good reason to outlaw them: undefined behaviour sucks.
-- Terry
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: rtld + static linking
Mark Murray wrote: > Terry Lambert writes: > > Since I have patches to make dlopen work with static binaries, and > [ snip ] > > As to inevitable "where are the patches?", please check the -current > > list archives, you will find at least one set there. > > I've looked without much success. Could you give a timeframe, a subject > and/or something? Note that the part you snipped indicated that the patches were posted by a third party, and that my own patches had been offered, but were not posted in their entirety to the mailing list. In actuality, I only ever posted portions of my own patches, since they also required compiler and linker changes. Here is the reference for Max Khon's patches, which were posted, by reference, in their entirety: | Date: Tue, 5 Nov 2002 13:21:42 +0600 | From: Max Khon <[EMAIL PROTECTED]> | To:Jake Burkholder <[EMAIL PROTECTED]> | Cc:[EMAIL PROTECTED] | Subject: Re: libc size | |http://docs.freebsd.org/cgi/getmsg.cgi?fetch=379714+0+archive/2002/freebsd-current/20021110.freebsd-current Portions of my patches were posted as part of the discussion which took place in the same thread. There did not seem to be much buy-in on the idea of modifying the compiler tools. The context of the discussion was for dealing with moving the resolver and other things out of libc, so that they could be more easily upgraded. For the purposes of handling PAM/NSS modules, Max's patches would work to move everything back to a static linking footing. FWIW, I dislike this idea, since static ELF libraries do not properly link against other static ELF libraries, and that type of chaning only works for dynamically linked ELF libraries (for no good reason). -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: rtld + static linking
On Wed, Nov 26, 2003 at 04:16:06AM -0800, Terry Lambert wrote:
> >
> > Dynamic linking works because the kernel loads and runs the dynamic
> > linker when it sees that the executable defines an interpeter.
>
> Since I have patches to make dlopen work with static binaries, and
> it doesn't work this way, I must conclude you have not really looked
> deeply into solving the problem.
True. I did not look at aout.
> In any case, I point you to /usr/src/lib/csu/i386/crt0.c, which
> contains these lines of code:
This is aout specific code, not ELF code.
> Ether way, you still need to deal with the linker changes necessary
> to export the symbol set for all statically linked objects, and to
> force the inclusion of all archive members when statically linking,
> if one of the linked libraries is libdl, if you wanted a full
> implementation.
I think the cure is worse than the decease in this case. You don't
want the full libc linked into a static sh, simply because we need
to be able to load a shared library at runtime.
If you can get gcc and binutils to add the necessary support, then
we can talk further. Until then it's academic.
> BTW: IEEE 1003.1-2003 requires a full implementation of dlopen, and
> does not permit an exception for statically linked binaries:
>
> http://www.opengroup.org/onlinepubs/007904975/functions/dlopen.html
Yes it does:
\begin{quote}
... Implementations may also impose specific constraints on the
construction of programs that can employ dlopen() and its related
services.
\end{quote}
> I'll also point out that the ELF specification does not define static
> linking *at all*.
I think that's because it doesn't need any special mention.
Note that staticly linked executables can be in violation of platform
runtime specifications.
--
Marcel Moolenaar USPA: A-39004 [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: rtld + static linking
Terry Lambert writes: > Since I have patches to make dlopen work with static binaries, and : [ snip ] : > As to inevitable "where are the patches?", please check the -current > list archives, you will find at least one set there. I've looked without much success. Could you give a timeframe, a subject and/or something? M -- Mark Murray iumop ap!sdn w,I idlaH ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: rtld + static linking
Marcel Moolenaar wrote: > On Tue, Nov 25, 2003 at 05:44:18PM -0800, Terry Lambert wrote: > > "E.B. Dreger" wrote: > > > Dynamic linking works by the kernel running the dynamic linker, > > > which loads shared objects and fixes the symbol tables, yes? > > > > No. > > > > Dynamic linking works because the crt0 mmap's the /usr/libexec/ld.so > > file as executable, and then points known stub offsets into it. > > No. > > Dynamic linking works because the kernel loads and runs the dynamic > linker when it sees that the executable defines an interpeter. Since I have patches to make dlopen work with static binaries, and it doesn't work this way, I must conclude you have not really looked deeply into solving the problem. While the ELF specification, and the SVID III, specifically set aside "enough space" that you can leave the first page unmapped and have room for the kernel to load in the ld.so (and thereby save yourself recreating that part of the address space on exec of non-setuid/setgid binaries, where it's not a security issue), FreeBSD doesn't do this. I can only conclude that either no one has gotten around to it since I pointed it out 6 years ago, or someone is intent on FreBSD's "exec" beinbg slower than it needs to be. In any case, I point you to /usr/src/lib/csu/i386/crt0.c, which contains these lines of code: [...] /* Map in ld.so */ crt.crt_ba = (int)_mmap(0, hdr.a_text, PROT_READ|PROT_EXEC, MAP_FILE|MAP_PRIVATE, crt.crt_ldfd, N_TXTOFF(hdr)); [...] > A complete executable (i.e. staticly linked) does not export any > symbols, or at least not in the same way a shared executable and > shared libraries do. This is why I keep saying that compiler work would be needed to export symbols from the original binary and any statically linked libraries. > If I try to dynamicly link libbar into a > complete executable foo and libbar depends on libc, then there's > no guarantee that all the required bits from libc are in foo, Correct, for the current (flawed) implementation of the linker. > nor is there any guarantee that the bits are actually visible or even > accessable (no linkage table). In fact, there is a guaranteed that they are not there, for the current (flawed) implementation of the linker. > Dynamicly loading libc for use by libbar can work, but it's not > guaranteed. One failure mode is suddenly having two instances of > a common variable instead of one. Another is the clobbering of > data caused reinitializations. The first problem is a non-issue, at least for FreeBSD supplied libraries. For other libraries, yes, it could be an issue. The second is always a non-issue, since the loading of shared objects occurs in a hierarchy, and you are perfectly allowed to have multiple instances of different versions of shared objects existing simultaneously. In fact, if you read the dlsym(3) manual page, you will see that there exists a special handle identifier, RTLD_NEXT, specifically to enable a program to traverse this hierarchy in order to obtain the correct symbol. If you are still interested in the details of obtaining the correct subhierarchy from an arbitrary graph of such hierarchies, the way in which this is normally accomplished is to look for a symbol at your own hierarchy level, for which the address is known (since both the program iterating and the symbol you are looking for come from the same .so file and can be compared), and thereafter descend into your own branch to find the symbol in queation. > So, the problem of dynamic linking a shared library into a static > process is non-trivial and probably cannot be solved genericly. And yet both Solaris and SVR4 manage to accomplish this. Of course, they are not using the crt0.o code supplied by the GCC people, who have so far failed to ship a set of tools (gcc, binutils) that can do the job. The problem can be resolved generically. > Under restricted and controlled conditions you can make it work. > I would call it the ability to have plugins, not the ability to > load dynamic libraries. Then you are probaly using the implementation that was posted to -current about half a year ago, which failed to take into account the symbol sets, and made no changes to the constructor argument list. If you look in crt0.c again, you will see a number of lines that look like this: #ifdef DYNAMIC /* ld(1) convention: if DYNAMIC = 0 then statically linked */ /* sometimes GCC is too smart/stupid for its own good */ x = (caddr_t)&_DYNAMIC; if (x) __do_dynamic_link(argv); #endif /* DYNAMIC */ This is the conditional compilation unit for the dynamic crt0.o, as the same source file is use for both dynamic and static linking. The value of 'x' and 'argv' in this case are, for the first, the location of the 'struct _dynamic' that's is used to gate the call to __do_dynamic_link(), and the enviro
Re: rtld + static linking
On Tue, Nov 25, 2003 at 05:44:18PM -0800, Terry Lambert wrote: > "E.B. Dreger" wrote: > > After watching the recent shared/dynamic threads, and reading the > > archives from five or six years ago, I have a question... > > > > Dynamic linking works by the kernel running the dynamic linker, > > which loads shared objects and fixes the symbol tables, yes? > > No. > > Dynamic linking works because the crt0 mmap's the /usr/libexec/ld.so > file as executable, and then points known stub offsets into it. No. Dynamic linking works because the kernel loads and runs the dynamic linker when it sees that the executable defines an interpeter. > > Is there some reason that a statically-linked program couldn't > > include some "ld-elf.a" type of intelligence? Would that be > > necessary and sufficient to allow statically-linked programs to > > load shared objects? > > Yes, and yes. No, and no. A complete executable (i.e. staticly linked) does not export any symbols, or at least not in the same way a shared executable and shared libraries do. If I try to dynamicly link libbar into a complete executable foo and libbar depends on libc, then there's no guarantee that all the required bits from libc are in foo, nor is there any guarantee that the bits are actually visible or even accessable (no linkage table). Dynamicly loading libc for use by libbar can work, but it's not guaranteed. One failure mode is suddenly having two instances of a common variable instead of one. Another is the clobbering of data caused reinitializations. So, the problem of dynamic linking a shared library into a static process is non-trivial and probably cannot be solved genericly. Under restricted and controlled conditions you can make it work. I would call it the ability to have plugins, not the ability to load dynamic libraries. -- Marcel Moolenaar USPA: A-39004 [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: rtld + static linking
"E.B. Dreger" wrote: > After watching the recent shared/dynamic threads, and reading the > archives from five or six years ago, I have a question... > > Dynamic linking works by the kernel running the dynamic linker, > which loads shared objects and fixes the symbol tables, yes? No. Dynamic linking works because the crt0 mmap's the /usr/libexec/ld.so file as executable, and then points known stub offsets into it. It then passes this as part of the environment, at a negative offset, into the _main, which fills out a little glue table. After all this is set up, then _main calls the location .entry in the executable, which is usally main, but can be set to something else at link time. This all works because the crt0.o has some self-knowledge for a number of symbol offsets, and because _main is called with the environment descriptor. Basically, the environment descriptor is lost in the static linking case. > Is there some reason that a statically-linked program couldn't > include some "ld-elf.a" type of intelligence? Would that be > necessary and sufficient to allow statically-linked programs to > load shared objects? Yes, and yes. The main reason that there is no dlopen is that the environment descriptor is lost. This is pretty trivial to remedy, but it means passing the environment descriptor to something that can use it to set up the startup. This is complicated by the fact that only a single .init entry point is usable, so if you were to override it, you would lose library initialization for C libraries, and you would fail to call constructor code for statically declare class instances in C++ code, and lose out on other linker set stuff, such as per thread exception stacks, etc.. The trivial fix for this is to add a void * parameter to the constructor iterator in the crt0, and then pass the environment there, so that you could implement a libdlopen that took that and used it to obtain the self-knowledge of the crt0 that was needed to (1) adjust the stub pointers to an mmap'ed ld.so, and (2) provide the access to the symbol table needed so that when you loaded modules, they would link properly vs. the symbols in the executable itself. You would either need to change the list to specifically reference the libc symbols, OR you would need to reload libc.so (the problem with doing the latter is that you might get a different libc out of it, and the executable symbols that may replace the libc symbols wouldn't do so for modules, so that's a non-starter). It's actually a pretty trivial crt0 and ld change to deal with this, the sticking point is that you have to also change the libgcc and other GNU code to pass a NULL value to the void * constructor parameter in the default case. This would make the code minorly incompatible with Linux, etc., unless you could get the GCC people to pick up your change. -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
