On Sat, Feb 15, 2003, [EMAIL PROTECTED] wrote:
> [...]
> o the resulting binaries are still not portable between different versions of
> the OS due to the fact that not all libraries are static.
That just partly holds true: in OpenPKG we link our executables not
_statically_. The are still linked _dynamically_ against all available
system libraries. It is just our own provided libraries we link
in statically. And for them, different OS versions usually do not
make great difference. What you tell here is the general problem of
_statically_ linked (think "gcc -static") executables which then have
no chance to pick up a different OS libc, etc. So one has to clearly
differentiate between static libraries part of the game and real static
executable linking. The first is what we're doing, the second is what
we're not doing at all.
Can you give an example of an OpenPKG provided library which causes
problems after linking into an OpenPKG executable when running under a
newer OS version?
> o it often happens that packages pick up the libraries provided by the host OS
> instead of the OpenPKG counterparts during build because the standard
> configure scripts are satisified with finding the required libs in standard
> places (like /lib or /usr/lib).
>
> oo modifiying the configure stuff is a maintainance nightmare and violates the
> minimal impact philosophy.
Although correct and a nasty problem with configure scripts, this has
nothing to do with shared versus static libraries. Independent whether
we provide a <prefix>/lib/libfoo.a or a <prefix>/lib/foo.so, it would
be not found under build-time correctly in this situation. So, AFAIK
these two points are neither points for or against using static
libraries. It is an independent problem we're faced in OpenPKG.
> oo it is rather difficult to detect if the wrong library or the wrong header
> was used during build when looking at the final binaries. The simple ldd
> approach usable for shared libraries fails here.
Correct, with shared libraries it is easier to determine the result.
But that's just a developer issue and, sorry, IMHO again not a point
against static libraries. Because even if you see with "ldd" that the
wrong library was linked in, you still have the same problem than with
a statically linked in library: you have to rebuild anyway to solve
the problem. So I count this not a real advantage (especially not for
the end user), especially because at the same time you get a major
disadvantage (see below).
> oo When doing the recommended build from source the resulting binaries differ
> depending on many factors. This is a big maintainane problem for us because
> we can not easily help people on mailing lists in this case and only get
> useless bug reports.
Sorry, I do not understand this. Independent whether you link in
libraries dynamically or statically, the resulting binaries will differ,
of course. Because of different OS, different OS versions, different
patchlevels, different underlying architectures, etc. So, for me this
reduces to the "ldd" point above, right?
> I therefor want to propose to drop the static building approach in OpenPKG. In
> the end it looks like there is more to loose than to win with static linking.
> All platforms which are supported by OpenPKG and also most unsupported
> platforms have in the meantime decent support for shared libraries.
Until now the only addressed advantage for using dynamic libraries
is the "ldd" issue above. But for this advantage you get a big
disadvantage: you no longer can easily run more than one OpenPKG
instance on the same machine! This multiple instance approach is one
of the core ideas of OpenPKG, because it allows you to run multiple
software instances on the same system -- usually to give each project
its own environment, etc.
To see the point: In your Kroupware/Kolab project you establish a /kolab
instance for your own purposes. Let's say another project Foo does the
same nice and useful approach and uses /foo. For both all is fine,
even if OpenPKG would use dynamically linked libraries. Now assume a
user finds both Kolab and Foo cool and wants to run both of them. If
dynamic libraries are part of the game, one instance with break the
other all the time, because executables of both instance will search
for libraries sequentially with a by-user LD_LIBRARY_PATH or by-system
ldconfig approach. BANG! Do you see the point?
-
So, yes, I fully agree that optionally supporting dynamic libraries is
one of the goals for OpenPKG 2.0. But IMHO not really for the above
points you mentioned. More because security upgrades get easier (think
back to the OpenPKG SA about zlib), OpenPKG instances get smaller
(reduced disk size), etc.
But to be able to provide optional support for dynamic libraries we
first have to solve the following issues:
1. Building shared libraries today really required GNU libtool or
you end up with lots of platform ld fiddling on your own. For most
packages we can do in foo.spec:
%if "%{with_dso}" == "yes"
--enable-shared \
%else
--disable-shared \
%endif
while for the others we have to add a dependency to the "libtool"
package and patch the Makefiles to use it.
2. Once a a DSO <prefix>/lib/foo.so is installed and a package "bar"
linking against it, we have to make sure that the RTLD of the system
when loading <prefix>/bin/bar really picks up <prefix>/lib/foo.so
and not <other>/lib/foo.so. Under Unix there exists mainly three
ways to ensure this: "ld" and "-rpath/-R" under link-time of
<prefix>/bin/bar, a by-system ldconfig RTLD hint that <prefix>/lib
is a library directory or a by-user hint via LD_LIBRARY_PATH that
<prefix>/lib is a library directory.
The last two approaches are the most portable ones but (as mentioned
above) cause conflicts in case more than one OpenPKG instance is
installed on the same system. OTOH the "ld" + "-rpath/-R" stuff is
very elegant and usually allows resolving conflicts, but requires
a modern Unix (with usually something like ELF plus a smart RTLD)
and especially means lots of fiddling while packaging (at least if
libtool is not already part of the game).
As I see it currently, the best and less intrusive approach to add
optional DSO support for libraries is to develop some RPM macro
magic which (when called at end if %install) automatically wraps all
<prefix>/bin and <prefix>/sbin binaries to ensure that on execution of
them an LD_LIBRARY_PATH exists with has <prefix>/lib in front of it.
This way we resolve conflicts between multiple OpenPKG instances on a
by-call basis and at the same time do not have to fiddle around with the
build process of all the various applications linking against shared
libraries.
The wrapping could be done this way:
Input: <prefix>/bin/foo
Output: <prefix>/bin/.foo (moved <prefix>/bin/foo)
<prefix>/bin/foo -> <prefix>/bin/.env
The <prefix>/bin/.env is a small program provided by the bootstrap
package "openpkg" which determines the <prefix>/bin/foo path (via
resolving argv[0]), sets LD_LIBRARY_PATH=<prefix>/lib:$LD_LIBRARY_PATH
and execve(2)'s <prefix>/bin/.foo with its own given argument vector.
This way we can provide full backward compatibility and have a totally
less intrusive DSO support while packaging. The question just is whether
all this really is worth the effort? Ok, we get dynamic shared libaries,
but that's all. Instead we have some load-time overhead, twice as many
executables staying around, etc.
Keep also in mind that saying "Ok, if I just want to run a single
OpenPKG instance, we just can ignore all this and use a system-wide
ldconfig RTLD setup". This does also not work, because this way OpenPKG
would conflict with the system libraries (think again about libz,
libssl, libcrypto, etc. which usually all modern Unix vendors ship in
/usr/lib nowadays). So, IMHO using dynamic shared libraries in OpenPKG
really means _all_ of the above wrapping. And I'm still far away from
being convinced that all the disadvantages or outweight by the few
advantages.
Keep also in mind that all this is not an argument against DSOs in
operating systems. They are one of the greatest inventions of the last
decade and I like them very much (I was even they guy writing the DSO
support for Apache, including mod_so), but applying them to an add-on
sub-system like OpenPKG is a major problem, especially because OpenPKG
provides software potentially conflicting with the one the vendor
already ships.
What is the opinion of others on this subtle "dynamic shared libraries
in OpenPKG" issue?
The OpenPKG Project Ralf S. Engelschall
http://www.openpkg.org/ [EMAIL PROTECTED]
Cross-Platform Software Packaging www.engelschall.com
______________________________________________________________________
The OpenPKG Project www.openpkg.org
Developer Communication List [EMAIL PROTECTED]
