Re: Policy on static linking ?
Interesting reading the responses to this from over the weekend, and I think that Stuart Barkley's comment below strikes the biggest chord with me: Today, I probably wouldn't fight using dynamic linking. I do wish things would continue to provide static libraries unless there are specific reasons static libraries won't work. I would like to see libc remain fully functional when statically linked. I would like documentation about functionality lost when statically linking with libc. That's kind of my position too - for 99.9% of cases I (like everyone else) link dynamicly. But for those 0.1% of cases where static linking is a useful and good idea then I want to be able to do so - and I worry that we are heading for a situation where it's not going to be possible to link staticly with things in ports. If we had a standardised know to define then I could have submitted a pr including a patch and that might have been acccepted - after all the intent of removing statics was to prevent people linking with them without knowing, so if they have to explicitly enable ot then I assume that would be acceptable. -pete. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Policy on static linking ?
On Friday, 14 January 2011, Pete French petefre...@ticketswitch.com wrote: I build code using static linking for deployment across a set of machines. For me this has a lot of advantages - I know that the code will run, no matter what the state of the ports is on the machine, and if there is a need to upgrade a library then I do it once on the build machine, rebuild the executable, and rsync it out to the leaf nodes. Only one place to track security updates, only one place where I need to have all the porst the code depends on installed. I actually tried to compile a port against another and have it link statically, but I couldn't find a way to do so without hacking the configure script. I was wondering if there was another (and easier) way to do so... I use ldap for authentication purposes, along with pam_ldap and nss_ldap If I compile openldap-client against openssl from ports, then it creates massive problems elsewhere. For example, base ssh server will now crash due to using different libcrypto. compiling ports will also become impossible as bsd tar itself crash (removing ldap call from nsswitch.conf is required to work again) I was then advised in the freebsd forums to uninstall openssl port, compile openldap against openssl base, install it, then re-install openssl port. (I have to use openssl from ports with apache/subversion to fix a bug with TLSv1 making svn commit crash under some circumstances) I dislike this method, because should openldap gets upgraded again and be linked against openssl port, I will lock myself out of the machine again due to sshd crashing. Just like what happened today :( So how can I configure openldap-client to link against libssl and libcrypto statically? Thanks ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Policy on static linking ?
On 1/15/2011 5:11 AM, Jean-Yves Avenard wrote: If I compile openldap-client against openssl from ports, then it creates massive problems elsewhere. [snip problems] I dislike this method, because should openldap gets upgraded again and be linked against openssl port, I will lock myself out of the machine again due to sshd crashing. Just like what happened today :( Problems like that are why I do my package compiling in a jail, or a VM, and not on a live system. Jails are simple to setup these days and it's handy to be able to just blow away the jail and recreate it if things go wonky with dependencies when experimenting with package builds. (Or snapshot a VM before trying) I've taken a stab or two at compiling ports static in the past and also came up empty. It would be really nice to be able to build a single tbz package that would run once installed and that didn't have to pull down every other dependency individually. There are a number of ways that dependency tracking with packages can go sideways, and it isn't fun when trying to ensure that said packages install OK when transplanting them to other machines... Jim ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Policy on static linking ?
On Sat, Jan 15, 2011 at 09:11:50PM +1100, Jean-Yves Avenard wrote:
On Friday, 14 January 2011, Pete French petefre...@ticketswitch.com wrote:
I build code using static linking for deployment across a set of
machines. For me this has a lot of advantages - I know that the
code will run, no matter what the state of the ports is on the
machine, and if there is a need to upgrade a library then I do it
once on the build machine, rebuild the executable, and rsync it out
to the leaf nodes. Only one place to track security updates, only
one place where I need to have all the porst the code depends on
installed.
I actually tried to compile a port against another and have it link
statically, but I couldn't find a way to do so without hacking the
configure script. I was wondering if there was another (and easier)
way to do so...
I use ldap for authentication purposes, along with pam_ldap and nss_ldap
If I compile openldap-client against openssl from ports, then it
creates massive problems elsewhere.
For example, base ssh server will now crash due to using different
libcrypto. compiling ports will also become impossible as bsd tar
itself crash (removing ldap call from nsswitch.conf is required to
work again)
I was then advised in the freebsd forums to uninstall openssl port,
compile openldap against openssl base, install it, then re-install
openssl port.
(I have to use openssl from ports with apache/subversion to fix a bug
with TLSv1 making svn commit crash under some circumstances)
I dislike this method, because should openldap gets upgraded again and
be linked against openssl port, I will lock myself out of the machine
again due to sshd crashing. Just like what happened today :(
So how can I configure openldap-client to link against libssl and
libcrypto statically?
I think this can be solved with a symbol versioning trick. By applying a
different version to all symbols from each OpenSSL version, the dynamic
linker will be able to distinguish between different versions of OpenSSL
and will use the correct OpenSSL version each object was linked to, even
if there are multiple OpenSSL versions in the process. Note that each
OpenSSL version should still have its own SONAME (the security/openssl
port does this).
The version script can be as simple as (substituting the version)
OPENSSL_0.9.8 {
global:
*;
};
and needs to be in the top level and in the engines directory.
For it to work completely, both base and the port need to be patched.
Old binaries continue to work but the benefit only appears after
recompilation.
The SONAME needs to be bumped when the version string is changed or
deleted (but not when it is initially added), otherwise binaries will
stop working. This also means that making the change cannot be undone
without breaking binary compatibility.
What will not work is allocating an OpenSSL structure in one object
linked to one OpenSSL version and then using it in another object linked
to another OpenSSL version. That would require true symbol versioning,
keeping compatibility with old versions in the same library with the
same SONAME. Unlike the approach I propose, that would be a lot of work
and can only be done by the OpenSSL project, and I think their policy is
not to do such extra work for ABI compatibility. If they change their
mind they will probably start with the symver version of the previous
release so as to remain compatible with what various Linux distributions
are doing.
Also, a side effect is that it is no longer possible to cheat by
symlinking different OpenSSL versions.
The approach has been used by Debian for some time.
Links:
http://chris.dzombak.name/blog/2010/03/building-openssl-with-symbol-versioning/
http://chris.dzombak.name/files/openssl/openssl-0.9.8l-symbolVersioning.diff
http://rt.openssl.org/Ticket/Display.html?id=1222user=guestpass=guest
--
Jilles Tjoelker
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Policy on static linking ?
Hi On 15 January 2011 23:48, Jilles Tjoelker jil...@stack.nl wrote: The approach has been used by Debian for some time. Links: http://chris.dzombak.name/blog/2010/03/building-openssl-with-symbol-versioning/ http://chris.dzombak.name/files/openssl/openssl-0.9.8l-symbolVersioning.diff http://rt.openssl.org/Ticket/Display.html?id=1222user=guestpass=guest This sounds very interesting. I do have trouble understanding on how this would make a difference with how it's currently working. base openssl uses libssl.so.6 and libcrypto.so.6 current port openssl is using .so.7 So they too have different sonames; How could changing this to .so.0.9.8 for base and so.1.0.2 for port make things behave differently? JY ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Policy on static linking ?
Hi On 16 January 2011 02:17, Jean-Yves Avenard jyaven...@gmail.com wrote: Hi On 15 January 2011 23:48, Jilles Tjoelker jil...@stack.nl wrote: The approach has been used by Debian for some time. Links: http://chris.dzombak.name/blog/2010/03/building-openssl-with-symbol-versioning/ http://chris.dzombak.name/files/openssl/openssl-0.9.8l-symbolVersioning.diff http://rt.openssl.org/Ticket/Display.html?id=1222user=guestpass=guest This sounds very interesting. I do have trouble understanding on how this would make a difference with how it's currently working. base openssl uses libssl.so.6 and libcrypto.so.6 current port openssl is using .so.7 So they too have different sonames; How could changing this to .so.0.9.8 for base and so.1.0.2 for port make things behave differently? Replying to myself.. Looking at the symbols in the openssl libraries found on a Ubuntu machine, I see what is going on: symbol name: X509_NAME_cmp@OPENSSL_0.9.8 vs X509_NAME_cmp This sounds like a great approach.. and should definitely resolve my problems I think. Are you sure both base and port needs to be patched? I would have assumed that patching only port would be sufficient (provided all tools depending on it are also recompiled) Jean-Yves ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Policy on static linking ?
I build code using static linking for deployment across a set of machines. For me this has a lot of advantages - I know that the code will run, no matter what the state of the ports is on the machine, and if there is a need to upgrade a library then I do it once on the build machine, rebuild the executable, and rsync it out to the leaf nodes. Only one place to track security updates, only one place where I need to have all the porst the code depends on installed. I recently wanted to use libdespatch, but I found that the port didn't install the static libraries. I filed a PR, and found out from the reponse that this was deliberate, and that a number of other ports were deliberately excluding static libraries too. Some good reasons where given, which I wont reporduce here, as you can read them at: http://www.freebsd.org/cgi/query-pr.cgi?pr=151306 Today I finally hit the problem where a critical library I am using has stopped working with static libraries (or so it appears at first glance). I was wondering what the general policy here was - should I just bite the bullet and go dynamic, and accept the maintannance headache that cases, or could we define something like 'WITH_STATIC_LIBRARIES' that could be set which would make ports install a set of static libraries (maybe into a separate /usr/local/lib/static?) so that the likes of me could continue to build static code ? I'd very much like to be able to continue to ship single executables that just run, but if theres some policy to only have dynamic libraries in ports going forward then fair enough... -pete. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Policy on static linking ?
On Fri, Jan 14, 2011 at 02:07:37PM +, Pete French wrote: I build code using static linking for deployment across a set of machines. For me this has a lot of advantages - I know that the code will run, no matter what the state of the ports is on the machine, and if there is a need to upgrade a library then I do it once on the build machine, rebuild the executable, and rsync it out to the leaf nodes. Only one place to track security updates, only one place where I need to have all the porst the code depends on installed. I recently wanted to use libdespatch, but I found that the port didn't install the static libraries. I filed a PR, and found out from the reponse that this was deliberate, and that a number of other ports were deliberately excluding static libraries too. Some good reasons where given, which I wont reporduce here, as you can read them at: http://www.freebsd.org/cgi/query-pr.cgi?pr=151306 Today I finally hit the problem where a critical library I am using has stopped working with static libraries (or so it appears at first glance). I was wondering what the general policy here was - should I just bite the bullet and go dynamic, and accept the maintannance headache that cases, or could we define something like 'WITH_STATIC_LIBRARIES' that could be set which would make ports install a set of static libraries (maybe into a separate /usr/local/lib/static?) so that the likes of me could continue to build static code ? I'd very much like to be able to continue to ship single executables that just run, but if theres some policy to only have dynamic libraries in ports going forward then fair enough... Various features do not work with static linking because dlopen() does not work from static executables. Libraries that are also used by dlopen()ed modules should generally be linked dynamically, particularly if these libraries have global state. Things that use dlopen() include NSS (getpwnam() and the like), PAM and most plugin systems. If libc is statically linked, NSS falls back to a traditional mode that only supports the traditional things (e.g. no LDAP user information); I think PAM and most plugin systems do not work at all. For some system libraries, there can be kernel compatibility problems that prevent old libraries from working, although an ABI-compatible shared library is available. This has happened with 6.x's libkse: binaries statically linked to it do not run on 8.x or newer, while libkse can be remapped to libthr for binaries dynamically linked to libkse. For these reasons, static linking to libc, libpthread and similar system libraries should be reserved for /rescue/rescue and similar programs, and not used in general. Another feature only available with dynamic linking is hidden symbols that are available only inside the shared object. Compiling a library that uses this feature as a static library will make the hidden symbols visible to the application or other libraries. This may cause name clashes that otherwise wouldn't have been a problem or invite API abuse. Proper use of hidden symbols can also speed up linking and load times considerably, particularly if the code is written in C++. Other issues are static linking's requirement to list all libraries a library depends on and in the correct order. With dynamic linking, listing the indirect dependencies is unnecessary and best avoided. This is generally not very hard to fix but still needs extra effort. (For example, pkg-config has Libs.private to help with it.) If you want to link dynamically but avoid too much management overhead, consider using PCBSD's PBI system which allows you to ship all necessary .so files (except system ones) with your application. -- Jilles Tjoelker ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Policy on static linking ?
On Fri, Jan 14, 2011 at 7:07 AM, Pete French petefre...@ticketswitch.com wrote: I build code using static linking for deployment across a set of machines. For me this has a lot of advantages - I know that the code will run, no matter what the state of the ports is on the machine, and if there is a need to upgrade a library then I do it once on the build machine, rebuild the executable, and rsync it out to the leaf nodes. Only one place to track security updates, only one place where I need to have all the porst the code depends on installed. You still have to track security updates for the rest of your system. ssh for example. Kernel bugs too (though admittedly those are much more rare). ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Policy on static linking ?
Being an old curmudgeon, I have historically preferred static linking for some of the points raised (no external installation dependencies, I know the application is complete and will work, use of believed good library versions, custom library patches are possible, etc). The other response about needing to be explicit at link time about the libraries in use is something I like in general. A programmer should know all the dependencies of his application. I find it really disgusting when some little application uses a library with a huge dependency base. This is especially true when the application doesn't use most of that base. Some applications pull needed library code into their own source tree and build their own versions. This can be useful in some cases. Some of the bloatware application do this to excess. If it is necessary, I don't know how anyone can make a proper and usable security declaration about an application which is dynamically linked against other libraries. On the other hand, the state of shared libraries is probably much better than the last time I actually cared. People have learned about maintaining backward compatibility, API versioning, etc. If you make good decisions about the libraries you depend upon life can be much better. Don't just select libraries that are flavor of the day but instead use libraries with good solid history and apparent ongoing support. The other response indicates that even libc with some of the core functionality doesn't work well with static linking. I'm disappointed about that state of things, but the reasons seem valid. It is also interesting about system version compatibility and providing shims for older ABIs. Also, in this day of interpreted code (perl, etc) much of that code base is equivalent to dynamically linked. At $DAYJOB I've seen a few things with their own versions of certain common libraries included. I've been curious about the reasons but haven't had time to investigate. The use of dlopen() also looks to make plugins much easier to do than with statically linked programs. Plugin equivalents in some of my earlier applications where much trickier to write since they also needed to be statically linked with a very reduced set of available support routines (basically none for the things I was doing at the time). Today, I probably wouldn't fight using dynamic linking. I do wish things would continue to provide static libraries unless there are specific reasons static libraries won't work. I would like to see libc remain fully functional when statically linked. I would like documentation about functionality lost when statically linking with libc. Mostly just some ramblings, Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Policy on static linking ?
On Fri 14 Jan 2011 at 06:07:37 PST Pete French wrote: I recently wanted to use libdespatch, but I found that the port didn't install the static libraries. I filed a PR, and found out from the reponse that this was deliberate, and that a number of other ports were deliberately excluding static libraries too. Some good reasons where given, which I wont reporduce here, as you can read them at: http://www.freebsd.org/cgi/query-pr.cgi?pr=151306 Interesting reading. One thing bothers me, however, about the reasons given against static linking. Surely, if a port statically links to a library, it calls out that library on a LIB_DEPENDS line and the dependency is reflected in the package database? So, if a security issue comes up with the library, it wouldn't be difficult to flag the dependent port as one that needs to be recompiled using the newly-patched library? The user only gets the patches to the shared library after he reads and responds to the security notice, or when he's doing a normal update of his ports. Correct? Well then, what's different about the scenario when it's a static library? What am I missing here? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
