Re: sort and -lm
On 07/10/2023 22:29, Paul Eggert wrote:
On 2023-10-07 04:42, Pádraig Brady wrote:
The auto linking is globally controlled with the --with-openssl
cofigure option, but you could build sort (and md5sum)
without that dependency with:
./configure ac_cv_lib_crypto_MD5=no
Thanks, I was thinking more along the lines that Bruno suggested, which
to continue to link to libcrypto, but do it with dlopen/dlsym in 'sort'
only when need_random is true.
It's not clear to me offhand whether this should be done entirely in
Coreutils, or whether we should add some Gnulib support to make it
easier to do this sort of lazier linking.
I was wondering if this was worth worrying about at all,
but it is a significant overhead that's worth improving.
To quantify the overhead I compared optimized builds,
with and without the above configure option, giving:
$ time seq 1 | xargs -I'{}' src/sort /dev/null -k'{}'
real0m7.009s
user0m3.462s
sys 0m3.578s
$ time seq 1 | xargs -I'{}' src/sort-lc /dev/null -k'{}'
real0m12.950s
user0m3.754s
sys 0m9.200s
So we should do something. Now dlopening libcrypto on demand
would work, but there may be better solutions.
sort doesn't have to use md5. It could use blake2 routines
already in coreutils to avoid the issue (and get some speed ups).
Alternatively it might use some other hash function.
For example see the other 128 bit functions compared at:
https://github.com/Cyan4973/xxHash
BTW there was mention of static linking as an option in this thread.
That's is an option to provide better speed an isolation for binaries,
however it's best left to the system builders to use this for their builds.
There can be security implications for prompt library updating,
and libcrypto is particularly sensitive in this regard.
Also, why doesn't coreutils/configure's ---with-linux-crypto option
cause 'sort' to use Linux kernel crypto? Is this because the syscall
overhead would be too high and it's significantly faster to do it in
user space?
I naively thought that --with-linux-crypto would mean we wouldn't need
to link to libcrypto. Evidently not.
Well --with-linux-crypto is for specialized setups
and not really for general usage, due to syscall overhead,
differing network syscall class (consider locked down environments),
and generally slower than libcrypto. That was detailed at:
https://git.sv.gnu.org/cgit/gnulib.git/commit/?id=71cc633a0f
cheers,
Pádraig
Re: sort and -lm
On 2023-10-07 04:42, Pádraig Brady wrote: The auto linking is globally controlled with the --with-openssl cofigure option, but you could build sort (and md5sum) without that dependency with: ./configure ac_cv_lib_crypto_MD5=no Thanks, I was thinking more along the lines that Bruno suggested, which to continue to link to libcrypto, but do it with dlopen/dlsym in 'sort' only when need_random is true. It's not clear to me offhand whether this should be done entirely in Coreutils, or whether we should add some Gnulib support to make it easier to do this sort of lazier linking. Also, why doesn't coreutils/configure's ---with-linux-crypto option cause 'sort' to use Linux kernel crypto? Is this because the syscall overhead would be too high and it's significantly faster to do it in user space? I naively thought that --with-linux-crypto would mean we wouldn't need to link to libcrypto. Evidently not.
Re: sort and -lm
> > Why would that be a problem? The average run time of a 'sort' invocation > > is long enough that an additional link dependency to libm shouldn't matter. > > We've avoided -lm in the past, I think more to avoid the dependency. You could alternatively link with libm.a instead of -lm. This won't increase the startup time of 'sort'. Bruno
Re: sort and -lm
On 07/10/2023 04:38, Paul Eggert wrote: On 2023-10-06 19:33, Bruno Haible wrote: Why would that be a problem? The average run time of a 'sort' invocation is long enough that an additional link dependency to libm shouldn't matter. We've avoided -lm in the past, I think more to avoid the dependency. If you really want to minimize the startup time, you could load the libcrypto, needed for the option '--random', using dlopen() ? I wouldn't advocate it as a solution on all platforms, since libltdl surely has its own set of portability problems. But guarded by a '#ifdef __GLIBC__', why not? Yes, that sounds like it might be a win too. I hadn't noticed the dependency on libcrypto (and libz). FYI. coreutils auto links with libcrypto >=3 since it's GPL compat, and quite a bit faster than the fallback routines. libz is a transitive dependency of libcrypto. The auto linking is globally controlled with the --with-openssl cofigure option, but you could build sort (and md5sum) without that dependency with: ./configure ac_cv_lib_crypto_MD5=no cheers, Pádraig
Re: sort and -lm
On 2023-10-06 19:33, Bruno Haible wrote: Why would that be a problem? The average run time of a 'sort' invocation is long enough that an additional link dependency to libm shouldn't matter. We've avoided -lm in the past, I think more to avoid the dependency. If you really want to minimize the startup time, you could load the libcrypto, needed for the option '--random', using dlopen() ? I wouldn't advocate it as a solution on all platforms, since libltdl surely has its own set of portability problems. But guarded by a '#ifdef __GLIBC__', why not? Yes, that sounds like it might be a win too. I hadn't noticed the dependency on libcrypto (and libz).
Re: sort and -lm
Paul Eggert wrote: > The win for GNU sort, once I get around to fixing totalorder's -lm > dependency Why would that be a problem? The average run time of a 'sort' invocation is long enough that an additional link dependency to libm shouldn't matter. 'sort' already links against libcrypto: $ nm sort | grep ' U ' | grep -v @GLIBC U MD5_Final@OPENSSL_3.0.0 U MD5_Init@OPENSSL_3.0.0 U MD5_Update@OPENSSL_3.0.0 If you really want to minimize the startup time, you could load the libcrypto, needed for the option '--random', using dlopen() ? I wouldn't advocate it as a solution on all platforms, since libltdl surely has its own set of portability problems. But guarded by a '#ifdef __GLIBC__', why not? Bruno
