Re: [R] dyn.load(now = FALSE) not actually lazy?
Hi Ivan,
Thanks for the suggestion.
I found that libpqwalreceiver.so, which is part of pgsql-15.1, made for a
simpler example of my issue. So, using your advice, I ran the following in
my shell:
LD_DEBUG=libs R -e
'dyn.load("/stornext/System/data/tools/pgsql/pgsql-15.1/lib/libpqwalreceiver.so",
now=FALSE)'
I have attached the relevant part of the output from this command below.
The gist of it is that, indeed, R starts to look for dependent libraries
even if now=FALSE. Maybe it is the case that, as Duncan suggests, this is
intended behaviour? But I think it's not ideal behaviour because otherwise
it requires you to topologically sort all library dependencies and load
them in the optimal order, which is very difficult to do. Is there someone
involved in R development who can explain this behaviour to me?
Cheers.
***
>
dyn.load("/stornext/System/data/tools/pgsql/pgsql-15.1/lib/libpqwalreceiver.so",
now=FALSE)
1655: find library=libpq.so.5 [0]; searching
1655: search
path=/stornext/System/data/apps/gcc/gcc-11.1.0/lib64:/stornext/System/data/apps/pcre2/pcre2-10.36-gcc-11.1.0/lib:/stornext/System/data/tools/openSSL/openSSL-1.1.1n/lib:/stornext/System/data/apps/R/R-4.2.1/lib64/R/lib:/usr/local/lib64:/stornext/System/data/tools/openjdk/openjdk-13.0.2/lib/server:/stornext/System/data/apps/hdf5/hdf5-1.8.20/lib
(LD_LIBRARY_PATH)
1655: trying
file=/stornext/System/data/apps/gcc/gcc-11.1.0/lib64/libpq.so.5
1655: trying
file=/stornext/System/data/apps/pcre2/pcre2-10.36-gcc-11.1.0/lib/libpq.so.5
1655: trying
file=/stornext/System/data/tools/openSSL/openSSL-1.1.1n/lib/libpq.so.5
1655: trying
file=/stornext/System/data/apps/R/R-4.2.1/lib64/R/lib/libpq.so.5
1655: trying file=/usr/local/lib64/libpq.so.5
1655: trying
file=/stornext/System/data/tools/openjdk/openjdk-13.0.2/lib/server/libpq.so.5
1655: trying
file=/stornext/System/data/apps/hdf5/hdf5-1.8.20/lib/libpq.so.5
1655: search
path=/usr/pgsql-15/lib/tls/x86_64:/usr/pgsql-15/lib/tls:/usr/pgsql-15/lib/x86_64:/usr/pgsql-15/lib
(RUNPATH from file
/stornext/System/data/tools/pgsql/pgsql-15.1/lib/libpqwalreceiver.so)
1655: trying file=/usr/pgsql-15/lib/tls/x86_64/libpq.so.5
1655: trying file=/usr/pgsql-15/lib/tls/libpq.so.5
1655: trying file=/usr/pgsql-15/lib/x86_64/libpq.so.5
1655: trying file=/usr/pgsql-15/lib/libpq.so.5
1655: search cache=/etc/ld.so.cache
1655: trying file=/lib64/libpq.so.5
1655:
1655: find library=libssl.so.10 [0]; searching
1655: search
path=/stornext/System/data/apps/gcc/gcc-11.1.0/lib64:/stornext/System/data/apps/pcre2/pcre2-10.36-gcc-11.1.0/lib:/stornext/System/data/tools/openSSL/openSSL-1.1.1n/lib:/stornext/System/data/apps/R/R-4.2.1/lib64/R/lib:/usr/local/lib64:/stornext/System/data/tools/openjdk/openjdk-13.0.2/lib/server:/stornext/System/data/apps/hdf5/hdf5-1.8.20/lib
(LD_LIBRARY_PATH)
1655: trying
file=/stornext/System/data/apps/gcc/gcc-11.1.0/lib64/libssl.so.10
1655: trying
file=/stornext/System/data/apps/pcre2/pcre2-10.36-gcc-11.1.0/lib/libssl.so.10
1655: trying
file=/stornext/System/data/tools/openSSL/openSSL-1.1.1n/lib/libssl.so.10
1655: trying
file=/stornext/System/data/apps/R/R-4.2.1/lib64/R/lib/libssl.so.10
1655: trying file=/usr/local/lib64/libssl.so.10
1655: trying
file=/stornext/System/data/tools/openjdk/openjdk-13.0.2/lib/server/libssl.so.10
1655: trying
file=/stornext/System/data/apps/hdf5/hdf5-1.8.20/lib/libssl.so.10
1655: search cache=/etc/ld.so.cache
1655: trying file=/lib64/libssl.so.10
1655:
1655: find library=libcrypto.so.10 [0]; searching
1655: search
path=/stornext/System/data/apps/gcc/gcc-11.1.0/lib64:/stornext/System/data/apps/pcre2/pcre2-10.36-gcc-11.1.0/lib:/stornext/System/data/tools/openSSL/openSSL-1.1.1n/lib:/stornext/System/data/apps/R/R-4.2.1/lib64/R/lib:/usr/local/lib64:/stornext/System/data/tools/openjdk/openjdk-13.0.2/lib/server:/stornext/System/data/apps/hdf5/hdf5-1.8.20/lib
(LD_LIBRARY_PATH)
1655: trying
file=/stornext/System/data/apps/gcc/gcc-11.1.0/lib64/libcrypto.so.10
1655: trying
file=/stornext/System/data/apps/pcre2/pcre2-10.36-gcc-11.1.0/lib/libcrypto.so.10
1655: trying
file=/stornext/System/data/tools/openSSL/openSSL-1.1.1n/lib/libcrypto.so.10
1655: trying
file=/stornext/System/data/apps/R/R-4.2.1/lib64/R/lib/libcrypto.so.10
1655: trying file=/usr/local/lib64/libcrypto.so.10
1655: trying
file=/stornext/System/data/tools/openjdk/openjdk-13.0.2/lib/server/libcrypto.so.10
1655: trying
file=/stornext/System/data/apps/hdf5/hdf5-1.8.20/lib/libcrypto.so.10
1655: search cache=/etc/ld.so.cache
1655: trying
Re: [R] Detpack package
On 01/02/2023 11:41 a.m., Nick Wray wrote: I did use "detpack" ie not with a capital detpack::chi2testuniform(vals,0.05) gives this: Error: 'chi2testuniform' is not an exported object from 'namespace:detpack' ?? Thanks Nick The only exports from detpack are these: > library(detpack) > ls(2) [1] "det.construct" "det.cut" "det.de""det.leafs" "det.query" "det.rnd" "det1" [8] "det2" It looks as though the author of detpack documented chi2testuniform, but forgot to export it. You can see the source of the version on CRAN here: https://github.com/cran/detpack The author used Roxygen2, but apparently forgot to say @export before that function (and some others). The package tests perfectly OK on CRAN, because none of the examples made it into the help pages, where they would have failed. Duncan Murdoch __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Detpack package
I did use "detpack" ie not with a capital detpack::chi2testuniform(vals,0.05) gives this: Error: 'chi2testuniform' is not an exported object from 'namespace:detpack' ?? Thanks Nick On Wed, 1 Feb 2023 at 16:29, Eric Berger wrote: > Detpack or detpack? > > What happens when you try detpack::chi2testuniform(...) ? > > > On Wed, Feb 1, 2023 at 6:08 PM Nick Wray wrote: > >> Hello I've successfully installed the package "Detpack" and remembered >> to >> call it via library() But when I try to use the functions i get this: >> >> >> chi2testuniform(vals, 0.05) >> >> Error in chi2testuniform(vals, 0.05) : >> could not find function "chi2testuniform" >> >> >> And yet this function (amongst other which also are not recognised) are >> all >> listed on the CRAN page: >> http://cran.nexr.com/web/packages/detpack/detpack.pdf >> >> I can't remember anything like this happening before - as far as I can see >> I don't need to load or write anything else before executing the function >> and I can't find references to any problems with this package on the net >> Can anyone cast light on what's happening? Thanks Nick Wray >> >> [[alternative HTML version deleted]] >> >> __ >> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> > [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Detpack package
Detpack or detpack? What happens when you try detpack::chi2testuniform(...) ? On Wed, Feb 1, 2023 at 6:08 PM Nick Wray wrote: > Hello I've successfully installed the package "Detpack" and remembered to > call it via library() But when I try to use the functions i get this: > > > chi2testuniform(vals, 0.05) > > Error in chi2testuniform(vals, 0.05) : > could not find function "chi2testuniform" > > > And yet this function (amongst other which also are not recognised) are all > listed on the CRAN page: > http://cran.nexr.com/web/packages/detpack/detpack.pdf > > I can't remember anything like this happening before - as far as I can see > I don't need to load or write anything else before executing the function > and I can't find references to any problems with this package on the net > Can anyone cast light on what's happening? Thanks Nick Wray > > [[alternative HTML version deleted]] > > __ > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Detpack package
Hello I've successfully installed the package "Detpack" and remembered to call it via library() But when I try to use the functions i get this: chi2testuniform(vals, 0.05) Error in chi2testuniform(vals, 0.05) : could not find function "chi2testuniform" And yet this function (amongst other which also are not recognised) are all listed on the CRAN page: http://cran.nexr.com/web/packages/detpack/detpack.pdf I can't remember anything like this happening before - as far as I can see I don't need to load or write anything else before executing the function and I can't find references to any problems with this package on the net Can anyone cast light on what's happening? Thanks Nick Wray [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] dyn.load(now = FALSE) not actually lazy?
В Wed, 1 Feb 2023 14:16:54 +1100 Michael Milton пишет: > Is this a bug in the `dyn.load` implementation for R? If not, why is > it behaving like this? What should I do about it? On Unix-like systems, dyn.load forwards its arguments to dlopen(). It should be possible to confirm with a debugger that R passes RTLD_NOW to dlopen() when calling dyn.load(now = TRUE) and RTLD_LAZY when calling dyn.load(now = FALSE). I don't know for sure why the symbols are being resolved despite you asked the linker not to. Did something in the system set the LD_BIND_NOW environment variable? Do any of the libraries in the dependency tree have any constructors (C++ or __attribute__((constructor)) or otherwise mentioned in .ini* sections) that rely on MKL being available at initialisation time? If you launch R with the environment variable LD_DEBUG=libs set, the debugging output may shine some light on the problem. -- Best regards, Ivan __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] dyn.load(now = FALSE) not actually lazy?
According to the help page for dyn.load, I think this is within the
allowed behaviour:
"now:
a logical controlling whether all symbols are resolved (and relocated)
immediately the library is loaded or deferred until they are used. This
control is useful for developers testing whether a library is complete
and has all the necessary symbols, and for users to ignore missing
symbols. Whether this has any effect is system-dependent."
It appears to be intended for a DLL that doesn't define all the symbols
that your program will use, not for a DLL that has external references
that can't be resolved. And there's that last sentence.
I think for what you want, you'd have to write the DLL (i.e. libtorch)
in such a way that it does delayed loading of its dependencies.
Duncan Murdoch
On 31/01/2023 10:16 p.m., Michael Milton wrote:
On Linux, if I have a .so file that has a dependency on another .so, and I
`dyn.load(now=FALSE)` the first one, R seems to try to resolve the symbols
immediately, causing the load to fail.
For example, I have `libtorch` installed on my HPC. Note that it links to
various libs such as `libcudart.so` and `libmkl_intel_lp64.so.2` which
aren't currently in my library path:
➜ ~ ldd
/stornext/System/data/nvidia/libtorch-gpu/libtorch-gpu-1.12.1/lib/libtorch_cpu.so
linux-vdso.so.1 => (0x7ffcab58c000)
libgomp.so.1 =>
/stornext/System/data/apps/gcc/gcc-11.2.0/lib64/libgomp.so.1
(0x7f8cb22bf000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x7f8cb20a3000)
libc10.so =>
/stornext/System/data/nvidia/libtorch-gpu/libtorch-gpu-1.12.1/lib/libc10.so
(0x7f8cb1e2d000)
libnuma.so.1 => /lib64/libnuma.so.1 (0x7f8cb1c21000)
librt.so.1 => /lib64/librt.so.1 (0x7f8cb1a19000)
libgcc_s.so.1 =>
/stornext/System/data/apps/gcc/gcc-11.2.0/lib64/libgcc_s.so.1
(0x7f8cb1801000)
libdl.so.2 => /lib64/libdl.so.2 (0x7f8cb15fd000)
libmkl_intel_lp64.so.2 => not found
libmkl_gnu_thread.so.2 => not found
libmkl_core.so.2 => not found
libm.so.6 => /lib64/libm.so.6 (0x7f8cb12fb000)
libcudart.so.11.0 => not found
Then in R, if I try to load that same file:
dyn.load("/stornext/System/data/nvidia/libtorch-gpu/libtorch-gpu-1.12.1/lib/libtorch_cpu.so",
now=FALSE)
Error in
dyn.load("/stornext/System/data/nvidia/libtorch-gpu/libtorch-gpu-1.12.1/lib/libtorch_cpu.so",
:
unable to load shared object
'/stornext/System/data/nvidia/libtorch-gpu/libtorch-gpu-1.12.1/lib/libtorch_cpu.so':
libmkl_intel_lp64.so.2: cannot open shared object file: No such file or
directory
Is this a bug in the `dyn.load` implementation for R? If not, why is it
behaving like this? What should I do about it?
For reference, I'm on CentOS 7, with Linux
kernel 3.10.0-1160.81.1.el7.x86_64.
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
