[julia-users] Unable to dlopen openblas and dSFMT by default when using embedded Julia
This is on OS/X I do have a workaround, but I'd like to get to the bottom of it. I don't see anywhere in the code where either of these 2 libraries are loaded explicitly --- maybe it's in loading sys.ji? My workaround can be to set an env variable, but it would be better to understand and address the problem directly. If DYLD_LIBRARY_PATH is set to point to the Julia library directory, then all works, but if this environment variable is not set, then the following occurs Warning: error initializing module LinAlg: ErrorException(error compiling __init__: error compiling check_blas: error compiling blas_vendor: could not load module libopenblas: dlopen(libopenblas.dylib, 1): image not found) Entropy pool not available to seed RNG; using ad-hoc entropy sources. Warning: error initializing module Random: ErrorException(could not load module libdSFMT: dlopen(libdSFMT.dylib, 1): image not found) === Those additional errors I'm sure are just side effects cascading form failing to load those libraries. The engine still does function, but the linear algebra stuff is unavailable. I've tried various combinations of jl_init and jl_init_with_image to no effect. -Jeff
Re: [julia-users] Unable to dlopen openblas and dSFMT by default when using embedded Julia
Since Julia's libraries are in a non-standard directory, (usually prefix/lib/julia, where prefix is the root of your julia installation) we rely on a dynamic linker feature known as the RPATH to find these libraries at runtime. You can inspect the rpath of the julia executable with otool: $ otool -l julia | grep -A 2 LC_RPATH cmd LC_RPATH cmdsize 48 path @executable_path/../lib/julia (offset 12) -- cmd LC_RPATH cmdsize 40 path @executable_path/../lib (offset 12) This is encoded into the binary calling julia by putting in extra linker commands when compiling julia. Things like -Wl,-rpath,@executable_path/../lib I haven't played around with whether encoding RPATHS into libraries will work, so you either have to try one of the following: * Change the RPATH of the executable you're embedding julia into * Change the RPATH of libjulia itself * Tack things onto DYLD_FALLBACK_LIBRARY_PATH (Try to stay away from DYLD_LIBRARY_PATH, it's better to mess with the FALLBACK variant http://stackoverflow.com/questions/3146274/is-it-ok-to-use-dyld-library-path-on-mac-os-x-and-whats-the-dynamic-library-s when you can) * Move the libraries into an easier-to-find place. Either a place that is already on your executable's RPATH, or a location that is searched by default by the linker. The default locations are the default list of DYLD_FALLBACK_LIBRARY_PATH entries, listed in this man page https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/dyld.1.html, and are $(HOME)/lib:/usr/local/lib:/lib:/usr/lib. -E On Tue, Aug 12, 2014 at 3:12 AM, Jeff Waller truth...@gmail.com wrote: This is on OS/X I do have a workaround, but I'd like to get to the bottom of it. I don't see anywhere in the code where either of these 2 libraries are loaded explicitly --- maybe it's in loading sys.ji? My workaround can be to set an env variable, but it would be better to understand and address the problem directly. If DYLD_LIBRARY_PATH is set to point to the Julia library directory, then all works, but if this environment variable is not set, then the following occurs Warning: error initializing module LinAlg: ErrorException(error compiling __init__: error compiling check_blas: error compiling blas_vendor: could not load module libopenblas: dlopen(libopenblas.dylib, 1): image not found) Entropy pool not available to seed RNG; using ad-hoc entropy sources. Warning: error initializing module Random: ErrorException(could not load module libdSFMT: dlopen(libdSFMT.dylib, 1): image not found) === Those additional errors I'm sure are just side effects cascading form failing to load those libraries. The engine still does function, but the linear algebra stuff is unavailable. I've tried various combinations of jl_init and jl_init_with_image to no effect. -Jeff
Re: [julia-users] Unable to dlopen openblas and dSFMT by default when using embedded Julia
Either of those variables an be modified, but in both cases you should be attentive to the side effects. For example, IIRC, if the fallback path is set, the link won't search the usual fallback locations, like /usr/lib (unless you explicitly add them to your fallback path) The primary failure case for usage of either environment variable is the same: having multiple possible libraries that could satisfy a dependency, and ending up with multiple copies of the dylib loaded, with independent global state. On Tuesday, August 12, 2014, Elliot Saba staticfl...@gmail.com wrote: Since Julia's libraries are in a non-standard directory, (usually prefix/lib/julia, where prefix is the root of your julia installation) we rely on a dynamic linker feature known as the RPATH to find these libraries at runtime. You can inspect the rpath of the julia executable with otool: $ otool -l julia | grep -A 2 LC_RPATH cmd LC_RPATH cmdsize 48 path @executable_path/../lib/julia (offset 12) -- cmd LC_RPATH cmdsize 40 path @executable_path/../lib (offset 12) This is encoded into the binary calling julia by putting in extra linker commands when compiling julia. Things like -Wl,-rpath,@executable_path/../lib I haven't played around with whether encoding RPATHS into libraries will work, so you either have to try one of the following: * Change the RPATH of the executable you're embedding julia into * Change the RPATH of libjulia itself * Tack things onto DYLD_FALLBACK_LIBRARY_PATH (Try to stay away from DYLD_LIBRARY_PATH, it's better to mess with the FALLBACK variant http://stackoverflow.com/questions/3146274/is-it-ok-to-use-dyld-library-path-on-mac-os-x-and-whats-the-dynamic-library-s when you can) * Move the libraries into an easier-to-find place. Either a place that is already on your executable's RPATH, or a location that is searched by default by the linker. The default locations are the default list of DYLD_FALLBACK_LIBRARY_PATH entries, listed in this man page https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/dyld.1.html, and are $(HOME)/lib:/usr/local/lib:/lib:/usr/lib. -E On Tue, Aug 12, 2014 at 3:12 AM, Jeff Waller truth...@gmail.com javascript:_e(%7B%7D,'cvml','truth...@gmail.com'); wrote: This is on OS/X I do have a workaround, but I'd like to get to the bottom of it. I don't see anywhere in the code where either of these 2 libraries are loaded explicitly --- maybe it's in loading sys.ji? My workaround can be to set an env variable, but it would be better to understand and address the problem directly. If DYLD_LIBRARY_PATH is set to point to the Julia library directory, then all works, but if this environment variable is not set, then the following occurs Warning: error initializing module LinAlg: ErrorException(error compiling __init__: error compiling check_blas: error compiling blas_vendor: could not load module libopenblas: dlopen(libopenblas.dylib, 1): image not found) Entropy pool not available to seed RNG; using ad-hoc entropy sources. Warning: error initializing module Random: ErrorException(could not load module libdSFMT: dlopen(libdSFMT.dylib, 1): image not found) === Those additional errors I'm sure are just side effects cascading form failing to load those libraries. The engine still does function, but the linear algebra stuff is unavailable. I've tried various combinations of jl_init and jl_init_with_image to no effect. -Jeff
Re: [julia-users] Unable to dlopen openblas and dSFMT by default when using embedded Julia
Yes, Jameson brings up a really important point; if you do end up mucking around with any of the LIBRARY_PATH variables, they have silent defaults that should be added in. You can read about it in the man page linked above, but modifying the RPATH of the binary is a better choice in virtually every case. -E On Tue, Aug 12, 2014 at 1:14 PM, Jameson Nash vtjn...@gmail.com wrote: Either of those variables an be modified, but in both cases you should be attentive to the side effects. For example, IIRC, if the fallback path is set, the link won't search the usual fallback locations, like /usr/lib (unless you explicitly add them to your fallback path) The primary failure case for usage of either environment variable is the same: having multiple possible libraries that could satisfy a dependency, and ending up with multiple copies of the dylib loaded, with independent global state. On Tuesday, August 12, 2014, Elliot Saba staticfl...@gmail.com wrote: Since Julia's libraries are in a non-standard directory, (usually prefix/lib/julia, where prefix is the root of your julia installation) we rely on a dynamic linker feature known as the RPATH to find these libraries at runtime. You can inspect the rpath of the julia executable with otool: $ otool -l julia | grep -A 2 LC_RPATH cmd LC_RPATH cmdsize 48 path @executable_path/../lib/julia (offset 12) -- cmd LC_RPATH cmdsize 40 path @executable_path/../lib (offset 12) This is encoded into the binary calling julia by putting in extra linker commands when compiling julia. Things like -Wl,-rpath,@executable_path/../lib I haven't played around with whether encoding RPATHS into libraries will work, so you either have to try one of the following: * Change the RPATH of the executable you're embedding julia into * Change the RPATH of libjulia itself * Tack things onto DYLD_FALLBACK_LIBRARY_PATH (Try to stay away from DYLD_LIBRARY_PATH, it's better to mess with the FALLBACK variant http://stackoverflow.com/questions/3146274/is-it-ok-to-use-dyld-library-path-on-mac-os-x-and-whats-the-dynamic-library-s when you can) * Move the libraries into an easier-to-find place. Either a place that is already on your executable's RPATH, or a location that is searched by default by the linker. The default locations are the default list of DYLD_FALLBACK_LIBRARY_PATH entries, listed in this man page https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/dyld.1.html, and are $(HOME)/lib:/usr/local/lib:/lib:/usr/lib. -E On Tue, Aug 12, 2014 at 3:12 AM, Jeff Waller truth...@gmail.com wrote: This is on OS/X I do have a workaround, but I'd like to get to the bottom of it. I don't see anywhere in the code where either of these 2 libraries are loaded explicitly --- maybe it's in loading sys.ji? My workaround can be to set an env variable, but it would be better to understand and address the problem directly. If DYLD_LIBRARY_PATH is set to point to the Julia library directory, then all works, but if this environment variable is not set, then the following occurs Warning: error initializing module LinAlg: ErrorException(error compiling __init__: error compiling check_blas: error compiling blas_vendor: could not load module libopenblas: dlopen(libopenblas.dylib, 1): image not found) Entropy pool not available to seed RNG; using ad-hoc entropy sources. Warning: error initializing module Random: ErrorException(could not load module libdSFMT: dlopen(libdSFMT.dylib, 1): image not found) === Those additional errors I'm sure are just side effects cascading form failing to load those libraries. The engine still does function, but the linear algebra stuff is unavailable. I've tried various combinations of jl_init and jl_init_with_image to no effect. -Jeff
Re: [julia-users] Unable to dlopen openblas and dSFMT by default when using embedded Julia
If you don't have control over the compilation process of the binary, you can use `install_name_tool` to change the rpath of the binary after it's been compiled, but that is dependent on how long the new path is and how much slack space the compiler added to the dynamic library header beforehand.
Re: [julia-users] Unable to dlopen openblas and dSFMT by default when using embedded Julia
On Tuesday, August 12, 2014 12:40:02 PM UTC-4, Elliot Saba wrote: Since Julia's libraries are in a non-standard directory, (usually prefix/lib/julia, where prefix is the root of your julia installation) we rely on a dynamic linker feature known as the RPATH to find these libraries at runtime. You can inspect the rpath of the julia executable with otool: $ otool -l julia | grep -A 2 LC_RPATH cmd LC_RPATH cmdsize 48 path @executable_path/../lib/julia (offset 12) -- cmd LC_RPATH cmdsize 40 path @executable_path/../lib (offset 12) Nice, this is a useful. I knew about rpath and am using it, but not the option and the grep to pull it out of compiled binary for verification. I've verified that rpath is part of the object code created and is the right value (or at least corresponds to a value of DYLD_LIBRARY_PATH) that makes things work. This is encoded into the binary calling julia by putting in extra linker commands when compiling julia. Things like -Wl,-rpath,@executable_path/../lib I haven't played around with whether encoding RPATHS into libraries will work, so you either have to try one of the following: * Change the RPATH of the executable you're embedding julia into I think this is the critical problem. I did try that, BUT. I'm not creating an executable but another dynamically loaded library -- a module -- which is loaded by another system at runtime. I have no control of that other system, only the module. * Change the RPATH of libjulia itself I have no control over that; this is for others. This has to work with any deployment, but I am able to control how my thing is compiled. I can insert any flags, constants, or run any sort configuration gathering script I want as a pre-compilation step (within reason of course). * Tack things onto DYLD_FALLBACK_LIBRARY_PATH (Try to stay away from DYLD_LIBRARY_PATH, it's better to mess with the FALLBACK variant http://stackoverflow.com/questions/3146274/is-it-ok-to-use-dyld-library-path-on-mac-os-x-and-whats-the-dynamic-library-s when you can) Well this looks like what I will have to do for the time being at least. I agree this is the least desirable. * Move the libraries into an easier-to-find place. Either a place that is already on your executable's RPATH, or a location that is searched by default by the linker. The default locations are the default list of DYLD_FALLBACK_LIBRARY_PATH entries, listed in this man page https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/dyld.1.html, and are $(HOME)/lib:/usr/local/lib:/lib:/usr/lib. -E Likewise as above, I have no control over that. On Tue, Aug 12, 2014 at 3:12 AM, Jeff Waller trut...@gmail.com javascript: wrote: This is on OS/X I do have a workaround, but I'd like to get to the bottom of it. I don't see anywhere in the code where either of these 2 libraries are loaded explicitly --- maybe it's in loading sys.ji? My workaround can be to set an env variable, but it would be better to understand and address the problem directly. If DYLD_LIBRARY_PATH is set to point to the Julia library directory, then all works, but if this environment variable is not set, then the following occurs Warning: error initializing module LinAlg: ErrorException(error compiling __init__: error compiling check_blas: error compiling blas_vendor: could not load module libopenblas: dlopen(libopenblas.dylib, 1): image not found) Entropy pool not available to seed RNG; using ad-hoc entropy sources. Warning: error initializing module Random: ErrorException(could not load module libdSFMT: dlopen(libdSFMT.dylib, 1): image not found) === Those additional errors I'm sure are just side effects cascading form failing to load those libraries. The engine still does function, but the linear algebra stuff is unavailable. I've tried various combinations of jl_init and jl_init_with_image to no effect. -Jeff
Re: [julia-users] Unable to dlopen openblas and dSFMT by default when using embedded Julia
Alright, I worked up a little proof-of-concept demo here https://github.com/staticfloat/dltest/tree/master, but the long and the short of it is that you should be able to alter the rpath of YOUR library to include whatever paths you need, (probably relative to @loader_path so that your library just needs to know the relative path to openblas and friends) and you'll then be able to dlopen these guys to your heart's content. If you have any questions about this, please feel free to ask, but it should be as simple as adding a -Wl,-rpath,@loader_path/../lib or whatever to the linker options for your library. -E On Tue, Aug 12, 2014 at 2:02 PM, Jeff Waller truth...@gmail.com wrote: On Tuesday, August 12, 2014 12:40:02 PM UTC-4, Elliot Saba wrote: Since Julia's libraries are in a non-standard directory, (usually prefix/lib/julia, where prefix is the root of your julia installation) we rely on a dynamic linker feature known as the RPATH to find these libraries at runtime. You can inspect the rpath of the julia executable with otool: $ otool -l julia | grep -A 2 LC_RPATH cmd LC_RPATH cmdsize 48 path @executable_path/../lib/julia (offset 12) -- cmd LC_RPATH cmdsize 40 path @executable_path/../lib (offset 12) Nice, this is a useful. I knew about rpath and am using it, but not the option and the grep to pull it out of compiled binary for verification. I've verified that rpath is part of the object code created and is the right value (or at least corresponds to a value of DYLD_LIBRARY_PATH) that makes things work. This is encoded into the binary calling julia by putting in extra linker commands when compiling julia. Things like -Wl,-rpath,@executable_path/../lib I haven't played around with whether encoding RPATHS into libraries will work, so you either have to try one of the following: * Change the RPATH of the executable you're embedding julia into I think this is the critical problem. I did try that, BUT. I'm not creating an executable but another dynamically loaded library -- a module -- which is loaded by another system at runtime. I have no control of that other system, only the module. * Change the RPATH of libjulia itself I have no control over that; this is for others. This has to work with any deployment, but I am able to control how my thing is compiled. I can insert any flags, constants, or run any sort configuration gathering script I want as a pre-compilation step (within reason of course). * Tack things onto DYLD_FALLBACK_LIBRARY_PATH (Try to stay away from DYLD_LIBRARY_PATH, it's better to mess with the FALLBACK variant http://stackoverflow.com/questions/3146274/is-it-ok-to-use-dyld-library-path-on-mac-os-x-and-whats-the-dynamic-library-s when you can) Well this looks like what I will have to do for the time being at least. I agree this is the least desirable. * Move the libraries into an easier-to-find place. Either a place that is already on your executable's RPATH, or a location that is searched by default by the linker. The default locations are the default list of DYLD_FALLBACK_LIBRARY_PATH entries, listed in this man page https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/dyld.1.html, and are $(HOME)/lib:/usr/local/lib:/lib:/usr/lib. -E Likewise as above, I have no control over that. On Tue, Aug 12, 2014 at 3:12 AM, Jeff Waller trut...@gmail.com wrote: This is on OS/X I do have a workaround, but I'd like to get to the bottom of it. I don't see anywhere in the code where either of these 2 libraries are loaded explicitly --- maybe it's in loading sys.ji? My workaround can be to set an env variable, but it would be better to understand and address the problem directly. If DYLD_LIBRARY_PATH is set to point to the Julia library directory, then all works, but if this environment variable is not set, then the following occurs Warning: error initializing module LinAlg: ErrorException(error compiling __init__: error compiling check_blas: error compiling blas_vendor: could not load module libopenblas: dlopen(libopenblas.dylib, 1): image not found) Entropy pool not available to seed RNG; using ad-hoc entropy sources. Warning: error initializing module Random: ErrorException(could not load module libdSFMT: dlopen(libdSFMT.dylib, 1): image not found) === Those additional errors I'm sure are just side effects cascading form failing to load those libraries. The engine still does function, but the linear algebra stuff is unavailable. I've tried various combinations of jl_init and jl_init_with_image to no effect. -Jeff
Re: [julia-users] Unable to dlopen openblas and dSFMT by default when using embedded Julia
On Tuesday, August 12, 2014 2:36:03 PM UTC-4, Elliot Saba wrote: Alright, I worked up a little proof-of-concept demo here https://github.com/staticfloat/dltest/tree/master, but the long and the short of it is that you should be able to alter the rpath of YOUR library to include whatever paths you need, (probably relative to @loader_path so that your library just needs to know the relative path to openblas and friends) and you'll then be able to dlopen these guys to your heart's content. If you have any questions about this, please feel free to ask, but it should be as simple as adding a -Wl,-rpath,@loader_path/../lib or whatever to the linker options for your library. -E Hmm ok. What needs to dlopen openblas, etc? dlopen returns a value and needs to be saved does it not? If dlopen occurs on a library that is already loaded, it doesn't error, but does it succeed in the way that Julia expects it to? Or are you saying that openblas, etc just needs to be loaded somehow and then it all works out, my library can invoke dlopen brute-force like with maybe an arbitrary location. on these things? To be honest, I don't quite understand why this is even happening except for possibly side effects of resolving symbols at link time. After all libopenblas and libdSFMT are in the same exact directory as libjulia, and libjulia does get loaded no problem. Hmm @loader_path is a value eh? I don't see that documented. -Jeff
Re: [julia-users] Unable to dlopen openblas and dSFMT by default when using embedded Julia
I wrote up a huge long response talking about dynamic loading, why you need to alter libjulia, etc... and then about a page and a half in I realized that there was a very simple change on our end that could fix this. So I pushed this commit https://github.com/JuliaLang/julia/commit/1a18e8782f39dabfb68e75892cd29f27a565baae. Any julia version newer than today should automatically have `@loader_path/` added to libjulia's RPATH, which means that you should be able to have everything just work when embedding. If you clone and compile your own julia to test this out, pulling the latest and rebuilding should fix this for you. If you live off of binaries, a new nightly should be up tonight with this fix included. @loader_path is documented in the dyld man page https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/dyld.1.html, which is hardly light reading, I know. -E On Tue, Aug 12, 2014 at 5:01 PM, Jeff Waller truth...@gmail.com wrote: On Tuesday, August 12, 2014 2:36:03 PM UTC-4, Elliot Saba wrote: Alright, I worked up a little proof-of-concept demo here https://github.com/staticfloat/dltest/tree/master, but the long and the short of it is that you should be able to alter the rpath of YOUR library to include whatever paths you need, (probably relative to @loader_path so that your library just needs to know the relative path to openblas and friends) and you'll then be able to dlopen these guys to your heart's content. If you have any questions about this, please feel free to ask, but it should be as simple as adding a -Wl,-rpath,@loader_path/../lib or whatever to the linker options for your library. -E Hmm ok. What needs to dlopen openblas, etc? dlopen returns a value and needs to be saved does it not? If dlopen occurs on a library that is already loaded, it doesn't error, but does it succeed in the way that Julia expects it to? Or are you saying that openblas, etc just needs to be loaded somehow and then it all works out, my library can invoke dlopen brute-force like with maybe an arbitrary location. on these things? To be honest, I don't quite understand why this is even happening except for possibly side effects of resolving symbols at link time. After all libopenblas and libdSFMT are in the same exact directory as libjulia, and libjulia does get loaded no problem. Hmm @loader_path is a value eh? I don't see that documented. -Jeff
Re: [julia-users] Unable to dlopen openblas and dSFMT by default when using embedded Julia
Be sure to let me know if it doesn't work! On Tue, Aug 12, 2014 at 6:12 PM, Jeff Waller truth...@gmail.com wrote: On Tuesday, August 12, 2014 5:54:47 PM UTC-4, Elliot Saba wrote: I wrote up a huge long response talking about dynamic loading, why you need to alter libjulia, etc... and then about a page and a half in I realized that there was a very simple change on our end that could fix this. So I pushed this commit https://github.com/JuliaLang/julia/commit/1a18e8782f39dabfb68e75892cd29f27a565baae. Any julia version newer than today should automatically have `@loader_path/` added to libjulia's RPATH, which means that you should be able to have everything just work when embedding. ... ! That's awesome! If you clone and compile your own julia to test this out, pulling the latest and rebuilding should fix this for you. If you live off of binaries, a new nightly should be up tonight with this fix included. No problem, I have that already. @loader_path is documented in the dyld man page https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/dyld.1.html, which is hardly light reading, I know. -E Well that explains that. I was man-ing ld, not dyld
Re: [julia-users] Unable to dlopen openblas and dSFMT by default when using embedded Julia
In fact this does indeed just work. I did need to specify rpath, but didn't need to do anything else, this one value of rpath during the link step was enough to resolve both libjulia and libopenblas, etc. e.g. -Wl,-rpath,/usr/local/julia/lib/julia, Also. *Version 0.4.0-dev+92 (2014-08-12 22:52 UTC) --- congratz!*
