Re: [CMake] Multiple occurrences of a library on linux (ldd)
export LD_DEBUG=files` to debug linking issues > on linux. It provides more detail on the ldd output, as below: > > 18843: file=libc10.so [0]; needed by > /home/dev/miniconda3/datareaders_py37/build/stage/datareader/lib/libdatareader.so > [0] >(no linking information) > 18843: file=libdatareader.so [0]; needed by > subprojects/Build/datareaders_core_test/standalone_gtests [0] > 18843: file=libdatareader.so [0]; generating link map > (linking information) > (...) > 18843: file=libc10.so [0]; needed by > /home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib/libcaffe2.so > [0] > 18843: file=libc10.so [0]; generating link map >(linking information) > 18843: file=libcaffe2.so [0]; needed by > subprojects/Build/datareaders_core_test/standalone_gtests [0] > 18843: file=libcaffe2.so [0]; generating link map >(linking information) > > Now I can see that the missing libc10.so is needed by > `libdatareader.so`, which was linked against `standalone_gtests`. > > However, RUNPATH for both `libdatareader.so` and `standalone_gtests` > seems to be correct and point to the same torch/lib folder that has > libc10.so: > libdatareader.so: > RUNPATH=/home/dev/miniconda3/datareaders_py37/build/stage/boost/lib:/home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib: > standalone_gtests: > RUNPATH=/home/dev/miniconda3/datareaders_py37/build/stage/boost/lib:/home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib:/home/dev/miniconda3/datareaders_py37/build/stage/datareader/lib > > So, libdatareader.so links to libcaffe2.so which links to libc10.so > successfully; standalone_gtests links to libcaffe2.so which links to > libc10.so successully too; however, when standalone_gtests links to > libdatareader.so, there is a transitive linking issue with > libdatareader.so dependency on libc10.so, like the diagram below: > > libdatareader.so ---> libcaffe2.so ---> libc10.so (ok) > > standalone_gtests 0 ---> libcaffe2.so ---> libc10.so (ok) > | > ---> libdatareader.so ---> libc10.so (not found) > > > Thanks again, > Thiago > > > > On Thu, Feb 14, 2019 at 6:13 AM Thompson, KT wrote: > > > > Thiago, > > > > > > > > I haven’t see the double entry pattern that you mention below. However, > > you might want to tell CMake to embed a BUILD_RPATH in your libraries. > > This should get around the issue of manually setting LD_LIBRARY_PATH. > > > > > > > > https://cmake.org/cmake/help/latest/prop_tgt/BUILD_RPATH.html?highlight=rpath > > > > > > > > -kt > > > > > > > > From: CMake On Behalf Of Thiago Crepaldi > > Sent: Thursday, February 14, 2019 12:43 AM > > To: cmake@cmake.org > > Subject: [CMake] Multiple occurrences of a library on linux (ldd) > > > > > > > > Hello all, > > > > > > > > After reading CMake Cookbook I have written my first "complex" CMake build > > script based on the superbuild pattern. I am excited to heave a better > > understanding on CMake, but I definitely will learn much more from > > experience and your kind help. > > > > In summary, the standalone google test application `standalone_gtests` > > publicly links to `libdatareader.so` and to pytorch libraries > > (`libc10.so`,`libcafee2.so`, `libtorch.so`). > > > > `libdatareader.so` also publicly links to pytorch libraries (I have a > > theoretical question on why `standalone_gtests` had to link to pytorch > > libraries if `libdatareader.so` already did, but that can wait). > > > > > > > > Compilation finishes successfully, but when I try to run > > `standalone_gtests`, it aborts because it cant find `libc10.so`. > > > > After executing `ldd standalone_gtests`, the weird result was that there > > were two entries for `libc10.so`. > > > > The first one maps to "not found" while the second had the correct path to > > the library. `libcaffe2.so`, which is also a pytorch library, has a single > > occurrence with full path. > > > > If I add the (...)/pytorch_external/(...) (see ldd output below) path to > > LD_LIBRARY_PATH, then everything works, but I would like to avoid this, if > > possible. > > > > > > > > `ldd ./subprojects/Build/datareaders_core_test/standalone_gtests > > > > libdatareader.so => > > /home/dev/miniconda3/datareaders_py37/build/stage/datareader/lib/libdatar
Re: [CMake] Multiple occurrences of a library on linux (ldd)
Thanks, Thompson, I will look into BUILD_RPATH and possibly INSTALL_RPATH. I just learned about `export LD_DEBUG=files` to debug linking issues on linux. It provides more detail on the ldd output, as below: 18843: file=libc10.so [0]; needed by /home/dev/miniconda3/datareaders_py37/build/stage/datareader/lib/libdatareader.so [0] (no linking information) 18843: file=libdatareader.so [0]; needed by subprojects/Build/datareaders_core_test/standalone_gtests [0] 18843: file=libdatareader.so [0]; generating link map (linking information) (...) 18843: file=libc10.so [0]; needed by /home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib/libcaffe2.so [0] 18843: file=libc10.so [0]; generating link map (linking information) 18843: file=libcaffe2.so [0]; needed by subprojects/Build/datareaders_core_test/standalone_gtests [0] 18843: file=libcaffe2.so [0]; generating link map (linking information) Now I can see that the missing libc10.so is needed by `libdatareader.so`, which was linked against `standalone_gtests`. However, RUNPATH for both `libdatareader.so` and `standalone_gtests` seems to be correct and point to the same torch/lib folder that has libc10.so: libdatareader.so: RUNPATH=/home/dev/miniconda3/datareaders_py37/build/stage/boost/lib:/home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib: standalone_gtests: RUNPATH=/home/dev/miniconda3/datareaders_py37/build/stage/boost/lib:/home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib:/home/dev/miniconda3/datareaders_py37/build/stage/datareader/lib So, libdatareader.so links to libcaffe2.so which links to libc10.so successfully; standalone_gtests links to libcaffe2.so which links to libc10.so successully too; however, when standalone_gtests links to libdatareader.so, there is a transitive linking issue with libdatareader.so dependency on libc10.so, like the diagram below: libdatareader.so ---> libcaffe2.so ---> libc10.so (ok) standalone_gtests 0 ---> libcaffe2.so ---> libc10.so (ok) | ---> libdatareader.so ---> libc10.so (not found) Thanks again, Thiago On Thu, Feb 14, 2019 at 6:13 AM Thompson, KT wrote: > > Thiago, > > > > I haven’t see the double entry pattern that you mention below. However, you > might want to tell CMake to embed a BUILD_RPATH in your libraries. This > should get around the issue of manually setting LD_LIBRARY_PATH. > > > > https://cmake.org/cmake/help/latest/prop_tgt/BUILD_RPATH.html?highlight=rpath > > > > -kt > > > > From: CMake On Behalf Of Thiago Crepaldi > Sent: Thursday, February 14, 2019 12:43 AM > To: cmake@cmake.org > Subject: [CMake] Multiple occurrences of a library on linux (ldd) > > > > Hello all, > > > > After reading CMake Cookbook I have written my first "complex" CMake build > script based on the superbuild pattern. I am excited to heave a better > understanding on CMake, but I definitely will learn much more from experience > and your kind help. > > In summary, the standalone google test application `standalone_gtests` > publicly links to `libdatareader.so` and to pytorch libraries > (`libc10.so`,`libcafee2.so`, `libtorch.so`). > > `libdatareader.so` also publicly links to pytorch libraries (I have a > theoretical question on why `standalone_gtests` had to link to pytorch > libraries if `libdatareader.so` already did, but that can wait). > > > > Compilation finishes successfully, but when I try to run `standalone_gtests`, > it aborts because it cant find `libc10.so`. > > After executing `ldd standalone_gtests`, the weird result was that there were > two entries for `libc10.so`. > > The first one maps to "not found" while the second had the correct path to > the library. `libcaffe2.so`, which is also a pytorch library, has a single > occurrence with full path. > > If I add the (...)/pytorch_external/(...) (see ldd output below) path to > LD_LIBRARY_PATH, then everything works, but I would like to avoid this, if > possible. > > > > `ldd ./subprojects/Build/datareaders_core_test/standalone_gtests > > libdatareader.so => > /home/dev/miniconda3/datareaders_py37/build/stage/datareader/lib/libdatareader.so > > libcaffe2.so => > /home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib/libcaffe2.so > > libc10.so => not found > > libc10.so => > /home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib/libc10.so > > (...)` > > > > Have anyone seen multiple entries for the same library on ldd before? Why is > that? Is it because `standalone_gte
Re: [CMake] Multiple occurrences of a library on linux (ldd)
Thiago, I haven’t see the double entry pattern that you mention below. However, you might want to tell CMake to embed a BUILD_RPATH in your libraries. This should get around the issue of manually setting LD_LIBRARY_PATH. https://cmake.org/cmake/help/latest/prop_tgt/BUILD_RPATH.html?highlight=rpath -kt From: CMake On Behalf Of Thiago Crepaldi Sent: Thursday, February 14, 2019 12:43 AM To: cmake@cmake.org Subject: [CMake] Multiple occurrences of a library on linux (ldd) Hello all, After reading CMake Cookbook I have written my first "complex" CMake build script based on the superbuild pattern. I am excited to heave a better understanding on CMake, but I definitely will learn much more from experience and your kind help. In summary, the standalone google test application `standalone_gtests` publicly links to `libdatareader.so` and to pytorch libraries (`libc10.so`,`libcafee2.so`, `libtorch.so`). `libdatareader.so` also publicly links to pytorch libraries (I have a theoretical question on why `standalone_gtests` had to link to pytorch libraries if `libdatareader.so` already did, but that can wait). Compilation finishes successfully, but when I try to run `standalone_gtests`, it aborts because it cant find `libc10.so`. After executing `ldd standalone_gtests`, the weird result was that there were two entries for `libc10.so`. The first one maps to "not found" while the second had the correct path to the library. `libcaffe2.so`, which is also a pytorch library, has a single occurrence with full path. If I add the (...)/pytorch_external/(...) (see ldd output below) path to LD_LIBRARY_PATH, then everything works, but I would like to avoid this, if possible. `ldd ./subprojects/Build/datareaders_core_test/standalone_gtests libdatareader.so => /home/dev/miniconda3/datareaders_py37/build/stage/datareader/lib/libdatareader.so libcaffe2.so => /home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib/libcaffe2.so libc10.so => not found libc10.so => /home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib/libc10.so (...)` Have anyone seen multiple entries for the same library on ldd before? Why is that? Is it because `standalone_gtests` links to libc10.so and to `libdatareader.so`, which in turn also links to `libc10.so`? Both CMakeLists.txt (libdatareader and standalone_gtests) succeeds at find_package(Torch REQUIRED QUIET) commands (${TORCH_LIBRARY} returns the correct path). I run the same build on my Mac and everything works fine, so that is confined to linux environment. I have destroyed my conda environment and performed multiple clean builds in the process and no luck :( Hoping this was some sort of ldconfig issue, I tried `sudo ldconfig` and `sudo rm /etc/ld.so.cache`, but that doesn't fix it. Any ideas? best regards, Thiago -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
[CMake] Multiple occurrences of a library on linux (ldd)
Hello all, After reading CMake Cookbook I have written my first "complex" CMake build script based on the superbuild pattern. I am excited to heave a better understanding on CMake, but I definitely will learn much more from experience and your kind help. In summary, the standalone google test application `standalone_gtests` publicly links to `libdatareader.so` and to pytorch libraries (`libc10.so`,`libcafee2.so`, `libtorch.so`). `libdatareader.so` also publicly links to pytorch libraries (I have a theoretical question on why `standalone_gtests` had to link to pytorch libraries if `libdatareader.so` already did, but that can wait). Compilation finishes successfully, but when I try to run `standalone_gtests`, it aborts because it cant find `libc10.so`. After executing `ldd standalone_gtests`, the weird result was that there were two entries for `libc10.so`. The first one maps to "not found" while the second had the correct path to the library. `libcaffe2.so`, which is also a pytorch library, has a single occurrence with full path. If I add the (...)/pytorch_external/(...) (see ldd output below) path to LD_LIBRARY_PATH, then everything works, but I would like to avoid this, if possible. `ldd ./subprojects/Build/datareaders_core_test/standalone_gtests libdatareader.so => /home/dev/miniconda3/datareaders_py37/build/stage/datareader/lib/libdatareader.so libcaffe2.so => /home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib/libcaffe2.so libc10.so => not found libc10.so => /home/dev/miniconda3/datareaders_py37/build/subprojects/Source/pytorch_external/torch/lib/libc10.so (...)` Have anyone seen multiple entries for the same library on ldd before? Why is that? Is it because `standalone_gtests` links to libc10.so and to `libdatareader.so`, which in turn also links to `libc10.so`? Both CMakeLists.txt (libdatareader and standalone_gtests) succeeds at find_package(Torch REQUIRED QUIET) commands (${TORCH_LIBRARY} returns the correct path). I run the same build on my Mac and everything works fine, so that is confined to linux environment. I have destroyed my conda environment and performed multiple clean builds in the process and no luck :( Hoping this was some sort of ldconfig issue, I tried `sudo ldconfig` and `sudo rm /etc/ld.so.cache`, but that doesn't fix it. Any ideas? best regards, Thiago -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake