On 2022-04-01 00:30 -0400, M. Zhou wrote: > On Fri, 2022-04-01 at 02:32 +0100, Wookey wrote: > > > > > > So it tries quite hard to find it, but doesn't know about multiarch > > and thus fails to look in the right place: > > /usr/lib/<triplet>/ (/usr/lib/x86_64-linux-gnu/ on this box) > > dlopen should know the multiarch triplet on debian. They have written > their own ffi loader, so I think it is an upstream bug. The upstream > should detect and add multiarch directory to the paths.
Agreed. I also don't think it should use the $PATH paths for finding
libraries (but maybe upstream have some reason for doing that)
I made this patch but it's debian-specific, using dpkg-architecture.
@@ -48,7 +49,8 @@ def get_dll_directories():
# $PREFIX/lib/python3.6/site-packages/tvm/_ffi
ffi_dir = os.path.dirname(os.path.realpath(os.path.expanduser(__file__)))
source_dir = os.path.join(ffi_dir, "..", "..", "..")
- install_lib_dir = os.path.join(ffi_dir, "..", "..", "..", "..")
+ multiarch_name = subprocess.run(['dpkg-architecture', '-q',
'DEB_HOST_MULTIARCH'], stdout=subprocess.PIPE).stdout.decode('utf-8').rstrip()
+ install_lib_dir = os.path.join(ffi_dir, "..", "..", "..", "..",
multiarch_name)
(and it took me _ages_ to work out that suprocess.run without that
.rstrip() leaves the trailing newline in the string which stops it
working!)
A correct implemntation really should use the full ldconfig set of search paths.
> > OK, but that mostly reveals a second issue: it's looking for
> > libtvm.so, but that unversioned link is only provoded in the dev
> > package
> > libtvm-dev. The library package has the versioned filenames
> > /usr/lib/x86_64-linux-gnu/libtvm.so.0
> > /usr/lib/x86_64-linux-gnu/libtvm_runtime.so.0
>
> I think it is fine to let it dlopen the libtvm.so, as it says
> itself as some sort of "compiler".
>
> Take pytorch as example, python3-torch has some functionalities
> for extending itself with C++. As a result, libtorch-dev is
> a dependency of python3-torch.
OK. I see there is also a find_include_path in libinfo.py so I guess
if it needs the headers too then depending on the -dev package is
indeed correct. I've reverted the change to look for libtvm.so.0.
> > What it should actually be adding is what's in /etc/ld.so.conf.d/*
> > That can be listed with
> > /sbin/ldconfig -v 2>/dev/null | grep -v ^$'\t' | cut -d: -f1
> > (yuk? is there really no better way?)
OK. I tried this, and given that I don't know any python it went better than I
expected.
So this code makes an array of paths (as strings) from ldconfig -v output.
However I fell at the last hurble of joining the lib_search_dirs array
to the dll_paths list such that I get one list of all the paths, not a
list where the first entry still has multiple entries. My reading of
the docs says that using extend() instead of append() should merge the
lists, but it isn't for some reason. I made them both strings, rather
than one lot of byte array and one lot of strings, but it still
doesn't work. I'm sure this is trivial to fix for someone who actually
knows some python, hence this mail.
So I get this nice set of paths:
search_dirs [['/usr/lib/x86_64-linux-gnu/libfakeroot:', '/usr/local/lib:',
'/lib/x86_64-linux-gnu:', '/usr/lib/x86_64-linux-gnu:', '/lib:', '/usr/lib:']]
which is combined with the other paths to get this incorrect data structure:
dll_path: [['/usr/lib/x86_64-linux-gnu/libfakeroot:', '/usr/local/lib:',
'/lib/x86_64-linux-gnu:', '/usr/lib/x86_64-linux-gnu:', '/lib:', '/usr/lib:'],
'/usr/lib/python3/dist-packages/tvm/_ffi/..',
'/usr/lib/python3/dist-packages/tvm/_ffi/../../../build',
'/usr/lib/python3/dist-packages/tvm/_ffi/../../../build/Release',
'/usr/lib/python3/dist-packages/tvm/_ffi/../../../lib']
Here is the code:
def get_lib_search_dirs():
"""Get unix library search path from ldconfig -v"""
# loads of output, only lines starting with / are relevant
output = subprocess.run(["/sbin/ldconfig","-v"],capture_output=True)
paths = output.stdout.split(b'\n')
filtered = []
for path in paths:
if path[:1] == b'/':
filtered.append(path.split()[0].decode())
return [filtered]
def get_dll_directories():
"""Get the possible dll directories"""
# NB: This will either be the source directory (if TVM is run
# inplace) or the install directory (if TVM is installed).
# An installed TVM's curr_path will look something like:
# $PREFIX/lib/python3.6/site-packages/tvm/_ffi
ffi_dir = os.path.dirname(os.path.realpath(os.path.expanduser(__file__)))
source_dir = os.path.join(ffi_dir, "..", "..", "..")
lib_search_dirs = get_lib_search_dirs()
print ("search_dirs",lib_search_dirs)
dll_path = []
if os.environ.get("TVM_LIBRARY_PATH", None):
dll_path.append(os.environ["TVM_LIBRARY_PATH"])
if sys.platform.startswith("linux") or sys.platform.startswith("freebsd"):
dll_path.extend(split_env_var("LD_LIBRARY_PATH", ":"))
dll_path.extend(lib_search_dirs)
elif sys.platform.startswith("darwin"):
dll_path.extend(split_env_var("DYLD_LIBRARY_PATH", ":"))
dll_path.extend(split_env_var("PATH", ":"))
elif sys.platform.startswith("win32"):
dll_path.extend(split_env_var("PATH", ";"))
# Pip lib directory
dll_path.append(os.path.join(ffi_dir, ".."))
# Default cmake build directory
dll_path.append(os.path.join(source_dir, "build"))
dll_path.append(os.path.join(source_dir, "build", "Release"))
# Default make build directory
dll_path.append(os.path.join(source_dir, "lib"))
print("dll_path:",dll_path)
If it helps with understanding this is what my changes look like as a patch:
Index: tvm-0.8.0/python/tvm/_ffi/libinfo.py
===================================================================
--- tvm-0.8.0.orig/python/tvm/_ffi/libinfo.py
+++ tvm-0.8.0/python/tvm/_ffi/libinfo.py
@@ -17,6 +17,7 @@
"""Library information."""
import sys
import os
+import subprocess
def split_env_var(env_var, split):
@@ -40,6 +41,17 @@ def split_env_var(env_var, split):
return []
+def get_lib_search_dirs():
+ """Get unix library search path from ldconfig -v"""
+ # loads of output, only lines starting with / are relevant
+ output = subprocess.run(["/sbin/ldconfig","-v"],capture_output=True)
+ paths = output.stdout.split(b'\n')
+ filtered = []
+ for path in paths:
+ if path[:1] == b'/':
+ filtered.append(path.split()[0].decode())
+ return [filtered]
+
def get_dll_directories():
"""Get the possible dll directories"""
# NB: This will either be the source directory (if TVM is run
@@ -48,7 +60,8 @@ def get_dll_directories():
# $PREFIX/lib/python3.6/site-packages/tvm/_ffi
ffi_dir = os.path.dirname(os.path.realpath(os.path.expanduser(__file__)))
source_dir = os.path.join(ffi_dir, "..", "..", "..")
- install_lib_dir = os.path.join(ffi_dir, "..", "..", "..", "..")
+ lib_search_dirs = get_lib_search_dirs()
+ print ("search_dirs",lib_search_dirs)
dll_path = []
@@ -57,7 +70,7 @@ def get_dll_directories():
if sys.platform.startswith("linux") or sys.platform.startswith("freebsd"):
dll_path.extend(split_env_var("LD_LIBRARY_PATH", ":"))
- dll_path.extend(split_env_var("PATH", ":"))
+ dll_path.extend(lib_search_dirs)
elif sys.platform.startswith("darwin"):
dll_path.extend(split_env_var("DYLD_LIBRARY_PATH", ":"))
dll_path.extend(split_env_var("PATH", ":"))
@@ -71,8 +84,7 @@ def get_dll_directories():
dll_path.append(os.path.join(source_dir, "build", "Release"))
# Default make build directory
dll_path.append(os.path.join(source_dir, "lib"))
-
- dll_path.append(install_lib_dir)
+ print("dll_path:",dll_path)
if os.path.isdir(source_dir):
dll_path.append(os.path.join(source_dir, "web", "dist", "wasm"))
Wookey
--
Principal hats: Debian, Wookware, ARM
http://wookware.org/
signature.asc
Description: PGP signature
