Re: [LAD] installing python extensions
Le 2022-09-22 à 10 h 51, Fons Adriaensen a écrit : That was it, many thanks ! Yay! To it looked as if pip didn't know the bdist_wheel command, and indeed pip help-commands didn't include it. No indication at all that something else was missing... The relations and dependencies between the various python tools - pip, setup, wheel, ... remain a mistery to me, and there seems to be little up to date documentation. Examples all assume you want to make a package for the PPI and no other use cases... The wheel package must be installed... it came after pip and setuptools, that both added support for wheel (not requiring it): https://peps.python.org/pep-0427/ I notice that audiotools is used by jacktools, so a requirement could be added to the setup function (in setup.py): install_requires=[ "zita-audiotools >= 1.3.0", ], Also, in order to create a valid source distribution, a MANIFEST.in file is required. Here's a starting point: global-exclude *.py[cod] include Makefile graft source First line is to exclude bytecode files. Second line is to include your (optional) Makefile (because it works). Third line is to include all files from the source directory (including .h files). More info: https://docs.python.org/3/distutils/sourcedist.html There's a newer packaging method that replace setup.py with a pyproject.toml file: https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/ More info on wheels and distribution on pypi.org: https://realpython.com/python-wheels/ The documentation is all over the place, and I'm also learning... With Archlinux you always (only) get the latest and greatest :-) Latest version of Python is 3.10.7, so 3.10.5 is behind by two patch versions. ;-) Marc ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org https://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] installing python extensions
On Thu, Sep 22, 2022 at 09:49:41AM -0400, Marc Lavallée wrote: > Le 2022-09-22 à 03 h 56, Fons Adriaensen a écrit : > >error: invalid command 'bdist_wheel' > It looks like wheel is not installed (locally or globally). Try installing > it with "pip install wheel", or install it on the system (python3-wheel on > Debian); it could be enough to fix the issue. That was it, many thanks ! To it looked as if pip didn't know the bdist_wheel command, and indeed pip help-commands didn't include it. No indication at all that something else was missing... The relations and dependencies between the various python tools - pip, setup, wheel, ... remain a mistery to me, and there seems to be little up to date documentation. Examples all assume you want to make a package for the PPI and no other use cases... > > Have things changed again > Python is changing faster now, so testing on different versions is a good > idea. With Archlinux you always (only) get the latest and greatest :-) Ciao, -- FA ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org https://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] installing python extensions
Hi Fons, Le 2022-09-22 à 03 h 56, Fons Adriaensen a écrit : error: invalid command 'bdist_wheel' It looks like wheel is not installed (locally or globally). Try installing it with "pip install wheel", or install it on the system (python3-wheel on Debian); it could be enough to fix the issue. python version is 3.10.5 My system still use python version 3.8, so for a test I installed python version 3.10.5 using pyenv (https://github.com/pyenv/pyenv) : pyenv install 3.10.5 To create and activate a virtualenv for my test I used: https://github.com/pyenv/pyenv-virtualenv pyenv virtualenv 3.10.5 zita pyenv activate zita But that was not enough; compiling your extension failed because wheel is missing in the virtualenv, so I installed it using "python3 -m pip install wheel", then compiling succeeded. There was one warning about the version of pip; to avoid it (before compiling): "python3 -m pip install --upgrade pip" After my test session, I deactivated the virtualenv using "source deactivate", then I deleted the virtualenv using "pyenv virtualenv-delete zita" Another difference IIRC is that on my main system pip would set up a virtual environment, while this doesn't seem to happen here. To my knowledge, pip does not install a virtual environment, but a virtual environment includes pip. Because wheel is not installed by default, adding a build step (in the makefile) could help: "$(PIP) install wheel". Have things changed again Python is changing faster now, so testing on different versions is a good idea. Marc ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org https://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] installing python extensions
On Mon, Aug 15, 2022 at 05:48:34PM -0400, Marc Lavallée wrote: > So the Makefile could be: > > PIP = python3 -m pip > PKG = zita_audiotools > > build: > $(PIP) wheel . > > install: > $(PIP) install --force-reinstall $(PKG)*.whl > > uninstall: > $(PIP) uninstall $(PKG) > > clean: > rm -rf build $(PKG)*.egg-info $(PKG)*.whl This worked perfectly a month ago on my main system. Now I'm in holidays with only my laptop, and it fails: python3 -m pip wheel . Processing /home/fons/python/jackimpfilt Preparing metadata (setup.py) ... [?25ldone [?25hBuilding wheels for collected packages: jackimpfilt Building wheel for jackimpfilt (setup.py) ... [?25lerror error: subprocess-exited-with-error × python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [6 lines of output] usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help error: invalid command 'bdist_wheel' [end of output] python version is 3.10.5 Another difference IIRC is that on my main system pip would set up a virtual environment, while this doesn't seem to happen here. Have things changed again -- FA ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org https://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] installing python extensions
Am 15.08.22 um 23:14 schrieb Marc Lavallée: Le 2022-08-15 à 15 h 21, Fons Adriaensen a écrit : sudo pip install . will install to /usr/local/lib When installing with the root user, the default prefix is /usr/local because /usr is usually managed by the system. Actually, it will install to sys.prefix, which is usually /usr, if the Python version used to run pip is installed via a distro package. BTW, the recommended way to run pip is actually python -m pip because then you can control (via PATH or the command name you use) which Python interpreter is used. Chris ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org https://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] installing python extensions
Am 15.08.22 um 21:21 schrieb Fons Adriaensen: I find the destination directory depending on who the user pretends to be a bit strange, but it works ! "pip install" as a normal user used to fail with an (apparently for most users) incomprehensible error message about not being able to write to the system-wide site-packages (doh!) and you needed the --user option to install into ~/.local/lib. That changed a while ago, so it automatically assumes --user, if not run as root and not told otherwise. I'm on the fence about whether I agree with this change or not. Unless there is a cleaner solution. $ python setup.py build $ sudo python setup.py install --skip-build (not sure about the appropriate options for other build backends) "install" implies build (and build_ext, if necessary) unless you tell it otherwise, and that step creates the .egg-info dir. Chris ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org https://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] installing python extensions
On Mon, Aug 15, 2022 at 10:54:06AM -0400, Marc Lavallée wrote: > Christopher Arndt sent a long and detailed answer, here's a shorter one. One (metric) unit of Eternal Gratitude to both of you. So > pip install . will install to ~/.local/lib while > sudo pip install . will install to /usr/local/lib I find the destination directory depending on who the user pretends to be a bit strange, but it works ! Also tested this with a package that includes data files (*.npy) that should be found by the installed code, and also this works. The only minor problem is that the sudo version leaves two directories (build and *.egg-info) that can only be cleaned up by root. No problem on systems that allow sudo everything, but I may keep the Makefile just to offer 'sudo make clean', assuming most system will allow this. Unless there is a cleaner solution. Ciao -- FA ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org https://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] installing python extensions
Am 15.08.22 um 14:30 schrieb Fons Adriaensen: I have some mixed python/C++ packages, e.g. zita-audiotools and zita-jacktools. To install these I expect the following to happen: 1. The C++ parts are compiled and combined into a *.so file which is a python extension. 2. The *.so and the python parts, any data etc. get installed into the user's /usr/lib/python*.*/site-packages. The way to install a python package nowadays should be via a wheel package (unless installed via a (Linux) distro package). These wheel files are basically Zip archives with some additional meta-data and can contain compiled Python extensions (which makes them Python version, OS and architecture-dependent, of course). The advantages of using wheels are: - no code from the package is executed at installation time (e.-g. setup.py etc.) - users can choose to install system-wide, under ~/.local or into a virtual environment. - installation is generally faster and wheels can be cached for speedup of repeated installations (e.g. in a development or CI environment). Originally this used distutils, when that got 'deprecated' this changed to setuptools. [...] Then I got warnings telling me that calling setup.py directly is now also deprecated, and that I should use 'official tools' to build and install. What exactly that means I was unable to find out, but the following seems to work: This guide explains the up-to-date process to define a package, at least for pure Python packages: https://packaging.python.org/en/latest/tutorials/packaging-projects/ If you have Python extension, you need to look into the documentation of your chosen build backend (e.g. poetry, flit, etc.). In short, you write a pyproject.toml file, in which you define which build backend to use and then, depending on that backend, you either define the package meta data and build instructions also in that file or (if you still want to use setuptools as a build backend) in a pair of setup.py / setup.cfg files. The goal is to have as little actual python code in setup.py as possible and define all static meta data elsewhere. WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead. What this basically says is that you shouldn't normally mess with the system-wide Python packages in /usr/lib/pythonX.Y/site-packages, because this is used by the Python packages installed via the distribution packaging system. Programs installed also via distro packages may rely on packages installed there and specific versions of them. If you install a Python package via pip as root into this location, pip will resolve dependencies, which may result in some existing packages being upgraded to newer versions. As the warning says , this may cause conflicts (or subtly break things). Now clearly installing things in site-packages requires root, so what is then the recommended method ?? And why the virtual environment (which is used by build anyway) ?? IMO you should distinguish between these scenarios: a) A program depends on, say, zita-xyz and is installed via a distro package. It should depend on the respective distro-package for zita-xyz. This installs into /usr/lib/pythonX.Y/site-packages. You can let the distro packages worry about the pip warning, you just provide the correct pyproject.toml and other build files. b) A program depends on zita-xyz and is installed via pip. Then it should define this dependency in pyproject.toml and pip will take care of installing zita-xyz. The user running pip can decide whether to install the program system-wide (not recommended), under ~/.local or into a virtual environment (e.g. using pipx [1]). zita-xyz will be installed in the same location. If both are installed in a dedicated virtual environment, it ensures that there are no conflicts with other programs, possibly depending on a different version of zita-xyz. c) A developer (or "power-user") uses zita-xyz for the implementation of a program or some utility scripts. He/she uses pip to install zita-xyz (and its dependencies, if any). If he/she want zita-xyz to be importable from any Python interpreter with a matching version started *by that user*, he installs it with the --user option into ~/.local. If he/she wants to make sure that a certain program always has access to and uses a certain zita-xyz version it depends on, he/she installs it into dedicated virtual environment for that (collection) of program(s) and always starts the program (or the Python interpreter) from the bin directory of that virtual env. There are several ways to facilitate / automate the latter, e.g. via poetry [2], which I have mostly good experiences with. And oh, btw, any virtual environment created by the build process is ephemeral and not relevant to the actual installation. One last thing: extension
