Re: Python, poetry, and pandas
Sébastien Lerique writes: > Now my wish is to find ways to collaborate with others on this kind of > project, who unfortunately often use macOS. I was exploring the > possibility of Guix on Darwin, but the best way to do that looks like > running Guix System inside Docker: > I haven't tried with MacOS yet but a few ideas: - Try using the QEMU image available (or create your own VirtualBox image - I've tried this and it worked OK). - If running a full graphical env is too clunky for you could experiment with multipass. I haven't tried this yet on multipass but I've had some success running Guix inside WSL2 (very similar to multipass) on Windows using busybox and these instructions: https://github.com/giuliano108/guix-packages/blob/master/notes/Guix-on-WSL2.md It's a bit 'beta', but I use this when I want fast local access to a Guix environment from a Windows desktop without having yet-another-foreign-distro (eg Ubuntu-on-WSL) in-between. - I'm not expert on this but I suspect you could produce a slimmed down version of Guix more akin to the WSL experience using the 'guix system vm-image' command. Check-out the manual. Having something that worked with multipass and WSL2 directly on the Guix website would be extremely useful in drawing a crowd from non-Linux users! So this would be a noble endeavour IMHO. If you get anything working I'd be very interested. Of course this isn't substitute for running on Darwin as a foreign distro, but I'm unaware of any current project to do that. I suspect it would be challenging to get package definitions working for substantially different compiler flags etc, it would almost become a separate project. It has been discussed tho - follow the links in this post to see discussions on guix-devel too: https://www.reddit.com/r/GUIX/comments/c2xei5/any_way_to_have_guix_running_as_a_package_manager/
Re: Python, poetry, and pandas
Hi Phil and Simon, Thanks both for the input and advice (not citing it all here). First the details. Simon wrote: How did you installed poetry? Via Guix or something else? Which python packages do you have in your profile? Is Guix System or Guix on foreign distro? I'm using Guix System, and installed poetry via Guix. I have the following python packages installed: python, poetry, python-jupyterlab (from guix-science), and python-black. And guix-jupyter. (I am aware that python-pandas is packaged for guix, but some packages I will need down the road might not be; this project is also in collaboration with non guix users, so I am looking for a workflow which will work for them too.) What are these packages? I don't know yet, but I suspect the situation might crop up. Phil pointed to the PyPi importer, and I haven't used it yet, but it does seem to make things simpler if I need to add a package to Guix. Phil wrote: ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory If it's only missing a reference to the underlying C++ library you could install this library using Guix, or if you're using a foreign distro point to the non-Guix version under /usr/lib which, at a guess, Poetry may have picked up - but this is going to cause you more trouble down the line than repackaging in Guix in my experience. Yes, my experience with trying this for some Julia packages (GLMakie) convinced me not to try LD_LIBRARY_PATH hacks any more. Now the broader view. Phil: Guix offers pretty much everything that virtualenvs offers, but it does things a little differently - you'll need to look at environments, profiles, manifests, and use of a guix.scm to recreate (IMHO) a better experience in the end. I just set up a simple guix.scm for my current project, which indeed does the job, and of course provides all the great guarantees of Guix! Given the number of python packages currently, and the existence of the PyPi importer, I'm pretty convinced it covers my personal needs. Now my wish is to find ways to collaborate with others on this kind of project, who unfortunately often use macOS. I was exploring the possibility of Guix on Darwin, but the best way to do that looks like running Guix System inside Docker: http://logs.guix.gnu.org/guix/2021-03-11.log#044004 https://lists.gnu.org/archive/html/guix-devel/2017-10/msg00111.html So it seems that to collaborate with non-guix and non-GNU/Linux users, the options are 1) maintain a second environment management configuration (e.g. using poetry) to provide a stop-gap, or 2) convince colleagues to run Guix in Docker. The situation for Windows might be a bit better using the Linux Subsystem. Thanks again all, Sébastien On 10 Mar 2021 at 19:02, Phil wrote: Hi Sébastien, Sébastien Lerique writes: Dear all, (I am aware that python-pandas is packaged for guix, but some packages I will need down the road might not be; this project is also in collaboration with non guix users, so I am looking for a workflow which will work for them too.) Under the covers Poetry is just using regular Python virtualenvs I think. These don't work well with Guix for a number of reasons - but in my previous attempts to use them I found that Guix's use of PYTHONPATH meant that virtualenvs where never isolated from packages installed by Guix. Guix offers pretty much everything that virtualenvs offers, but it does things a little differently - you'll need to look at environments, profiles, manifests, and use of a guix.scm to recreate (IMHO) a better experience in the end. When I started using Guix I went to some lengths to harmonize virtualenvs and Guix - and others have made more substantial efforts than I did - but you end up creating as many problems as you fix: https://lists.gnu.org/archive/html/help-guix/2020-11/msg00237.html If you search the mail archive a couple of other people have asked similar questions to do with poetry, pyenv, pip, etc - they're all interesting reading and easy to grep with those keywords: https://lists.gnu.org/archive/html/help-guix/ The advice I was given at the time was don't mix package managers, and in hindsight I'm glad repackaged what I needed under Guix. I've packaged quite a few python libraries now, and most are pretty trivial to do - if they are in PyPi you can just use this: guix import pypi If you need a later version than Guix provides (eg pandas), my advice is to use the ability to "inherit" the current definition in Guix and tweak the version. For Pandas I did this for a more recent version I needed and all I had to do was disable 2 unit tests (which failed for a non-Guix reason). Apologies - not exactly the answer you want (I have a stab at that below), but it's worth considering moving to Guix packaging exclusively to keep your life simple. File
Re: Python, poetry, and pandas
Hi, On Wed, 10 Mar 2021 at 16:38, Sébastien Lerique wrote: > In the course of trying out poetry for data science projects on > guix, I am running into the problem of some poetry-installed > packages not finding their desired libraries. (See example below > with pandas.) I suppose this is because poetry will download > pre-built binary wheels which have pre-coded library paths. How did you installed poetry? Via Guix or something else? Which python packages do you have in your profile? Is Guix System or Guix on foreign distro? > If this is the problem, would there be any clean way to wrap those > binary wheels, or patch poetry-on-guix to mandatorily rebuild > binaries? (Would that make sense?) > > (I am aware that python-pandas is packaged for guix, but some > packages I will need down the road might not be; this project is > also in collaboration with non guix users, so I am looking for a > workflow which will work for them too.) What are these packages? Cheers, simon
Re: Python, poetry, and pandas
Hi Sébastien, Sébastien Lerique writes: > Dear all, > > (I am aware that python-pandas is packaged for guix, but some packages > I will need down the road might not be; this project is also in > collaboration with non guix users, so I am looking for a workflow > which will work for them too.) > Under the covers Poetry is just using regular Python virtualenvs I think. These don't work well with Guix for a number of reasons - but in my previous attempts to use them I found that Guix's use of PYTHONPATH meant that virtualenvs where never isolated from packages installed by Guix. Guix offers pretty much everything that virtualenvs offers, but it does things a little differently - you'll need to look at environments, profiles, manifests, and use of a guix.scm to recreate (IMHO) a better experience in the end. When I started using Guix I went to some lengths to harmonize virtualenvs and Guix - and others have made more substantial efforts than I did - but you end up creating as many problems as you fix: https://lists.gnu.org/archive/html/help-guix/2020-11/msg00237.html If you search the mail archive a couple of other people have asked similar questions to do with poetry, pyenv, pip, etc - they're all interesting reading and easy to grep with those keywords: https://lists.gnu.org/archive/html/help-guix/ The advice I was given at the time was don't mix package managers, and in hindsight I'm glad repackaged what I needed under Guix. I've packaged quite a few python libraries now, and most are pretty trivial to do - if they are in PyPi you can just use this: guix import pypi If you need a later version than Guix provides (eg pandas), my advice is to use the ability to "inherit" the current definition in Guix and tweak the version. For Pandas I did this for a more recent version I needed and all I had to do was disable 2 unit tests (which failed for a non-Guix reason). Apologies - not exactly the answer you want (I have a stab at that below), but it's worth considering moving to Guix packaging exclusively to keep your life simple. > File > "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/window/ewm.py", > line 9, in >import pandas._libs.window.aggregations as window_aggregations > ImportError: libstdc++.so.6: cannot open shared object file: No such > file or directory If it's only missing a reference to the underlying C++ library you could install this library using Guix, or if you're using a foreign distro point to the non-Guix version under /usr/lib which, at a guess, Poetry may have picked up - but this is going to cause you more trouble down the line than repackaging in Guix in my experience.
Python, poetry, and pandas
Dear all, In the course of trying out poetry for data science projects on guix, I am running into the problem of some poetry-installed packages not finding their desired libraries. (See example below with pandas.) I suppose this is because poetry will download pre-built binary wheels which have pre-coded library paths. If this is the problem, would there be any clean way to wrap those binary wheels, or patch poetry-on-guix to mandatorily rebuild binaries? (Would that make sense?) (I am aware that python-pandas is packaged for guix, but some packages I will need down the road might not be; this project is also in collaboration with non guix users, so I am looking for a workflow which will work for them too.) (I am also aware of guix-jupyter, but that is another topic I'd like to return to later as it needs some more bugsquashing for usability.) Happy hacking! Sébastien #+BEGIN_SRC $ poetry new poetry-test Created package poetry_test in poetry-test $ cd poetry-test/ $ poetry add pandas Creating virtualenv poetry-test-vOWX4_Vr-py3.8 in /home/sl/.cache/pypoetry/virtualenvs Using version ^1.2.3 for pandas Updating dependencies Resolving dependencies... (0.2s) Writing lock file Package operations: 4 installs, 9 updates, 0 removals • Updating pyparsing (2.4.6 /home/sl/.guix-profile/lib/python3.8/site-packages -> 2.4.7) • Updating six (1.14.0 /home/sl/.guix-profile/lib/python3.8/site-packages -> 1.15.0) • Updating attrs (19.3.0 /home/sl/.guix-profile/lib/python3.8/site-packages -> 20.3.0) • Updating more-itertools (8.2.0 /home/sl/.guix-profile/lib/python3.8/site-packages -> 8.7.0) • Updating numpy (1.17.3 /home/sl/.guix-profile/lib/python3.8/site-packages -> 1.20.1) • Updating packaging (20.0 /home/sl/.guix-profile/lib/python3.8/site-packages -> 20.9) • Installing pluggy (0.13.1) • Installing py (1.10.0) • Updating python-dateutil (2.8.1 /home/sl/.guix-profile/lib/python3.8/site-packages -> 2.8.1) • Updating pytz (2020.4 /home/sl/.guix-profile/lib/python3.8/site-packages -> 2021.1) • Updating wcwidth (0.1.8 /home/sl/.guix-profile/lib/python3.8/site-packages -> 0.2.5) • Installing pandas (1.2.3) • Installing pytest (5.4.3) $ poetry run which python /home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/bin/python $ poetry run python Python 3.8.2 (default, Jan 1 1970, 00:00:01) [GCC 7.5.0] on linux Type "help", "copyright", "credits" or "license" for more information. import pandas Traceback (most recent call last): File "", line 1, in File "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/__init__.py", line 51, in from pandas.core.api import ( File "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/api.py", line 31, in from pandas.core.groupby import Grouper, NamedAgg File "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/groupby/__init__.py", line 1, in from pandas.core.groupby.generic import DataFrameGroupBy, NamedAgg, SeriesGroupBy File "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/groupby/generic.py", line 65, in from pandas.core.frame import DataFrame File "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/frame.py", line 119, in from pandas.core import algorithms, common as com, generic, nanops, ops File "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/generic.py", line 111, in from pandas.core.window import Expanding, ExponentialMovingWindow, Rolling, Window File "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/window/__init__.py", line 1, in from pandas.core.window.ewm import ( # noqa:F401 File "/home/sl/.cache/pypoetry/virtualenvs/poetry-test-vOWX4_Vr-py3.8/lib/python3.8/site-packages/pandas/core/window/ewm.py", line 9, in import pandas._libs.window.aggregations as window_aggregations ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory #+END_SRC