Re: Concerns/questions around Software Heritage Archive
Hey, > I have heard folks in the Guix maintenance sphere claim that we never rewrite > git history in Guix, as a matter of policy. I believe we should revisit that > policy (is it actually written anywhere?) with an eye towards possible > exceptions, and develop a mechanism for securely maintaining continuity of > Guix installations after history has been rewritten so that we maintain this > as a technical possibility in the future, even if we should choose to use it > sparingly. the fallout of rewriting Guix’ git history would be devastating. It would break every single Guix installation, because a) `guix pull` authenticates commits and we might lose our trust anchor if we rewrite history earlier than the introduction of this feature, b) `guix pull` outright rejects changes to the commit history to prevent downgrade attacks. Additionally it would break every single existing usage of the time machine and thereby completely defeat the goal of providing reproducible software environments since the commit hash is used to identify the point in time to jump to. I doubt developing “mechanisms” – whatever they look like – would be worth the effort. Our contributors matter, but so do our users. Never ever rewriting our git history is a tradeoff we should make for our users. Lars
Re: Should commits rather be buildable or small
Hi, > I'm planning on refreshing Guix's haskell packages as my fix for > https://issues.guix.gnu.org/66347 requires rebuilding all of them > anyway. Should I try to keep commits small with only one update per > commit (which is more work but managable if I don't care about the > commits being buildable) or should I try to keep them buildable (i.e. > update everything in one commit)? so far I’ve been updating Haskell packages in bulk in a single commit. See 49a320aaa6fb4c20d6b30c56c35a8c7ffceed822 or b97f549b14402421fcfb360ddd4cff7de93b9af0 for example. I also used custom scripts last time, because `guix refresh` was not sufficient to update all fields required (arguments, inputs, …). This is hopefully different this time. Cheers, Lars
Re: Cadence For Merging python-team into master
Hi, > […] or is it just a matter of successfully building all dependents that you > are introducing against master? basically this and running the system tests. Lars
Re: Cadence For Merging python-team into master
Hi, > What is the cadence for merging the python-team branch into master? > Should you do it, me, or someone else? there’s no timeline or schedule. It happens when someone does the integration and testing work. I don’t have the time to do this right now, so please go ahead. Note there’s also https://issues.guix.gnu.org/65010 (i.e. the pyproject-toml branch), which would be nice to have integrated as well, since it also requires a world rebuild. Cheers, Lars
Re: Python Team: Keeping Branch Up To Date Question
Hi, > What is the git approach for keeping the Python branch up to date? 閭 > Should I be rebasing off of master or something else? yeah, that’s generally what I would do before working on it. Note that you cannot force-push into Savannah. You have to remove the remote branch and create it again. Lars
Re: poetry: python-poetry?
Hi Andy, > curious poetry is not named python-poetry in Guix as following > convention of most python packages packages providing a binary instead of a library usually skip the language-specific prefix, because the programming language does not matter in that case. Lars
Re: poetry not building
Hi Reza, > Poetry is not building on ci.guix.gnu.org [1]. There is a pending patch > [2] on the issue tracker. What is missing to apply this patch and how > can I help? both contributors to that issue – John and me – are busy with other things right now. That’s all. My proposal in the python-team branch is rather intrusive and requires a world rebuild. I wrote down what needs to be done here: https://issues.guix.gnu.org/63139#23 Some of that work has been merged into the python-team branch already, but I think the 3rd point is missing entirely and the 2nd one is not properly tested yet. Cheers, Lars
Re: branch master updated: gnu: eudev: Use new package style.
Hi, > a less frequent drop in substitute availability on master would considerably > elevate my satisfaction with guix as a user. and in my reading that is the > cost that Christopher talks about here. I was also unhappy enough to pull after the change was made, but substitutes were not available yet. This is pretty bad if you don’t have the local compute power (laptop) to build the entire Java bootstrap chain. Or just imagine needing Haskell… I just ended up using `guix pull --roll-back`, because that revision was unusable for me. Woudn’t it make sense to create a “feature branch” for mass rebuilds like this one (even if it’s just one package), get the substitutes ready and then merge to master? Cheers, Lars
Re: GHC 8.6.5 build failure (bootstrapping)
Hi, > Everything looks great, except that GHC failed to build from source > locally. (I know GHC bootstraps separately.) I don't think my code > caused the failure, but it's possible. Looks like it failed in the > tests. could you share your changes? The logs are rather unhelpful. Cheers, Lars
Re: Python feature branch
Hi Andreas, > I wanted to set up automatic building on cuirass for the Python updates > branch, but was not sure which one it is: > $ git branch -a | grep python > remotes/origin/python-updates > remotes/origin/wip-python-graphviz > remotes/origin/wip-python-mne > remotes/origin/wip-python-pep517 I doubt there is one yet. Per #63139 I was going to set up “python-team”, which seems to be the naming consesus right now. > Some of these have recent commits, others not; it would be nice if you > could clean them up and agree on which one to build. @Ricardo, Vivien: Can you confirm wip-python-graphviz and wip-python-mne are stale/unused? Cheers, Lars
Re: evince: Is this a bug?
Hi, > Gtk:ERROR:gtkiconhelper.c:494:ensure_surface_for_gicon: assertion failed > (error == NULL): Failed to load > /gnu/store/vkcl29s7qcfgqz1cs6lab98fyxsxyczx-adwaita-icon-theme-43/share/icons/Adwaita/48x48/status/image-missing.png: > Unrecognized image file format (gdk-pixbuf-error-quark, 3) > Bail out! Gtk:ERROR:gtkiconhelper.c:494:ensure_surface_for_gicon: assertion > failed (error == NULL): Failed to load > /gnu/store/vkcl29s7qcfgqz1cs6lab98fyxsxyczx-adwaita-icon-theme-43/share/icons/Adwaita/48x48/status/image-missing.png: > Unrecognized image file format (gdk-pixbuf-error-quark, 3) I’ve noticed this too with several GTK+ programs. The problem seems to be that GDK_PIXBUF_MODULE_FILE is set to /home//.config/guix/current/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache. When I change it to refer to ~/.guix-profile instead it works. Not sure how ~/.config/guix got in there though… Cheers, Lars
Re: Core-updates after the staging merge
Hi Andreas, > Probably so! I will let you decide whether to apply it or to drop the 9.0 > version, both are fine from the core-updates point of view. I fixed 9.0 in commit dc9c09023a5258de035424169b8e804acfd38cb2, but 9.4 still fails :( Will investigate. Lars
Re: Core-updates after the staging merge
Hi Andreas,
> Well, I would see this as rather an action for a later feature branch.
> Except for removing 9.0 by building 9.4 with 9.2, since 9.0 does not
> build now anyway.
shouldn’t this snippet from 8.10 also work for 9.0?
(modules '((guix build utils)))
(snippet
;; collections.Iterable was moved to collections.abc in Python 3.10.
'(substitute* "testsuite/driver/testlib.py"
(("collections\\.Iterable")
"collections.abc.Iterable")
Cheers,
Lars
Re: Core-updates after the staging merge
Hi Simon, > and instead we could try this shorter one: > >7.8.4 > -> 8.0.2 (needs >= 7.10) > -> 8.4.4 (needs >= 8.0) > -> 8.8.4 (needs >= 8.4) > -> 9.2.5 (needs >= 8.6) > > WDYT? If no objection, I will give a try. note that the information on haskell.org is not always accurate and thus this shorter chain may not actually work. Please give it a try and send a patch. > I mean I propose to have both: ghc-x.y (with tests) and > ghc.x.y/bootstrap (without tests), it would ease the maintenance of the > Haskell ecosystem on several architectures. WDYT? I have a bad feeling about turning the testsuites of intermediate versions off. Yes, they take a long time, but then they also ensure the resulting compiler actually works (with high confidence) and we don’t silently propagate issues into the next one. Cheers, Lars
Re: i686 core-updates failure.
Hi Andreas, > This also did not work - timeout after 3600 seconds of silence, > the property is overwritten by the guix daemon, I think. > Anyway, as said before, I rather think this is a real problem with > the tests (maybe depending on some special situation on berlin if you > could build it on your machine) than a genuine silence. do you have a log file indicating where exactly it failed? Thanks, Lars
Re: i686 core-updates failure.
Hi Simon, > Thanks for checking. It also builds for me locally. So I guess, > > +(properties > + ;; 3 hours to avoid time-out in the check phase. > + `((max-silent-time . 10800))) > > would be helpful. And it should be inherited by 9.2.5 so it should > also build on CI. WDYT? last time I tried this I was told Cuirass does not respect this property, so I’ll leave your patch untouched, hoping someone more knowledgable will push it – if it works. Lars
Re: i686 core-updates failure.
Hi Simon, > About GHC, I am trying to build ghc-8.10.7 on core-updates for i686. It > could be nice to fix it. Well, I will be happy if someone™ beats me. ;-) GHC 8.10.7 on i686 built fine for me locally with all timeouts disabled. 9.2.5 is not done yet, but is slowly processing through the test suite as well for me. Can someone build both manually on the CI servers maybe? Lars
Re: PyQt in core-updates
Hi Andreas, > I have just fixed calibre. It failed to build because .sip fails are > now in a subdirectory /lib/python3.10/site-packages/PyQt5/bindings > instead of /share/sip (or maybe before, they were in both directories). no, it was definitely in /share/sip before, but the pyproject-based build system does not expose any option to move it there :( If anyone can figure out how to move it back *and* successfully compile pyqt, please, change it back to /share/sip. Lars
Re: Python
Hi again, > I just updated python-ipython to the first version that passes all tests > without any change, in the hope that this would have the least impact on > its dependents. Without luck, now python-trio fails. Updating this to the > latest version 0.22 does not help. I copy-paste the error message below. > If you could have a look, that would definitely be welcome. this one’s also fixed, see cf26ee1f99f84f817f96269541073846d546026b gnu: python-trio: Run pytest on tests directory only. Lars
Re: Python
Hello Andreas, > I looked at it and it seems that python-sip and python-pyqt-builder need a > version bump and we should maybe switch to pyproject-build-system. However > python-sip does not expose any switches to specify a different path for > .sip file includes. That requires custom patches for python-sip. > python-pyqt builds, but everything else is work in progress… python-pyqt and python-pyqtwebengine should build on core-updates now. I’ve tested qutebrowser, which works. Other packages depending on python-pyqt fail due to unrelated problems. See d9dc32b8716e5213fa127587dce64a6bbe5daee8 gnu: python-pyqtwebengine: Update to 5.15.9. 628eefa3695fad74f8fb8daca1f243b1684fe9f8 gnu: python-pyqt: Update to 5.15.9. 9b7e8365e228407c4f6f7071f6b36046be5e20b7 gnu: python-pyqt-builder: Update to 1.14.1. eebb0bfd68a863f6ef6c18e20daad7aaa6d3ba2a gnu: python-sip: Update to 6.7.7. Cheers, Lars
Re: Python
Hi, > Well, there is pyqt as written elsewhere, but this is not only python. I looked at it and it seems that python-sip and python-pyqt-builder need a version bump and we should maybe switch to pyproject-build-system. However python-sip does not expose any switches to specify a different path for .sip file includes. That requires custom patches for python-sip. python-pyqt builds, but everything else is work in progress… Lars
Re: Python
Hello Andreas, > excellent! It contradicts the feedparser author's statement that everything > works well, so if you have the courage, maybe you could report the issue > upstream. I just saw it was fixed on the develop branch already, but there’s no new release: https://github.com/kurtmckee/feedparser/commit/c55bd8ad37db89bd219783bc514d600c9523ed38 Cheers, Lars
Re: Python
Hi again, > > > Right now I am left with a number of test failures that look real and > > > cannot > > > easily be solved by an upgrade (either because we are already on the > > > latest > > > version or because the tests still fail): python-sgmllib3k, > > > python-typeguard > > > and python-coveralls. See messages below. > > I don’t know for sure why any of these packages’ tests fail. typeguard > > looks like it expects specific strings from Python or one of its libraries > > – safe to ignore. > > would you mind having a look how we should proceed? Fix tests or disable > specific ones that you deem safe to ignore? python-typeguard and python-coveralls are functional again: d9a31cd34eb9f63460d74d3cfff0190f964f48f4 gnu: python-coveralls: Disable failing test and relax version requirements. a7c96167ae8748a8b76fabaf74a997094cdf7fc5 gnu: python-typeguard: Python 3.10 compatibility. Anything else that needs work? Cheers, Lars
Re: Python
Hello Andreas, > This one is used for python-feedparser, used for calibre and quodlibet. > The feedparser author is not enclined to work on it: >https://github.com/kurtmckee/feedparser/issues/328 > I would suggest to try compiling python-sgmllib3k (and potentially > python-feedparser) without the tests, and see whether calibre still works. > But for this, we first need to get all other inputs of calibre into working > shape. I fixed both python-sgmllib3k and python-feedparser. Disabling tests would not do much in this case, since the packages really *were* broken by a Python upstream change. f9bff8614be32f9c2c1a54da80eaed1365b49be3 gnu: python-feedparser: Add Python >=3.9 compatibility. cfccd6fe5ae00d7e81cd755be55d51ff3bf17186 gnu: python-sgmllib3k: Add Python >=3.9 compatibility. Cheers, Lars
Re: GHC in core-updates
Hi Andreas, > But I confirm that ghc@9.2.5 now fails its tests, it has happened a second > time for me. Nothing in the recent commits to core-updates since I tried > the last time springs to mind as obviously having a connection to these > failures. fixed by commit ec79901a33b6463ad77dcaf4c37e538645b4d7b5 on core-updates. I also verified pandoc builds. Cheers, Lars
Re: GHC in core-updates
Hello Andreas, > Maybe you had a transient test failure? "async" sounds like the kind of > tests that could fail randomly... unfortunately it is not a transient failure. I can reproduce it every time I try to build python2 on core-updates, even with `--cores=1`. And it also reproduces on master. But, now that I have upgraded to kernel 6.1.12 it works. So I’m assuming it might have been something like [1]. Building up to GHC 9.2.5 might take a while though… Cheers, Lars [1] https://discuss.python.org/t/linux-kernel-6-0-16-6-0-18-breaking-python-tests-allows-to-bind-a-socket-twice/22701
Re: GHC in core-updates
Hello Andreas, > the latest trial to compile ghc@9.2.5 resulted in many test errors, > see below. I tried to compile GHC on core-updates recently, but didn’t even get to GHC 9.2.5, because python2 failed to build (test test_asyncore failed, without any further information). Do you have any extra patches applied? Cheers, Lars
Re: Merging branch wip-haskell
Hello Felix, > I see build failures [1] for Ghc 9.2.5 in the Cuirass job set [2] for > my own feature branch. [3] They are caused by a one-hour timeout. Did > you folks figure out how to extend the timeout limit on Cuirass when > working on your own branch? Thanks! I don’t see GHC being rebuild in [2] and GHC 9.2 should have a higher max-silent-time than 1 hour on master since 3bb2078a12d78a13f1e1520fe3705333a74ef6e3 (which is part of your branch). So I’m slightly confused. Cheers, Lars > [1] https://ci.guix.gnu.org/build/478772/log/raw > [2] https://ci.guix.gnu.org/eval/262309 > [3] https://git.savannah.gnu.org/cgit/guix.git/log/?h=wip-go-updates
Re: Python
Hi, > I updated it to its latest version under its current name python-cheetah, > but would suggest to rename it to python-ct3. What do you think? I don’t think we should follow PyPi’s names strictly. python-cheetah3 makes much more sense than python-ct3. That’s what upstream-name is for. Lars
Re: Python
Hi, > Do we have a list of packages in the python importer that can be removed > from inputs? Like already exists for hackage (and maybe others)? I’m not aware of any list like that and to compile it we’d probably have to build all python-* packages and check whether any of their installed modules collides with anything the python package itself provides (where collision does not necessarily mean file collision, but package namespace collision in the Python world). Cheers, Lars
Re: Merging branch wip-haskell
Hi, > > the branch has been merged into master. > \o/ Nice, thank you! well, GHC fails a single testcase on i686 (which we did not test for wip-haskell) right now[1], but it’ll take some rounds of building it locally until I have this sorted out too. Lars [1] https://ci.guix.gnu.org/eval/230900?status=failed=0
Re: Merging branch wip-haskell
Hi, > just a heads-up: The long overdue Haskell update is finally rolling in, > bumping our packages to Stackage release 20.5 and the compiler to 9.2. The > corresponding issue is #61420. the branch has been merged into master. Lars
Re: Python
Hi, > Right now I am left with a number of test failures that look real and cannot > easily be solved by an upgrade (either because we are already on the latest > version or because the tests still fail): python-sgmllib3k, python-typeguard > and python-coveralls. See messages below. I don’t know for sure why any of these packages’ tests fail. typeguard looks like it expects specific strings from Python or one of its libraries – safe to ignore. sgmllib3k looks pretty dead upstream. Perhaps it’s not even needed any more? Updates to Python packages (via `guix refresh`) do not update dependencies and thus the list of inputs/native-inputs are most likely outdated. Well, there are reasons no-one is updating the Python ecosystem regularly… Cheers, Lars
Re: Python
Hi Andreas, > This version requires python-importlib-metadata; not its latest version 6, > but something at least 5 and less than 6. We were still at 4.something. > So I have just updated it to 5.2.0, the latest version 5 from last December. > This gives me python-json-spec, so I am one step closer to calibre, which > I am interested in. note that importlib-metadata is – again – part of the standard library, as the table on [1] points out. So if we would ship Python 3.12, we would not need it. Bumping it to version 5.2 seems like the correct approach right now, since we’re at 3.10.7 on core-updates (as far as I see). > But python-importlib-metadata has 892 dependents (among which interesting > looking ones, such as freecad and gnome-terminal). I wonder whether I am not > breaking 891 dependents by enabling, maybe, the one that I am interested in. Besides the 'sanity-check phase, which checks for the most common issues, the only thing we can do is rebuild and run tests. Since Python is a dynamic language that still does not guarantee that it will run, but it’s all we can realisically do -.- Cheers, Lars [1] https://pypi.org/project/importlib-metadata/
Re: Python
Hi Andreas, sorry, I can’t quite keep up with the Python issues on core-updates right now :( > Yet another python failure: python-pathlib this is a backport of Python’s built-in pathlib library. It should be dropped as a dependency for all of these packages, since our Python is >= 3.4 – the version pathlib was introduced into the standard library. And then drop the package entirely. Lars
Re: Python (was: Merging core-updates?)
Hi, > Except that we have to decide what to do about its dependents... upgrade or drop if not possible. pycryptodome does not provide an entirely compatible interface (see https://www.pycryptodome.org/src/vs_pycrypto), so we cannot simply switch existing packages from pycrypto to pycryptdome without manual testing and (possibly) patching. eolie upstream looks dead, same with jrnl. The rest seems to be alive without any references to python-pycrypto. So these should be upgradable and then we can drop python-pycrypto. Lars
Re: Python (was: Merging core-updates?)
Hi Andreas,
> *** File
> "/gnu/store/5i3yqwaqd8mayl2vr9lmrihxwv8203b1-python-pycrypto-2.6.1/lib/python3.10/site-packages/Crypto/Util/number.py",
> line 139
> value |= 2L ** (N-1)# Ensure high bit is set
> ^
> SyntaxError: invalid decimal literal
> error: in phase 'install': uncaught exception:
> %exception #< program: "python" arguments: ("-m" "compileall"
> "--invalidation-mode=unchecked-hash"
> "/gnu/store/5i3yqwaqd8mayl2vr9lmrihxwv8203b1-python-pycrypto-2.6.1")
> exit-status: 1 term-signal: #f stop-signal: #f>
> phase `install' failed after 0.5 seconds
> command "python" "-m" "compileall" "--invalidation-mode=unchecked-hash"
> "/gnu/store/5i3yqwaqd8mayl2vr9lmrihxwv8203b1-python-pycrypto-2.6.1" failed
> with status 1
this particular line looks different with Python 3.9, since the package
is built with an automated Python 2 to Python 3 converter, which does
not seems to work correctly on 3.10 (build_py_2to3 in setup.py). Not
sure why though. Given the warning on their homepage it’s probably
safe to drop the package.
> from collections import Mapping
> ImportError: cannot import name 'Mapping' from 'collections'
> (/gnu/store/blals34ar25fiifvm17m2b504waxzys0-python-3.10.7/lib/python3.10/collections/__init__.py)
This is trivial to fix and should be
from collections.abc import Mapping
which seems to be the only change in attrdict3, see
https://github.com/pirofti/AttrDict3/commit/f6678b627b469c9aeddca2a9e4ba4e1ee9e3ccbb
Cheers,
Lars
Re: Merging branch wip-haskell
Hi Andreas, > I do not think we should coordinate, this was part of the motivation for > considering feature branches in the first place, to avoid entanglement > with different updates. ah, alright. I’ll give everyone more time for reviews and then just merge it. > However, QA has not run yet: >https://qa.guix.gnu.org/issue/61420 > I assume it will start in a few days, so maybe you could wait a little bit > until it has run and hopefully provided substitutes on bordeaux? There’s a job on CI already: https://ci.guix.gnu.org/jobset/wip-haskell > By the way, there are also branches wip-haskell-updates and > wip-haskell-updates-2 from 2020; maybe they can be deleted? Yeah, I guess we can remove them. Cheers, Lars
Merging branch wip-haskell
Hi, just a heads-up: The long overdue Haskell update is finally rolling in, bumping our packages to Stackage release 20.5 and the compiler to 9.2. The corresponding issue is #61420. Is there anything preventing a merge into currently? Can we coordinate the merge with some other big world-rebuilding changes waiting to happen (apart from core-updates)? Cheers, Lars
Re: Licence of the Guix blog posts
Hi, > Simon, what do you think about emailing the authors of the “10 years of > stories” post asking if they agree with the licensing? :-) No rush, > though the sooner the more likely we are to get an answer. I’m also fine with the terms proposed here. Lars
Re: FOSDEM’s coming!
Hi Ludo, > Please take a look and send patches if needed! If someone can come up > with some kind of a logo for the Guix Days, that’d be great; otherwise > we’ll just reuse the one from last year. note that the Guix Days take place on February *2nd and 3rd* (correctly identified as Thursday and Friday in the post). Cheers, Lars
Re: GHC >= 9.0?
Hi jgart, > Hi, what's the status on packaging GHC >= 9.0? > Is anyone working on that? there is commit 7a9350c208be14484d4fa6d90c0169f40fcf500e on wip-haskell, which adds GHC 9.0.2. No further work has been done to update the Haskell ecosystem. Help is welcome though, because it’s very time-consuming. Lars
Re: 01/03: guix-install.sh: Improve prompt_yes_no procedure.
Hi Maxim, > commit 6a2e303d3a49baf7c222a70b91f453e9efd456c6 > Author: Maxim Cournoyer > AuthorDate: Wed Oct 5 21:48:25 2022 -0400 > > guix-install.sh: Improve prompt_yes_no procedure. > > * etc/guix-install.sh (_flush): New function. > (prompt_yes_no): Clear input, then only read the first character, > silently. > Add the [Yes/no] string to the message. When a newline is entered by the > user, treat it as the default value, which is "yes". > (chk_gpg_keyring): Remove "(yes/no)" from the prompt message. > (configure_substitute_discovery): Likewise. > (sys_authorize_build_farms): Likewise. I got a report here[1] that this refactoring broke my GitHub action, which simply pipes `yes` into guix-install.sh. Thus all pipelines depending on https://github.com/PromyLOPh/guix-install-action are (again) broken. Shall we revert this or do you have a quick fix at hand? Lars [1] https://github.com/PromyLOPh/guix-install-action/issues/16
Re: Pick your team!
Hi Mathieu, > 1. Defining your team scope, if you are already part of a team I pushed scopes for the Haskell and Python teams, but especially for the latter packages are all over the place and barely covered by file-based scopes unfortunately. Cheers, Lars
Re: 02/04: services: Add samba service.
Hi Mathieu, > I noticed that while the "samba" test > works fine on my machine, it fails on the CI: > https://ci.guix.gnu.org/build/1495525/log/raw > any idea why? it works on my machine too. The log above says ---snip--- This is the GNU system. Welcome. komputilo login: In execvp of /gnu/store/996q9xpmk73l1z8mjx1yix3ri629jprb-samba-4.16.4/bin/smbclient: Invalid argument In execvp of /gnu/store/996q9xpmk73l1z8mjx1yix3ri629jprb-samba-4.16.4/bin/smbclient: Invalid argument ---snap--- and the test is locally using a different smbclient (/gnu/store/mmnawpyk0zn08cvhypj8a7ir95mk5fag-samba-4.16.4/bin/smbclient) for me. Lars
Re: State of the 'core-updates' branch
Hi Marius, > I expect many Python packages need to be updated for 3.10. To ease this > process it would be great to get the revamped build system from > 'wip-python-pep517' merged. I plan to look at it "soon", but volunteers > welcome. :-) not to discourage you from taking a look, but wip-python-pep517 is most likely not in a good state. I haven’t worked on it for quite a while and don’t expect the branch to apply cleanly to either master or core-updates. It also forces us to touch alot of Python packages manually, since the behavior of the 'check phase is slightly different and lots of packages with previously not run test suites now fail due to running it. I’d love to get this done, but my time is still very limited. Cheers, Lars
Re: Cabal mismatch in ghc-lucid; long-term archiving Haskell?
Hi zimoun, > Maybe it could be better to move the ’cabal-revision’ from the > ’arguments’ field to the ’origin’ field. > Perhaps, we could have a ’cabal-revision’ procedure returning a G-exp > and put it under the ’snippet’ field. > WDYT? it would be awesome to have that functionality and I agree that a “combining” origin putting the .cabal file in the right spot would be ideal. Not sure how to build that though, any ideas? > That’s said, it is a core-updates change because it requires to modify > the Haskell build-system and, a rough estimate about the number of > impacted packages: There’s a branch wip-haskell waiting to be worked on. Lars
Re: Cabal mismatch in ghc-lucid; long-term archiving Haskell?
Hi zimoun, > Is it a manual in-place replacement? Or is it an automatic in-place > update by Hackage infrastructure? Or do I miss a point? > > In all cases, these revised Cabal files are not archived elsewhere than > in Hackage, right? The question is thus, where could we archive them? > > Note that the both items have the same store-address hash > (/gnu/store/rj33x41i86vgw6c0iwffzwrgzrib1shb-ghc-lucid-2.9.12.1-1.cabal) > but not the same content hash. fear not, this is a mistake on our side – mine in particular. `git blame` indicates the package was updated in my last big Haskell bump as commit b97f549b14402421fcfb360ddd4cff7de93b9af0. And while I updated the version number, I did not touch #:cabal-revision, which is obviously a mistake. Unfortunately we chose to encode this information into arguments and not the version and so this happens (alot), because – alas – `guix refresh` touches the version only, but not #:cabal-revision. As for Haskell/Hackage/Stackage: All of their .cabal files are versioned and never overwritten. I think they also keep them permanently. Cheers, Lars
Re: Who owns guix-mirror?
Hi, > Thanks :-) Keeps the PRs away, or at least you can't prove > otherwise. you could also enable “temporary interaction limits”[1] in your org account, which applies to all of its repositories. It’s only valid up to 6 months, but maybe better than nothing…? Cheers, Lars [1] https://github.com/settings/interaction_limits
Re: Welcome to our new committer!
Hello Andrew, > The Guix Maintainer Collective (tm) would like to welcome Andrew Tropin, > aka abcdw, as our newest committer! I'm sure many of you recognize them > from their work on Guix Home and their regular videos hacking on Guix, > among other things. welcome! Good to have you here as a committer! Cheers, Lars
Re: python-pytest in references graph
Hi, > It should, but sometimes there are bugs in the package definition or > build system, in this case causing python-rdflib to refer to the > native-input python-pytest. Likely it's the 'add-install-to-path' phase > adding too much, a known issue, which could be solved by separating > inputs and native-inputs on the build side when compiling natively (and > not only when cross-compiling) (currently they are merged together into > 'inputs'), though non-trivial. the issue is indeed a known bug https://issues.guix.gnu.org/25235 Lars
Re: Why is greetd greeter user in so many groups?
Hi, > Sounds good, thanks for the fix! d921516f50a946e92f9d5dc6d3bd49aca9788ac2 services: greetd: Remove unnecessary user groups. Cheers, Lars
Re: Why is greetd greeter user in so many groups?
Hi, > Since greetd is currently being run as root, it doesn't need any > extra group membership. indeed, agreety works fine with that patch. I’d still keep the video supplementary group, so one can run gtkgreet/wlgreet (if they ever pop up in Guix). Any objections? Cheers, Lars
Re: Why is greetd greeter user in so many groups?
Hi, > Only root can write to /var/log, so wheel is irrelevant. And, indeed, greetd > logs are being written as root: oh, I guess they are written by greetd, not the greeter itself. Does greetd work without the groups in questions? (I don’t have access to a powerful machine right now to test it.) Thanks, Lars
Re: Why is greetd greeter user in so many groups?
Hi,
I merged greetd.
> (group "wheel")
> (supplementary-groups '("users" "tty" "input" "video"
> "audio"))
> […]
> I can understand the need for tty and input, but why does the
> greeter user need the wheel and audio?
I believe wheel is necessary to write logs to /var/log, because they
don’t go through the syslog. audio maybe for GTK-based greeter with
accessibility (i.e. TTS), but I’m not sure to be honest.
Lars
Re: python-libcst upgrade and python-flake8
Hi jgart,
> validating 'flake8'
> /gnu/store/zaw5z708sldm6v3qxjczcia7gl65dw5x-python-flake8-3.9.2/lib/python3.9/site-packages
> ...checking requirements: ERROR: flake8==3.9.2
> ContextualVersionConflict(pyflakes 2.4.0
> (/gnu/store/mxkp1zl64cd3i247shp7njkxad8v13x9-python-pyflakes-2.4.0/lib/python3.9/site-packages),
> Requirement.parse('pyflakes<2.4.0,>=2.3.0'), {'flake8'})
I believe this was fixed in Guix proper already by upgrading to 4.0.1
and loosening the version checks. In any case it’s not a problem with
the library you are trying to package.
Cheers,
Lars
Re: Teams
Hi Ricardo, > As a first step I’d suggest collecting teams, setting up the email > aliases, and updating the website to show the existing teams. Here’s > a draft of three teams: * Python Team Lars-Dominik Braun * Haskell Team Lars-Dominik Braun Anyone interested to join these with me? Cheers, Lars
Re: Test Failure when upgrading python-rope
Hi jgart,
> ==
> FAIL: test_search_submodule (ropetest.contrib.autoimporttest.AutoImportTest)
> --
> Traceback (most recent call last):
> File
> "/tmp/guix-build-python-rope-1.1.1.drv-0/rope-1.1.1/ropetest/contrib/autoimporttest.py",
> line 121, in test_search_submodule
> self.assertIn(import_statement, self.importer.search("env",
> exact_match=True))
> AssertionError: ('from build import env', 'env') not found in []
this exact failure also appears outside of Guix (i.e. virtualenv and then
`python setup.py test`), so I’d assume it’s a bug, disable the test
and move on.
Cheers,
Lars
Re: Haskell updates: GHC 9 and Stackage 19
Hi zimoun, > In the light of «‘staging’ branch is open!» [1], it could be nice! :-) there’s no way we’ll get this done by your proposed date May 8th. Last time it took me two weeks full time to update everything. I don’t want to delay the merge of staging for this. > About the importer, I do not know. But we can share the workload. > Well, I will give a try next week. Sure, go ahead. But for starters we don’t even have GHC 9.0 as a package yet. Meanwhile I’m working on improvements for the cabal importer, so it does not choke on certain .cabal files. Cheers, Lars
Haskell updates: GHC 9 and Stackage 19
Hi everyone, it’s time to update our Haskell environment – again. GHC 9.0 has been out for a while and Stackage updated its LTS distribution to version 19 recently, providing a new set of packages for GHC 9.0. Additionally there some issues/patches regarding haskell-build-system and the importer, which we should address in this cycle: - build: haskell-build-system: Support packages w. multiple libraries https://issues.guix.gnu.org/54729 (applied to wip-haskell, fixes https://issues.guix.gnu.org/52152) - build-system: haskell: Add ‘package-with-explicit-haskell’ procedure https://issues.guix.gnu.org/51655 (probably a world rebuild including Python, maybe core-updates?) - import: hackage: `elif` conditionals not supported https://issues.guix.gnu.org/54752 (needs patch) - guix import hackage does not support build-tools and build-tool-depends stanzas https://issues.guix.gnu.org/49320 (?) Are there any other issues that need to be adressed that require a world rebuild? To ease the update it would also be very nice if the importer could update the entire package definition, including inputs and arguments (#:cabal-revision, in particular). Is that somehow possible? I will not be able to perform the upgrade if it is not possible. The amount of manual work required is too much for me. Thanks, Lars
Re: How to use Guix with sssd, not nscd, on a foreign distro?
Hi Chris, > The Fedora document explains that at least the hosts cache will be > handled by systemd-resolved. Can I expect Guix-built programs to "try > to use systemd" when resolving host names, or is additional > configuration likely to be required? as far as I know systemd also plugs into the nss mechanism, i.e. it’s a shared library libnss_systemd.so from the host system, which would be dlopen’ed. > Regarding sssd specifically, how can I arrange for a Guix-built program > to "try to use sssd" first? I believe nscd is a glibc internal mechanism, which – if enabled – will be queried before using the nsswitch mechanism. Thus it works to circumvent the shared library dilemma on foreign systems. Programs do not use sssd directly (that’s the whole point of nss/sssd). It plugs into nsswitch just like systemd and therefore does nothing for our use case. nixOS seems to have the same problem, with suggestions for solutions like [1]. So in conclusion I don’t think sssd is an appropriate replacement for our use-case. Cheers, Lars [1] https://github.com/NixOS/nixpkgs/pull/155655
Re: GNU Guix maintainer rotation
Hi Guix, and a warm welcome to Efraim! Maybe update the about part of the website too? (See attached patch.) Cheers, Lars diff --git a/website/apps/base/templates/about.scm b/website/apps/base/templates/about.scm index 427b908..3d72adf 100644 --- a/website/apps/base/templates/about.scm +++ b/website/apps/base/templates/about.scm @@ -107,8 +107,8 @@ system|GNU Hurd|GNU Guix package manager") #\|) ,(G_ `(p - "Guix is currently maintained by Ludovic Courtès, Marius Bakke, Maxim -Cournoyer, Tobias Geerinckx-Rice and Mathieu Othacehe. Please use the " + "Guix is currently maintained by Efraim Flashner, Mathieu +Othacehe, Maxim Cournoyer and Tobias Geerinckx-Rice. Please use the " ,(G_ `(a (@ (href ,(guix-url "contact/"))) "mailing lists")) " for contact. For sensitive issues, you can reach them " "using the " (code "guix-maintain...@gnu.org") " private alias."))
Re: python-sqlalchemy-utils sanity-check phase broken
Hi, > I just wanted to mention that python-sqlalchemy-utils sanity-check phase > is broken. > […] > ImportError: cannot import name '_literal_as_text' from > 'sqlalchemy.sql.expression' > (/gnu/store/b11zrncrhbpslivk5qpzxsy36b6hsfpw-python-sqlalchemy-1.4.27/lib/python3.9/site-packages/sqlalchemy/sql/expression.py) this look alot like 'sanity-check caught an API compatibility problem and indeed, python-sqlalchemy-utils is at version 0.32.21, which is four years old. It would probably help to upgrade that package to a version compatible with our python-sqlalchemy. Cheers, Lars
Re: Formalizing teams
Hi Ricardo, > FWIW, mumi also lets you search patches as all contents are indexed: > > https://issues.guix.gnu.org/search?query=%22%28gnu+packages+python-web%29%22+is%3Aopen+tag%3Apatch thanks, I didn’t think about that. I tried searching for python-build-system, but not all patches – especially trivial ones – include enough context (for example https://issues.guix.gnu.org/52595). Searching for files yields too many packages, since Python packages are scattered across dozens of files (e.g. gnu/packages/music.scm). So unfortunately it doesn’t solve my problem. The only reasonable thing to do right now seems to be setting up package name based filters, because commit messages have a format known in advance. Cheers, Lars
Re: Formalizing teams
Hi Maxim, > I've grown to like our apparent lack of structure; we interact globally > on any topic of interest and the discussions all happen in a shared > space, which makes it easy to stay informed with everything that's going > on (do we really need more mailing lists to follow? I don't think so -- > our current volume doesn't warrant it). to me this is a disadvantage. I don’t have enough time at my hands to look at every single thread and patch in my inbox and decide whether I can/want to work on it. Thus I’d prefer if I could unsubscribe from -patches (I’m not even subscribed to -bug/-commit) and instead only look at bugs/patches/commits related to packages/components I’m interested into/I actually use. To that end having more structure would help. Teams are just one option, but we need a way to efficiently connect people with skills to review/help with those who need the review/help. This sounds similar to what Gentoo’s Bug Wranglers[1] do. I believe someone brought up to automate this by suggesting reviewers for patches based on the commit history of packages/components. [1] https://wiki.gentoo.org/wiki/Project:Bug-wranglers Cheers, Lars
Re: sanity-check: Don't understand the error
Hi Hartmut,
> ...checking requirements: ERROR: trytond-party==6.0.2 (python-stdnum
> 1.14 (/gnu/
> store/04i1p7rw5583g0la8d66qwzwlfs9rvhg-python-stdnum-1.14/lib/python3.9/site-pac
> kages), Requirement.parse('python-stdnum>=1.15'), {'trytond-party'})
it means that the package trytond-party depends on python-stdnum>=1.15,
but we only have version 1.14 available.
> PS: Any change to get a improved sanity-check.py into
> core-udates-frozen? Then we could add more explainations to the output.
Any change to this script unfortunately means a full rebuild of everything
depending on python-build-system.
Hope that helps,
Lars
Re: [core-updates-frozen] Different variants of Python packages in the same profile?
Hello Ricardo, > So… since numpy 1.20 is the exception here, could we perhaps … > rename it? And then have python-numba import that renamed module > “totally-not-numpy” instead of “numpy”? Could we thus avoid this > conflict? If renaming is an option — how would it be done? Is it > enough to rename the “numpy” directory with “numpy-1.20”, the > “numpy.py” file with “numpy-1.20.py”, and then update all “import” > statements both by numpy itself and by python-numba? I feel this is a dangerous idea. Python is dynamically typed and if we – somehow – end up with objects from both, numpy 1.20 and numpy 1.21 in the same program, which can still happen when renaming, things may go wrong very badly. [1] says versions 1.* are ABI compatible, but the API changes between releases, which is probably why numba cannot be used with a newer numpy. Can we rewrite the entire graph to use numpy 1.20 whenever a package imports numba? Cheers, Lars [1] https://numpy.org/devdocs/user/depending_on_numpy.html
Re: python: setuptools version doens’t match guix package version / packaging python-ipydatawidgets
Hi Théo, > Ok I see, it makes more sense now. But then I wonder why I could package > "python-ipydatawidgets" with "python-setuptools" and > "python-jupyter-packaging" as native-inputs. I had no collision warning > but didn’t change anything in "python-jupyter-packaging" or anything. as far as I know there are no collision checks/warnings in the build environment. Which of the two will be picked up there merely depends on the order of packages in PYTHONPATH. There are also no file collision checks in `guix shell`/`guix environment`, which is why you end up with a mix between v52 and v42 when running guix shell --pure python-jupyter-packaging python -- python3 -c 'import jupyter_packaging' It fails with an ImportError for me due to the wrong setuptools version, but works fine if I add python-setuptools to the command line, despite python-setuptools being propagated already. Cheers, Lars
Re: python: setuptools version doens’t match guix package version / packaging python-ipydatawidgets
Hi Théo, > I don’t understand how this could happen though since the definition of > python-setuptools seems just fine in python-xyz.scm ... unfortunately our python package bundles setuptools 41.2.0, which will collide with python-setuptools. I ran into the same issue with python-jupyter-packaging a while ago too[1]. I tried adding python-setuptools as a native-input, but it failed, because of said collision. Cheers, Lars [1] https://issues.guix.gnu.org/46848#9
Re: Commit ‘gnu: ghc-8.10: Disable failing test on i686.’ has a cross-compilation bug
Hi Maxime, > Currently, non-Linux is not supported by the GHC package. However, > people learn how to package things by example (and by reading > documentation, etc.), so I'd prefer to avoid (accidentally) teaching > people to make their package definitions Linux-specific. do we have any documentation/guidelines on how to do this properly? I remember I just copied code from a different package, because I couldn’t find anything else. Thanks, Lars
Re: Commit ‘gnu: ghc-8.10: Disable failing test on i686.’ has a cross-compilation bug
Hi Maxime, > https://git.savannah.gnu.org/cgit/guix.git/commit/?id=89de1924cb535fc2c97d3654e21badaebd43518e > > > + ,@(if (string=? "i686-linux" (%current-system)) > […] > > Barring any reports of the contrary, I'd presume the same would > apply for the Hurd. Also, %current-target-system seems more > appropriate, though here it doesn't matter because it's only > for tests. the GHC package declares support for x86 and x86-64 on Linux only, because we don’t have a bootstrap path for the Hurd (there are no prebuilt binaries) and the Hurd is not officially supported by upstream. Unless someone puts some effort into that I doubt it’ll ever run on that platform. > I suggest: ,@(if (target-x86-64?) '(...) '()) I don’t see `target-x86-64?` being defined on master and I assume you meant `target-x86-32?`, right? Cheers, Lars
Re: Accuracy of importers?
Hi Ludo’, > Right. PyPI/setup.py/.whl doesn’t contain info as to how to run tests, > right? technically setup.py has a standard test target, but it’s been deprecated for years and it must be enabled manually by the project. I’m not aware of any standard pyproject.toml approach to this. It might be possible to parse tox.ini. > >>hackage ? > >>stackage (Lars?) > > I’ve mostly used the updater, not the importer, so I can’t say a > > number unfortunately. > Did the updater suggest input changes? yes, I added it in 127828ddd74fc950c0403ca58a6f650355e3d67d, but it cannot update #:cabal-revision, which is a common source for errors. Would be nice if updaters could just return an entirely new package and the generic updater code would modify/merge the existing package definition as needed. Cheers, Lars
Re: Accuracy of importers?
Hi Ludo’, > My understanding is that most of them require manual intervention—i.e., > one has to tweak what ‘guix import’ produces, even if we ignore > synopsis/description/license, to set the right inputs, etc. If we were > to estimate the fraction of imported packages for which manual changes > are needed, what would it look like? > >importer fraction of imported packages needing changes >pypi 50% (some miss source distro, “sdist”; some have > non-Python deps) that seems right, although the most common modification I do nowadays is replacing 'check with a pytest phase. >hackage ? >stackage (Lars?) I’ve mostly used the updater, not the importer, so I can’t say a number unfortunately. >cran 5% (Ricardo? Simon? seems to almost always work?) In my experience the number of interventions here goes towards zero actually, except for description. It’s pretty good :) >npm (WIP)(Jelle? Timothy?) Maybe 5%? But the imported packages do not build anything and don’t run tests either, so chances for failure are pretty low. Would it be possible to just run the importer again for existing packages and compare the result (minus synopsis/description) with what’s available in Guix? That should give you much more accurate numbers than our guesswork. Cheers, Lars
eudev deprecation
Hi everyone, it looks like eudev, which we heavily rely on, is dead upstream: https://github.com/gentoo/eudev/issues/199 Looking at Gentoo’s ebuilds it should be possible to “extract” and build udev from systemd’s sources. Cheers, Lars
Re: [core-updates-frozen] python2-google-api-client and sanity-check
Hi, > From my point of view, yes you can push because it is a fix. pushed as bac072e09bb80c172672e13782e1ed25ce04d6d5. Please don’t hesitate to ping me if there’s any other issuse with the sanity check phase. Cheers, Lars
Re: [core-updates-frozen] python2-google-api-client and sanity-check
Hi zimoun,
> Because this ’sanity-check’ is new, IIUC, my question is: is it
> expected? And is it worth to fix such Python 2 packages or simply
> remove them because ’sanity-check.py’ uses Python 3 only features?
setup.py of this package is broken, because the package list contains a slash.
See attached patch for a fix.
Can I simply push that to core-updates-frozen or is there anything else I
should do?
Lars
--
Lars-Dominik Braun
Wissenschaftlicher Mitarbeiter/Research Associate
www.leibniz-psychology.org
ZPID - Leibniz-Institut für Psychologie /
ZPID - Leibniz Institute for Psychology
Universitätsring 15
D-54296 Trier - Germany
Tel.: +49–651–201-4964
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 91b68cf5f4..4763abce8b 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -4338,7 +4338,15 @@ Google search engine. Its module is called
@code{googlesearch}.")
"1wpbbbxfpy9mwxdy3kn352cb590ladv574j1aa2l4grjdqw3ln05"
(build-system python-build-system)
(arguments
- '(#:tests? #f)) ; tests require internet access
+ `(#:tests? #f ; tests require internet access
+ #:phases
+ (modify-phases %standard-phases
+ (add-after 'unpack 'fix-setup-py
+ (lambda _
+ (substitute* "setup.py"
+ (("googleapiclient/discovery_cache")
+"googleapiclient.discovery_cache"))
+ #t)
(native-inputs
`(("python-httplib2" ,python-httplib2)
("python-six" ,python-six)
signature.asc
Description: PGP signature
Re: Project direction with testing changes (branches and patches)
Hi Chris, > So, I think I've recently switched to thinking about the problem as one > of testing changes, rather than just testing patches. Since both patch > series, and branches are used to propose changes, I think this makes > sense. I agree with this analysis and this was always something that bugged me about Patchwork: That it knew about patch series (most of the time), but seemed to operate on individual patches only. > - Then there's the general UI component, ideally a first time > contributor would be able to take advantage of automatic feedback > about a patch they submit. There's multiple other groups of users > though, like patch reviewers, and committers for example. > […] > The UI part is much less certain, I've done some work with Patchwork, > and I do have some ideas in mind, but there's still more thinking and > work to do in this area. This is the main problem right now: UI. As a reviewer I need the following information about a changeset: 1 Is anyone else reviewing it already and what were the comments? (Do I need to take a look at it?) 2 Which packages/components are affected? (Do I have the knowledge to review this changeset?) 3 Are there any obvious issues with it? (Does it pass linting? Does it build? Can it be rebased onto master without conflicts?) 4 Where can I pull the changeset from to do further tests? (Patch-over-email is super-fragile, from encoding/newline issues to merging problems) 5 Who is the contributor? (Does he/she have commit access or do I have to commit on his behalf?) I also want to subscribe to certain changesets (new comments, new versions, …) and filter them to find those, where I can actually help. And debbugs certainly can’t provide any of this. I believe it would be wise not to spend too much time to recreate what GitLab/gitea/GitHub/… do already and just use them to our advantage, filling the holes with bridges/bots to some CI infrastructure (it does not matter whether this is Cuirass, Chris’ build coordinator, the Data Service, … – we just have to be able to get the data in and out somehow). Cheers, Lars
Re: #f as a package description, gnu: Add rocminfo.
Hi Christopher, > Anyway, I wouldn't like for this change to lower the standard though, > it's currently the only package in Guix with an invalid description (as > far as I'm aware), is there some reason why it doesn't have one? it simply fell through the cracks[1]. Commit 0a379de3249d5e9ff66fb404f7e5aa8ce2cb3d24 adds a proper descripton. Sorry for the trouble, Lars [1] Unfortunately I cannot run `guix lint` on an entire git changeset, so instead I have to check each package by hand and I probably missed rocminfo. I wish someday we can have a branch/pull-request-based workflow with automated CI checks (linting, `guix pull`, signature verification, …) *before* merging to master.
Re: Questions regarding Python packaging
Hi again,
> No, try
>
> git clone https://github.com/pypa/pep517.git
> cd pep517
> pip wheel --use-pep517 -v .
>
> which has no setup.py and uses flit instead. pip can build it, because
> it supports PEP 517-based builds. As I said, if we decide to keep it
> bundled with our python package, there’s no good reason to choose
> pypa-build.
I now have a proof of concept for a pip-based python-build-system, see
1st patch. The changeset is quite small, but it does not handle testing
at all right now.
Note that alot of setuptools-based packages lack a dependency on
python-wheel (2nd patch; pardon the whitespace errors), which is
required when using pyproject.toml (and also declared there, but our
Python importer cannot read that file yet). But – as expected – that’s
really the only issue I’ve encountered while rebuilding alot of Python
packages so far.
Cheers,
Lars
>From a7c6750917f5dc2e1eaf34520f7e6b0e3d5e0d3c Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun
Date: Tue, 6 Jul 2021 14:13:51 +0200
Subject: [PATCH 1/2] dirty: build: Build Python packages using pip.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
* guix/build/python-build-system.scm …
---
guix/build/python-build-system.scm | 109 +++--
1 file changed, 42 insertions(+), 67 deletions(-)
diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index 8ade1d5911..e4b63e131e 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -109,30 +109,6 @@
;; and the scripts defined in entry-points will always be created.
-(define setuptools-shim
- ;; Run setup.py with "setuptools" being imported, which will patch
- ;; "distutils". This is needed for packages using "distutils" instead of
- ;; "setuptools" since the former does not understand the
- ;; "--single-version-externally-managed" flag.
- ;; Python code taken from pip 9.0.1 pip/utils/setuptools_build.py
- (string-append
- "import setuptools, tokenize;__file__='setup.py';"
- "f=getattr(tokenize, 'open', open)(__file__);"
- "code=f.read().replace('\\r\\n', '\\n');"
- "f.close();"
- "exec(compile(code, __file__, 'exec'))"))
-
-(define (call-setuppy command params use-setuptools?)
- (if (file-exists? "setup.py")
- (begin
- (format #t "running \"python setup.py\" with command ~s and parameters ~s~%"
-command params)
- (if use-setuptools?
- (apply invoke "python" "-c" setuptools-shim
-command params)
- (apply invoke "python" "./setup.py" command params)))
- (error "no setup.py found")))
-
(define* (sanity-check #:key tests? inputs outputs #:allow-other-keys)
"Ensure packages depending on this package via setuptools work properly,
their advertised endpoints work and their top level modules are importable
@@ -142,26 +118,6 @@ without errors."
(with-directory-excursion "/tmp"
(invoke "python" sanity-check.py (site-packages inputs outputs)
-(define* (build #:key use-setuptools? #:allow-other-keys)
- "Build a given Python package."
- (call-setuppy "build" '() use-setuptools?)
- #t)
-
-(define* (check #:key tests? test-target use-setuptools? #:allow-other-keys)
- "Run the test suite of a given Python package."
- (if tests?
- ;; Running `setup.py test` creates an additional .egg-info directory in
- ;; build/lib in some cases, e.g. if the source is in a sub-directory
- ;; (given with `package_dir`). This will by copied to the output, too,
- ;; so we need to remove.
- (let ((before (find-files "build" "\\.egg-info$" #:directories? #t)))
-(call-setuppy test-target '() use-setuptools?)
-(let* ((after (find-files "build" "\\.egg-info$" #:directories? #t))
- (inter (lset-difference string=? after before)))
- (for-each delete-file-recursively inter)))
- (format #t "test suite not run~%"))
- #t)
-
(define (python-version python)
(let* ((version (last (string-split python #\-)))
(components (string-split version #\.))
@@ -195,31 +151,40 @@ running checks after installing the package."
"/bin:"
(getenv "PATH"
-(define* (install #:key inputs outputs (configure-flags '()) use-setuptools?
+;; Some packages expect 'build and 'check exist. If they don’t replacing them
+;; or adding phases before/after will fail. Preserve them as dummy-phases.
+(define* (build #:key outputs (configure-flags '()) use-setuptools?
+ #:allow-other-key
Re: Questions regarding Python packaging
Hi Hartmut, > What should be the use of having a package without pip? Anything else > than saving a few KB? saving some space and unvendoring components that we also have separate packages for. (As I understand it, ensurepip, which installs both pip and setuptools, is merely a convenience tool if you install Python from source yourself – not for distributions.) > AFAIK this might not be true if other build systems not using setuptools > at all might show up. And isn't this the main reason for all your work? No, try git clone https://github.com/pypa/pep517.git cd pep517 pip wheel --use-pep517 -v . which has no setup.py and uses flit instead. pip can build it, because it supports PEP 517-based builds. As I said, if we decide to keep it bundled with our python package, there’s no good reason to choose pypa-build. Cheers, Lars
Re: Questions regarding Python packaging
Hi Hartmut, > sorry for being late for commenting on this (the time I can spend on > guix is rather limited atm). no problem, same thing on my side. > * > > Not installing pip by default might break some user's environments. > Anyhow, since using pip in guix is not such a good idea anyway, this > should be okay. True. We could rename python→python-minimal-interpreteronly (or similar; given that python-minimal already exists) and python-toolchain→python to work around that. > * > > "use-setuptools" is gone. There are still about 10 packages with > "#:use-setuptools #f" - which means they are (expected to be) > incompatible with setuptools for some reason. You might want to > check whether these packages actually still can't be packages with > setuptools. Yeah, I’ve seen those, but the number was too small to bother for now. I’ll have a look later. > * > > setuptools-shim has been removed. I don't think this is a good idea, > since this peace of code enforces packages to be actually build with > setuptools instead of old distutils. This code is still in current > pip, so I assume it is still required. > > (This shim ensures setuptools is used, even if setup.py only imports > distutils. And setuptools is required for some options like > ""--single-version-externally-managed" - as the comment for the shim > says.) Is this relevant though? I doubt many packages are still importing distutils and the few that do can be patched. > * > > set-SOURCE-DATE-EPOCH: Please keep the verbose rational. It's much > more helpful than the new one-line comment. You mean the one from the now-removed ensure-no-mtimes-pre-1980? Sure. > * > > set-SOURCE-DATE-EPOCH: This implementation makes the code depend on > wheel and wheel being used for installation. Technically it depends on the wheel builder understanding SOURCE_DATE_EPOCH (not necessarily python-wheel). I’d say that’s acceptable and it’d be preferable to fix build systems not respecting this variable imo. > * > > Why has rename-pth-file been removed? Are you sure .pth-files are > never created anymore nowerdays? Given that easy-install has been deprecated I think it’s safe to remove this phase and flag any packages creating this easy-install.pth as broken. (There are, however, legitimate packages creating files like ruamel.yaml-0.15.83-py3.8-nspkg.pth.) > * > > python-hashbang: Isn't this done already by the normal > "patch-shebangs" phase after install in gnu-build-system? (BTW: > these are called *she*bangs). Afaik the function patch-shebang expects a leading slash and thus it does not replace this “special” shebang (see https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl; Spread, point 3). > * > > I suggest to have phase compile-bytecode still honor older versions > of python I’m not sure what you mean. compileall is also part of Python 2. > pypa bulld is where the PyPA is pushing towards. Anyhow, as of today, as > far as I can see, adoption is low. Of pypa build? That is true. > AFAIK fhere is no standard way for running tests in python. pytest seems > to be the most modern test-system. Anyhow packages still use nose or tox > (which again might run pytest or nose, with parameters fetched from > tox.ini). So I'm afraid, there is no general rule. > > Did the PyPA publish some recommendations or PEP on this? I’m not aware of any accepted PEP’s. There is a discussion about the removal of `python setup.py test`: https://github.com/pypa/setuptools/issues/931 And a proposal for pyproject.toml going nowhere: https://discuss.python.org/t/proposal-for-tests-entry-point-in-pyproject-toml/2077/2 > As I Python developer I nowerdays would expect pip and venv (which is > part of the std-lib - but not the virualenv, which is a separate module) > to be availalbe when installing "python". Anyhow I could live with pip > being a separate package. If we keep setuptools/pip bundled, we don’t have to do any of this pypa-build dance. We could also modernize python-build-system around `pip install` and just be done with it. (I don’t have a proof-of-concept for that yet.) > "python-toolchain" sounds oversized for me. Would this include the > C-compiler, too (which one? maybe I want to build cross). I'd rather not > have such a package. See suggestion above wrt renaming. > The gnu-build-system already provides the "unzip" binary (used in phase > "unpack"). So we could simply use this. Otherwise I recommend using the > Python zip module, as this is what is used for creating the zip-archives > :-) I’m using Python’s zipfile module already. Cheers, Lars
Re: Questions regarding Python packaging
Hi Tanguy, (cross-posting this to the issue itself too) > Sorry if I'm (very) late, but apprently this hasn't made it to master > yet, so… What the status? Do you still need a willing-but-maybe-not-qualified > person to review or discuss your patch? the patch set works, I can build many Python packages, although some require changes. Still, multiple things need to be done before merging is possible imo: 1) Validate the general idea of using pypa-build is viable and sustainable in the long run – ideally through review by someone else than me. We can’t touch python-build-system every week to solve structural issues, so it needs to be bullet-proof. 2) Figure out how to run testing code. Currently python-build-system just picks pytest, if available – not sure this is the best option we have. How do we deal with other test systems? How do we pass options? 3) Determine the fate of Python 2, which is probably broken through this patch set. Shall we remove it entirely? Is it worth to keep support? 4) Iron out minor details like including pip in the python package or create a new python-toolchain package? What do we include in that meta-package? pip? virtualenv? …? 5) Fix my awkward Scheme code, especially regarding unpacking of the built wheels. Should we be using Python’s unzip module or can be assumed unzip is available in the build environment? (Should we add it?) I’m by no means a Python packaging expert, so any help would be appreciated, even if it’s just a question or thumbs-up/thumbs-down on my ideas. Cheers, Lars
Re: Questions regarding Python packaging
Hi everyone, just a quick reminder that an updated version (includes python-toolchain) of this proposal is still looking for a code review or further discussion. So if you feel confident about touching python-build-system, please have a look at https://issues.guix.gnu.org/46848#1 I’d be nice to get this into core-updates before the next merge. Otherwise we’ll have to keep adding workarounds (see for example python-testpath in master) to Python packages not using setuptools as their build system. Cheers, Lars
Re: Improving the quality of Python packages
Hi Ricardo, > Is there a way we can improve the quality of those Python packages that > have tests disabled? I’d like to gain more confidence in these packages > and be sure that at least all dependencies are among the inputs. there is a new sanity check phase on core-updates, which should solve this problem. See 09448c0994390697e876db235a3b773311795238. Cheers, Lars
Re: guix environment --profile with --ad-hoc
Hi Pierre,
> Instead, I'd like the following behaviour:
> […]
hm, I feel that’s unnecessarly complex with lots of if’s and else’s. If
I could design the frontend from scratch, I’d have one command that does
profile/environment manipulation (because they’re essentially the same)
and one that can set them up for use. In that world you’d do
$ guix environment $(guix profile -m manifest.scm -i additional-package)
to get a temporary profile/environment and
# add a few packages
$ guix profile -p /path -m manifest.scm -i additional-package
/path
# remove one
$ guix profile -p /path -r additional-package
/path
# enter env
$ guix environment /path
$ ^D
# enter using container
$ guix environment -C --user=joeuser /path
$ ^D
to get a persistent environment and entering it without fiddling with -r
vs. -p vs. package options. In this case `guix environment` does not
know anything about packages and can only operate on existing profiles.
Maybe it’s possible to do profile inheritance like this too:
$ guix profile -p /new/profile -i /old/profile additional-package
/new/profile
which would be the union of /old/profile and {additional-package}.
Cheers,
Lars
Re: guix environment --profile with --ad-hoc
Hi Pierre, > My only complaint is that it's still a bit slow: > Is there anything we can do to speed this up? yeah, true. I’m think it’s still computing and building derivations, for example using manifest->derivation for `prof-drv` and then again built-derivations for `prof-drv`. Maybe if we make these conditional it’ll be faster…? Lars signature.asc Description: PGP signature
Re: guix environment --profile with --ad-hoc
Hi Pierre, > Do you have a link? sorry, I meant, I wrote the patch that added the --profile switch, see https://issues.guix.gnu.org/46291 > I'd love to see this merged! :) The patch above is already merged. Cheers, Lars signature.asc Description: PGP signature
Joining the Guix family
Hi everyone, today I’m joining the Guix family as a new committer. Some of you might know me already from guix-patc...@gnu.org or the last Guix Days in Brussels, which I also attended. I’m mainly working on Python and R packaging as part of my job at leibniz-psychology.org. Apart from that I’ll be looking into improving package quality, for example through my changes to python-build-system. On the technical side I’m using two PGP keys, the one this email is signed with 017D 74E2 7F58 5696 3801 781D F663 943E 08D8 092A and CA4F 8CF4 37D7 478F DA05 5FD4 4213 7701 1A37 8446 at work. Cheers, Lars signature.asc Description: PGP signature
Re: guix environment --profile with --ad-hoc
Hi, > guix environment doesn't allow --ad-hoc to be used when --profile is > set. I propose to have --ad-hoc add more packages to the container when > using it with --profile. I wrote the patch that adds this feature to `guix environment`. The idea was to have a way to turn an existing profile into an environment, specifically a container (-C), which is not possible with `source etc/profile`. It’s also very quick, because it does not compute anything to do that. (I think it still does, which is my bad.) Can I ask: What is your use-case? Why not extend the existing profile using `guix package -p /path -i foobar`? Cheers, Lars
Re: Mitigating "dependency confusion" attacks on Guix users
Hi, very interesting read. > However, I'm still thinking about how to attack Guix users. Somebody who > adds an internal channel for their own packages could still be > vulnerable to a dependency confusion attack via a compromised or > manipulated Guix maintainer. The target of the attack could install > packages they believed would be provided by their internal channel but > actually get another package provided upstream. Usually you’d use module imports and variable names inside your channel’s packages. Wouldn’t that defeat this attack? (Depending on Guix’/Guile’s module loading order of course.) What about substitute servers? As far as I understand as soon as they’re authorized they can deliver substitutes for *any* package. Lars
Re: Questions regarding Python packaging
Hi Ryan, > I think if we do that then Python will need a treatment similar to GCC, > where we don't expose the package and instead offer a compound package > (could be called "python-toolchain" or just "python") which includes pip > and setuptools. The last decades of python packaging have trained people > that when you install python you get pip and setuptools as well, I think > we should not try to be too clever and violate that assumption. this change is actually more about moving forward and adding PEP 517 support to python-build-system. Being able to demote setuptools and pip to ordinary packages is merely a side-effect, because they’re not essential any more. Your mail seems to be incomplete, it stopped after: > Also, for what it's worth, we already have python-minimal which doesn't > have pip, and it's only Cheers, Lars
Re: Questions regarding Python packaging
Hi Tanguy,
> Done! :-)
> I've eventually succeeded in ("properly") packaging a software managed
> with Poetry. And I've learned quite a lot on the way!
oh, I see. I’ve actually been trying to replace python-build-system with
a python-build based build. Attached is my current work in progress. I
cannot quite build python-build, because I’m lacking support for
python-flit, but I think the general idea is clear: Remove pip and
setuptools from python (saves almost 20MiB from the closure and avoids
weird conflicts between python’s setuptools and python-setuptools) and
turn them into (almost) ordinary packages. Then use setuptools to
bootstrap itself, bootstrap python-build with setuptools and use
python-build to build evrey other packages using python-build-system.
Cheers,
Lars
diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index d08e23936c..a9843bd9cb 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -1123,14 +1123,14 @@ other machines, such as over the network.")
(define-public python-setuptools
(package
(name "python-setuptools")
-(version "41.0.1")
+(version "51.3.1")
(source
(origin
(method url-fetch)
- (uri (pypi-uri "setuptools" version ".zip"))
+ (uri (pypi-uri "setuptools" version))
(sha256
(base32
-"04sns22y2hhsrwfy1mha2lgslvpjsjsz8xws7h2rh5a7ylkd28m2"))
+"07qk6ykhq1q4k33x1m96r39f9qpfzp2xfl5ly9zww0180gssrvqc"))
(modules '((guix build utils)))
(snippet
'(begin
@@ -1143,7 +1143,26 @@ other machines, such as over the network.")
;; FIXME: Tests require pytest, which itself relies on setuptools.
;; One could bootstrap with an internal untested setuptools.
(arguments
- `(#:tests? #f))
+ `(#:tests? #f
+ #:need-setuptools #f
+ #:python-build-variant #f
+ #:phases (modify-phases %standard-phases
+ (replace 'build
+ (lambda _
+ (invoke "python" "setup.py" "build")))
+ (replace 'install
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let ((out (assoc-ref outputs "out")))
+ (invoke "python" "setup.py" "install" "--prefix"
out "--single-version-externally-managed" "--root=/"
+ ;; Use this setuptools’ sources to bootstrap themselves.
+ (add-before 'build 'set-PYTHONPATH
+(lambda _
+ (format #t "current working dir ~s~%" (getcwd))
+ (setenv "PYTHONPATH"
+ (string-append ".:" (getenv "PYTHONPATH")))
+ #t)
+;; Not required when not building a wheel
+;(propagated-inputs `(("python-wheel" ,python-wheel)))
(home-page "https://pypi.org/project/setuptools/;)
(synopsis
"Library designed to facilitate packaging Python projects")
@@ -1822,7 +1841,7 @@ Python file, so it can be easily copied into your
project.")
(package
(inherit python-six)
(name "python-six-bootstrap")
-(native-inputs `())
+(native-inputs '())
(arguments `(#:tests? #f
(define-public python2-six-bootstrap
@@ -5675,7 +5694,8 @@ by pycodestyle.")
;; NOTE: Any value works, the variable just has to be present.
(setenv "SKIP_ONLINE" "1")
#t)
-(native-inputs `(("unzip" ,unzip)))
+(native-inputs
+ `(("unzip" ,unzip)))
(home-page "https://bitbucket.org/pypa/distlib;)
(synopsis "Distribution utilities")
(description "Distlib is a library which implements low-level functions
that
@@ -15272,7 +15292,7 @@ protocols.")
(package
(inherit python-attrs)
(name "python-attrs-bootstrap")
-(native-inputs `())
+(native-inputs '())
(arguments `(#:tests? #f
(define-public python2-attrs-bootstrap
@@ -23451,3 +23471,292 @@ Qt applications.")
"Pivy provides python bindings for Coin, a 3D graphics library with an
Application Programming Interface based on the Open Inventor 2.1 API.")
(license license:isc)))
+
+(define-public python-build
+ (package
+(name "python-build")
+(version "0.1.0")
+(source
+ (origin
+(method url-fetch)
+(uri (pypi-uri "build" version))
+(sha256
+ (base32
+"1d6m21lijwm04g50nwgsgj7x3vhblzw7jv05ah8psqgzk20bbch8"
+(build-system python-build-system)
+(arguments `(#:python-build-variant ,python-build-bootstrap))
+(propagated-inputs
+ `(("python-packaging" ,python-packaging)
+("python-pep517" ,python-pep517)
+("python-toml" ,python-toml)
+;; For fallback option.
+("python-setuptools" ,python-setuptools)))
+(native-inputs
+ `(("python-filelock" ,python-filelock)
+("python-pytest" ,python-pytest)
+("python-pytest-cov" ,python-pytest-cov)
+
Re: [RFC] Improve Python package quality
Hi, for reference, the patchset is now tracked by issue 45712 (http://issues.guix.gnu.org/45712) Cheers, Lars
Re: [RFC] Improve Python package quality
Hi Vincent, > I like the idea of better testing for our python packages, but would it be > possible to avoid embedding the python code as scheme strings ? > Like in a separate pure-python file. I don’t know how unfortunately. Any ideas? I moved it into a separate top-level variable now and turned it into a single multi-line Scheme string. That makes it easier to read. Cheers, Lars
Re: Questions regarding Python packaging
Hi Tanguy, > So, I've tried packaging `python-keyring` with those two… > > `pep517` keeps on trying to download dependencies, which won't work. > > `build` crashes with "ZIP does not support timestamps before 1980", > which, I guess is related to the fact that everything in the store is > timestamped to January 1st 1970. have you been looking into a python-build-system using `build`[1]? I’ve had the same issue with egg versions set to 0.0.0 and think in the long run moving to a PEP 517-style build is the way forward. Cheers, Lars [1] https://github.com/pypa/build
Re: [RFC] Improve Python package quality
Hi Hartmut, > this is a good idea. (Since you where mentioning setuptools, I first was > afraid your solution would be tightened to setuptools, but it is not. > Well done!) afaik pkg_resources is technically a part of setuptools, although it is distributed with Python. > This comment should go behind the line of code, as it only related to > that single line. > […] > I suggest putting the comments into the python source. This would allow > to indent them according the the python code, which would make it easier > to understand. This would also allow to use a single multi-line > guile-string, which allows to easiyl copy the script out and in from the > guile-source for testing it. > […] > Please follow PEP8 (no space before opening parentheses) - also at other > places. > […] > Add `end=""`, thus the "result" can be printed on the same line. > Print result terse, on same line, without repeating the name: You’re right, all fixed. I’ll send a non-hacky patch (with test-cases!) to guix-patches@ for review once we’ve figured out a path to merge it. I guess it would be best to fix packages directly on master and merge this new phase to core-updates? Shall I apply for commit access or can you (or Tobias?) review and “proxy” required changes? Right now I have fixes for about 10 packges, but there will be more. > Would is be better to use mkdtemp here to ge a fresh, empty directory? I tried that, but mkdtemp! is not available and I’m not confident enough to add that module to the closure. Any ideas? Cheers, Lars
[RFC] Improve Python package quality
Hi,
I’d like to propose adding an additional phase to python-build-system to
improve Guix’ Python package quality.
Python is an interpreted language without any compile-time checks. Any
errors are only visible at run-time, including missing or wrong imports.
Additionally most Python packages use an additional packaging layer
through setuptools, which also declares mandatory and optional
dependencies.
This can result in buildable, but broken packages. An example of a
botched upgrade is fixed in commit
22e06297b1982f75aaadddba616b1052e506e4a0. The package python-httpx was
upgraded, but that new version declared a dependency to a newer
python-httpcore via setuptools, which wasn’t available and thus packages
depending on python-httpx were broken.
My proposal adds some build-time checks to guarentee three properties:
1) Depending on the package (distribution in Python language) via
setuptools works. This ensures dependencies are complete and have the
correct version as determined by setup.{py,cfg}.
2) Console entry points can be loaded. This ensures scripts installed to
bin/ actually work.
3) Top-level modules can be imported. This ensures the package is
actually usable and has no undeclared dependencies.
The attached patch implements all three. I’ve been rebuilding a few
Python packages using the patch and uncovered some issues already. I’m
sure it’s not perfect yet and thus I’m open to improvements. If this
idea is well received I’m willing to rebuild and fix *all* Python
packages broken by this new phase. I’m also aware this change needs to
go to core-updates.
Cheers,
Lars
>From 919fd78b1aff6c277baca396ef962a5dcd4e23ae Mon Sep 17 00:00:00 2001
From: Lars-Dominik Braun
Date: Sun, 3 Jan 2021 10:30:29 +0100
Subject: [PATCH] [WIP] build-system/python: Validate installed package
* guix/build/python-build-system.scm (validate-loadable): New phase.
(%standard-phases): Use it.
---
guix/build/python-build-system.scm | 51 ++
1 file changed, 51 insertions(+)
diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index 09bd8465c8..edb772a7a4 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -148,6 +148,56 @@
(format #t "test suite not run~%"))
#t)
+(define* (validate-loadable #:key tests? inputs outputs #:allow-other-keys)
+ "Ensure packages depending on this package via setuptools work properly,
+their advertised endpoints work and their top level modules are importable
+without errors."
+ (let ((script (string-join
+'(
+;; Python 2 support.
+"from __future__ import print_function"
+"import pkg_resources, sys, importlib"
+;; Only check site-packages installed by this package, but not dependencies
+;; (which pkg_resources.working_set would include). Path supplied via argv.
+"ws = pkg_resources.find_distributions (sys.argv[1])"
+"for dist in ws:"
+"print ('validating', repr (dist.project_name), dist.location)"
+"try:"
+"req = str (dist.as_requirement ())"
+;; dist.activate() is not enough to actually check requirements, we have to
+;; .require() it.
+"pkg_resources.require (req)"
+"except Exception as e:"
+"print (req, e)"
+"sys.exit (1)"
+;; Try to load entry points of console scripts too, making sure they work. They
+;; should be removed if they don’t. Other groups may not be safe, as they can
+;; depend on optional packages.
+"for group, v in dist.get_entry_map ().items ():"
+" if group not in {'console_scripts', }:"
+" continue"
+" for name, ep in v.items ():"
+" print ('...trying to load endpoint', group, name)"
+" ep.load ()"
+;; And finally try to load top level modules. This should not have any
+;; side-effects.
+"for name in dist.get_metadata_lines ('top_level.txt'):"
+"print ('...trying to load module', name)"
+"try:"
+"importlib.import_module (name)"
+;; Ignore non-existent modules, we only want to know if the existing ones work.
+"except ModuleNotFoundError:"
+"print ('..WARNING: module', name, 'not found, continuing')"
+"continue")
+"\n")))
+(add-installed-pythonpath inputs outputs)
+;; Make sure the working directory is empty (i.e. no Python modules in it)
+(with-directory-excursion "/tmp"
+;; XXX: Cloak command run. Long and unreadable if it fails, provide an
+;; explanation instead.
+ (invoke "python" "-c" script (site-packages inputs outputs
+ #t)
+
(define (python-version python)
(let* ((version (last (string-split python #\-)))
(components (string-split version #\.))
@@ -2
Re: Guix Front End (GUI) and making it more mainstream, popular in scientific community.
Hi, > […] from my point of view, the good direction would be a “web-app frontend”, > similarly to git-annex-assistant [1]. This design is more flexible because > it could be used locally *and* could also be the front-end of some servers > (e.g., build farms). I have something like this on my todo list as well. It would probably be more of a manifest file editor though, since we use manifest files for reproducibility. Cheers, Lars signature.asc Description: PGP signature
Re: [Python] pypy3 integration
Hi, > > Also, what about .pyc files? Does pypy create compatible .pyc files? > Well, the .pyc generated by CPython should be compatible with the ones > generated by Pypy, both VM targeting say Python 3.6. > But there is no necessary compatibility between .pyc of Python 3.6 and > Python 3.8, at least for CPython as Lars wrote. both C extensions and .pyc files are guarded by encoding implementation and Python version in the file name. If we look at python-numpy for example, we can see files like decorators.cpython-38.pyc and common.cpython-38-x86_64-linux-gnu.so. If I build numpy with pypy the identifier (cpython-38) changes to pypy36 and pypy36-pp73 respectively. Thus pypy won’t load CPython’s files. > > Perhaps we could have a package transformation option to turn a > > ‘python-build-system’ package into a pypy package? > ...Yes it could be nice to be able to change the "package builder" of > the build-system. (Obviously, without any guarantee that the build > would be correct for all combinatorial :-)) > The issue is the same for emacs vs emacs-next, GCC versions (without > saying gcc vs clang ;-)), OCaml 4.07 vs OCaml 4.09 etc.. > We already discussed this kind of issue when discussing "package parameters". I think that would be really helpful. What’s the status of this work? Is there a PoC I can extend to cover CPython vs. PyPy? Cheers, Lars
Re: [Python] pypy3 integration
Hi, > pypy3 works somewhat well for me already in this regard: indeed, you’re right. This will probably break for some packages, because python provides Python 3.8 whereas pypy3 provides Python 3.6. (They’ve always lagged behind and given that we’re going towards 3.10, well…) One example are packages depending on importlib.resources, which only became available with Python 3.7. Unfortunately this includes the widely-used pytest (or rather: its dependency python-pluggy). Also Python’s C ABI is not stable[1] and thus extensions compiled for 3.8 can fail in unpredictable ways with 3.6. And looking at python-numpy, it seems they won’t even load. So, does this justify creating pypy3-* packages? Cheers, Lars [1] https://docs.python.org/3/c-api/stable.html
