Re: cython 3.x (for Python 3.12)
On Sat, Nov 25, 2023 at 04:23:46PM +, Stefano Rivera wrote: > As part of preparing for Python 3.12 in Debian, I've uploaded cython 3 > to experimental. > [...] > > So, that's 71 regressions with cython3. dd-list below. Please help us > port to cython 3. If this isn't possible, Graham is preparing a > cython-legacy package, to help the stragglers. But we're expecting that > this won't have great Python 3.12 support... > https://ftp-master.debian.org/new/cython-legacy_0.29.36-1~exp1.html Here's an update on cython-legacy. I just tried running autopkgtest on cython-legacy with Python 3.12, and unfortunately it fails. I can't quite work out how many tests fail (I don't understand the format of the test output), but there are 25 lines containing the string "FAIL" in the output of autopkgtest with the latest version of cython-legacy (0.29.36-2, uploaded today). We could just skip these tests - some of them are probably harmless - but they probably indicate incompatibilty (unsurprisingly) with Python 3.12, though some of these incompatibilties may be restricted to the tests themselves rather than the underlying Cython 0.x code. I'd be happy to try uploading a version with these tests skipped - I just wanted to check here first what the thoughts on this are. (Note that it is much better to modify other packages to work with Cython 3.x!) Best wishes, Julian
Re: cython 3.x (for Python 3.12)
Le jeudi 30 novembre 2023 à 08:25 +0100, julien.pu...@gmail.com a écrit : > Hi, > > Le samedi 25 novembre 2023 à 16:23 +, Stefano Rivera a écrit : > > > > Julien Puydt > > fpylll (U) > > pplpy (U) > > > For pplpy, I found the package was pointing to an archived repo, but > the author had a newer one with a version which is supposed to be > okay > (to be confirmed). I found the time to check: the last uploaded pplpy is good. > For fpylll, there is a newer version compatible with recent cython > but not with our current one. I'm discussing with upstream on how to > have a version compatible with both, but of course if cython 3 gets > updated the issue becomes moot. The next fpylll will be ok ; and it's packaged in the git repo already - it's just a 'dch -r', 'git tag -s' and 'dput' away. Cheers, J.Puydt
Re: cython 3.x (for Python 3.12)
On Mon, Dec 11, 2023 at 08:06:41PM +0100, Matthias Klose wrote: > > [...] > > Excellent - I didn't know about that. Are you OK for me to upload > > versions of cython and cython-legacy which depend on this to fix the > > Python 3.12 breakage? > > not for cython, which won't need that soonish for 3.0.x. and if you have to > update the b-d to cython3-legacy, why not add the zombie-imp dependency as > well manually for the few packages that need it? The problem is not with other packages importing imp (which need to be fixed in such a case), rather that pyximport (in cython/cython-legacy) imports imp. So it's cython that needs to be patched. I'm wondering what the provisional timescale for cython 3.0.x is? Should I just leave my package with an autopkgtest failure until cython 3.0.x is in unstable or ideally testing? That's why I was thinking of also patching cython in the short term until we are ready for cython 3.0.x to enter unstable. Best wishes, Julian
Re: cython 3.x (for Python 3.12)
On 11.12.23 19:55, Julian Gilbey wrote: On Mon, Dec 11, 2023 at 04:34:17PM +0100, Matthias Klose wrote: On 11.12.23 16:19, Julian Gilbey wrote: On Mon, Dec 11, 2023 at 08:09:31AM +0100, Matthias Klose wrote: [...] You could package a non-conflicting cython-legacy, however that would require more changes, also how to build it. Hi Matthias, Unfortunately, at least some of cython3-legacy doesn't currently work with Python 3.12, and is the primary cause of (at least) #1056531. cython3 provides the pyximport module, and that uses the imp module which has been removed from Python 3.12. Two possible ways forward on this particular bug: - Disable all of the cython tests for this package for the time being, until cython 3.x migrates to testing - this is simple and effective. - Patch cython3-legacy to use importlib rather than imp. This is probably a good thing to do anyway. (It may also be good to do this with cython3 version 0.x currently in testing/unstable until cython 3.x is able to be uploaded to unstable.) Then have my package's autopkgtest depend on cython3-legacy (unless cython3 0.x is also patched). I won't working on this. Have you tried to depend on the python3-zombie-imp instead? Excellent - I didn't know about that. Are you OK for me to upload versions of cython and cython-legacy which depend on this to fix the Python 3.12 breakage? not for cython, which won't need that soonish for 3.0.x. and if you have to update the b-d to cython3-legacy, why not add the zombie-imp dependency as well manually for the few packages that need it?
Re: cython 3.x (for Python 3.12)
On Mon, Dec 11, 2023 at 04:34:17PM +0100, Matthias Klose wrote: > On 11.12.23 16:19, Julian Gilbey wrote: > > On Mon, Dec 11, 2023 at 08:09:31AM +0100, Matthias Klose wrote: > > > [...] > > > You could package a non-conflicting cython-legacy, however that would > > > require more changes, also how to build it. > > > > Hi Matthias, > > > > Unfortunately, at least some of cython3-legacy doesn't currently work > > with Python 3.12, and is the primary cause of (at least) #1056531. > > cython3 provides the pyximport module, and that uses the imp module > > which has been removed from Python 3.12. > > > > Two possible ways forward on this particular bug: > > > > - Disable all of the cython tests for this package for the time being, > >until cython 3.x migrates to testing - this is simple and effective. > > > > - Patch cython3-legacy to use importlib rather than imp. This is > >probably a good thing to do anyway. (It may also be good to do this > >with cython3 version 0.x currently in testing/unstable until cython > >3.x is able to be uploaded to unstable.) Then have my package's > >autopkgtest depend on cython3-legacy (unless cython3 0.x is also > >patched). > > I won't working on this. Have you tried to depend on the python3-zombie-imp > instead? Excellent - I didn't know about that. Are you OK for me to upload versions of cython and cython-legacy which depend on this to fix the Python 3.12 breakage? Best wishes, Julian
Re: cython 3.x (for Python 3.12)
On 11.12.23 16:19, Julian Gilbey wrote: On Mon, Dec 11, 2023 at 08:09:31AM +0100, Matthias Klose wrote: [...] You could package a non-conflicting cython-legacy, however that would require more changes, also how to build it. Hi Matthias, Unfortunately, at least some of cython3-legacy doesn't currently work with Python 3.12, and is the primary cause of (at least) #1056531. cython3 provides the pyximport module, and that uses the imp module which has been removed from Python 3.12. Two possible ways forward on this particular bug: - Disable all of the cython tests for this package for the time being, until cython 3.x migrates to testing - this is simple and effective. - Patch cython3-legacy to use importlib rather than imp. This is probably a good thing to do anyway. (It may also be good to do this with cython3 version 0.x currently in testing/unstable until cython 3.x is able to be uploaded to unstable.) Then have my package's autopkgtest depend on cython3-legacy (unless cython3 0.x is also patched). I won't working on this. Have you tried to depend on the python3-zombie-imp instead?
Re: cython 3.x (for Python 3.12)
On Mon, Dec 11, 2023 at 08:09:31AM +0100, Matthias Klose wrote: > [...] > You could package a non-conflicting cython-legacy, however that would > require more changes, also how to build it. Hi Matthias, Unfortunately, at least some of cython3-legacy doesn't currently work with Python 3.12, and is the primary cause of (at least) #1056531. cython3 provides the pyximport module, and that uses the imp module which has been removed from Python 3.12. Two possible ways forward on this particular bug: - Disable all of the cython tests for this package for the time being, until cython 3.x migrates to testing - this is simple and effective. - Patch cython3-legacy to use importlib rather than imp. This is probably a good thing to do anyway. (It may also be good to do this with cython3 version 0.x currently in testing/unstable until cython 3.x is able to be uploaded to unstable.) Then have my package's autopkgtest depend on cython3-legacy (unless cython3 0.x is also patched). Thoughts? Julian
Re: cython 3.x (for Python 3.12)
On Mon, Dec 11, 2023 at 08:09:31AM +0100, Matthias Klose wrote: > [...] > > > > I find that there's also a significant issue with relying on > > > > cython3-legacy: it conflicts with cython3, meaning that it will be > > > > impossible to simultaneously install packages depending on cython3 and > > > > cython3-legacy. Once cython 3.x moves from experimental to unstable > > > > to testing and packages start depending on it, this will become a > > > > significant issue. I assume that the aim will be for everything to be > > > > ported to cython 3.x and for cython3-legacy to be dropped from testing > > > > before the trixie freeze? > [...] > You could package a non-conflicting cython-legacy, however that would > require more changes, also how to build it. I had a very quick look at the packaged cython3-legacy. One could change the name of the module to Cython_legacy or probably better, Cython0, and cython.py to cython0.py, pyximport to pyximport0. Just a handful of changes would be needed, I'm guessing. Then it would be co-installable with Cython 3.x, and any packages that actually need cython3-legacy rather than Cython 3.x would have to be patched to call the legacy version. It's probably easier to just patch them for Cython 3.x ;-) Best wishes, Julian
Re: cython 3.x (for Python 3.12)
On 11.12.23 08:09, Matthias Klose wrote: On 10.12.23 21:32, Andrey Rakhmatullin wrote: On Sun, Dec 10, 2023 at 09:30:03PM +0100, Andrey Rakhmatullin wrote: I find that there's also a significant issue with relying on cython3-legacy: it conflicts with cython3, meaning that it will be impossible to simultaneously install packages depending on cython3 and cython3-legacy. Once cython 3.x moves from experimental to unstable to testing and packages start depending on it, this will become a significant issue. I assume that the aim will be for everything to be ported to cython 3.x and for cython3-legacy to be dropped from testing before the trixie freeze? I wonder how many packages actually need a runtime dep on cython. I quickly checked three packages from `reverse-depends cython3` and while python3-pysph probably uses cython to generate code at the run time (not sure), python3-pyzoltan seems to only use it at the build time and python3-epimodels doesn't seem to use it at all. (I may be terribly wrong, I just thought cython is almost always a build time only dep and there are around 30 packages depending on it which was surprising) yes, many setup.py's list that under install_requires. I hope we now identified most of them, and fixed these. At least we were now able to do all the binNMUs. You could package a non-conflicting cython-legacy, however that would require more changes, also how to build it. Graham created a tracker for these, and I filed bug reports: https://release.debian.org/transitions/html/cython-runtime.html https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=cython-rt-dep&users=debian-python@lists.debian.org
Re: cython 3.x (for Python 3.12)
On 10.12.23 21:32, Andrey Rakhmatullin wrote: On Sun, Dec 10, 2023 at 09:30:03PM +0100, Andrey Rakhmatullin wrote: I find that there's also a significant issue with relying on cython3-legacy: it conflicts with cython3, meaning that it will be impossible to simultaneously install packages depending on cython3 and cython3-legacy. Once cython 3.x moves from experimental to unstable to testing and packages start depending on it, this will become a significant issue. I assume that the aim will be for everything to be ported to cython 3.x and for cython3-legacy to be dropped from testing before the trixie freeze? I wonder how many packages actually need a runtime dep on cython. I quickly checked three packages from `reverse-depends cython3` and while python3-pysph probably uses cython to generate code at the run time (not sure), python3-pyzoltan seems to only use it at the build time and python3-epimodels doesn't seem to use it at all. (I may be terribly wrong, I just thought cython is almost always a build time only dep and there are around 30 packages depending on it which was surprising) yes, many setup.py's list that under install_requires. I hope we now identified most of them, and fixed these. At least we were now able to do all the binNMUs. You could package a non-conflicting cython-legacy, however that would require more changes, also how to build it. Matthias
Re: cython 3.x (for Python 3.12)
On Sun, Dec 10, 2023 at 09:30:03PM +0100, Andrey Rakhmatullin wrote: > > I find that there's also a significant issue with relying on > > cython3-legacy: it conflicts with cython3, meaning that it will be > > impossible to simultaneously install packages depending on cython3 and > > cython3-legacy. Once cython 3.x moves from experimental to unstable > > to testing and packages start depending on it, this will become a > > significant issue. I assume that the aim will be for everything to be > > ported to cython 3.x and for cython3-legacy to be dropped from testing > > before the trixie freeze? > I wonder how many packages actually need a runtime dep on cython. I > quickly checked three packages from `reverse-depends cython3` and while > python3-pysph probably uses cython to generate code at the run time (not > sure), python3-pyzoltan seems to only use it at the build time and > python3-epimodels doesn't seem to use it at all. (I may be terribly wrong, I just thought cython is almost always a build time only dep and there are around 30 packages depending on it which was surprising)
Re: cython 3.x (for Python 3.12)
On Sun, Dec 10, 2023 at 08:12:40PM +, Julian Gilbey wrote: > On Sat, Nov 25, 2023 at 04:23:46PM +, Stefano Rivera wrote: > > As part of preparing for Python 3.12 in Debian, I've uploaded cython 3 > > to experimental. > > [...] > > > > So, that's 71 regressions with cython3. dd-list below. Please help us > > port to cython 3. If this isn't possible, Graham is preparing a > > cython-legacy package, to help the stragglers. But we're expecting that > > this won't have great Python 3.12 support... > > https://ftp-master.debian.org/new/cython-legacy_0.29.36-1~exp1.html > > I find that there's also a significant issue with relying on > cython3-legacy: it conflicts with cython3, meaning that it will be > impossible to simultaneously install packages depending on cython3 and > cython3-legacy. Once cython 3.x moves from experimental to unstable > to testing and packages start depending on it, this will become a > significant issue. I assume that the aim will be for everything to be > ported to cython 3.x and for cython3-legacy to be dropped from testing > before the trixie freeze? I wonder how many packages actually need a runtime dep on cython. I quickly checked three packages from `reverse-depends cython3` and while python3-pysph probably uses cython to generate code at the run time (not sure), python3-pyzoltan seems to only use it at the build time and python3-epimodels doesn't seem to use it at all.
Re: cython 3.x (for Python 3.12)
On Sat, Nov 25, 2023 at 04:23:46PM +, Stefano Rivera wrote: > As part of preparing for Python 3.12 in Debian, I've uploaded cython 3 > to experimental. > [...] > > So, that's 71 regressions with cython3. dd-list below. Please help us > port to cython 3. If this isn't possible, Graham is preparing a > cython-legacy package, to help the stragglers. But we're expecting that > this won't have great Python 3.12 support... > https://ftp-master.debian.org/new/cython-legacy_0.29.36-1~exp1.html I find that there's also a significant issue with relying on cython3-legacy: it conflicts with cython3, meaning that it will be impossible to simultaneously install packages depending on cython3 and cython3-legacy. Once cython 3.x moves from experimental to unstable to testing and packages start depending on it, this will become a significant issue. I assume that the aim will be for everything to be ported to cython 3.x and for cython3-legacy to be dropped from testing before the trixie freeze? Best wishes, Julian
Re: cython 3.x (for Python 3.12)
On Sat, Nov 25, 2023 at 04:23:46PM +, Stefano Rivera wrote: > [...] > So, that's 71 regressions with cython3. dd-list below. Please help us > port to cython 3. If this isn't possible, Graham is preparing a > cython-legacy package, to help the stragglers. But we're expecting that > this won't have great Python 3.12 support... > https://ftp-master.debian.org/new/cython-legacy_0.29.36-1~exp1.html Indeed; there's already one Python 3.12 related bug in the legacy version: it tries to import imp in pyximport.py. Best wishes, Julian
Re: cython 3.x (for Python 3.12)
Hi, Le samedi 25 novembre 2023 à 16:23 +, Stefano Rivera a écrit : > > Julien Puydt > fpylll (U) > pplpy (U) For pplpy, I found the package was pointing to an archived repo, but the author had a newer one with a version which is supposed to be okay (to be confirmed). For fpylll, there is a newer version compatible with recent cython but not with our current one. I'm discussing with upstream on how to have a version compatible with both, but of course if cython 3 gets updated the issue becomes moot. Cheers, J.Puydt
Re: cython 3.x (for Python 3.12)
Hi, Le samedi 25 novembre 2023 à 16:23 +, Stefano Rivera a écrit : > > Julien Puydt > fpylll (U) > pplpy (U) > I had a look. For fpylll: it has a newer upstream which works with more recent cython versions, so I'm actually eager to see the move. For pplpy, I'm a bit annoyed as I have no recollection of that package... According to its homepage, the project has been archived, so it's probably dead. But the sagemath package still depends on it (or at least its d/control says it does) so we can't drop it yet. Cheers, JP
Re: cython 3.x (for Python 3.12)
Hi debian-python (2023.11.25_16:23:46_+) > As part of preparing for Python 3.12 in Debian, I've uploaded cython 3 > to experimental. > > I did some test-building of reverse dependencies, and quite a lot of > them fail. I should have said, all the build logs are here: https://people.debian.org/~stefanor/cython3/cython-3.0.5/ > Stefano Rivera >pystemmer (U) Fixed in 2.2.0.1-2 Stefano -- Stefano Rivera http://tumbleweed.org.za/ +1 415 683 3272
cython 3.x (for Python 3.12)
As part of preparing for Python 3.12 in Debian, I've uploaded cython 3 to experimental. I did some test-building of reverse dependencies, and quite a lot of them fail. Out of 291 packages that build-depend on cython3: 104 attempted <- FTBFS 6 given-back<- build-deps aren't installable 1 skipped <- not for arm64 180 successful https://people.debian.org/~stefanor/cython3/cython-3.0.5/summary.txt https://people.debian.org/~stefanor/cython3/cython-3.0.5/stats.txt I re-tried those "attempted" failures, with cython 0.29.36 to find the regressions and: 32 attempted 1 given-back 71 successful https://people.debian.org/~stefanor/cython3/cython-0.29.36/summary.txt https://people.debian.org/~stefanor/cython3/cython-0.29.36/stats.txt So, that's 71 regressions with cython3. dd-list below. Please help us port to cython 3. If this isn't possible, Graham is preparing a cython-legacy package, to help the stragglers. But we're expecting that this won't have great Python 3.12 support... https://ftp-master.debian.org/new/cython-legacy_0.29.36-1~exp1.html Stefano Regression dd-list: Adrian Vondendriesch peewee (U) Alastair McKinstry adios Andreas Tille atropos (U) macs (U) python-cykhash (U) python-cytoolz (U) python-skbio (U) python-srsly (U) python-thinc (U) tombo (U) Anton Gladky sfepy (U) Antonio Valentino cyarray (U) pysph (U) python-ltfatpy (U) pyzoltan (U) B. Clausius pybik (U) Bastian Venthur kivy (U) Benjamin Drung rdma-core Boyuan Yang py-libzfs (U) ChangZhuo Chen (陳昌倬) python-feather-format (U) Dean Serenevy kivy (U) Debian Astro Team healpy montage pyregion Debian Fonts Task Force compreffor pyclipper Debian Games Team pygame-sdl2 python-sfml Debian Med Packaging Team atropos cyvcf2 macs nipy python-cykhash python-skbio tombo Debian Multimedia Maintainers pyliblo Debian PaN Maintainers opentsne Debian Python Modules Team fpylll Debian Python Team azure-uamqp-python indexed-gzip kivy mayavi2 numpy (U) peewee py-libzfs pybik pymssql pyreadstat (U) pystemmer python-cassandra-driver python-cytoolz python-feather-format python-hidapi python-line-profiler python-llfuse (U) python-openstep-plist python-pcl python-pomegranate python-srsly python-stack-data python-thriftpy uvloop (U) Debian QA Group xmms2 Debian Science Maintainers cyarray gensim h5py libgpuarray mpi4py-fft opentsne (U) petsc4py pyfftw pysph python-ltfatpy python-thinc pyzoltan sfepy statsmodels Debian Science Team cypari2 pandas pplpy Debian ZFS on Linux maintainers py-libzfs (U) Debichem Team mdtraj Diane Trout statsmodels (U) Drew Parsons mdtraj (U) mpi4py-fft (U) petsc4py (U) Emmanuel Arias python-cassandra-driver (U) Free Ekanayaka pyliblo (U) Gard Spreemann gudhi python-pyspike Geoffrey Thomas pymssql (U) Ghislain Antony Vaillant h5py (U) libgpuarray (U) pyfftw (U) python-line-profiler (U) Gianfranco Costamagna python-esmre Gijs Molenaar montage (U) Gordon Ball python-stack-data (U) gtkpod Maintainers libimobiledevice Hilko Bengen python-acora James Cowgill python-sfml (U) Jaromír Mikeš pyliblo (U) Jeremy Bicha compreffor (U) pyclipper (U) Jerome Benoit fpylll (U) Jochen Sprickerhof python-pcl (U) Jonas Smedegaard python-av Joost van Baal-Ilić pyreadstat Josue Ortega python-rtmidi Julien Puydt fpylll (U) pplpy (U) Kevin Murray python-skbio (U) Kunal Mehta python-libzim Leo Singer healpy (U) Liubov Chuprikova cyvcf2 (U) Luca Boccassi azure-uamqp-python (U) Markus Koschany pygame-sdl2 (U) Matthias Klose pygccjit Michael Hanke pandas (U) statsmodels (U) Michael Hanke indexed-gzip (U) Michael R. Crusoe macs (U) python-pomegranate (U) Mo Zhou h5py (U) Nikolaus Rath python-llfuse Nilesh Patra python-cytoolz (U) Ole Streicher montage (U) Olivier Sallou python-thriftpy (U) Paul Wise gensim (U) Picca Frédéric-Emmanuel opentsne (U) Piotr Ożarowski uvloop Rebecca N. Palmer libgpuarray (U) pandas (U) statsmodels (U) Richard Ulrich python-hidapi (U) Roland Mas python-orderedset Sandro Tosi numpy Sebastien Delafond opentsne (U) Stefano Rivera pystemmer (U) Steffen Moeller cyvcf2 (U) python-pomegranate (U) Tobias Hansen cypari2 (U) pplpy (U) Varun Hiremath mayavi2 (U) Vincent Cheng kivy (U) Vincent Prat pyregion (U) Ximin Luo cypari2 (U) fpylll (U) Yao Wei (魏銘廷) python-openstep-plist (U) Yaroslav Halchenko indexed-gzip (U) pandas (U) statsmodels (U) Yves-Alexis Perez libimobiledevice (U) Étienne Moll
