Re: Python 3.11 for bookworm?
Am Tue, Jan 17, 2023 at 01:04:50AM +0100 schrieb Thomas Goirand: > > > #1023965 [src:pandas] pandas FTBFS with Python 3.11 as supported > > > version > > > #1024031 [src:numba] numba FTBFS with Python 3.11 as supported version > > I saw the above 2 were fixed. Fixed in the sense that there was an upload closing the bug. If you look at https://tracker.debian.org/pkg/pandas https://tracker.debian.org/pkg/numba you see multiple blockers for a testing migration. So the problem for the release persists. I have confirmations of upstream of several rdepends of these packages that they do not support 3.11 since these two packages do not officially support Python 3.11 yet (the Debian packages are coming with several patches - just one example of an answer from upstream for python-cogent[1]) > > I'd like to add > > > >#1027851 [src:pytorch] FTBFS with Python 3.11 as default version > > > > also with lots of rdepends. > > So we're back with one single bug. I remember seeing something similar in > another package ... (scratching my head...). The latest version of the > upstream code has some modifications to this broken Stream.cpp, have you > tried to apply them? No, I have not dealt with torch. I just know that this package is in the very competent hands of Mo Zhou who will know what to do. > Do you have more to share? It's harder to help if you don't ask for it. > IMO, feel free to give a full list of problematic packages in this list, so > others may help. As I said above the packages above are far from testing. If someone could lend helping hands to let these packages migrate this would be really helpful. > > I did not received any response to my "small" list. What does this > > mean for the transition to 3.11 process? > > As much as I know, we're moving toward having Python 3.11 only in Bookworm. > I'm not the person driving it though, and I am not in the best position to > make such choice, but I support it (as I would prefer having the nice > enhancements of Python 3.11 rather than lagging behind). Hopefully, I wont > regret it and wont find more failures in "my" packages. I understand the advantages of Python 3.11 and I'm not against releasing Bookworm with it. I'm against overlong freezing periods which I see as the consequence of pushing Python 3.11 while sticking to the current freeze shedule. If we would decide that Python 3.11 is really important I would consider shifting the testing period by one or two months. We have just seen that every full rebuild of the archive as Lucas recently did creates quite some new RC bugs that are related to the Python version bump and I expect more of these at the next rebuild. > > > You mean you are fixing Python 3.10 manually in some packages diverging > > > what will be default Python? > > > > Any answer to this question? > > All of my packages hopefully always test with all available versions, and > most run autopkgtest. So I was warned early of Py 3.11 failures. They are > all fixed, as much as I know (no opened RC bug remaining...). And no, > forcing Python 3.10 is *NOT* an option, one must fix failures in Python > 3.11. Since you said above that we want to release with 3.11 only this becomes clear. Luckily the teams where I'm working in have also quite good autopkgtest coverage so I'm positive about sensible warnings. > > I keep on thinking that the timing is unfortunate and that it > > will spoil the whole release process. > > I'm sad to read this. Hopefully, this is truth only for some of the packages > you care, and the vast majority of the packages are fine? I'm unfortunately > not in a good position to tell (I didn't run any survey of broken > packages...). We are just a couple of people who are fighting hard through scientific packages with a complex depency tree. Some of them (like numba) take ages to build even on amd64 (salsa CI is set to 6h here) and thus take quite some time to fix. As I said above I'm not against Python 3.11 per see. I'm simply afraid that this decision will delay the release process and IMHO it makes sense to shift the schedule if we decide that Python 3.11 is something we want. Kind regards Andreas. [1] https://github.com/cogent3/cogent3/issues/1263 -- http://fam-tille.de
Re: Python 3.11 for bookworm?
On 1/16/23 17:23, Andreas Tille wrote: Am Sat, Jan 07, 2023 at 09:05:21AM +0100 schrieb Andreas Tille: If I would create a list mine whould be way longer. Please share it in this list! #1023965 [src:pandas] pandas FTBFS with Python 3.11 as supported version #1024031 [src:numba] numba FTBFS with Python 3.11 as supported version I saw the above 2 were fixed. I'd like to add #1027851 [src:pytorch] FTBFS with Python 3.11 as default version also with lots of rdepends. So we're back with one single bug. I remember seeing something similar in another package ... (scratching my head...). The latest version of the upstream code has some modifications to this broken Stream.cpp, have you tried to apply them? Do you have more to share? It's harder to help if you don't ask for it. IMO, feel free to give a full list of problematic packages in this list, so others may help. I did not received any response to my "small" list. What does this mean for the transition to 3.11 process? As much as I know, we're moving toward having Python 3.11 only in Bookworm. I'm not the person driving it though, and I am not in the best position to make such choice, but I support it (as I would prefer having the nice enhancements of Python 3.11 rather than lagging behind). Hopefully, I wont regret it and wont find more failures in "my" packages. We are constantly beaten by removal from testing warnings even with Python3.11 as an option and sometimes we simply need to remove that option as a temporary means for bookworm. Same over here. It's finally looking good for me after 2 months of heavy efforts. You mean you are fixing Python 3.10 manually in some packages diverging what will be default Python? Any answer to this question? All of my packages hopefully always test with all available versions, and most run autopkgtest. So I was warned early of Py 3.11 failures. They are all fixed, as much as I know (no opened RC bug remaining...). And no, forcing Python 3.10 is *NOT* an option, one must fix failures in Python 3.11. Bug #1026825: python3.11 as default filed right before (21.12.) a series of holidays in the region of the world where lots of developers will typically reduce their activity which is closely followed by the first freeze step is IMHO something else. Since I realised that the transition was started[3] our discussion is a bit useless. I just want to explain the motivation why I sounded "astonished" since you said that you do not understood astonishment since we are following the suggested plan. I keep on thinking that the timing is unfortunate and that it will spoil the whole release process. I'm sad to read this. Hopefully, this is truth only for some of the packages you care, and the vast majority of the packages are fine? I'm unfortunately not in a good position to tell (I didn't run any survey of broken packages...). Cheers, Thomas Goirand (zigo)
Re: Python 3.11 for bookworm?
Am Sat, Jan 07, 2023 at 09:05:21AM +0100 schrieb Andreas Tille: > Hi Thomas, > > Am Thu, Jan 05, 2023 at 01:57:43PM +0100 schrieb Thomas Goirand: > > > > This has since already been discussed here: the final decision was to "at > > least try 3.11", which is exactly what we're doing. > > I admit I was not at site and may be I missunderstood what was finally > decided. From my perspective this "at least try" is that we are > actually trying by having 3.11 as additional supported version which has > triggered quite some work. We are approaching the Transition and > Toolchain Freeze in 5 days[1]. Given that switching default Python is a > transition I wonder how you can assume that this is the right time to > suggest this. I would not have been that astonished if you would have > done so say at first December last year. But now its absolutely clear > that a migration (with the "option" to revert which will cause extra > work) will pour sand into the gears of the release process. How will we decide whether the "at least try 3.11" is success or fail? Did we defined anything I might have missed in terms of number of packages that need to pass or number of packages we shoule not loose? > > FYI, I'm down with only 2 major bugs (I don't mind much if the other 3 RC > > bugs push the 3 affected packages away from the release, it's just a "would > > be nice" thing ATM): > > That's a nice situation for the field you are working in. > > > > If I would create a list mine whould be way longer. > > > > Please share it in this list! > >#1023965 [src:pandas] pandas FTBFS with Python 3.11 as supported version >#1024031 [src:numba] numba FTBFS with Python 3.11 as supported version I'd like to add #1027851 [src:pytorch] FTBFS with Python 3.11 as default version also with lots of rdepends. > These packages have a sufficient amount of rdepends and usually trigger > lots of other autopkgtest failures in other packages which will keep > them out of testing for quite some time. We could need all helping > hands to fix these ... all noise that will happen afterwards will keep > the relevant teams busy enough. I did not received any response to my "small" list. What does this mean for the transition to 3.11 process? > > > We are constantly beaten by removal from testing warnings > > > even with Python3.11 as an option and sometimes we simply need to remove > > > that option as a temporary means for bookworm. > > > > Same over here. It's finally looking good for me after 2 months of heavy > > efforts. > > You mean you are fixing Python 3.10 manually in some packages diverging > what will be default Python? Any answer to this question? > > > No better solution but better timing which means after bookworm release. > > > > I've read *many* people saying it would be disappointing for them to see > > their package removed because of Python 3.11. Well, please consider that it > > would also be very disappointing to *not* have Python 3.11 for those who > > managed constantly fix issues for it. > > I can understand that we can never satisfy the needs of everybody. My > main problem is the extremely unfortunate timing that is happening now. > > > The timing was exactly what was discussed during Debconf: it's very annoying > > that this year, upstream Python release was one month late... we're only > > trying to deal with it. > > I do not remember that the scchedule was discussed. > >* Add 3.11 as a supported Python3 version > > was done in python3-defaults (3.10.6-2) at Fri, 11 Nov 2022 11:10:31 > +0200. At 12. December Graham suggested on the behalf of the release > team[2] to switch to 3.11. If we would have acted upon this at that > very time I would have considered this quite dense but the last chance > to consider this in line with the "lets try" attempt discussed at > DebConf. > > Bug #1026825: python3.11 as default filed right before (21.12.) a series > of holidays in the region of the world where lots of developers will > typically reduce their activity which is closely followed by the first > freeze step is IMHO something else. Since I realised that the transition > was started[3] our discussion is a bit useless. I just want to explain > the motivation why I sounded "astonished" since you said that you do > not understood astonishment since we are following the suggested plan. I keep on thinking that the timing is unfortunate and that it will spoil the whole release process. Kind regards Andreas. > > [1] https://release.debian.org/testing/freeze_policy.html > [2] https://lists.debian.org/debian-python/2022/12/msg00074.html > [3] https://release.debian.org/transitions/html/python3.11-default.html > > -- > http://fam-tille.de > > -- http://fam-tille.de
Re: Python 3.11 for bookworm?
Hi, For information I tagged [#1027851] with help needed for the python3.11 port of pytorch, following a message from its main uploader. To people who might like to have a look at it: please be careful as pytorch may be quite the resource hog and time sink if one's time is better spent somewhere else. #1027851: pytorch FTBFS with Python 3.11 as default version [#1027851]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1027851 Have a nice day, :) -- Étienne Mollier Fingerprint: 8f91 b227 c7d6 f2b1 948c 8236 793c f67e 8f0d 11da Sent from /dev/pts/1, please excuse my verbosity. On air: A Liquid Landscape - Secret isle signature.asc Description: PGP signature
Re: Python 3.11 for bookworm?
Hi Simon,
On 2023-01-07 13:24, Simon McVittie wrote:
On Sat, 07 Jan 2023 at 10:23:19 +0200, Andrius Merkys wrote:
If I may, I would as well be grateful if someone could give a look at:
#1023972 [src:python-ase] FTBFS with Python 3.11 due to
pathlib.Path.__enter__() deprecation
I have no idea how to fix this and the upstream is silent.
My first thought on seeing this was: why were Path objects a context
manager in the first place? What would that mean?
Looking at the code in python3.10 and python3.11 pathlib, it seems the
reason this is deprecated is indeed that it didn't make sense:
def __enter__(self):
# In previous versions of pathlib, __exit__() marked this path as
# closed; subsequent attempts to perform I/O would raise an IOError.
# This functionality was never documented, and had the effect of
# making Path objects mutable, contrary to PEP 428.
# In Python 3.9 __exit__() was made a no-op.
# In Python 3.11 __enter__() began emitting DeprecationWarning.
# In Python 3.13 __enter__() and __exit__() should be removed.
warnings.warn("pathlib.Path.__enter__() is deprecated and scheduled "
"for removal in Python 3.13; Path objects as a context "
"manager is a no-op",
DeprecationWarning, stacklevel=2)
return self
def __exit__(self, t, v, tb):
pass
So the solution seems to be that if your package contains this:
with some_path_object:
do_stuff(some_path_object)
replace it with:
do_stuff(some_path_object)
and if it contains:
with Path(...) as my_path:
do_stuff(my_path)
replace with:
my_path = Path(...)
do_stuff(my_path)
I hope that should be a relatively straightforward change.
Thanks a lot for the hint, this indeed worked. Failure in __enter__()
threw me off tracks, but now I recall how it is related to 'with'
construction.
Best wishes,
Andrius
Re: Python 3.11 for bookworm?
On Sat, 07 Jan 2023 at 10:23:19 +0200, Andrius Merkys wrote:
> If I may, I would as well be grateful if someone could give a look at:
>
> #1023972 [src:python-ase] FTBFS with Python 3.11 due to
> pathlib.Path.__enter__() deprecation
>
> I have no idea how to fix this and the upstream is silent.
My first thought on seeing this was: why were Path objects a context
manager in the first place? What would that mean?
Looking at the code in python3.10 and python3.11 pathlib, it seems the
reason this is deprecated is indeed that it didn't make sense:
def __enter__(self):
# In previous versions of pathlib, __exit__() marked this path as
# closed; subsequent attempts to perform I/O would raise an IOError.
# This functionality was never documented, and had the effect of
# making Path objects mutable, contrary to PEP 428.
# In Python 3.9 __exit__() was made a no-op.
# In Python 3.11 __enter__() began emitting DeprecationWarning.
# In Python 3.13 __enter__() and __exit__() should be removed.
warnings.warn("pathlib.Path.__enter__() is deprecated and scheduled "
"for removal in Python 3.13; Path objects as a context "
"manager is a no-op",
DeprecationWarning, stacklevel=2)
return self
def __exit__(self, t, v, tb):
pass
So the solution seems to be that if your package contains this:
with some_path_object:
do_stuff(some_path_object)
replace it with:
do_stuff(some_path_object)
and if it contains:
with Path(...) as my_path:
do_stuff(my_path)
replace with:
my_path = Path(...)
do_stuff(my_path)
I hope that should be a relatively straightforward change.
smcv
Re: Python 3.11 for bookworm?
Hello, On 2023-01-07 10:05, Andreas Tille wrote: Am Thu, Jan 05, 2023 at 01:57:43PM +0100 schrieb Thomas Goirand: Please share it in this list! #1023965 [src:pandas] pandas FTBFS with Python 3.11 as supported version #1024031 [src:numba] numba FTBFS with Python 3.11 as supported version If I may, I would as well be grateful if someone could give a look at: #1023972 [src:python-ase] FTBFS with Python 3.11 due to pathlib.Path.__enter__() deprecation I have no idea how to fix this and the upstream is silent. Best, Andrius
Re: Python 3.11 for bookworm?
Hi Thomas, Am Thu, Jan 05, 2023 at 01:57:43PM +0100 schrieb Thomas Goirand: > > This has since already been discussed here: the final decision was to "at > least try 3.11", which is exactly what we're doing. I admit I was not at site and may be I missunderstood what was finally decided. From my perspective this "at least try" is that we are actually trying by having 3.11 as additional supported version which has triggered quite some work. We are approaching the Transition and Toolchain Freeze in 5 days[1]. Given that switching default Python is a transition I wonder how you can assume that this is the right time to suggest this. I would not have been that astonished if you would have done so say at first December last year. But now its absolutely clear that a migration (with the "option" to revert which will cause extra work) will pour sand into the gears of the release process. > FYI, I'm down with only 2 major bugs (I don't mind much if the other 3 RC > bugs push the 3 affected packages away from the release, it's just a "would > be nice" thing ATM): That's a nice situation for the field you are working in. > > If I would create a list mine whould be way longer. > > Please share it in this list! #1023965 [src:pandas] pandas FTBFS with Python 3.11 as supported version #1024031 [src:numba] numba FTBFS with Python 3.11 as supported version These packages have a sufficient amount of rdepends and usually trigger lots of other autopkgtest failures in other packages which will keep them out of testing for quite some time. We could need all helping hands to fix these ... all noise that will happen afterwards will keep the relevant teams busy enough. > > We are constantly beaten by removal from testing warnings > > even with Python3.11 as an option and sometimes we simply need to remove > > that option as a temporary means for bookworm. > > Same over here. It's finally looking good for me after 2 months of heavy > efforts. You mean you are fixing Python 3.10 manually in some packages diverging what will be default Python? > > No better solution but better timing which means after bookworm release. > > I've read *many* people saying it would be disappointing for them to see > their package removed because of Python 3.11. Well, please consider that it > would also be very disappointing to *not* have Python 3.11 for those who > managed constantly fix issues for it. I can understand that we can never satisfy the needs of everybody. My main problem is the extremely unfortunate timing that is happening now. > The timing was exactly what was discussed during Debconf: it's very annoying > that this year, upstream Python release was one month late... we're only > trying to deal with it. I do not remember that the scchedule was discussed. * Add 3.11 as a supported Python3 version was done in python3-defaults (3.10.6-2) at Fri, 11 Nov 2022 11:10:31 +0200. At 12. December Graham suggested on the behalf of the release team[2] to switch to 3.11. If we would have acted upon this at that very time I would have considered this quite dense but the last chance to consider this in line with the "lets try" attempt discussed at DebConf. Bug #1026825: python3.11 as default filed right before (21.12.) a series of holidays in the region of the world where lots of developers will typically reduce their activity which is closely followed by the first freeze step is IMHO something else. Since I realised that the transition was started[3] our discussion is a bit useless. I just want to explain the motivation why I sounded "astonished" since you said that you do not understood astonishment since we are following the suggested plan. Kind regards Andreas. [1] https://release.debian.org/testing/freeze_policy.html [2] https://lists.debian.org/debian-python/2022/12/msg00074.html [3] https://release.debian.org/transitions/html/python3.11-default.html -- http://fam-tille.de
Re: Python 3.11 for bookworm?
Hi Andreas, On 12/30/22 10:22, Andreas Tille wrote: True, I remember the DebConf Python BoF. My memory tells me that the plan was to keep 3.10 as default. Thus Python 3.11 would be really a surprise. From a maintainers team with lots of Python packages that will need heavy work I can't say I'd be happy about a "migrate and see what might happen afterwards" strategy as it was proposed here. We have lots to do even without such an experiment. This has since already been discussed here: the final decision was to "at least try 3.11", which is exactly what we're doing. Well, that's not the first time we are trying to push the latest interpreter close to a release. The fact that we did so before does not make the idea better. I also very much would appreciate help on these (which all look like probably related to py3.11): #1026524 ironic-inspector: FTBFS: AttributeError: '_thread.RLock' object has no attribute 'is_locked' #1026547 python-pypowervm: FTBFS: AssertionError: Expected 'warn' to be called once. Called 2 times. #1026549 python-pytest-xprocess: FTBFS: tests failed #1026591 cinder: FTBFS: make[1]: pyversions: No such file or directory #1026595 python-murano-pkg-check: FTBFS: TypeError: load_all() missing 1 required positional argument: 'Loader' #1026610 horizon: FTBFS: tests failed #1026691 bandit: FTBFS: TypeError: load() missing 1 required positional argument: 'Loader' #1026693 cloudkitty: FTBFS: touch: cannot touch '/<>/debian/cloudkitty-doc/usr/share/doc/cloudkitty-doc/html/_static/toggle.js': No such file or directory #1026702 python-cursive: FTBFS: AttributeError: '_RSAPrivateKey' object has no attribute 'signer' #1026705 python-pecan: FTBFS: E AttributeError: 'code' object has no attribute 'co_endlinetable' #1026707 jenkins-job-builder: FTBFS: E: pybuild pybuild:386: test: plugin custom failed with: exit code=1: PYTHON=python3.10 stestr run FYI, I'm down with only 2 major bugs (I don't mind much if the other 3 RC bugs push the 3 affected packages away from the release, it's just a "would be nice" thing ATM): #1026524 ironic-inspector: FTBFS: AttributeError: '_thread.RLock' object has no attribute 'is_locked' #1026591 cinder: FTBFS: make[1]: pyversions: No such file or directory The first one also appears in Python 3.10.9, but seems not present in Python 3.10.6, as per a discussion with upstream on IRC. It looks like the default threading thingy has changed a little bit, leading to this... I know already how to fix the actual executed code, but this leaves some wrong tests (assert called), so I'm waiting to see if upstream can do better than me. For the Cinder bug, it looks like a test-only issue, which upstream is already working on. Worst case: blacklist these 250 failing tests, which is better than what it sounds (ie: just like Ironic, the actual production code is ok...). If I would create a list mine whould be way longer. Please share it in this list! We are constantly beaten by removal from testing warnings even with Python3.11 as an option and sometimes we simply need to remove that option as a temporary means for bookworm. Same over here. It's finally looking good for me after 2 months of heavy efforts. No better solution but better timing which means after bookworm release. I've read *many* people saying it would be disappointing for them to see their package removed because of Python 3.11. Well, please consider that it would also be very disappointing to *not* have Python 3.11 for those who managed constantly fix issues for it. The timing was exactly what was discussed during Debconf: it's very annoying that this year, upstream Python release was one month late... we're only trying to deal with it. Cheers, Thomas Goirand (zigo)
Re: Python 3.11 for bookworm?
Am Wed, Dec 28, 2022 at 01:54:13AM +0100 schrieb Thomas Goirand: > > While I can agree that it may be harsh, and that some packages wont be fixed > in time, I can't let you say that there was only 1 month given, and that all > of this comes as a surprise. We've been talking about this since last > summer. True, I remember the DebConf Python BoF. My memory tells me that the plan was to keep 3.10 as default. Thus Python 3.11 would be really a surprise. From a maintainers team with lots of Python packages that will need heavy work I can't say I'd be happy about a "migrate and see what might happen afterwards" strategy as it was proposed here. We have lots to do even without such an experiment. > Well, that's not the first time we are trying to push the latest interpreter > close to a release. The fact that we did so before does not make the idea better. > I also very much would appreciate help on these (which all look like > probably related to py3.11): > > #1026524 ironic-inspector: FTBFS: AttributeError: '_thread.RLock' object has > no attribute 'is_locked' > #1026547 python-pypowervm: FTBFS: AssertionError: Expected 'warn' to be > called once. Called 2 times. > #1026549 python-pytest-xprocess: FTBFS: tests failed > #1026591 cinder: FTBFS: make[1]: pyversions: No such file or directory > #1026595 python-murano-pkg-check: FTBFS: TypeError: load_all() missing 1 > required positional argument: 'Loader' > #1026610 horizon: FTBFS: tests failed > #1026691 bandit: FTBFS: TypeError: load() missing 1 required positional > argument: 'Loader' > #1026693 cloudkitty: FTBFS: touch: cannot touch > '/<>/debian/cloudkitty-doc/usr/share/doc/cloudkitty-doc/html/_static/toggle.js': > No such file or directory > #1026702 python-cursive: FTBFS: AttributeError: '_RSAPrivateKey' object has > no attribute 'signer' > #1026705 python-pecan: FTBFS: E AttributeError: 'code' object has no > attribute 'co_endlinetable' > #1026707 jenkins-job-builder: FTBFS: E: pybuild pybuild:386: test: plugin > custom failed with: exit code=1: PYTHON=python3.10 stestr run If I would create a list mine whould be way longer. (The list of successful fixes we did in the Debian Med team is also quite impressive.) We are constantly beaten by removal from testing warnings even with Python3.11 as an option and sometimes we simply need to remove that option as a temporary means for bookworm. > To sum-up: while I'm not 100% on the side of breaking things that close to a > release, and would also feel it very painful if one of the above bugs isn't > fixed in time, I don't feel like you guys are giving good point of > argumentation, or a solution to improve the process. Doko already explained > that switching the interpreter (the hard way) is the only viable way to find > out the remaining bugs. Do you have a better solution in mind? No better solution but better timing which means after bookworm release. Kind regards Andreas. -- http://fam-tille.de
Re: Python 3.11 for bookworm?
On 12/22/22 02:21, Nicholas D Steeves wrote: 100% +1 I'm especially concerned about how a clear plan was not communicated to other teams--whose work will be broken by the proposed transition, were an exception to be granted. Debian is not a paragon of community if it makes late, unannounced changes that result in a yet-undetermined number of projects being dropped from bookworm's release. If Python 3.11 as the only supported version is a release goal, then the freeze schedule would need to be modified. Regards, Nicholas Hi Nicholas, While I can agree that it may be harsh, and that some packages wont be fixed in time, I can't let you say that there was only 1 month given, and that all of this comes as a surprise. We've been talking about this since last summer. On 12/22/22 05:48, M. Zhou wrote: > A significant Python performance improvement in 3.11 is good. > But note, when python performance has really become an issue, > people already have mature solutions, e.g. offloading the > performance critical part onto a compiled language. I don't agree with this either. I'm running large-ish OpenStack swift clusters, where we run up to 18 physical nodes as proxies (eating 100s of requests per seconds). Even a 10% gain would be greatly appreciated for us. There's no "C" improvement here, it's fully in Python... I tried running all Swift services over uwsgi, but it didn't work well (we reverted this type of setup because many things broke). The fact that a new python process can spawn faster is also really a good thing (so that's not only the interpreter pure speed that I would like to have). On 12/22/22 05:48, M. Zhou wrote: > Apart from that, package maintainers have their own plans as well. > I believe that at the current stage, many people have assumed that > the next stable will ship python3.10, and have their packages > finalized for freeze already. Making that transition at the current > stage will push a number of maintainers into a rush of updating > their packages again -- in the worst case, the package upstreams > might be not even ready for python 3.11. Well, that's not the first time we are trying to push the latest interpreter close to a release. In fact, this happens on nearly each freeze. Yes, sometimes, there's no upstream fixes yet and you have to write your own. But that's what we do: we maintain packages... and fix bugs when we can. Since last summer, I fixed maybe about 20 to 30 Python 3.11 issues, sometimes with, and sometimes without help from upstream. And I have about a dozen more to fix in my TODO (see below). I invite everyone to try to do the same, so we can reach that goal. I also very much would appreciate help on these (which all look like probably related to py3.11): #1026524 ironic-inspector: FTBFS: AttributeError: '_thread.RLock' object has no attribute 'is_locked' #1026547 python-pypowervm: FTBFS: AssertionError: Expected 'warn' to be called once. Called 2 times. #1026549 python-pytest-xprocess: FTBFS: tests failed #1026591 cinder: FTBFS: make[1]: pyversions: No such file or directory #1026595 python-murano-pkg-check: FTBFS: TypeError: load_all() missing 1 required positional argument: 'Loader' #1026610 horizon: FTBFS: tests failed #1026691 bandit: FTBFS: TypeError: load() missing 1 required positional argument: 'Loader' #1026693 cloudkitty: FTBFS: touch: cannot touch '/<>/debian/cloudkitty-doc/usr/share/doc/cloudkitty-doc/html/_static/toggle.js': No such file or directory #1026702 python-cursive: FTBFS: AttributeError: '_RSAPrivateKey' object has no attribute 'signer' #1026705 python-pecan: FTBFS: E AttributeError: 'code' object has no attribute 'co_endlinetable' #1026707 jenkins-job-builder: FTBFS: E: pybuild pybuild:386: test: plugin custom failed with: exit code=1: PYTHON=python3.10 stestr run BTW, thanks a lot to Lucas Nussbaum for doing the archive rebuilt and finding these out early! To sum-up: while I'm not 100% on the side of breaking things that close to a release, and would also feel it very painful if one of the above bugs isn't fixed in time, I don't feel like you guys are giving good point of argumentation, or a solution to improve the process. Doko already explained that switching the interpreter (the hard way) is the only viable way to find out the remaining bugs. Do you have a better solution in mind? Cheers, Thomas Goirand (zigo) OpenPGP_signature Description: OpenPGP digital signature
Re: Python 3.11 for bookworm?
Hi Matthias On Wed, 21 Dec 2022 at 18:24, Matthias Klose wrote: > while we have not an 100% agreement to go ahead, I think we should aim for > 3.11. Action speaks louder than words, and there's been a whole lot of work done to push this forward. > The following steps would be: > > - accept the current python3-defaults into > testing (adding 3.11 as supported) This has been done. > - ask for a transition slot to upload (see #1026825) > python3-defaults with 3.11 as the default We're currently waiting on the PHP 8.2 (#1014460) and qtbase-opensource-src (#1025863) transitions. Although, there's been no progress on PHP 8.2, so this may be reconsidered. qtbase-opensource-src is currently blocked on a pyqt5 FTBFS on mipsel (#1026980), but there is a possible workaround. We hope to be able to start with the remaining steps soon. > - start the no-change NMUs > - file bug reports. > - fix issues > - let 3.11 as default migrate to testing. > If things don't go as planned, we could default back to 3.10. Mostly that > would > be no-change uploads, however in the case of a 3.11 specific fix that doesn't > work for 3.10, these fixes would need reverting. I have no idea who many of > these we will introduce with this transition. OK, good to know we can still back out if we have to. > Also we might want to ask for some freeze exceptions for third party > libraries, > that we can't fix before the feature freeze, unfortunately at this point we > cannot say which and how many packages would be affected. The release team is prepared to consider these on a case-by-case basis. Regards Graham
Re: Python 3.11 for bookworm?
Hi Timo, Stefano On Thu, 22 Dec 2022 at 18:46, Stefano Rivera wrote: > > Hi Timo (2022.12.22_12:56:20_+) > > > There have been rebuilds in Ubuntu that give us some idea of how much > > > work remains. I think it's tractable, but also will have some package > > > casualties. > > I have some spare time right now, and I am happy to help > > work on problematic cases, so hopefully nobody will feel left out in > > the cold with their favorite packages. > > Offhand, the one I most expect trouble with is numba. We were reliant on > upstream for the 3.10 transition, and probably will be for 3.11. > > Thanks for your help with pony ORM, Timo. I didn't think we'd be able to > port that without upstream, but it did end up being tractable. > > I'm expecting to have more time in the upcoming weeks, too. > > So, release team, I still think we should go ahead! Thanks for committing your time to this, and for the fixes you've already uploaded (and to everyone else who has helped). Please let the release team know if you need things unstuck. Regards Graham
Re: Python 3.11 for bookworm?
It seems I was a little bit out of date. Diane Trout has tried with an unreleased snapshot which looks good with llvm-14 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024795 Will work on it soon. On Thu, 2022-12-22 at 18:04 -0500, M. Zhou wrote: > I'm the regular uploader of python3-llvmlite. > > Please give up with numba. Its core dependency llvmlite is not even > ready for llvm != 11, while Sid had already get llvm-11 removed. > I have tried to cherry-pick an upstream fix to bump llvmlite's > llvm dependency to 12/14, but the autopkgtest shows numba would > be vastly broken. > > Unless they bump LLVM dependency to a newer version: > https://github.com/numba/llvmlite/issues/897 > https://github.com/numba/llvmlite/pull/830 > there is zero chance to get numba in stable. I do not want to > bump LLVM by force and leave a broken package in stable. > > llvmlite's python 3.11 support is still on the way: > https://github.com/numba/llvmlite/issues/885 > > One possibility is that we may apply for freeze exception > and wait for the llvmlite v0.40.0 release and see whether > they will bump llvm dependency. > > > On Thu, 2022-12-22 at 18:46 +, Stefano Rivera wrote: > > Hi Timo (2022.12.22_12:56:20_+) > > > > There have been rebuilds in Ubuntu that give us some idea of how much > > > > work remains. I think it's tractable, but also will have some package > > > > casualties. > > > I have some spare time right now, and I am happy to help > > > work on problematic cases, so hopefully nobody will feel left out in > > > the cold with their favorite packages. > > > > Offhand, the one I most expect trouble with is numba. We were reliant on > > upstream for the 3.10 transition, and probably will be for 3.11. > > > > Thanks for your help with pony ORM, Timo. I didn't think we'd be able to > > port that without upstream, but it did end up being tractable. > > > > I'm expecting to have more time in the upcoming weeks, too. > > > > So, release team, I still think we should go ahead! > > > > SR > > >
Re: Python 3.11 for bookworm?
I'm the regular uploader of python3-llvmlite. Please give up with numba. Its core dependency llvmlite is not even ready for llvm != 11, while Sid had already get llvm-11 removed. I have tried to cherry-pick an upstream fix to bump llvmlite's llvm dependency to 12/14, but the autopkgtest shows numba would be vastly broken. Unless they bump LLVM dependency to a newer version: https://github.com/numba/llvmlite/issues/897 https://github.com/numba/llvmlite/pull/830 there is zero chance to get numba in stable. I do not want to bump LLVM by force and leave a broken package in stable. llvmlite's python 3.11 support is still on the way: https://github.com/numba/llvmlite/issues/885 One possibility is that we may apply for freeze exception and wait for the llvmlite v0.40.0 release and see whether they will bump llvm dependency. On Thu, 2022-12-22 at 18:46 +, Stefano Rivera wrote: > Hi Timo (2022.12.22_12:56:20_+) > > > There have been rebuilds in Ubuntu that give us some idea of how much > > > work remains. I think it's tractable, but also will have some package > > > casualties. > > I have some spare time right now, and I am happy to help > > work on problematic cases, so hopefully nobody will feel left out in > > the cold with their favorite packages. > > Offhand, the one I most expect trouble with is numba. We were reliant on > upstream for the 3.10 transition, and probably will be for 3.11. > > Thanks for your help with pony ORM, Timo. I didn't think we'd be able to > port that without upstream, but it did end up being tractable. > > I'm expecting to have more time in the upcoming weeks, too. > > So, release team, I still think we should go ahead! > > SR >
Re: Python 3.11 for bookworm?
Hi Timo (2022.12.22_12:56:20_+) > > There have been rebuilds in Ubuntu that give us some idea of how much > > work remains. I think it's tractable, but also will have some package > > casualties. > I have some spare time right now, and I am happy to help > work on problematic cases, so hopefully nobody will feel left out in > the cold with their favorite packages. Offhand, the one I most expect trouble with is numba. We were reliant on upstream for the 3.10 transition, and probably will be for 3.11. Thanks for your help with pony ORM, Timo. I didn't think we'd be able to port that without upstream, but it did end up being tractable. I'm expecting to have more time in the upcoming weeks, too. So, release team, I still think we should go ahead! SR -- Stefano Rivera http://tumbleweed.org.za/ +1 415 683 3272
Re: Python 3.11 for bookworm?
* Stefano Rivera [2022-12-22 12:44]: There have been rebuilds in Ubuntu that give us some idea of how much work remains. I think it's tractable, but also will have some package casualties. I have some spare time right now, and I am happy to help work on problematic cases, so hopefully nobody will feel left out in the cold with their favorite packages. Cheers Timo -- ⢀⣴⠾⠻⢶⣦⠀ ╭╮ ⣾⠁⢠⠒⠀⣿⡁ │ Timo Röhling │ ⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │ ⠈⠳⣄ ╰╯ signature.asc Description: PGP signature
Re: Python 3.11 for bookworm?
Hi Sandro (2022.12.22_00:13:36_+) > It appears there has been little work in preparing the work to > introduce python3.11 from its maintainer, instead that works has been > pushed downstream to maintainers. That is, I'm afraid, the only realistic approach for handling new Python versions. It is too much work for one or two people to do. It needs the help of the team and upstreams to make it happen. Yes, a maintainer could take all this work on their shoulders, but if we require them to, I don't think we'll ship even vaguely current Python versions. > if we continue with the plan as described above, several python > libraries/applications maintainers will be left with the short end of > the stick and deal with an unknown amount of issues (upstream fixes > not available, not ready and or/ not released, rushed, etc) with less > than a month from the beginning of the transition freeze[2] That will almost certainly be the case, yes. So we have a trade-off to make between shipping a new Python upstream release, that many of our users would definitely appreciate, and having some libraries / apps miss the release, that many of our users would probably be affected by. > [2] also highlights at the very beginning "Plan your changes for > bullseye", this change appears as if it was not planned and we should > be skeptical to proceed without further (and in advance) understanding > of the impact it may have on Bullseye. We discussed this transition at DebConf 22, and decided to approach it the way that it has been approached. Where we currently are in the release, I would lean towards going through with the transition. So far, it seems to have been roughly as difficult as previous Python transitions. There have been rebuilds in Ubuntu that give us some idea of how much work remains. I think it's tractable, but also will have some package casualties. SR -- Stefano Rivera http://tumbleweed.org.za/ +1 415 683 3272
Re: Python 3.11 for bookworm?
There is not yet an accurate estimate of time required to fix the python packages during the transition -- and I still remember the transition from python3.9 -> python3.10 took a very long period that does not seem short enough to be covered by the freeze schedule. Apart from that, package maintainers have their own plans as well. I believe that at the current stage, many people have assumed that the next stable will ship python3.10, and have their packages finalized for freeze already. Making that transition at the current stage will push a number of maintainers into a rush of updating their packages again -- in the worst case, the package upstreams might be not even ready for python 3.11. A significant Python performance improvement in 3.11 is good. But note, when python performance has really become an issue, people already have mature solutions, e.g. offloading the performance critical part onto a compiled language. I think the risk of greatly breaking the release plan outweighs the gain. On Wed, 2022-12-21 at 20:21 -0500, Nicholas D Steeves wrote: > Sandro Tosi writes: > > > thoughts from a concerned maintainer > > > > Sandro, thank you for writing this email. > > > > > it seems this email advocates for a "let's wing it"[1] type of > > transition. > > > > [1] https://en.wiktionary.org/wiki/wing_it > > > > It appears there has been little work in preparing the work to > > introduce python3.11 from its maintainer, instead that works has > > been > > pushed downstream to maintainers. > > > > if we continue with the plan as described above, several python > > libraries/applications maintainers will be left with the short end > > of > > the stick and deal with an unknown amount of issues (upstream fixes > > not available, not ready and or/ not released, rushed, etc) with > > less > > than a month from the beginning of the transition freeze[2] > > > > Agreed. At a bare minimum, complete data from ratt (Rebuild All The > Things) should be required at this point. > > > [2] https://release.debian.org/bullseye/freeze_policy.html > > > > [2] also highlights at the very beginning "Plan your changes for > > bullseye", this change appears as if it was not planned and we > > should > > be skeptical to proceed without further (and in advance) > > understanding > > of the impact it may have on Bullseye. > > > > 100% +1 I'm especially concerned about how a clear plan was not > communicated to other teams--whose work will be broken by the > proposed > transition, were an exception to be granted. > > Debian is not a paragon of community if it makes late, unannounced > changes that result in a yet-undetermined number of projects being > dropped from bookworm's release. If Python 3.11 as the only > supported > version is a release goal, then the freeze schedule would need to be > modified. > > Regards, > Nicholas
Re: Python 3.11 for bookworm?
Sandro Tosi writes: > thoughts from a concerned maintainer > Sandro, thank you for writing this email. > > it seems this email advocates for a "let's wing it"[1] type of transition. > > [1] https://en.wiktionary.org/wiki/wing_it > > It appears there has been little work in preparing the work to > introduce python3.11 from its maintainer, instead that works has been > pushed downstream to maintainers. > > if we continue with the plan as described above, several python > libraries/applications maintainers will be left with the short end of > the stick and deal with an unknown amount of issues (upstream fixes > not available, not ready and or/ not released, rushed, etc) with less > than a month from the beginning of the transition freeze[2] > Agreed. At a bare minimum, complete data from ratt (Rebuild All The Things) should be required at this point. > [2] https://release.debian.org/bullseye/freeze_policy.html > > [2] also highlights at the very beginning "Plan your changes for > bullseye", this change appears as if it was not planned and we should > be skeptical to proceed without further (and in advance) understanding > of the impact it may have on Bullseye. > 100% +1 I'm especially concerned about how a clear plan was not communicated to other teams--whose work will be broken by the proposed transition, were an exception to be granted. Debian is not a paragon of community if it makes late, unannounced changes that result in a yet-undetermined number of projects being dropped from bookworm's release. If Python 3.11 as the only supported version is a release goal, then the freeze schedule would need to be modified. Regards, Nicholas signature.asc Description: PGP signature
Re: Python 3.11 for bookworm?
thoughts from a concerned maintainer On Wed, Dec 21, 2022 at 1:24 PM Matthias Klose wrote: > > while we have not an 100% agreement to go ahead, I think we should aim for > 3.11. > > The following steps would be: > > - accept the current python3-defaults into > testing (adding 3.11 as supported) > - ask for a transition slot to upload (see #1026825) > python3-defaults with 3.11 as the default > - start the no-change NMUs > - file bug reports. > - fix issues > - let 3.11 as default migrate to testing. > > If things don't go as planned, we could default back to 3.10. Mostly that > would > be no-change uploads, however in the case of a 3.11 specific fix that doesn't > work for 3.10, these fixes would need reverting. I have no idea who many of > these we will introduce with this transition. > > Also we might want to ask for some freeze exceptions for third party > libraries, > that we can't fix before the feature freeze, unfortunately at this point we > cannot say which and how many packages would be affected. from expressions like * "If things don't go as planned" * "no idea who many of these we will introduce with this transition." * "cannot say which and how many packages would be affected" it seems this email advocates for a "let's wing it"[1] type of transition. [1] https://en.wiktionary.org/wiki/wing_it It appears there has been little work in preparing the work to introduce python3.11 from its maintainer, instead that works has been pushed downstream to maintainers. if we continue with the plan as described above, several python libraries/applications maintainers will be left with the short end of the stick and deal with an unknown amount of issues (upstream fixes not available, not ready and or/ not released, rushed, etc) with less than a month from the beginning of the transition freeze[2] [2] https://release.debian.org/bullseye/freeze_policy.html [2] also highlights at the very beginning "Plan your changes for bullseye", this change appears as if it was not planned and we should be skeptical to proceed without further (and in advance) understanding of the impact it may have on Bullseye. Regards, -- Sandro "morph" Tosi My website: http://sandrotosi.me/ Me at Debian: http://wiki.debian.org/SandroTosi Twitter: https://twitter.com/sandrotosi
Re: Python 3.11 for bookworm?
while we have not an 100% agreement to go ahead, I think we should aim for 3.11. The following steps would be: - accept the current python3-defaults into testing (adding 3.11 as supported) - ask for a transition slot to upload (see #1026825) python3-defaults with 3.11 as the default - start the no-change NMUs - file bug reports. - fix issues - let 3.11 as default migrate to testing. If things don't go as planned, we could default back to 3.10. Mostly that would be no-change uploads, however in the case of a 3.11 specific fix that doesn't work for 3.10, these fixes would need reverting. I have no idea who many of these we will introduce with this transition. Also we might want to ask for some freeze exceptions for third party libraries, that we can't fix before the feature freeze, unfortunately at this point we cannot say which and how many packages would be affected. Matthias
Re: Python 3.11 for bookworm?
Hi Jochen, On Mon, Dec 19, 2022 at 04:53:58PM +0100, Jochen Sprickerhof wrote: > Hi Julian, > > * Julian Gilbey [2022-12-19 09:41]: > > Quick update: with the updating of python3-bytecode from 0.13 to 0.14 > > in unstable/testing, which allows it to handle Python 3.11, something > > has changed and now pydevd doesn't even pass the tests on Python > > 3.10. The python3-bytecode underwent a major restructuring, so it is > > entirely possible that something has changed that wasn't part of the > > advertised API or something like that. But that's for upstream pydevd > > developers to deal with. > > I've uploaded 0.14.0-2 that should fix this. As far as I've found that was > only a minor fix in the Debian specific offset patch, sorry for the trouble. Phew! I didn't think to check that. Unfortunately, though, there are still numerous pydevd test errors even with 0.14.0-2, so I think something has changed in bytecode that the pydevd maintainers will have to adapt to. So either I skip 14 newly failing tests on Python 3.10 (they're mostly skipped on 3.11 as the current pydevd version skips bytecode tests on Python 3.11) or wait for a new version of pydevd. Hmmm. Anyway, thanks so much for all your work updating this package - it's been really helpful, as I've been a bit overloaded and Spyder 5.4.0 together with the Python 3.11 transition has been a lot to handle. I also learnt a lot from your changes! Best wishes, Julian
Re: Python 3.11 for bookworm?
Hi Julian, * Julian Gilbey [2022-12-19 09:41]: Quick update: with the updating of python3-bytecode from 0.13 to 0.14 in unstable/testing, which allows it to handle Python 3.11, something has changed and now pydevd doesn't even pass the tests on Python 3.10. The python3-bytecode underwent a major restructuring, so it is entirely possible that something has changed that wasn't part of the advertised API or something like that. But that's for upstream pydevd developers to deal with. I've uploaded 0.14.0-2 that should fix this. As far as I've found that was only a minor fix in the Debian specific offset patch, sorry for the trouble. Cheers Jochen signature.asc Description: PGP signature
Re: Python 3.11 for bookworm?
On Sun, Dec 18, 2022 at 06:02:58PM +, Julian Gilbey wrote: > On Thu, Dec 15, 2022 at 04:10:05PM +0100, Thomas Goirand wrote: > > On 12/13/22 13:34, Julian Gilbey wrote: > > > If Python 3.11 is the default, then it is highly likely that Spyder > > > will not be included: debugpy, which is a dependency of Spyder and > > > python3-ipykernel (and lots of things that depend on that) seems to > > > require major work upstream to make it fully compatible with Python > > > 3.11. This is work in progress, but I don't know whether it will be > > > ready in time for the freeze. At the moment, I have worked around > > > this problem by just skipping the failing tests, but that is far from > > > an ideal solution. > > > > It's probably ok if it's a *TEMPORARY* solution until upstream fixes > > everything in time for the release (which is months after the freeze). The > > question is: do you believe this may happen for let's say next March? > > The truth is that I don't know. Upstream is very good, but the Python > 3.11 internal changes are very significant, and debugpy (along with > pydevd, which is part of debugpy) are deeply affected by this, as they > work at the level of Python's internals. So I don't know how long it > will take them to make the required changes (and it's far beyond my > capability or capacity to do myself). I can hope they'll do it in > time for the freeze, but I wouldn't like to place a bet on it. Quick update: with the updating of python3-bytecode from 0.13 to 0.14 in unstable/testing, which allows it to handle Python 3.11, something has changed and now pydevd doesn't even pass the tests on Python 3.10. The python3-bytecode underwent a major restructuring, so it is entirely possible that something has changed that wasn't part of the advertised API or something like that. But that's for upstream pydevd developers to deal with. One possibility is that we revert to the situation in bullseye if pydevd is not ready in time for the freeze. Bullseye didn't have bytecode/pydevd/debugpy, and it meant that debugging was limited in Spyder: we remove python3-debugpy from the dependencies of python3-ipykernel and then the rest of the dependency chain will be OK. It's certainly not an ideal solution, but it may be the best we can do if we're sticking with Python 3.11. It may actually be worth doing that at this point so that Spyder can stay in testing for the time being, even though pydevd and debugpy won't. Best wishes, Julian
Re: Python 3.11 for bookworm?
On Thu, Dec 15, 2022 at 04:10:05PM +0100, Thomas Goirand wrote: > On 12/13/22 13:34, Julian Gilbey wrote: > > If Python 3.11 is the default, then it is highly likely that Spyder > > will not be included: debugpy, which is a dependency of Spyder and > > python3-ipykernel (and lots of things that depend on that) seems to > > require major work upstream to make it fully compatible with Python > > 3.11. This is work in progress, but I don't know whether it will be > > ready in time for the freeze. At the moment, I have worked around > > this problem by just skipping the failing tests, but that is far from > > an ideal solution. > > It's probably ok if it's a *TEMPORARY* solution until upstream fixes > everything in time for the release (which is months after the freeze). The > question is: do you believe this may happen for let's say next March? The truth is that I don't know. Upstream is very good, but the Python 3.11 internal changes are very significant, and debugpy (along with pydevd, which is part of debugpy) are deeply affected by this, as they work at the level of Python's internals. So I don't know how long it will take them to make the required changes (and it's far beyond my capability or capacity to do myself). I can hope they'll do it in time for the freeze, but I wouldn't like to place a bet on it. Best wishes, Julian
Re: Python 3.11 for bookworm?
On 12/15/22 16:18, Thomas Goirand wrote: On 12/13/22 00:51, Graham Inggs wrote: Dear Python Team Looking at the current state of the 'adding Python 3.11 as a supported version' transition [1], the tracker [2] shows only 12 red packages (excluding unknowns and packages not in testing) remaining, copied below for reference. We believe all FTBFS and autopkgtest regression bugs have already been filed and tagged. The current state of bugs tagged 'python3.11' [3] is 116 resolved and 49 still open. Many thanks to everyone who has contributed to fixing these, and especially to the organizers of the recent Python sprint [4]. As this transition is non-blocking (i.e. uploaded packages are able to migrate ahead of python3-defaults), we could wait for the remaining bugs to be fixed, or for auto-removal to take its course. However, with the bookworm transition freeze only one month away [5], we'd like to hear from the Python Team within the next week whether they wish to proceed with Python 3.11 being the only supported version for bookworm (in which case we will allow python3-defaults to migrate right now) or, revert the changes in python3-defaults and have Python 3.10 as the only supported version for bookworm. Should it be the former, we'd like an undertaking from the Python Team that they will help resolve the remaining bugs against key packages [6], as these cannot easily be avoided by manual or auto-removals. On behalf of the Release Team Graham Hi Graham, For OpenStack, I believe I was able to fix all Python 3.11 bugs (often with the help of upstream, a few times, on my own but very trivial fixes like the easy getargspec -> getfullargspec). The remaining filled RC bugs because of Python 3.11, I don't really mind (even if these 5 packages are AUTORM, I'm fine with that, they aren't key packages from the OpenStack perspective). However, there's still one very annoying bug in flask-restful that makes it not rebuild under Python 3.11. Keystone needs flask-restful, so I *must* fix it. Note here that #1024913 is because of another issue (ie: the autopkgtest functional test doesn't support more than one available Python interpreter, though it's easy to fix: simply kill the server between runs...). Though it hides the real FTBFS issue (failure in unit tests), which I believe is probably due to my upgrade of werkzeug 2.2.2 rather than Python 3.11. Help would be greatly appreciated fixing this bug. Hopefully, I can get this done during my holidays (and avoid any other work...). Cheers, Thomas Goirand (zigo) FYI, I have just uploaded a new Debian release of this package that fixes all the issues, thanks to a colleague of mine that had time to help (which I didn't really have...). So, I believe I'm all good regarding OpenStack packages right now (even if that last one was werkzeug 2.2.2 related, not python 3.11). Cheers, Thomas Goirand (zigo)
Re: Python 3.11 for bookworm?
On Thu, Dec 15, 2022 at 04:08:29PM +0100, Thomas Goirand wrote: > It's unfortunately sometimes a bit harder than what one may think. Removing > Nose from (build-)depends in some cases is hard, and IMO we started this > process a way too late. Hopefully, Trixie will be nose-free! In the mean > time, it is unfortunately my opinion that it's too late for Bookworm and > that we must keep Nose for one more release. I forgot to reply here, but I uploaded fixed nose on Tuesday: https://tracker.debian.org/news/1398350/accepted-nose-137-9-source-into-unstable/ -- Dmitry Shachnev signature.asc Description: PGP signature
Re: Python 3.11 for bookworm?
On 12/13/22 00:51, Graham Inggs wrote: Dear Python Team Looking at the current state of the 'adding Python 3.11 as a supported version' transition [1], the tracker [2] shows only 12 red packages (excluding unknowns and packages not in testing) remaining, copied below for reference. We believe all FTBFS and autopkgtest regression bugs have already been filed and tagged. The current state of bugs tagged 'python3.11' [3] is 116 resolved and 49 still open. Many thanks to everyone who has contributed to fixing these, and especially to the organizers of the recent Python sprint [4]. As this transition is non-blocking (i.e. uploaded packages are able to migrate ahead of python3-defaults), we could wait for the remaining bugs to be fixed, or for auto-removal to take its course. However, with the bookworm transition freeze only one month away [5], we'd like to hear from the Python Team within the next week whether they wish to proceed with Python 3.11 being the only supported version for bookworm (in which case we will allow python3-defaults to migrate right now) or, revert the changes in python3-defaults and have Python 3.10 as the only supported version for bookworm. Should it be the former, we'd like an undertaking from the Python Team that they will help resolve the remaining bugs against key packages [6], as these cannot easily be avoided by manual or auto-removals. On behalf of the Release Team Graham Hi Graham, For OpenStack, I believe I was able to fix all Python 3.11 bugs (often with the help of upstream, a few times, on my own but very trivial fixes like the easy getargspec -> getfullargspec). The remaining filled RC bugs because of Python 3.11, I don't really mind (even if these 5 packages are AUTORM, I'm fine with that, they aren't key packages from the OpenStack perspective). However, there's still one very annoying bug in flask-restful that makes it not rebuild under Python 3.11. Keystone needs flask-restful, so I *must* fix it. Note here that #1024913 is because of another issue (ie: the autopkgtest functional test doesn't support more than one available Python interpreter, though it's easy to fix: simply kill the server between runs...). Though it hides the real FTBFS issue (failure in unit tests), which I believe is probably due to my upgrade of werkzeug 2.2.2 rather than Python 3.11. Help would be greatly appreciated fixing this bug. Hopefully, I can get this done during my holidays (and avoid any other work...). Cheers, Thomas Goirand (zigo)
Re: Python 3.11 for bookworm?
On 12/13/22 13:34, Julian Gilbey wrote: Hi Graham, On Mon, Dec 12, 2022 at 11:51:11PM +, Graham Inggs wrote: Dear Python Team Looking at the current state of the 'adding Python 3.11 as a supported version' transition [1], the tracker [2] shows only 12 red packages (excluding unknowns and packages not in testing) remaining, copied below for reference. [...] If Python 3.11 is the default, then it is highly likely that Spyder will not be included: debugpy, which is a dependency of Spyder and python3-ipykernel (and lots of things that depend on that) seems to require major work upstream to make it fully compatible with Python 3.11. This is work in progress, but I don't know whether it will be ready in time for the freeze. At the moment, I have worked around this problem by just skipping the failing tests, but that is far from an ideal solution. Best wishes, Julian Hi Julian, It's probably ok if it's a *TEMPORARY* solution until upstream fixes everything in time for the release (which is months after the freeze). The question is: do you believe this may happen for let's say next March? Cheers, Thomas Goirand (zigo)
Re: Python 3.11 for bookworm?
On 12/13/22 10:57, c.bu...@posteo.jp wrote: Am 13.12.2022 10:15 schrieb Timo Röhling: One remaining problem is the unmaintained nose package [...] done some work patching up nose This question is just for my learning: Why is nose patched? Upstream nose is unmaintained for years. I understand that you cannot drop nose from Debian in the current situation of a freeze in one months and so many dependencies. But isn't there a Debian process/workflow to "warn" package maintainers about an upcoming package drop of one of there dependend packages to put some pressure into it? Looking into the list of over 200 packages I see this also as a chance to clean out some other unmaintained (and maybe not so important) packages from the Debian repo. It's unfortunately sometimes a bit harder than what one may think. Removing Nose from (build-)depends in some cases is hard, and IMO we started this process a way too late. Hopefully, Trixie will be nose-free! In the mean time, it is unfortunately my opinion that it's too late for Bookworm and that we must keep Nose for one more release. Cheers, Thomas Goirand (zigo)
Re: Python 3.11 for bookworm?
Hi Andrius On Tue, 13 Dec 2022 at 07:13, Andrius Merkys wrote: > Am I right that whichever the choice, there will be only one supported > Python version in bookworm? Yes, I believe that was the decision made at DebConf 22. > I believe there are many packages that will > FTBFS with Python 3.11 as default (i.e., packages that use only the > default Python). Was there an attempt to rebuild the archive with that > setting? A typical test rebuild will only catch FTBFS in dependency level one (and maybe level two) of the transition tracker. In the higher levels, you'll get false positives due to failed imports of the modules that need rebuilding. Similarly, uploading python3-defaults to experimental and checking for autopkgtest failures will also result in false positives. For reference, the python3.11-add tracker lists 594 packages (excluding unknowns), and the python3.11-default tracker lists 351. With the ben files currently used in the trackers, packages still red on the first tracker also appear on the second. For what it's worth, an incremental test rebuild of the first three dependency levels was done in an Ubuntu PPA [3]. Roughly 80% of the packages involved in the python3.11-default transition were tested, and roughly 80% of the builds were successful. All build failures are counted here, including dependency-wait and architectures that have never had a successful build. A similar test rebuild was done in January 2022 for Python 3.10 [4] and I think the numbers indicate we are in a very similar state. Regards Graham [1] https://release.debian.org/transitions/html/python3.11-add.html [2] https://release.debian.org/transitions/html/python3.11-default.html [3] https://launchpad.net/~pythoneers/+archive/ubuntu/python3.11-default [4] https://launchpad.net/~pythoneers/+archive/ubuntu/python3.10-default
Re: Python 3.11 for bookworm?
On 2022-12-12 18 h 51, Graham Inggs wrote: Dear Python Team Looking at the current state of the 'adding Python 3.11 as a supported version' transition [1], the tracker [2] shows only 12 red packages (excluding unknowns and packages not in testing) remaining, copied below for reference. We believe all FTBFS and autopkgtest regression bugs have already been filed and tagged. The current state of bugs tagged 'python3.11' [3] is 116 resolved and 49 still open. Many thanks to everyone who has contributed to fixing these, and especially to the organizers of the recent Python sprint [4]. As this transition is non-blocking (i.e. uploaded packages are able to migrate ahead of python3-defaults), we could wait for the remaining bugs to be fixed, or for auto-removal to take its course. However, with the bookworm transition freeze only one month away [5], we'd like to hear from the Python Team within the next week whether they wish to proceed with Python 3.11 being the only supported version for bookworm (in which case we will allow python3-defaults to migrate right now) or, revert the changes in python3-defaults and have Python 3.10 as the only supported version for bookworm. Should it be the former, we'd like an undertaking from the Python Team that they will help resolve the remaining bugs against key packages [6], as these cannot easily be avoided by manual or auto-removals. On behalf of the Release Team Graham I still feel the move to 3.11 so late in the release cycle was cavalier and we should have used our energies to fix issues we had in the archive instead of trying to fix 3.11 bugs. I've said it already here, but it's very frustrating to work on packaging python libraries and apps for a whole release cycle, just to see all that work put in the bin at the last minute because upstream doesn't support 3.11. I hate to be put in a position where I have to tell upstreams (some with whom I've been collaborating for years) "ahem, by the way, you have 2 months to fix this or it won't be in the next Debian stable release". That said, 3.11 proponents certainly walked the walk and fixed a lot of stuff already. Kudos to them. I don't feel like I can take position on this. I'm certainly biased by one of the packages I really care about not being 3.11 compatible (and probably won't be before the release). All I know is this late transition leaves a bitter taste in my mouth. -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Louis-Philippe Véronneau ⢿⡄⠘⠷⠚⠋ po...@debian.org / veronneau.org ⠈⠳⣄ OpenPGP_0xE1E5457C8BAD4113.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: Python 3.11 for bookworm?
Hi Graham, On Mon, Dec 12, 2022 at 11:51:11PM +, Graham Inggs wrote: > Dear Python Team > > Looking at the current state of the 'adding Python 3.11 as a supported > version' transition [1], the tracker [2] shows only 12 red packages > (excluding unknowns and packages not in testing) remaining, copied > below for reference. > [...] If Python 3.11 is the default, then it is highly likely that Spyder will not be included: debugpy, which is a dependency of Spyder and python3-ipykernel (and lots of things that depend on that) seems to require major work upstream to make it fully compatible with Python 3.11. This is work in progress, but I don't know whether it will be ready in time for the freeze. At the moment, I have worked around this problem by just skipping the failing tests, but that is far from an ideal solution. Best wishes, Julian
Re: Python 3.11 for bookworm?
Hi all, On Tue, Dec 13, 2022 at 10:15:37AM +0100, Timo Röhling wrote: > One remaining problem is the unmaintained nose package, which is not > compatible with Python 3.11 and still a dependency of 200+ packages, > including ~40 key packages [1]. I've seen that crusoe has done some work > patching up nose, but AFAICT it is not building yet. > > Is this something we can resolve in time, either by fixing nose or > removing it altogether? I will try to fix nose. The remaining problem is just failing doctest. On Tue, Dec 13, 2022 at 09:57:57AM +, c.bu...@posteo.jp wrote: > This question is just for my learning: Why is nose patched? Upstream nose is > unmaintained for years. > > I understand that you cannot drop nose from Debian in the current situation > of a freeze in one months and so many dependencies. > > But isn't there a Debian process/workflow to "warn" package maintainers > about an upcoming package drop of one of there dependend packages to put > some pressure into it? Looking into the list of over 200 packages I see this > also as a chance to clean out some other unmaintained (and maybe not so > important) packages from the Debian repo. There are bugs filed against every package: https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=python-modules-t...@lists.alioth.debian.org;tag=nose-rm -- Dmitry Shachnev signature.asc Description: PGP signature
Re: Python 3.11 for bookworm?
Am 13.12.2022 10:15 schrieb Timo Röhling: One remaining problem is the unmaintained nose package [...] done some work patching up nose This question is just for my learning: Why is nose patched? Upstream nose is unmaintained for years. I understand that you cannot drop nose from Debian in the current situation of a freeze in one months and so many dependencies. But isn't there a Debian process/workflow to "warn" package maintainers about an upcoming package drop of one of there dependend packages to put some pressure into it? Looking into the list of over 200 packages I see this also as a chance to clean out some other unmaintained (and maybe not so important) packages from the Debian repo.
Re: Python 3.11 for bookworm?
* Graham Inggs [2022-12-12 23:51]: with the bookworm transition freeze only one month away [5], we'd like to hear from the Python Team within the next week whether they wish to proceed with Python 3.11 being the only supported version for bookworm [...] Should it be the former, we'd like an undertaking from the Python Team that they will help resolve the remaining bugs against key packages One remaining problem is the unmaintained nose package, which is not compatible with Python 3.11 and still a dependency of 200+ packages, including ~40 key packages [1]. I've seen that crusoe has done some work patching up nose, but AFAICT it is not building yet. Is this something we can resolve in time, either by fixing nose or removing it altogether? If yes, +1 for Python 3.11 Cheers Timo [1] https://udd.debian.org/bugs/?release=bookworm=ign=7=7=only=nose-rm=python-modules-team%40lists.alioth.debian.org=1=1=id=asc=html#results -- ⢀⣴⠾⠻⢶⣦⠀ ╭╮ ⣾⠁⢠⠒⠀⣿⡁ │ Timo Röhling │ ⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │ ⠈⠳⣄ ╰╯ signature.asc Description: PGP signature
Re: Python 3.11 for bookworm?
Hello, On 2022-12-13 01:51, Graham Inggs wrote: As this transition is non-blocking (i.e. uploaded packages are able to migrate ahead of python3-defaults), we could wait for the remaining bugs to be fixed, or for auto-removal to take its course. However, with the bookworm transition freeze only one month away [5], we'd like to hear from the Python Team within the next week whether they wish to proceed with Python 3.11 being the only supported version for bookworm (in which case we will allow python3-defaults to migrate right now) or, revert the changes in python3-defaults and have Python 3.10 as the only supported version for bookworm. Am I right that whichever the choice, there will be only one supported Python version in bookworm? I believe there are many packages that will FTBFS with Python 3.11 as default (i.e., packages that use only the default Python). Was there an attempt to rebuild the archive with that setting? [1] https://bugs.debian.org/1021984 [2] https://release.debian.org/transitions/html/python3.11-add.html [3] https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=python3.11=debian-python@lists.debian.org [4] https://veronneau.org/debian-python-team-2022-sprint-report.html [5] https://release.debian.org/bookworm/freeze_policy.html [6] https://udd.debian.org/bugs/?release=bookworm=ign=7=7=only=python3.11=debian-python%40lists.debian.org=1=1=id=asc=html#results Best, Andrius
