Re: Python Wheel Related Policy Change
On Thursday, April 30, 2020 5:49:20 PM EDT Stefano Rivera wrote: > Hi Scott (2020.04.30_20:33:59_+) > > > > That seems reasonable, although if we're going down that road, it > > > probably makes no sense for any of them to be universal. > > > > If we were talking about maintaining this for multiple release cycles with > > lots of version divergence, I would agree. Let's not do more than we have > > to until python2 is gone (whether it is before bullseye or after). > > I suspect pypy (2.7) will probably be around post-bullseye, unless > somebody funds pypy to migrate rpython to python 3. > > But yeah, we can change strategy later, if appropriate. Well, we have also talked about pypy vendoring as much of the python2.7 package as it needs to build itself so we don't have to support it in the archive as an active interpreter, but that's a different discussion. > > As a result, one could add some kind of py2 flag to dirtbike to tell it to > > look in /usr/lib/python2.7/dist-packages and make a 'universal' wheel > > from there. It could then rename the files from py2.py3-none-any.whl to > > py2-none-any.whl. The bonus Tag in the WHEEL file shouldn't hurt > > anything. > > > > They key is I think we can do this without actually running python2, we > > just need to be able to install python-setuptools/pkg-resources. That > > should be supportable ~as long as python2.7 is in the archive. > > Yep, that sounds good. > > > Agreed. As long as pip supports python2, we should try. Worst case we > > can > > tell people to find their own interpreter and do a --download install with > > setuptools pinned to 44.0.0 (that approach doesn't need ipaddr because > > they'll get the upstream pip in the virtualenv too. > > When we get to the download route: pip can parse Requires-Python, and it > looks like virtualenv uses the system pip to download pip, so the > pinning shouldn't be necessary, I think. I haven't really tested that yet, so I don't doubt that. It was an assumption. My first run at virtualenv ran aground on trying to have different package lists depending on if the install invoked --download or not (since a --download install with the upstream pip doesn't need all the unvendored wheels). So that's something to sort later. We'll want to document how to do it even if we get a working solution that doesn't require it. > > > So, the options are: > > > 1. pin pip dependencies at versions that support py2, forever. Obviously > > > > > >this is a no-go. > > > > > > 2. Ship py2 and py3 wheels. > > > > > >2.1. package the py2 weels with dirtbike. This requires bringing back > > > > > > python-wheel, or more work on dirtbike. > > > > > >2.2. Ship static py2 wheels (like we did pre-dirtbike). It's not like > > > > > > upstream is continuing development... > > > > > > 3. Let virtualenv always download pip from pypi for py2. Only ship py3 > > > > > >wheels. > > > > > > The easy options here are 2.2 and 3. > > > > 2.1a Where needed package 'py2' wheels with dirtbike running python3. > > Ah, yeah. > > Still requires keeping py2 source packages whenever a library in the pip > dependency stack drops py2 support. But if they move slowly, that's > fine. So far setuptools is the only one. It might pick up though now that they've done it. We've split packages into a source for python2 and a source for python3 in a number of cases, so I think this scales reasonably. We can just yank all the python2 sources the same time pip drops python2 or we do. > > 2.2 is using sourceless binaries, so I think it's got DFSG issues. We > > need > > the preferred form of modification and a bdist_wheel is not that. > > They could be bdist_wheeled at build time, within a single source > package. > > > I think 2.1a would be nice if someone can manage it. We'll need to > > support 3 eventually. I plan to implement 3 as an option when I upload > > pip 20.1 and virtualenv 2.0.18. > > Not necessarily. 2.2 is equivalent to 3, but without hitting PyPI. Let's focus on 2.1a so we don't have to have that argument ... I might actually have a little time tomorrow to work on it. Scott K signature.asc Description: This is a digitally signed message part.
Re: Python Wheel Related Policy Change
Hi Scott (2020.04.30_20:33:59_+) > > That seems reasonable, although if we're going down that road, it > > probably makes no sense for any of them to be universal. > > If we were talking about maintaining this for multiple release cycles with > lots of version divergence, I would agree. Let's not do more than we have to > until python2 is gone (whether it is before bullseye or after). I suspect pypy (2.7) will probably be around post-bullseye, unless somebody funds pypy to migrate rpython to python 3. But yeah, we can change strategy later, if appropriate. > As a result, one could add some kind of py2 flag to dirtbike to tell it to > look > in /usr/lib/python2.7/dist-packages and make a 'universal' wheel from there. > It could then rename the files from py2.py3-none-any.whl to py2-none-any.whl. > > The bonus Tag in the WHEEL file shouldn't hurt anything. > > They key is I think we can do this without actually running python2, we just > need to be able to install python-setuptools/pkg-resources. That should be > supportable ~as long as python2.7 is in the archive. Yep, that sounds good. > Agreed. As long as pip supports python2, we should try. Worst case we can > tell people to find their own interpreter and do a --download install with > setuptools pinned to 44.0.0 (that approach doesn't need ipaddr because > they'll > get the upstream pip in the virtualenv too. When we get to the download route: pip can parse Requires-Python, and it looks like virtualenv uses the system pip to download pip, so the pinning shouldn't be necessary, I think. > > So, the options are: > > 1. pin pip dependencies at versions that support py2, forever. Obviously > >this is a no-go. > > 2. Ship py2 and py3 wheels. > >2.1. package the py2 weels with dirtbike. This requires bringing back > > python-wheel, or more work on dirtbike. > >2.2. Ship static py2 wheels (like we did pre-dirtbike). It's not like > > upstream is continuing development... > > 3. Let virtualenv always download pip from pypi for py2. Only ship py3 > >wheels. > > > > The easy options here are 2.2 and 3. > > 2.1a Where needed package 'py2' wheels with dirtbike running python3. Ah, yeah. Still requires keeping py2 source packages whenever a library in the pip dependency stack drops py2 support. But if they move slowly, that's fine. > 2.2 is using sourceless binaries, so I think it's got DFSG issues. We need > the preferred form of modification and a bdist_wheel is not that. They could be bdist_wheeled at build time, within a single source package. > I think 2.1a would be nice if someone can manage it. We'll need to support 3 > eventually. I plan to implement 3 as an option when I upload pip 20.1 and > virtualenv 2.0.18. Not necessarily. 2.2 is equivalent to 3, but without hitting PyPI. SR -- Stefano Rivera http://tumbleweed.org.za/ +1 415 683 3272
joining the DPMT.
I would like to join the DPMT, there are a couple of reasons for this. Firstly I have been making an effort to try and get broken build-dependencies in testing fixed, and this often ends up involving python module packages. It would be easier to fix such packages as a member of the team than working through patches and NMUs as I have done so far. Secondly I maintain a couple of python modules, which it may make sense to bring into team maintainership, though I would have to figure out how to restructure the git repositories to fit the dpmt policy. I have read and accept the DPMT policy at https://salsa.debian.org/python-team/tools/python-modules/blob/master/policy.rst my salsa username is plugwash
Re: Python Wheel Related Policy Change
On Thursday, April 30, 2020 2:32:24 PM EDT Stefano Rivera wrote: > Hi Scott (2020.04.30_16:23:41_+) > > > Making dirtbike build a py3 wheel for setuptools (and pkg_resources), > > which is technically accurate at this point since dirtbike can't build > > wheels from a> > > python2 package was trivial to do: > > dirtbike/__init__.py | 6 +- > > 1 file changed, 5 insertions(+), 1 deletion(-) > > > > Three of the five insertions were comment. > > That seems reasonable, although if we're going down that road, it > probably makes no sense for any of them to be universal. If we were talking about maintaining this for multiple release cycles with lots of version divergence, I would agree. Let's not do more than we have to until python2 is gone (whether it is before bullseye or after). > > If someone wants to provide a patch to dirtbike so it can also build > > wheels > > from python-setuptools and python-pkg-resources, then we could add then to > > build-depends and provide both a set of py2 wheels and py3 wheels for as > > long as they are in the archive. > > The window for doing this is probably closing, too. > > I hacked this in Ubuntu 20.04, just before release: > https://launchpadlibrarian.net/475592029/python-pip_20.0.2-5_20.0.2-5ubuntu1 > .diff.gz > > As you can see, python-wheel is no longer available, and some of the > other libraries dirtbike uses are probably heading that way. > > So, to make that approach work, we have to introduce more py2-only > libraries again, or allow dirtbike to build py2 wheels while running > under python3. I had in mind the latter. I looked and there's not a lot of difference in the metadata of a pure python wheel that's universal and one that's not. The universal one has tags for both versions: $ cat WHEEL Wheel-Version: 1.0 Generator: bdist_wheel (0.34.2) Root-Is-Purelib: true Tag: py2-none-any Tag: py3-none-any As a result, one could add some kind of py2 flag to dirtbike to tell it to look in /usr/lib/python2.7/dist-packages and make a 'universal' wheel from there. It could then rename the files from py2.py3-none-any.whl to py2-none-any.whl. The bonus Tag in the WHEEL file shouldn't hurt anything. They key is I think we can do this without actually running python2, we just need to be able to install python-setuptools/pkg-resources. That should be supportable ~as long as python2.7 is in the archive. > > It still needs the Python policy change since they aren't universal, but > > it > > would allow setuptools to keep up for python3 in pip without dropping the > > local wheels from python2. We also have ipaddr back in the archive to > > support wheeling it so pip will run with python2. > > But, without the py2 setuptools wheels, I don't know much that buys us. Technically I think it's solvable and is supportable until the python2 interpreter goes away. > Long-term: pypy (2.7) is going to continue to be around, until rpython > moves to python3 (which is still officially never, but people are > starting to poke at it). > https://doc.pypy.org/en/latest/faq.html#how-long-will-pypy-support-python2 > > If we have a python 2 interpreter, it's really nice to have working > virtualenv for it. Makes it a lot more useful for users, esp. as we > don't have many debian libraries packaged for it. Agreed. As long as pip supports python2, we should try. Worst case we can tell people to find their own interpreter and do a --download install with setuptools pinned to 44.0.0 (that approach doesn't need ipaddr because they'll get the upstream pip in the virtualenv too. > So, the options are: > 1. pin pip dependencies at versions that support py2, forever. Obviously >this is a no-go. > 2. Ship py2 and py3 wheels. >2.1. package the py2 weels with dirtbike. This requires bringing back > python-wheel, or more work on dirtbike. >2.2. Ship static py2 wheels (like we did pre-dirtbike). It's not like > upstream is continuing development... > 3. Let virtualenv always download pip from pypi for py2. Only ship py3 >wheels. > > The easy options here are 2.2 and 3. 2.1a Where needed package 'py2' wheels with dirtbike running python3. 2.2 is using sourceless binaries, so I think it's got DFSG issues. We need the preferred form of modification and a bdist_wheel is not that. I think 2.1a would be nice if someone can manage it. We'll need to support 3 eventually. I plan to implement 3 as an option when I upload pip 20.1 and virtualenv 2.0.18. Scott K signature.asc Description: This is a digitally signed message part.
Re: Python Wheel Related Policy Change
Hi Scott (2020.04.30_16:23:41_+) > Making dirtbike build a py3 wheel for setuptools (and pkg_resources), which > is > technically accurate at this point since dirtbike can't build wheels from a > python2 package was trivial to do: > > dirtbike/__init__.py | 6 +- > 1 file changed, 5 insertions(+), 1 deletion(-) > > Three of the five insertions were comment. That seems reasonable, although if we're going down that road, it probably makes no sense for any of them to be universal. > If someone wants to provide a patch to dirtbike so it can also build wheels > from python-setuptools and python-pkg-resources, then we could add then to > build-depends and provide both a set of py2 wheels and py3 wheels for as long > as they are in the archive. The window for doing this is probably closing, too. I hacked this in Ubuntu 20.04, just before release: https://launchpadlibrarian.net/475592029/python-pip_20.0.2-5_20.0.2-5ubuntu1.diff.gz As you can see, python-wheel is no longer available, and some of the other libraries dirtbike uses are probably heading that way. So, to make that approach work, we have to introduce more py2-only libraries again, or allow dirtbike to build py2 wheels while running under python3. > It still needs the Python policy change since they aren't universal, but it > would allow setuptools to keep up for python3 in pip without dropping the > local wheels from python2. We also have ipaddr back in the archive to > support > wheeling it so pip will run with python2. But, without the py2 setuptools wheels, I don't know much that buys us. Long-term: pypy (2.7) is going to continue to be around, until rpython moves to python3 (which is still officially never, but people are starting to poke at it). https://doc.pypy.org/en/latest/faq.html#how-long-will-pypy-support-python2 If we have a python 2 interpreter, it's really nice to have working virtualenv for it. Makes it a lot more useful for users, esp. as we don't have many debian libraries packaged for it. So, the options are: 1. pin pip dependencies at versions that support py2, forever. Obviously this is a no-go. 2. Ship py2 and py3 wheels. 2.1. package the py2 weels with dirtbike. This requires bringing back python-wheel, or more work on dirtbike. 2.2. Ship static py2 wheels (like we did pre-dirtbike). It's not like upstream is continuing development... 3. Let virtualenv always download pip from pypi for py2. Only ship py3 wheels. The easy options here are 2.2 and 3. SR -- Stefano Rivera http://tumbleweed.org.za/ +1 415 683 3272
RFS: r4d/1.5-1 [ITP] -- Remote For Device-under-test (R4D) Daemon
Package: sponsorship-requests Severity: wishlist Dear mentors, I am looking for a sponsor for my package "r4d" * Package name: r4d Version : 1.5-1 Upstream Author : Benedikt Spranger * URL : https://github.com/ci-rt/r4d * License : GPL-3+ * Vcs : https://salsa.debian.org/python-team/applications/r4d Section : net It builds those binary packages: r4d - Remote For Device-under-test (R4D) Daemon To access further information about this package, please visit the following URL: https://mentors.debian.net/package/r4d Alternatively, one can download the package with dget using this command: dget -x https://mentors.debian.net/debian/pool/contrib/r/r4d/r4d_1.5-1.dsc Changes since the last upload: * Initial release (Closes: #955616) Regards, Bastian Germann
Re: Python Wheel Related Policy Change
On April 30, 2020 7:21:39 AM UTC, Matthias Klose wrote: >On 4/30/20 5:28 AM, Scott Kitterman wrote: >> I think Python policy changes should be discussed. I accidentally >committed a >> change to git [1] (I didn't realize I still had access, I thought it >would be >> a merge request) to allow Python 3 only wheels for packages that >require >> wheels, but that no longer support Python 2. This is going to happen >the next >> time I upload python-pip since the current version of setuptools is >python3 >> only. > >this is not true. python-setuptools is still built from the source >packages >python-setuptools. Is there a reason that you cannot use it? In the short run, the blocker is that dirtbike doesn't support it. In the longer run I think adding back python2 dependencies is not a good idea. At best it's a short-term avoidance mechanism. >> Hopefully this won't be controversial. There's really no way to >> avoid it. > >well, better don't assume that. It's true that there is a short-term alternative, but I think it's a waste of effort to try and implement it. At most it only buys you until the next pip vendored lib that we unbundle as a wheel goes python3 only. Are you volunteering to rewrite dirtbike so it can make a universal wheel from python-setuptools? Personally, I'm not interested. If you add the word reasonable to my original statement, I stand bye it. Scott K
Re: Python Wheel Related Policy Change
On 4/30/20 5:28 AM, Scott Kitterman wrote: > I think Python policy changes should be discussed. I accidentally committed > a > change to git [1] (I didn't realize I still had access, I thought it would be > a merge request) to allow Python 3 only wheels for packages that require > wheels, but that no longer support Python 2. This is going to happen the > next > time I upload python-pip since the current version of setuptools is python3 > only. this is not true. python-setuptools is still built from the source packages python-setuptools. Is there a reason that you cannot use it? > Hopefully this won't be controversial. There's really no way to avoid it. well, better don't assume that. Matthias
