Re: Python 3.9 for bullseye

2020-12-06 Thread Matthias Klose
On 11/9/20 10:19 AM, Matthias Klose wrote:
> On 10/23/20 1:07 PM, Matthias Klose wrote:
>> On 10/18/20 12:13 PM, Matthias Klose wrote:
>>> Python 3.9 as a supported Python3 version is now in unstable, and all 
>>> binNMUs
>>> are done (thanks to Graham for the work).   Bug reports should be all filed 
>>> for
>>> all known problems [1], and the current state of the 3.9 addition can be 
>>> seen at
>>> [2] (a few of the "bad" are false packages with b-d n python3-all-dev, but 
>>> not
>>> building for 3.9, bug reports also filed).
>>>
>>> The major outstanding issue is the pandas stack, all other problems are 
>>> found in
>>> leaf packages (leaf in the sense of that no other package for the 3.9 
>>> addition
>>> is blocked).
>>>
>>> Please help fixing the remaining issues!
>>>
>>> Matthias
>>>
>>> [1]
>>> https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=python3.9;users=debian-python@lists.debian.org
>>> [2] https://release.debian.org/transitions/html/python3.9.html
>>
>> Going on with a test rebuild of 3.9 as default to file bug reports for more
>> packages.  The first stage1 packages for 3.9 as default [1] can be found at
>>
>>   deb [trusted=yes] http://people.debian.org/~doko/tmp/python3.9 ./
>>   deb [trusted=yes] http://people.debian.org/~ginggs/python3.9-repo ./
>>
>> The first repo just having the python3-defaults packages from experimental. 
>> The
>> second repo of course will be outdated very soon ...  Bug reports for stage1 
>> are
>> filed, Graham is now running the test builds for stage2.
>>
>> The autopkg test results at [2] need checking. There's currently a britney 
>> bug
>> which marks things as bad, and only gets it right after a week. Plus there's 
>> no
>> way to select an "unrelated" package from unstable for a test, and have it
>> marked as a successful test.  So basically you need to wait until all the 3.9
>> related fixes migrate to testing for running a successful autopkg test.
>>
>> Matthias
>>
>> [1] https://release.debian.org/transitions/html/python3.9-default.html
>> [2] https://tracker.debian.org/pkg/python3-defaults
> 
>  - python3-defaults now migrated to testing.  The following packages were
>removed from testing with fastened hints.

python3-defaults 3.9.0-3 (supporting 3.8 and 3.9, defaulting to 3.9) is now in
testing.

python3-defaults 3.9.0-4 (supporting only 3.9, defaulting to 3.9) is now in
unstable.  After that version migrates to testing, we'll do the binNMUs to drop
the extensions for 3.8 (this way we avoid testing against 3.8 again).

python3-defaults 3.9.1-1 is expected next week with the upstream Python 3.9.1
release.

We are not yet finished, still having the list list of RC issues at [1].

If you think that your package needs a break, because it is likely to break with
partial upgrades, leave a message at https://bugs.debian.org/976655. I'll add
these breaks as we had them for 3.7 (but didn't have them for 3.8).

Maybe now is also the time to look at packages with outdated upstream versions
https://qa.debian.org/developer.php?email=python-modules-team%40lists.alioth.debian.org
lists 30-40% outdated source packages.

https://qa.debian.org/developer.php?email=team%2Bpython%40tracker.debian.org
looks a bit better.

Matthias

[1]
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=python3.9;users=debian-python@lists.debian.org



Re: Python 3.9 for bullseye

2020-11-13 Thread Matthias Klose
On 11/11/20 3:27 AM, Michael Hudson-Doyle wrote:
> On Mon, 9 Nov 2020 at 22:20, Matthias Klose  wrote:
> 
>> On 10/23/20 1:07 PM, Matthias Klose wrote:
>>> On 10/18/20 12:13 PM, Matthias Klose wrote:
>>>> Python 3.9 as a supported Python3 version is now in unstable, and all
>> binNMUs
>>>> are done (thanks to Graham for the work).   Bug reports should be all
>> filed for
>>>> all known problems [1], and the current state of the 3.9 addition can
>> be seen at
>>>> [2] (a few of the "bad" are false packages with b-d n python3-all-dev,
>> but not
>>>> building for 3.9, bug reports also filed).
>>>>
>>>> The major outstanding issue is the pandas stack, all other problems are
>> found in
>>>> leaf packages (leaf in the sense of that no other package for the 3.9
>> addition
>>>> is blocked).
>>>>
>>>> Please help fixing the remaining issues!
>>>>
>>>> Matthias
>>>>
>>>> [1]
>>>>
>> https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=python3.9;users=debian-python@lists.debian.org
>>>> [2] https://release.debian.org/transitions/html/python3.9.html
>>>
>>> Going on with a test rebuild of 3.9 as default to file bug reports for
>> more
>>> packages.  The first stage1 packages for 3.9 as default [1] can be found
>> at
>>>
>>>   deb [trusted=yes] http://people.debian.org/~doko/tmp/python3.9 ./
>>>   deb [trusted=yes] http://people.debian.org/~ginggs/python3.9-repo ./
>>>
>>> The first repo just having the python3-defaults packages from
>> experimental. The
>>> second repo of course will be outdated very soon ...  Bug reports for
>> stage1 are
>>> filed, Graham is now running the test builds for stage2.
>>>
>>> The autopkg test results at [2] need checking. There's currently a
>> britney bug
>>> which marks things as bad, and only gets it right after a week. Plus
>> there's no
>>> way to select an "unrelated" package from unstable for a test, and have
>> it
>>> marked as a successful test.  So basically you need to wait until all
>> the 3.9
>>> related fixes migrate to testing for running a successful autopkg test.
>>>
>>> Matthias
>>>
>>> [1] https://release.debian.org/transitions/html/python3.9-default.html
>>> [2] https://tracker.debian.org/pkg/python3-defaults
>>
>>  - python3-defaults now migrated to testing.  The following packages were
>>removed from testing with fastened hints.
>>
>>numba
>>python-executing
>>python-fabio
>>python-icecream
>>python-molotov
>>supysonic
>>
>>  - The Python related ftbfs issues from the last archive test rebuild
>>were user-tagged with 'python3.9', although I didn't make much
>>effort to determine if these are ftbfs seen for 3.9 only, or for
>>both 3.8 and 3.9.  dh-python now supports building and testing for
>>all supported python version before bailing out in case of errors,
>>but this came too late for the test rebuild.
>>
>>issues for key packages (those with lots of dependencies) are:
>>
>> https://bugs.debian.org/973056 src:sphinx-tabs
>>
> 
> Fixed.
> 
> 
>> https://bugs.debian.org/973057 src:python-py
>>
> 
> Fixed.
> 
> 
>> https://bugs.debian.org/973061 src:nototools
>>
> 
> Pasted a trivial fix to the bug. I guess it could be NMUed (it's not a
> Python team package).
> 
> 
>> https://bugs.debian.org/973072 src:python-kubernetes
>>
> 
> I patched the failures (see the bug) but then the build hangs for me.
> 
> 
>> https://bugs.debian.org/973087 src:python-fs
>>
> 
> Fixed, but I forgot to put the Closes: #xxx in the changelog.
> 
> 
>> https://bugs.debian.org/973114 src:python-future
>> https://bugs.debian.org/973121 src:cairocffi
>> https://bugs.debian.org/973126 src:responses
>> https://bugs.debian.org/973134 src:python-webob
>> https://bugs.debian.org/973165 src:pyflakes
>>
> 
> Fixed.
> 
> 
>> https://bugs.debian.org/973167 src:ufonormalizer
>> https://bugs.debian.org/973168 src:pylint
>>
> 
> This looks confusing! Upstream is thinking about it but I'm not sure what
> their ETA is.
> 
> 
>> https://bugs.debian.org/973193 src:parso
>> https://bugs.debian.org/973195 src:python-asyncssh
>> https://bugs.debian.

Re: Python 3.9 for bullseye

2020-11-09 Thread Matthias Klose
On 10/23/20 1:07 PM, Matthias Klose wrote:
> On 10/18/20 12:13 PM, Matthias Klose wrote:
>> Python 3.9 as a supported Python3 version is now in unstable, and all binNMUs
>> are done (thanks to Graham for the work).   Bug reports should be all filed 
>> for
>> all known problems [1], and the current state of the 3.9 addition can be 
>> seen at
>> [2] (a few of the "bad" are false packages with b-d n python3-all-dev, but 
>> not
>> building for 3.9, bug reports also filed).
>>
>> The major outstanding issue is the pandas stack, all other problems are 
>> found in
>> leaf packages (leaf in the sense of that no other package for the 3.9 
>> addition
>> is blocked).
>>
>> Please help fixing the remaining issues!
>>
>> Matthias
>>
>> [1]
>> https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=python3.9;users=debian-python@lists.debian.org
>> [2] https://release.debian.org/transitions/html/python3.9.html
> 
> Going on with a test rebuild of 3.9 as default to file bug reports for more
> packages.  The first stage1 packages for 3.9 as default [1] can be found at
> 
>   deb [trusted=yes] http://people.debian.org/~doko/tmp/python3.9 ./
>   deb [trusted=yes] http://people.debian.org/~ginggs/python3.9-repo ./
> 
> The first repo just having the python3-defaults packages from experimental. 
> The
> second repo of course will be outdated very soon ...  Bug reports for stage1 
> are
> filed, Graham is now running the test builds for stage2.
> 
> The autopkg test results at [2] need checking. There's currently a britney bug
> which marks things as bad, and only gets it right after a week. Plus there's 
> no
> way to select an "unrelated" package from unstable for a test, and have it
> marked as a successful test.  So basically you need to wait until all the 3.9
> related fixes migrate to testing for running a successful autopkg test.
> 
> Matthias
> 
> [1] https://release.debian.org/transitions/html/python3.9-default.html
> [2] https://tracker.debian.org/pkg/python3-defaults

 - python3-defaults now migrated to testing.  The following packages were
   removed from testing with fastened hints.

   numba
   python-executing
   python-fabio
   python-icecream
   python-molotov
   supysonic

 - The Python related ftbfs issues from the last archive test rebuild
   were user-tagged with 'python3.9', although I didn't make much
   effort to determine if these are ftbfs seen for 3.9 only, or for
   both 3.8 and 3.9.  dh-python now supports building and testing for
   all supported python version before bailing out in case of errors,
   but this came too late for the test rebuild.

   issues for key packages (those with lots of dependencies) are:

https://bugs.debian.org/973056 src:sphinx-tabs
https://bugs.debian.org/973057 src:python-py
https://bugs.debian.org/973061 src:nototools
https://bugs.debian.org/973072 src:python-kubernetes
https://bugs.debian.org/973087 src:python-fs
https://bugs.debian.org/973114 src:python-future
https://bugs.debian.org/973121 src:cairocffi
https://bugs.debian.org/973126 src:responses
https://bugs.debian.org/973134 src:python-webob
https://bugs.debian.org/973165 src:pyflakes
https://bugs.debian.org/973167 src:ufonormalizer
https://bugs.debian.org/973168 src:pylint
https://bugs.debian.org/973193 src:parso
https://bugs.debian.org/973195 src:python-asyncssh
https://bugs.debian.org/973239 src:python-fixtures

   For the other ftbfs, see [1].

Matthias

[1]
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=python3.9;users=debian-python@lists.debian.org



Re: "pytest" command is missing

2020-11-05 Thread Matthias Klose
On 11/5/20 1:03 PM, terce...@debian.org wrote:
> On Thu, Nov 05, 2020 at 09:28:56AM +0100, Thomas Goirand wrote:
>> On 11/4/20 9:27 PM, Novy, Ondrej wrote:
>>> Hi,
>>>
>>> Antonio Terceiro píše v St 04. 11. 2020 v 14:01 -0300:
 Could you ellaborate? Maybe we should have a discussion in the Python
 team so that we implement consistent practices. For example, `gunicorn`
 and `pip` now point to their python3 versions, but you are saying that
 pytest will not do that, what maybe creates more confusion given Debian
 bullseye will not support any other Python.
>>>
>>> "pytest" in buster now points to python2 version of pytest and
>>> "pytest-3" points to python3 version. To prevent confusion after upgrade
>>> I want to keep one stable release with pytest cmd "unoccupied" and keep
>>> pytest-3.
>>>
>>> Bullseye will support Python2 interpreter so user can keep python-pytest
>>> package installed from buster.
> 
> The python2 interpreter will be supported, but nothing else will, so I
> don't see the point of this compatibility.  I don't think that "keep you
> old packages from the previous release" is a great upgrade path.

well, afaiu, we also will ship python-pip, so that users are able to get their
python2 code from other places.

> Anyway, my point is that we should collective aim to be consistent
> across the Python packages. The fact that some packages have made their
> "not *3" binaries be the python3 versions, and others not, due to
> arbitrary individual maintainer decisions, is a mess.

it's a difficult decision to make. But if people want, then we could ship
pytest-is-python2 and pytest-is-python3 packages as well, as long as we don't
build-depend on those.

Matthias



Re: Python 3.9 for bullseye

2020-10-28 Thread Matthias Klose
On 10/23/20 1:07 PM, Matthias Klose wrote:
> On 10/18/20 12:13 PM, Matthias Klose wrote:
>> Python 3.9 as a supported Python3 version is now in unstable, and all binNMUs
>> are done (thanks to Graham for the work).   Bug reports should be all filed 
>> for
>> all known problems [1], and the current state of the 3.9 addition can be 
>> seen at
>> [2] (a few of the "bad" are false packages with b-d n python3-all-dev, but 
>> not
>> building for 3.9, bug reports also filed).
>>
>> The major outstanding issue is the pandas stack, all other problems are 
>> found in
>> leaf packages (leaf in the sense of that no other package for the 3.9 
>> addition
>> is blocked).
>>
>> Please help fixing the remaining issues!
>>
>> Matthias
>>
>> [1]
>> https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=python3.9;users=debian-python@lists.debian.org
>> [2] https://release.debian.org/transitions/html/python3.9.html
> 
> Going on with a test rebuild of 3.9 as default to file bug reports for more
> packages.  The first stage1 packages for 3.9 as default [1] can be found at
> 
>   deb [trusted=yes] http://people.debian.org/~doko/tmp/python3.9 ./
>   deb [trusted=yes] http://people.debian.org/~ginggs/python3.9-repo ./
> 
> The first repo just having the python3-defaults packages from experimental. 
> The
> second repo of course will be outdated very soon ...  Bug reports for stage1 
> are
> filed, Graham is now running the test builds for stage2.
> 
> The autopkg test results at [2] need checking. There's currently a britney bug
> which marks things as bad, and only gets it right after a week. Plus there's 
> no
> way to select an "unrelated" package from unstable for a test, and have it
> marked as a successful test.  So basically you need to wait until all the 3.9
> related fixes migrate to testing for running a successful autopkg test.
> 
> Matthias
> 
> [1] https://release.debian.org/transitions/html/python3.9-default.html
> [2] https://tracker.debian.org/pkg/python3-defaults

Update:

 - Graham finished the test rebuilds with 3.9 as the default,
   and the archive on p.d.o is updated.  We don't plan to keep
   that up-to-date. Bug reports for ftbfs are filed.

 - Bug reports are also filed for all failing autopkg tests triggered
   by the python3-defaults upload.

 - Lucas made an archive-wide test rebuild for unstable, adding more
   RC ftbfs issues, mostly for python binary-indep (all) packages.
   These don't have the python3.9 user tag.

Matthias



Re: Python 3.9 for bullseye

2020-10-23 Thread Matthias Klose
On 10/18/20 12:13 PM, Matthias Klose wrote:
> Python 3.9 as a supported Python3 version is now in unstable, and all binNMUs
> are done (thanks to Graham for the work).   Bug reports should be all filed 
> for
> all known problems [1], and the current state of the 3.9 addition can be seen 
> at
> [2] (a few of the "bad" are false packages with b-d n python3-all-dev, but not
> building for 3.9, bug reports also filed).
> 
> The major outstanding issue is the pandas stack, all other problems are found 
> in
> leaf packages (leaf in the sense of that no other package for the 3.9 addition
> is blocked).
> 
> Please help fixing the remaining issues!
> 
> Matthias
> 
> [1]
> https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=python3.9;users=debian-python@lists.debian.org
> [2] https://release.debian.org/transitions/html/python3.9.html

Going on with a test rebuild of 3.9 as default to file bug reports for more
packages.  The first stage1 packages for 3.9 as default [1] can be found at

  deb [trusted=yes] http://people.debian.org/~doko/tmp/python3.9 ./
  deb [trusted=yes] http://people.debian.org/~ginggs/python3.9-repo ./

The first repo just having the python3-defaults packages from experimental. The
second repo of course will be outdated very soon ...  Bug reports for stage1 are
filed, Graham is now running the test builds for stage2.

The autopkg test results at [2] need checking. There's currently a britney bug
which marks things as bad, and only gets it right after a week. Plus there's no
way to select an "unrelated" package from unstable for a test, and have it
marked as a successful test.  So basically you need to wait until all the 3.9
related fixes migrate to testing for running a successful autopkg test.

Matthias

[1] https://release.debian.org/transitions/html/python3.9-default.html
[2] https://tracker.debian.org/pkg/python3-defaults



Python 3.9 for bullseye

2020-10-18 Thread Matthias Klose
Python 3.9 as a supported Python3 version is now in unstable, and all binNMUs
are done (thanks to Graham for the work).   Bug reports should be all filed for
all known problems [1], and the current state of the 3.9 addition can be seen at
[2] (a few of the "bad" are false packages with b-d n python3-all-dev, but not
building for 3.9, bug reports also filed).

The major outstanding issue is the pandas stack, all other problems are found in
leaf packages (leaf in the sense of that no other package for the 3.9 addition
is blocked).

Please help fixing the remaining issues!

Matthias

[1]
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=python3.9;users=debian-python@lists.debian.org
[2] https://release.debian.org/transitions/html/python3.9.html



Re: Python 2 support for Bullseye

2020-10-16 Thread Matthias Klose
On 10/16/20 8:04 PM, Moritz Mühlenhoff wrote:
> There will be few core packages build-depending on Python 2 (for tests
> or building) which won't be ready for Python 3 for Bullseye (Chromium,
> qtwebkit and IIRC also Pypy), but those only need Python 2 (and a very
> small set of support packages like setuptools/jinja) to build and
> run their tests.
> 
> Apart from those there's only a handful of packages still in testing
> which have a run time dependency on Python 2 packages (and most are
> actively being worked on (e.g. Xen)).
> 
> All packages (and their test suites) shipped in Debian are trusted
> anyway (and consequently don't need any kind of updates during the
> bullseye cycle), so a lack of updates/upstream EOL doesn't matter.
> 
> As such, I'd propose to include Python 2 (plus the small set of
> support packages) in Bullseye

ok. I think you should explicitly name all these packages.

> but exempt it from support (and then
> remove it for good after the Bullseye release):

not ok. That would mean removing pypy3 from the archive as well.  If you don't
want to support Python2, then why do you care about it's removal for bullseye+1?

> - Mark src:python (plus related support packages) as unsupported in
>   debian-security-support and with a README.Debian in the source
>   package (and given how prominent Python is, also in the release notes)

That should list the binary packages, there's no src:python package.

> - Bump the severity of the few remaining packages in testing to RC
>   state (and force them out testing if auto removals were blocked
>   for some reason)

See above, I think it's unfriendly to push out pypy3.

Matthias



Re: Bug#972213: boost1.71: Please indicate some way which python versions you support

2020-10-16 Thread Matthias Klose
On 10/15/20 10:03 PM, Alastair McKinstry wrote:
> On 15/10/2020 08:13, Giovanni Mascellani wrote:
> 
>> Hi,
>>
>> Il 14/10/20 15:52, Alastair McKinstry ha scritto:
>>> I maintain the package "ecflow" which uses libboost-python-dev. Now
>>> with the transition to python3.9, ecflow will support (where
>>> possible) multiple python versions. Currently it supports python3.8
>>> but not python3.9 ; this may be fixed in a binNMU or next version,
>>> but I cannot tell whether my failure to build python3.9 support for
>>> ecflow is due to missing py3.9 support in boost, or a bug in my
>>> packaging.
>> BTW, a binNMU was just requested to add Python 3.9 support.
>>
>>> Can some mechanism be added to enable tracability ?
>> In general, Boost supports all the Python versions currently supported
>> by the Python team, except that someone has to file a binNMU for them to
>> appear once a new Python version is packaged. To check which Python
>> versions are supported by a specific package version, it is enough to
>> check the content of the libboost-python1.71.0 package (replace 1.71.0
>> with future versions where applicable):
>>
>> $ dpkg -L libboost-python1.71.0 | grep libboost_python
>> /usr/lib/x86_64-linux-gnu/libboost_python38.so.1.71.0
>> /usr/lib/x86_64-linux-gnu/libboost_python39.so.1.71.0
>>
>> (until yesterday you did not have the python39 variant)
>>
>> Does this answer your question?
>>
>> Giovanni.
> 
> Not really. This is probably better discussed on debian-python.
> 
> The point was that there is a lack of a good mechanism to see which packages
> provide py39 support, etc.
> 
> In principle my package ecflow just needs a rebuild after the boost is 
> rebuilt,
> but tracking if this actually works, or add a particular dependency to enable
> automatic rebuild/tracking.

I don't think that packages should care about that at all.  It usually is only
an issue when adding a new python version, and this happens once in a year.  The
transition tracker [1] provides a guidance how to build stuff in which order,
based on dependencies, but doesn't take care about build-dependencies,
autopkgtest dependencies, and dependency cycles.  And binNMUing 600 packages
takes time ...

Matthias

[1] https://release.debian.org/transitions/html/python3.9.html



Re: Timing of Python (3.9) upstream and Debian releases

2020-10-12 Thread Matthias Klose
On 7/6/20 9:04 PM, Matthias Klose wrote:
> Starting with Python 3.8, Python upstream changed to a time based yearly 
> release
> schedule, targeting the first release of a major Python version (3.x) for
> October of each year.  For the transition to 3.8:
> 
>  - we add 3.8 as supported in November
>  - made 3.8 the default in March
>  - dropped 3.7 in April
> 
> That's a little bit late looking at the Debian release schedule, so a little
> speedup would be needed if we want to target the current Python release for 
> the
> current Debian release.  For 3.8 Ubuntu started to prepare the switch a bit
> earlier, using the results for a better experience in Debian:
> 
>  - added 3.8 in mid October
>  - made 3.8 the default in mid December
> 
> Technically it would be possible to do the defaults change before the Debian
> freeze, however there's usually a tail of follow-up upstream releases required
> to support a new major Python version, unless you opt for actively backporting
> changes in various packages.  Having the confidence, that the switch is 
> feasible
> in another distro certainly helps doing the transition in Debian, however it
> adds a delay.  I don't think it's feasible to do the transition in 
> experimental
> first, or doing a large test rebuild, because it requires keeping the whole
> python stack in sync with testing/unstable.
> 
> So what I'm proposing here is to aim to support 3.9 as early as possible as a
> supported Python3 version, starting with the 3.9 upstream release, and fixing
> stuff on the go.  Then decide in November, if we can do the defaults change to
> 3.9, or if we drop 3.9 again, or ship with two supported Python3 versions.
> 
> Please note this will be a re-occuring situation with Python 3.11 and
> bullseye+1, so we should find out how to handle this on a regular basis,
> assuming that Debian release schedules won't change much.

3.9 was released last week and migrated to testing.  I did a test rebuild with
3.9 as a supported Python3 version, however not for unstable, but for the
current Ubuntu development version. Of course there is some version skew, but it
should give an idea what still needs work.

Bugs are filed at
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=python3.9;users=debian-python@lists.debian.org

which definitely looks better than the addition of 3.8 last year.

The PPAs used are based on groovy:
https://launchpad.net/~pythoneers/+archive/ubuntu/python3.9
https://launchpad.net/~pythoneers/+archive/ubuntu/python3.9-fixes

The first PPA just has no-change uploads, while the second one has some
sourceful uploads.  If you want to see some update in these PPAs, please ask
Emmanuel Arias, Graham Inggs or myself.

I'm planning to upload python3-defaults from experimental to unstable this
Wednesday (Oct 14). We will see some days of instability as we are doing the
binNMUs.

Matthias



Re: The python command in Debian

2020-09-17 Thread Matthias Klose
On 9/17/20 3:04 PM, Nicolas Dandrimont wrote:
> Hi Matthias, others,
> 
> On Thu, Jul 9, 2020, at 15:26, Matthias Klose wrote:
>> As written in [1], bullseye will not see unversioned python packages and the
>> unversioned python command being built from the python-defaults package.
>>
>> It seems to be a little bit more controversial what should happen to the 
>> python
>> command in the long term.  Some people argue that python should never point 
>> to
>> python3, because it's incompatible, however Debian will have difficulties to
>> explain that decision to users who start with Python3 and are not aware of 
>> the 2
>> to 3 transition.  So yes, in the long term, Debian should have a python 
>> command
>> again.
>>
>> One solution could be not to ship the python command in bullseye, forcing 
>> users
>> to adjust their local installations.  This has the advantage that the error 
>> of
>> an unknown interpreter should be pretty clear.  But leaves users without a
>> python command for the next two years until bullseye+1.
>>
>> Describing here a solution which is implemented for Ubuntu focal (20.04 
>> LTS).  A
>> new source package what-is-python (-perl-dont-hurt-me) ships binary packages
>> python-is-python2, python-dev-is-python2, python-is-python3 and
>> python-dev-is-python3.  The python-is-python2 package provides the python
>> package, such that packages that still depend on python are not removed on a
>> distro upgrade.  On new installs, python-is-python3 is not installed by 
>> default,
>> but the user gets a hint from command-not-found to install the package if he
>> tries to run python.  Package dependencies on the new four binary packages 
>> have
>> to be disallowed in the Python policy.  Note that such a package including 
>> the
>> Provides should only be uploaded once all dependencies on the unversioned 
>> python
>> packages are gone.
> 
> So I see that the removal of `/usr/bin/python`-shipped-by-python-defaults has 
> happened as planned. Thanks!
> 
> I've just got a friend ask me about what to do to get /usr/bin/python to 
> point at python3; Do you have any plan of uploading what-is-python for use in 
> bullseye, at least without the python-is-python2 Provides for python as a 
> first step (to keep the current "breakage")?
> 
> In any case I think the python packaging policy at 
> https://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html 
> should get an update to match the current status quo related to 
> /usr/bin/python; My friend looked at it and were confused not to find a 
> /usr/bin/python anymore.

the what-is-python package is now in NEW.  Yes, the policy needs an update.
I'll put that on my TODO list.  If there is too much disagreement about the
python-is-python3 package, then I plan to run it via the CTTE, and ask for an
advice how to move on.

Matthias



Re: Bug#967209: rgtk2: Unversioned Python removal in sid/bullseye

2020-08-04 Thread Matthias Klose
block 967209 by 967157
thanks

sorry, that's due to uninstallability of libglade2-dev.

On 8/4/20 2:49 PM, Dirk Eddelbuettel wrote:
> 
> Hi doko and Python folks,
> 
> On 4 August 2020 at 09:29, Matthias Klose wrote:
> | Package: src:rgtk2
> | Version: 2.20.36-2
> | Severity: serious
> | Tags: sid bullseye
> | User: debian-python@lists.debian.org
> | Usertags: py2unversioned
> | 
> | Python2 becomes end-of-live upstream, and Debian aims to remove
> | Python2 from the distribution, as discussed in
> | https://lists.debian.org/debian-python/2019/07/msg00080.html
> | 
> | We will keep some Python2 package as discussed in
> | https://lists.debian.org/debian-python/2020/07/msg00039.html
> | but removing the unversioned python packages python-minimal, python,
> | python-dev, python-dbg, python-doc.
> | 
> | Your package either build-depends, depends on one of those packages.
> | Please either convert these packages to Python3, or if that is not
> | possible, replaces the dependencies on the unversioned Python
> | packages with one of the python2 dependencies (python2, python2-dev,
> | python2-dbg, python2-doc).
> | 
> | Please check for dependencies, build dependencies AND autopkg tests.
> 
> I am lost.
> 
> Build-Depends:
>   Source: rgtk2
>   Section: gnu-r
>   Priority: optional
>   Maintainer: Dirk Eddelbuettel 
>   Build-Depends: debhelper (>= 10), dh-r (>= 20180615), r-base-dev (>= 
> 3.6.1), libgtk2.0-dev (>= 2.10.12), libglade2-dev, libpango1.0-dev, 
> libcairo2-dev
>   Standards-Version: 4.4.0
>   Vcs-Browser: https://salsa.debian.org/edd/r-cran-rgtk2
>   Vcs-Git: https://salsa.debian.org/edd/r-cran-rgtk2.git
>   Homepage: https://cran.r-project.org/package=RGtk2
> 
> Build-Depends: debhelper (>= 10), dh-r (>= 20180615), r-base-dev (>= 3.6.1), 
> libgtk2.0-dev (>= 2.10.12), libglade2-dev, libpango1.0-dev, libcairo2-dev
> 
> Depends:
>   Package: r-cran-rgtk2
>   Source: rgtk2 (2.20.36-2)
>   Version: 2.20.36-2+b1
>   Installed-Size: 18722
>   Maintainer: Dirk Eddelbuettel 
>   Architecture: amd64
>   Depends: r-base-core (>= 4.0.0-3), r-api-4.0, libatk1.0-0 (>= 1.12.4), 
> libc6 (>= 2.14), libcairo2 (>= 1.8.0), libgdk-pixbuf2.0-0 (>= 2.22.0), 
> libglib2.0-0 (>= 2.41.1), libgtk2.0-0 (>= 2.20.0), libpango-1.0-0 (>= 
> 1.25.5), libpangocairo-1.0-0 (>= 1.22.0)
>   Description-en: GNU R binding for Gtk2
>   [...]
> 
> Where is python coming in from, and what could I change?  Is it libglade2 ?
> 
> If we cannot change and must remove it that would be, I guess, defensible.
> The package and the related ones are a little end-of-life-ish, but have not
> yet been removed from CRAN either so there is need for imminent action for
> removal. We could remove it but that is likely to annoy a user or two, and
> "as users are our priority" I suggest to keep it ... but I need a helping
> hand here.
> 
> | If there are questions, please refer to the wiki page for the removal:
> | https://wiki.debian.org/Python/2Removal, or ask for help on IRC
> | #debian-python, or the debian-python@lists.debian.org mailing list.
> 
> d-p CC'ed.
> 
> Thanks,  Dirk
> 



Re: The python command in Debian

2020-07-13 Thread Matthias Klose
On 7/13/20 6:23 PM, Fabrice BAUZAC-STEHLY wrote:
> Hi,
> 
> Another solution would be to simply use the update-alternatives system
> to manage /usr/bin/python.  python3 would have a higher priority than
> python2.  Users would still have the possibility to switch
> /usr/bin/python to python2 explicitly if they require it...

No, never ever. update-alternatives cannot be used because it breaks the
dependency system.



The python command in Debian

2020-07-09 Thread Matthias Klose
As written in [1], bullseye will not see unversioned python packages and the
unversioned python command being built from the python-defaults package.

It seems to be a little bit more controversial what should happen to the python
command in the long term.  Some people argue that python should never point to
python3, because it's incompatible, however Debian will have difficulties to
explain that decision to users who start with Python3 and are not aware of the 2
to 3 transition.  So yes, in the long term, Debian should have a python command
again.

One solution could be not to ship the python command in bullseye, forcing users
to adjust their local installations.  This has the advantage that the error of
an unknown interpreter should be pretty clear.  But leaves users without a
python command for the next two years until bullseye+1.

Describing here a solution which is implemented for Ubuntu focal (20.04 LTS).  A
new source package what-is-python (-perl-dont-hurt-me) ships binary packages
python-is-python2, python-dev-is-python2, python-is-python3 and
python-dev-is-python3.  The python-is-python2 package provides the python
package, such that packages that still depend on python are not removed on a
distro upgrade.  On new installs, python-is-python3 is not installed by default,
but the user gets a hint from command-not-found to install the package if he
tries to run python.  Package dependencies on the new four binary packages have
to be disallowed in the Python policy.  Note that such a package including the
Provides should only be uploaded once all dependencies on the unversioned python
packages are gone.

Matthias

[1] https://lists.debian.org/debian-python/2020/07/msg00039.html
[2] https://launchpad.net/ubuntu/+source/what-is-python



Re: Python2 packages for bullseye

2020-07-09 Thread Matthias Klose
On 7/9/20 1:45 PM, Scott Kitterman wrote:
> On Thursday, July 9, 2020 7:21:45 AM EDT Matthias Klose wrote:
>> The removal of packages still depending on Python2 looks good [1], however
>> we have a bunch of packages that still require Python2, and where
>> maintainers explicitly asked to keep those in the distro [2].  Among those
>> are pypy and pypy3 which need Python2 for bootstrapping.  I'm going to keep
>> the Python2 packages for bullseye, and having those just as build
>> dependencies shouldn't really effect any end-user.  A different thing might
>> be the Python2 usage at runtime, however I'm not very passionate to remove
>> all of those packages.
>>
>> What still should be done for bullseye is the removal of the unversioned
>> python. I'm removing the packages python-minimal, python, python-dev,
>> python-dbg, python-doc, and stop shipping the /usr/bin/python symlink, so
>> that packages are required to either use python2 or python3 explicitly. 
>> Planning that change for late August / early September.  That should give
>> plenty of time to address any unversioned python usage before the release
>> freeze.
> 
> Are you going to keep python-setuptools?  If you do, it seems likely we'll be 
> able to keep pip and virtualenv so they support running python2 in a 
> virtualenv in bullseye, which seems the best way to do it for those that need 
> to.

yes, that would be a sensible thing to do.



Python2 packages for bullseye

2020-07-09 Thread Matthias Klose
The removal of packages still depending on Python2 looks good [1], however we
have a bunch of packages that still require Python2, and where maintainers
explicitly asked to keep those in the distro [2].  Among those are pypy and
pypy3 which need Python2 for bootstrapping.  I'm going to keep the Python2
packages for bullseye, and having those just as build dependencies shouldn't
really effect any end-user.  A different thing might be the Python2 usage at
runtime, however I'm not very passionate to remove all of those packages.

What still should be done for bullseye is the removal of the unversioned python.
I'm removing the packages python-minimal, python, python-dev, python-dbg,
python-doc, and stop shipping the /usr/bin/python symlink, so that packages are
required to either use python2 or python3 explicitly.  Planning that change for
late August / early September.  That should give plenty of time to address any
unversioned python usage before the release freeze.

Matthias


[1]
http://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=py2removal;users=debian-python@lists.debian.org
[2]
http://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=py2keep;users=debian-python@lists.debian.org



Timing of Python upstream and Debian releases

2020-07-06 Thread Matthias Klose
Starting with Python 3.8, Python upstream changed to a time based yearly release
schedule, targeting the first release of a major Python version (3.x) for
October of each year.  For the transition to 3.8:

 - we add 3.8 as supported in November
 - made 3.8 the default in March
 - dropped 3.7 in April

That's a little bit late looking at the Debian release schedule, so a little
speedup would be needed if we want to target the current Python release for the
current Debian release.  For 3.8 Ubuntu started to prepare the switch a bit
earlier, using the results for a better experience in Debian:

 - added 3.8 in mid October
 - made 3.8 the default in mid December

Technically it would be possible to do the defaults change before the Debian
freeze, however there's usually a tail of follow-up upstream releases required
to support a new major Python version, unless you opt for actively backporting
changes in various packages.  Having the confidence, that the switch is feasible
in another distro certainly helps doing the transition in Debian, however it
adds a delay.  I don't think it's feasible to do the transition in experimental
first, or doing a large test rebuild, because it requires keeping the whole
python stack in sync with testing/unstable.

So what I'm proposing here is to aim to support 3.9 as early as possible as a
supported Python3 version, starting with the 3.9 upstream release, and fixing
stuff on the go.  Then decide in November, if we can do the defaults change to
3.9, or if we drop 3.9 again, or ship with two supported Python3 versions.

Please note this will be a re-occuring situation with Python 3.11 and
bullseye+1, so we should find out how to handle this on a regular basis,
assuming that Debian release schedules won't change much.

Matthias



Python3 -dbg packages

2020-07-06 Thread Matthias Klose
Python 3.8 upstream now has a common ABI for normal and debug extension builds,
so it is technically possible to load a debug extension in the normal
interpreter, or to load a normal extension in the debug interpreter.  In Debian,
debug extensions are shipped with a different name, and only loaded by the
corresponding interpreter.  We could change / simply the current setup, but I
first wanted to know how many people are still using the debug builds.  The
reason for the separate debug builds allowed debugging of stuff in modules
further down the Python stack, without having to rebuild the whole stack. There
are several solutions how to simplify the packaging, I'm not sure how much the
dbg extensions are still used ... There are several scenarios:

 - Keep the current setup (-dbg packages need to be available to
   run them).

 - Allow the debug interpreter to load normal debug extensions (but
   load a debug extension if it's available by default).  That would
   allow building debug extensions without having debug extensions
   built for all it's dependencies, maybe requiring changes in the
   dependencies of a package.

 - Stop building debug extensions, and telling developers to
   build extensions in debug mode, if they need them.  That would
   probably be inline with everything else shipped in Debian.

 - Stop building debug extensions, and also stop building the Python
   debug interpreter.  You would need to rebuild the interpreter
   itself to have meaningful debug sessions.  I'm not preferring
   this solution.

I'm currently tending to implement the second scenario, but if people think that
having the -dbg packages available is still useful, then also opt for the third
option.

Matthias



Re: Python Wheel Related Policy Change

2020-04-30 Thread Matthias Klose
On 4/30/20 5:28 AM, Scott Kitterman wrote:
> I think Python policy changes should be discussed.  I accidentally committed 
> a 
> change to git [1] (I didn't realize I still had access, I thought it would be 
> a merge request) to allow Python 3 only wheels for packages that require 
> wheels, but that no longer support Python 2.  This is going to happen the 
> next 
> time I upload python-pip since the current version of setuptools is python3 
> only.

this is not true. python-setuptools is still built from the source packages
python-setuptools.  Is there a reason that you cannot use it?

> Hopefully this won't be controversial.  There's really no way to avoid it.

well, better don't assume that.

Matthias



Re: Python3 modules not built for all supported Python versions

2020-03-30 Thread Matthias Klose
On 3/30/20 1:24 PM, Emilio Pozuelo Monfort wrote:
> Hi,
> 
> We've just finished the transition to python3.8 as the default python3
> interpreter, which was a bit difficult due to some autopkgtest regressions in 
> a
> few rdeps, and to the fact that many modules only build their extensions for 
> the
> default python version, which means they have a strict dependency on the 
> python3
> version[1] and they need to be rebuilt and migrated in lockstep with
> python3-defaults.
> 
> I have analyzed this based on current sid amd64 contents and have come up with
> the following packages that don't ship extensions for both py3.7 and 3.8 
> (which
> are the currently supported versions). Note that pure python packages that 
> don't
> build C extensions are not affected.
> 
> It would be great if this situation can be improved in order to help with 
> future
> python transitions. Building for all the supported python versions can be done
> by build-depending on python3-all-dev and compiling your package (or just the
> python bits) with PYTHON pointing to each version. Depending on your package's
> build system, this could be largely automated using some helper, such as
> pybuild. If you don't know how to add support for your package, feel free to 
> ask.

Yes, that would be useful.  However there are only limited time windows when you
can do *and* check such work, i.e. when the python3-defaults package has two
supported python3 versions.  Filing and fixing these issues would be nice.

Plus for most of those packages you have to modify the build system to build the
package for each python3 version, or to just build the extension for the
non-default version.

I assume we still could keep 3.7 as a supported python3 version for some time,
if people want to work on those issues.  But people probably won't have the time
to work on that when we introduce 3.9.

Matthias



Re: Bug#949187: transition: python3.8

2020-02-03 Thread Matthias Klose
On 2/3/20 8:22 PM, Simon McVittie wrote:
> On Sun, 02 Feb 2020 at 09:35:04 +0100, Matthias Klose wrote:
>> I think this is now in shape to be started.
> 
> Please can this wait until the remaining bits of the libffi7 transition
> and the restructuring of the libgcc_s packaging have settled down?
> 
> I'm still trying to sort out the missing Breaks around
> gobject-introspection, as highlighted by autopkgtest failures: this has
> been delayed by needing coordinated action between multiple packages,
> some of them quite big (glib2.0), and by Paul and I being at FOSDEM.
> This is entangled with python3.8 via pygobject (which will fail tests
> with python3.8 as default - an upload is pending to fix that).

fine. I'm happy to see that fixed.

> Meanwhile, multiple packages seem to FTBFS on s390x with the new libgcc_s
> (I've just opened the bug for that, so no bug number known yet), which is
> going to limit the ability to get things into testing.

please retry your package builds. it's fixed in unstable.



Re: Bug#949187: transition: python3.8

2020-02-02 Thread Matthias Klose
On 2/2/20 5:53 PM, Rene Engelhard wrote:
> On Sun, Feb 02, 2020 at 09:35:04AM +0100, Matthias Klose wrote:
>>> On 17-01-2020 23:28, Matthias Klose wrote:
>>>> Please add a transition tracker to switch the python3 default to 3.8.  
>>>> It's not
>>>> yet ready, however it would be good to see affected packages. Please copy 
>>>> it
>>>> from the 3.7 defaults change if possible.
>>>
>>> Tracker is in place. Please remove the moreinfo tag when you're ready.
>>
>> I think this is now in shape to be started. bug reports have been filed for
> 
> I don't think so.
> 
> e.g. fontforge is still red in
> https://release.debian.org/transitions/html/python3.8.html.
> 
> That means that a rebuild of stuff using fontforge in the build will
> just FTBFS since it will be called with python3.8 and that has no module
> for 3.8 (yet).
> 
> (Seen in a python-3.8-as-default testbuild for libreoffice some time
> ago.)

you are wrong. fontforge just builds for the default python version.



Re: Bug#949187: transition: python3.8

2020-02-02 Thread Matthias Klose
Control: tags -1 - moreinfo

On 1/18/20 9:30 PM, Paul Gevers wrote:
> Control: tags -1 moreinfo
> 
> Hi Matthias,
> 
> On 17-01-2020 23:28, Matthias Klose wrote:
>> Please add a transition tracker to switch the python3 default to 3.8.  It's 
>> not
>> yet ready, however it would be good to see affected packages. Please copy it
>> from the 3.7 defaults change if possible.
> 
> Tracker is in place. Please remove the moreinfo tag when you're ready.

I think this is now in shape to be started. bug reports have been filed for
issues with 3.8 in [1], and there may be some which are missing the proper
tagging.  I also filed bug reports for a dozen missing dh-python build 
dependencies.

I would prefer a transition slot with less activity with transitions in the
science area, like opencv, openmpi, gnuradio, ros*, and others which only build
extensions for the default python3 version.

Matthias

[1]
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=python3.8;users=debian-python@lists.debian.org



Re: Updating pip

2020-01-24 Thread Matthias Klose
On 24.01.20 20:00, Scott Kitterman wrote:
> I started looking into updating pip to the current release

thanks for doing that.

> packaging

the recent version is in the archive but ftbfs.  it's a dh-python issue.

Matthias



Re: dh_python3 sets shebang to Python 2 -- is this a bug?

2020-01-23 Thread Matthias Klose
On 22.01.20 02:18, Nicholas D Steeves wrote:
> I don't think this is an assumption.  Debian adheres to PEP 394, which
> until recently said that /usr/bin/python is supposed to be Python 2, for
> backwards compatibility.  When I discovered this I felt increased trust
> in Debian and its developers for doing the sensible and standardised
> thing, rather than what some other distributions have done.  To everyone
> involved, thank you!  This also made me proud to be part of such a
> community :-)

well, looking a few years forward, Debian will be the only system not providing
a python command, incompatible with everybody else.  New users will not
understand that Debian doesn't have a python command when there's no Python2
anymore.

What we will do is to get rid off the unversioned python for all the packaging
in the archive, so that users can do what they want.

Matthias



Re: python-datrie: FTBFS with recent hypothesis version

2019-12-09 Thread Matthias Klose
On 09.12.19 11:46, Andreas Tille wrote:
> Hi,
> 
> I have upgraded python-datrie in Git[1] to latest upstream version
> (0.8).  It shows the same issue - so I admit I have no better clue than
> reporting the issue upstream which I'd rather leave to the official
> Uploader of the package.
> 
> BTW, we probably should expect build issues with Python 3.8[2]

https://patches.ubuntu.com/p/python-datrie/python-datrie_0.7.1-3ubuntu1.patch

didn't forward the hypothesis issues, because Debian didn't update that package
for a long time.


> Kind regards
> 
>  Andreas.
> 
> [1] https://salsa.debian.org/python-team/modules/python-datrie
> [2] https://github.com/pytries/datrie/issues/73
> 



Re: Severity bump script

2019-12-03 Thread Matthias Klose
On 02.12.19 20:28, Paul Gevers wrote:
> Hi all,
> 
> On 01-12-2019 22:45, Sandro Tosi wrote:
>> Paul, this is the thread i was talking about.
>>
>> you were copied in the original email:
>> https://lists.debian.org/debian-python/2019/10/msg00098.html
>>
>> if there is something the RT wants to discuss about this effort,
>> please do so here, not directly to me (i may be the mail address
>> sending those control commands, but the decision was taken here).
> 
> I understand the drive to push for Python 2 removal and sympathize with
> it. The issue I had yesterday with the process is that "leaf" was
> wrongly defined, it was looking at Depends, instead of also including
> Build-Depends.

that should be fixed.

> I don't want to stand in the way of Python 2 removal, but as I said
> before, pulling packages out from underneath maintainers isn't pretty so
> needs to be done with proper notifications and care. An RC bug to ones
> own package is acceptable in my opinion as it is a clear discussion
> forum, and can be (temporarily) down-graded while the discussion is
> ongoing. Being notified about some other package that I not even need
> directly but is going to pull "my" package out of testing isn't a nice
> experience and should be avoided. Yes, that slows down the process, but
> there are people, emotions and all those irrational things involved.

It's unfortunate that issues for some packages only get attention when the
severity of an issue is raised.  Following your proposal means that the issue is
probably ignored forever, and you don't propose a better way going forward, just
saying we should stop earlier.  I don't think that's the correct choice.  For
now these seem to be single packages, so please could you name those, and we can
look at those with a priority?  That's at least a path that is forward looking,
or feel free to propose another approach better than your current proposal for
not getting the attention of maintainers.

Matthias

PS: There's a RC issue for creduce now, not caused by the package itself, should
I downgrade it?



Re: Python 3.8 transition: CMake problems

2019-11-26 Thread Matthias Klose
On 26.11.19 11:02, Ole Streicher wrote:
> Hi,
> 
> I am currently updating the "casacore" package to the latest upstream
> version. Doing this, I discovered that it is not marked as part of the
> Python 3.8 transition, and was also not binNMUed for this, although it
> has the package libcasa-python3-3 (to be renamed to libcasa-python3-4)
> that depends on libpython3.7. Was this somehow forgotten?

No. We are currently adding Python3.8 support, that's not yet switching the
default.  And the tracker only tracks packages b-d on python3-all-dev, which
casacore doesn't match, only building for the default Python3.

Matthias



Re: Bug#942106: Adding Python 3.8 as a supported Python3 version

2019-11-14 Thread Matthias Klose
On 12.11.19 23:39, Matthias Klose wrote:
> On 07.11.19 15:08, Matthias Klose wrote:
>> This weekend, I am planning to upload python3-defaults, adding python3.8 as a
>> supported Python3 version.  This may introduce some churn in unstable until 
>> the
>> basic binNMUs are available as well.
>>
>> Details for the addition can be found at [1], known issues and patches are 
>> filed
>> [2].  There was no test rebuild in Debian itself, but the addition was made 
>> in
>> the current Ubuntu development series.  Things look good, and from my point 
>> of
>> view don't block any unrelated transition work.  python3-defaults will get a 
>> RC
>> issue to stay in unstable until the packages build-depending on 
>> python3-all-dev
>> are built, and after doing a test rebuild with the two supported Python3
>> versions.  Earlier test rebuilds don't make sense.
>>
>> There are a few packages, where the upstream versions used in Ubuntu diverge
>> from the ones currently in unstable (not naming those already updated in
>> unstable by the DPMT):
>>
>>  - hypothesis #942693, not sure if this is really needed,
>>    the testsuite stack might be fixed by the new pluggy
>>    version as well.
>>
>>  - python-xarray #944044. 1.4 is already broken with Python 3.7. For
>>    Ubuntu I fell back to 1.2.1. 1.2.3 might work as well, still seeing
>>    one or two test failures.
>>
>>  - Using numpy from experimental, and only building the Python2 numpy
>>    packages from the python-numpy source.
>>
>>  - Using python-scipy from experimental, building the Python3 packages,
>>    and renaming the python-scipy package to python2-scipy, only building
>>    the Python2 packages.
>>
>>  - matplotlib and pandas don't have Python2 packages in Ubuntu
>>    anymore, so I can't tell much what is needed here. Pandas needs
>>    a new upstream for 3.8.
>>
>> Packages building on top of numpy/scipy/pandas, like the PyAstro stack, 
>> continue
>> to work with these updates, despite some new test failures.
> 
> So this upload didn't happen, but thanks to Rebecca we now have an almost
> Python2 free pandas in unstable. And apologies to the science team for the 
> 1-day
> delayed NMUs for patsy and scikit-learn.
> 
> Now planning the python3-defaults upload for Thursday or Friday.

python3-defaults with 3.8 as a supported Python3 version is now built in
unstable.  Release team, please schedule binNMUs fpr stage1 and then stage2.
I'll raise severity of the 3.8 issues, and follow-up with 2-days delayed NMUs.

Matthias



Re: Adding Python 3.8 as a supported Python3 version

2019-11-12 Thread Matthias Klose
On 07.11.19 15:08, Matthias Klose wrote:
> This weekend, I am planning to upload python3-defaults, adding python3.8 as a
> supported Python3 version.  This may introduce some churn in unstable until 
> the
> basic binNMUs are available as well.
> 
> Details for the addition can be found at [1], known issues and patches are 
> filed
> [2].  There was no test rebuild in Debian itself, but the addition was made in
> the current Ubuntu development series.  Things look good, and from my point of
> view don't block any unrelated transition work.  python3-defaults will get a 
> RC
> issue to stay in unstable until the packages build-depending on 
> python3-all-dev
> are built, and after doing a test rebuild with the two supported Python3
> versions.  Earlier test rebuilds don't make sense.
> 
> There are a few packages, where the upstream versions used in Ubuntu diverge
> from the ones currently in unstable (not naming those already updated in
> unstable by the DPMT):
> 
>  - hypothesis #942693, not sure if this is really needed,
>    the testsuite stack might be fixed by the new pluggy
>    version as well.
> 
>  - python-xarray #944044. 1.4 is already broken with Python 3.7. For
>    Ubuntu I fell back to 1.2.1. 1.2.3 might work as well, still seeing
>    one or two test failures.
> 
>  - Using numpy from experimental, and only building the Python2 numpy
>    packages from the python-numpy source.
> 
>  - Using python-scipy from experimental, building the Python3 packages,
>    and renaming the python-scipy package to python2-scipy, only building
>    the Python2 packages.
> 
>  - matplotlib and pandas don't have Python2 packages in Ubuntu
>    anymore, so I can't tell much what is needed here. Pandas needs
>    a new upstream for 3.8.
> 
> Packages building on top of numpy/scipy/pandas, like the PyAstro stack, 
> continue
> to work with these updates, despite some new test failures.

So this upload didn't happen, but thanks to Rebecca we now have an almost
Python2 free pandas in unstable. And apologies to the science team for the 1-day
delayed NMUs for patsy and scikit-learn.

Now planning the python3-defaults upload for Thursday or Friday.

Matthias



Re: Python 2 removal in sid/bullseye: Progress and next steps

2019-11-11 Thread Matthias Klose

On 11.11.19 11:43, Yves-Alexis Perez wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On Mon, 2019-11-11 at 10:33 +0100, Ondřej Nový wrote:

We are going to raise the severity of the py2removal bugs to "serious"
in several steps.  In the
first phase we are going to raise severity of the py2removal bugs for
all leaf module packages and low popcon (< 300) application packages.
Bugs marked with the "py2keep" user tag will not have their severity
raised.  If nobody fixes that bug, the packages will be auto-removed
from testing.
We will also then file bug reports against ftp.debian.org to remove
such packages from unstable.  We are going to do this semi-
automatically as additional packages become leaf packages.


Hi,

sorry if this has been discussed already somewhere else (I stopped reading
- -devel@ a long time ago) but is there something done to improve NEW processing
here? I have two package sitting there because of the introduction of a
- -python3 binary package. I don't think it's really make sense to remove
python2 stuff if python3 stuff can't enter the archive…


well, at least you can name the packages in your email ...



Adding Python 3.8 as a supported Python3 version

2019-11-07 Thread Matthias Klose
This weekend, I am planning to upload python3-defaults, adding python3.8 as a 
supported Python3 version.  This may introduce some churn in unstable until the 
basic binNMUs are available as well.


Details for the addition can be found at [1], known issues and patches are filed 
[2].  There was no test rebuild in Debian itself, but the addition was made in 
the current Ubuntu development series.  Things look good, and from my point of 
view don't block any unrelated transition work.  python3-defaults will get a RC 
issue to stay in unstable until the packages build-depending on python3-all-dev 
are built, and after doing a test rebuild with the two supported Python3 
versions.  Earlier test rebuilds don't make sense.


There are a few packages, where the upstream versions used in Ubuntu diverge 
from the ones currently in unstable (not naming those already updated in 
unstable by the DPMT):


 - hypothesis #942693, not sure if this is really needed,
   the testsuite stack might be fixed by the new pluggy
   version as well.

 - python-xarray #944044. 1.4 is already broken with Python 3.7. For
   Ubuntu I fell back to 1.2.1. 1.2.3 might work as well, still seeing
   one or two test failures.

 - Using numpy from experimental, and only building the Python2 numpy
   packages from the python-numpy source.

 - Using python-scipy from experimental, building the Python3 packages,
   and renaming the python-scipy package to python2-scipy, only building
   the Python2 packages.

 - matplotlib and pandas don't have Python2 packages in Ubuntu
   anymore, so I can't tell much what is needed here. Pandas needs
   a new upstream for 3.8.

Packages building on top of numpy/scipy/pandas, like the PyAstro stack, continue 
to work with these updates, despite some new test failures.


Matthias

[1] https://wiki.debian.org/Python/Python3.8.
[2] 
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=python3.8;users=debian-python@lists.debian.org




Re: Bug#943666: python3: Update Python Policy for removal of the Python 2 stack

2019-11-06 Thread Matthias Klose

On 06.11.19 22:04, Nicholas D Steeves wrote:

Brian May  writes:


Stéphane Blondon  writes:


Perhaps there is a doubt how to read it?
- do not (remove python-foo-doc or rename it to python3-foo-doc)
- (do not remove python-foo-doc) or (rename it to python3-foo-doc)

Would it be better if we remove the indentation and use this sentence(?):
if documentation is in python-foo-doc, do not move it


Myself, I read it as the first option.

I would personally use:

- Do not remove python-foo-doc and do not rename it to python3-foo-doc.

Or maybe even expand as two bullet points:

- Do not remove python-foo-doc.
- Do not rename it to python3-foo-doc.

I think this makes it very explicit what was intended.


+1.  I also read it as (do (not (remove python-foo-doc) or (not (rename
to python3-foo-doc.  In natural language that "or" should be a
"nor", but breaking it into two negated bullet points may be clearer to
those whose first language doesn't possess a negative list operator.


please could one of you open an issue with a patch to track the change?

Thanks, Matthias



Re: Bug#943666: python3: Update Python Policy for removal of the Python 2 stack

2019-11-03 Thread Matthias Klose

On 03.11.19 15:09, Neil Williams wrote:

On Sun, 3 Nov 2019 15:00:17 +0100
Matthias Klose  wrote:


[discussing this outside the bug report on the ML]

On 03.11.19 14:39, Neil Williams wrote:

Actually, that's a good catch. I was mixing up the defaults package
with the general advice on python3 migration to not remove
python-foo-doc just to rename it to python3-foo-doc.


where did you read that? IMO we don't want to rename the -doc
packages.


https://wiki.debian.org/Python/2Removal

Check list

* if documentation was in python-foo - move it to python3-foo or python-foo-doc
   * if it was automatically placed there by dh_installdocs, it will 
automatically move to python-foo-doc: you don't need to do anything, unless you 
had links to it that need un-breaking
   * do not remove python-foo-doc or rename it to python3-foo-doc


yes, but this tells you not to rename it to python3-foo-doc.



Re: Bug#943666: python3: Update Python Policy for removal of the Python 2 stack

2019-11-03 Thread Matthias Klose

[discussing this outside the bug report on the ML]

On 03.11.19 14:39, Neil Williams wrote:

Actually, that's a good catch. I was mixing up the defaults package
with the general advice on python3 migration to not remove
python-foo-doc just to rename it to python3-foo-doc.


where did you read that? IMO we don't want to rename the -doc packages.


Removing the defaults package python-doc seems right to me. I've
updated that paragraph.




Re: Python module packages that don't bytecompile on installation?

2019-11-03 Thread Matthias Klose

On 03.11.19 02:20, Paul Wise wrote:

On Sat, Nov 2, 2019 at 6:04 PM Matthias Klose wrote:


At this point I'd ignore any Python2 related package, and concentrate on Python3
stuff only.


Yes, I was referring only to python3-* module packages.


Byte compilation is an optimization, speeding up a program start if
byte-compiled files are ready.  Packages are smaller without bc, take a bit
longer to install.  I think nobody is asking the question if .py files should be
dropped in favor of .pyc files.


As I understand it, the .pyc files are interpreter version specific
and doing a transition for each major Python update was deemed to be
too much work, so instead we offload building the .pyc files onto
user's systems.


yes, that's another reason. Not having to touch binary indep packages during a 
transition.



I'd say, there are currently more pressing issues than that, like the Python2
removal, or the introduction of Python 3.8.  3.8 also offers a central directory
for bc files, so that's maybe another thing to look at, but not a priority now
from my point of view.


Agreed re Python 2 etc. Eventually switching to using a central
directory in /var would be nice, right now the .pyc files are dumped
cruftily into /usr subdirs.


yes, but that's something when 3.7 is removed.

Matthias



Re: Python module packages that don't bytecompile on installation?

2019-11-02 Thread Matthias Klose

On 02.11.19 04:22, Paul Wise wrote:

Hi all,

I run adequate on my system, which means I notice when Python module
packages don't bytecompile when they are installed. So far I've just
been ignoring the warnings that adequate prints.

https://salsa.debian.org/debian/adequate

In addition I noticed:

  * some Python modules on my system weren't bytecompiled but did
bytecompile when I reinstalled them (blinker for example)
  * many of the Python modules on my system don't bytecompile their test
directories (django for example)
  * relatively few packages I have installed don't bytecompile when I
ignore test paths (lib2to3, tk, uno, distutils)

Some questions:

Should all module packages bytecompile?


At this point I'd ignore any Python2 related package, and concentrate on Python3 
stuff only.



Should all module packages bytecompile all their directories?

What are the downsides when module packages don't bytecompile?

What are the downsides when module packages do bytecompile?


Byte compilation is an optimization, speeding up a program start if 
byte-compiled files are ready.  Packages are smaller without bc, take a bit 
longer to install.  I think nobody is asking the question if .py files should be 
dropped in favor of .pyc files.



Do we need a lintian complaint about this issue?

Do we need any improvements in how module packages bytecompile?
For example using triggers instead of postinst scriptlets.

Any other thoughts?


I'd say, there are currently more pressing issues than that, like the Python2 
removal, or the introduction of Python 3.8.  3.8 also offers a central directory 
for bc files, so that's maybe another thing to look at, but not a priority now 
from my point of view.


Matthias



Re: Bug#936214: bleachbit: Python2 removal in sid/bullseye

2019-11-02 Thread Matthias Klose

On 02.11.19 09:05, Hugo Lefeuvre wrote:

Hi Matthias,

I see that you just raised the severity of this bug to serious, and
Bleachbit is now to be removed on 16.11.

I don't think this is the way to go. Upstream is actively working on this.
We have recently managed the GTK3 migration, meaning that Py3 is now top
priority.  Loosing Bleachbit would be a significant source of annoyance for
many Debian users (popcon 2754 at the moment).

May I add the py2keep flag, until the Bleachbit Py3 migration completes?


Hi,

thanks for reaching out to debian-python.  Unfortunately the pygtk py2removal 
bug (#937452) didn't see much attention, and I didn't raise the severity of this 
issue myself, just for the r-deps affecting the Python2 removal.  Pygtk will 
never be ported to Python3, so action is needed for every rdep.


You won't loose Bleachbit just because it's removed from testing for a few 
weeks/months, and I don't see any rdeps for the package itself.  Using the 
py2keep tag for packages which have a transition plan in place for bullseye 
seems to be odd, and distracts the view for packages which we apparently need to 
keep.


Lowering the severity for the bleachbit issue might be a work around, but then 
unexpected pygtk removal may bite you again.


Matthias



packages with the py2keep tag

2019-11-01 Thread Matthias Klose

https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=py2keep;users=debian-python@lists.debian.org

doesn't show show too many packages yet, which is good. Not sure if we should 
tag the dependencies of these packages with a different tag, e.g. py2keep-dep 
for python-setuptools.


Im unsure if I understand the rationale for the mercurial py2keep tag.



Re: Bug#942106: (some kind of) transition: add python3.8 as a supported python3 version

2019-10-26 Thread Matthias Klose

On 26.10.19 22:09, Rebecca N. Palmer wrote:
What should be done with modules where Python 3.8 compatibility requires moving 
to a new upstream release that doesn't support Python 2, but the Python 2 
package still has dependencies (so can't be removed yet under existing rules)?


- Split them into two source packages with different upstream versions, as was 
done for matplotlib and numpy?

- Remove the Python 2 package anyway?
- Let them be broken in Python 3.8 for now?

e.g. pandas dropped python2 support in 0.25.0, and gained python3.8 support in 
0.25.2:

https://github.com/pandas-dev/pandas/issues/29043


yes, that will be an ongoing problem, I see the same for pillow (latest 2.7 
supporting release is 6.2.1) and numpy (1.16 not supporting 3.8, and 1.17 not 
supporting 2.7).


Ubuntu got pandas 0.23 to build with python3.8, but only by ignoring 268 test 
failures (I haven't yet had time to assess their severity):

https://bugs.launchpad.net/ubuntu/+source/pandas/+bug/1849374
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/amd64/p/pandas/20191024_181815_7c017@/log.gz 


yes, https://bugs.launchpad.net/bugs/1849374 documents where I ignored test 
results for a first build, and numpy test results are ignored as well due to a 
packaging bug.


Ubuntu already dropped python-pandas, I wasn't involved with that. So this 
should be possible to do.  Please ask Steve Langasek for details. In the case 
for pandas it should be possible to remove it now with some work, avoiding a 
second Pandas source.


Having a first build in the archive allows you to get more packages built, and 
more people working on the stack. For example the whole astropy stack builds and 
passes tests (except astropy itself). So there is value. Lets enable to build 
stuff first for 3.8 as a supported non-default option.




dh-python now generates dependencies on python2 instead of python

2019-10-22 Thread Matthias Klose
For Python2 packages, dh-python 4.20191017 now rewrites any python shebang to 
/usr/bin/python2 and generates dependencies on python2 instead of python.


The py2removal bugs have a section

"""
- If the package has still many users (popcon >= 300), or is needed to
  build another package which cannot be removed, document that by
  adding the "py2keep" user tag (not replacing the py2remove tag),
  using the debian-python@lists.debian.org user.  Also any
  dependencies on an unversioned python package (python, python-dev)
  must not be used, same with the python shebang.  These have to be
  replaced by python2/python2.7 dependencies and shebang.
"""

As seen with the recent python-crypto upload, the package now has no dependency 
on python anymore in the binary packages, and it doesn't have any python, 
python-dev build-dependencies anymore.


Also seen in python-crypto, the python2 autopkg test calls python, and fails 
because python is not a dependency of python-crypto anymore.  This needs fixing.


If we need to ship Python2 in bullseye, then the python, python-dev, python-dbg, 
python-doc, libpython-stdlib, libpython-dev binary packages will be removed 
before the release.


With that change, python-crypto is in a form to ship with bullseye (if 
python-crypto is still needed for some application that we want to keep). 
Application packages are likely to need a change in the build dependencies to 
ship with bullseye, if needed.




Discussing next steps for the Python2 removal

2019-10-22 Thread Matthias Klose
Paul Gevers from the release team pointed out that the Python2 removal is 
causing some uninstall-ability issues in testing because some packages 
apparently are removed too early, but never the less are migrating to testing. 
He suggested to make the removal plan more concrete and having a timeline.


So maybe people interested in joining the discussion and wanting to help could 
come to #debian-python on this Thu, Oct 24, 18:00 UTC.  If needed, with a 
follow-up on Sat, Oct 27, 18:00 UTC.


Topics could be:

 - filing removal bugs for packages which don't have a bug
   report yet.

 - when to raise bug severity for which bugs, even now
   for leaf packages?

 - bumping lintian info to warn, warn to error, ask ftp-master
   to autoreject packages for some errors?

If you cannot join, please leave some comments in the "Discussions/Proposals" 
section at https://wiki.debian.org/Python/2Removal




Re: Python2 removal: package with low-popcon reverse dependencies

2019-10-11 Thread Matthias Klose

On 11.10.19 18:27, Christian Kastner wrote:

Hi,

python-cachetools provides modules for Python2 and Python3.

The Python2 module as two reverse dependencies, both with low installed
popcon:
 python-cachetools:  302
 mopidy-podcast: 109
 mopidy-internetarchive:  95

This would nevertheless be a case for the "py2keep", right?


No.

#933348 is another bug for removed packages (mopidy-scrobler). Do you really 
want to keep that?  Is the mopidy popcon really enough to keep it?  Don't look 
at the popcon of any python module. The relevant popocon is the one for the 
application.



Would this also be the case if python-cachetools itself had a lower
popcon, say 50? Or is any form of dependency a case for "py2keep"?

Furthermore, the instructions in the Python2 removal request #937627 say:

Also any dependencies on an unversioned python package (python,
python-dev) must not be used, same with the python shebang.  These
have to be replaced by python2/python2.7 dependencies and shebang.


The binary package has versioned dependencies generating during the
build process, so there is nothing else to do, right?


No, this is not yet done. Pending a dh-python upload.



Re: should Debian add itself to https://python3statement.org ?

2019-09-12 Thread Matthias Klose

On 12.09.19 17:01, Ian Jackson wrote:

Drew Parsons writes ("should Debian add itself to https://python3statement.org  
?"):

https://python3statement.org/ is a site documenting the projects which
are supporting the policy of dropping Python2 to keep Python3 only.


That statement is a *pledge* to drop support for python2 by the end of
2020.  Have we in fact made such a pledge ?  I think I may have missed
the memo that python2 would be removed from bullseye.

I did some searching and found this
   https://lists.debian.org/debian-python/2019/07/msg00080.html
which is a sane-looking transition plan but doesn't seem to have a
timeframe and doesn't seem to contemplate removal of the actual
python2 interpreter.

FTAOD I don't have an opinion about whether bullseye *should* ship
without python2.  Obviously dropping it would not be desirable from
users' pov, but maintaining an ancient thing by ourselves would not be
desirable either.  I think I trust the Debian Python team to make that
tradeoff.

But we need to be clear what's going on and communicate early.  If
python2 is going out of bullseye then there are a lot of bugs that
should probably be marked rc fairly soon...


it's communicated here:
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=py2removal;users=debian-python@lists.debian.org

derived from
https://release.debian.org/transitions/html/python2-rm.html (not perfect, we are 
still missing bug reports).



it's way too early to mark all of these as RC.

No, it's not yet decided if Python2 will be part of bullseye.  But the python 
command and the unversioned python packages won't be part of bullseye.






thanks,
Ian.





Re: Bug#936613: ginac: Python2 removal in sid/bullseye

2019-09-10 Thread Matthias Klose

On 10.09.19 20:31, Richard B. Kreckel wrote:

On 10.09.19 11:29, Matthias Klose wrote:

Please read the instructions, they mention to check dependencies, build
dependencies, and test dependencies ...


I have read https://wiki.debian.org/Python/2Removal and the linked
pages. Are there any other instructions?


$ apt-cache showsrc ginac|grep Depends
Build-Depends: cdbs (>= 0.4.28), debhelper (>= 9), libcln-dev,
libgmp-dev, libreadline-dev, pkg-config (>= 0.18) | pkgconf,
dh-autoreconf, python, texinfo


That 'python' package it depends on is a dependency package. Isn't it
going to depend on Debian's Python3, so /usr/bin/python will mean Python
3, as e.g. in Fedora 31?


No. bullseye will not have a python package, and no python command.


If the python command is going to be python3, then I suppose we can
close this bug. (I've checked that the package can be built on a box
where python3 has been made the default.)

If the python command isn't going to be python3, then I'll update the
build-dependencies ASAP. And then this fact ought to be prominently
documented and explained in the instructions so to support this transition.


please do. and feel free to clarify the wiki page.

Thanks, Matthias



Re: Should python-cloudpickle get a py2keep tag?

2019-09-05 Thread Matthias Klose

On 05.09.19 19:50, Diane Trout wrote:

Hi,

The py2removal bug says to discuss the py2keep tag first.

src:cloudpickle is a dependency of src:spyder and src:skimage.

python-skimage has a popcon inst score of 469
spyder has a popcon inst score of 1385
spyder3 has a popcon inst score of 1069

The removal bug says the popcon threshold for py2keep is >= 300. So add
it?


you are asking about the least preferred option, without telling why you can't 
convert to python3, or why you can't remove the affected packages.  If you have 
both spyder and spyder3 (Python3?), then why not drop spyder?  I didn't look at 
skimage, but maybe look there first if it needs to be Python2.


Matthias



Re: Python 3 transition question

2019-09-01 Thread Matthias Klose

On 01.09.19 21:48, Martin Kelly wrote:

Hi,

I maintain python-gmpy and python-gmpy2, which need to transition to Python 3. 
However, they have several packages that have Suggests or Recommends (not a hard 
dependency) pointing to python-gmpy/python-gmpy2. These other packages appear to 
be Python 2 only.


Should I stop building the Python 2 versions of these packages (and invalidate 
the Suggests/Recommends of these other packages), or should I instead just 
document this issue? If I document it, where should this documentation go?


I would just stop building these.  And if the reverse dependencies have a 
py2removal bug itself, then comment in these issues that the 
suggested/recommended package gets removed.  If they don't have a py2removal 
bug, please file the bugs for these packages.




Re: updating mechanize - help concerning tests with pybuild

2019-08-31 Thread Matthias Klose

On 01.09.19 01:59, Norbert Preining wrote:

Hi Raphael,

@Matthias, please read on.

On Sat, 31 Aug 2019, Raphael Hertzog wrote:

https://salsa.debian.org/python-team/modules/python-mechanize


Thanks, that is perfect. I pushed my work there, changed control VCS,
maintainer, and uploader.

ATM I only put me into the uploaders. Please, those who are interested,
put yourself in there, thanks!


Do we go through package salvaging?
https://wiki.debian.org/PackageSalvaging


I don't think it's required here. The bugs have been open for long enough
without any activity.


Hmmm... I don't feel confident simply uploading someone's else package.
Best would probably be if Matthias Klose, one of the current Uploaders,
agrees to that and uploads the current package thus passing over
maintainership.

Matthias?


did you see my comment in #936270?



Filing py2removal bugs with correct attributes

2019-08-28 Thread Matthias Klose

Please file Python2 removal bugs with the appropriate attributes.

It's

Package: ftp.debian.org
Severity: normal
User: debian-python@lists.debian.org
Usertags: py2removal

 - It's NOT "Usertag".  Usertag is only recognized by the bug server.

 - It's NOT "py2-removal". It's "py2removal".

See https://wiki.debian.org/Python/2Removal for the reference.



Re: py2-rm: a few leaf packages to work on

2019-08-25 Thread Matthias Klose
On 24.08.19 07:03, Scott Kitterman wrote:
> On Thursday, August 15, 2019 8:08:41 AM EDT Thomas Goirand wrote:
>> Hi there!
>>
>> According to the daily graph I built here:
>> http://py2graph.infomaniak.ch/py2.7.deps.svg
>>
>> we can work on Python 2 removal for the below packages. Note that I have
>> *not* checked for reverse dependencies, please do so before working on a
>> package. The list isn't exhaustive at all, and didn't check if a package
>> is just a remaining curft, though it's hopefully still helpful as a TODO
>> list.
> 
> The arch all decruft is caught up now, so it ought to be easier now to tell 
> what still has rdepends left.

is this now done automatically, or just a manual effort?

Matthias



Re: py2-rm: a few leaf packages to work on

2019-08-25 Thread Matthias Klose
On 25.08.19 00:08, Thomas Goirand wrote:
> On 8/24/19 10:38 AM, Neil Williams wrote:
>> How is that graph turned into a list of packages? It's too large to
>> scan manually.
> 
> Well, I did it manually... and this is only a short list, as a
> suggestion for a todo list, so nothing exhaustive... I very much would
> welcome something automated.
> 
> BTW, working on this (as I've assigned myself to do a Python 2 support
> removal for at least one package a day), I've seen that lots of the
> packages are just bit-rotting stuff that has sometimes been uploaded
> only once, and that nobody cares about anymore. We should have spotted
> these earlier, IMO, and probably ping the maintainers.
> 
> I'm still waiting on Piotr (or someone else) to do the mail to -devel
> and the mass-bug-filling ... Any news on this?

I'm on the mass bug filing, but I need the tracker updated again.  And I need to
go through the suggestions here on the list how to work on things to make the
py2-rm issues more readable.  I didn't hear back from Piotr after DebConf, but I
think what we need is dh-python generating python2/python2.7 dependencies
instead of python dependencies, and maybe make shebang rewriting to
python2/python2.7 the default.

Matthias



Re: Investigating the reverse dependencies of python-monotonic.

2019-08-13 Thread Matthias Klose
On 13.08.19 15:30, peter green wrote:
> IMO python-monotonic should be reinstated until it's reverse dependencies are
> sorted out.

I agree with that.



Re: Re: dropping python2 [was Re: scientific python stack transitions]

2019-07-16 Thread Matthias Klose
On 16.07.19 17:31, Julien Puydt wrote:
> Le 16/07/2019 à 17:21, Andrey Rahmatullin a écrit :
>> Python 2 is included in buster and so will be supported for several
>> years.
>>
> 
> The starting point of the thread was :
> 1. Ok, buster has Python 2 so even if upstream drops it, we will still
> support it for the years to come for our users ;
> 2. but the next stable will come out after upstream has EOL-ed Python 2,
> so we must kick it out as soon as possible from testing+unstable,

> because there's no way we'll support Python 2 in that next stable!

no, that's your interpretation, not mine.  If an important enough application
still needs it, we should ship it.



Re: Plan for DebCamp

2019-07-16 Thread Matthias Klose
On 13.07.19 23:18, Emmanuel Arias wrote:
> Hello everybody,
> 
> Next week will start the DebCamp and then DebConf. I would like to know
> to any debian python team member :-)
> 
> Exist any plan for debian packaging during the DebCamp/DebConf?
> 
> Unfortunately, I will arrive to Curitiba on 18th (Thursday), So I can
> help remotly
> and well, participate on site from Thursday.

I'll be there from Wednesday.  My work time will splitted among Python 3.8 and
GCC 9 work ;)

Matthias



Re: dropping python2 [was Re: scientific python stack transitions]

2019-07-07 Thread Matthias Klose
On 07.07.19 16:55, Drew Parsons wrote:
> On 2019-07-07 22:46, Mo Zhou wrote:
>> Hi science team,
>>
>> By the way, when do we start dropping python2 support?
>> The upstreams of the whole python scientific computing
>> stack had already started dropping it.
> 
> Good question.  I think it is on the agenda this cycle, but debian-python will
> have the call on it.

you can start dropping it now, however please don't drop anything yet with
reverse dependencies.  So leaf packages first.

Matthias



Re: Command to query “Python versions that are installed *with* standard library”

2019-01-24 Thread Matthias Klose
On 24.01.19 00:16, Ben Finney wrote:
> Howdy all,
> 
> What is a ‘py3versions’ (or alternative) command that can be run in
> AutoPkgTest, to query the Python versions that are installed on this
> machine *with* their standard library?
> 
> The ‘pythonX.Y-minimal’ packages can be installed *without* standard
> library, but will still appear in the ‘py3versions --installed’ output.
> 
> This means it's possible to have an AutoPkgTest test that attempts to
> run a module for all the ‘py3versions --installed’ versions, then fail
> because an import of a standard library module fails.
> 
> What command, hopefully as simple as ‘py3versions --installed’, can be
> used in AutoPkgTest to interrogate *only* those Python versions on the
> local machine that have their standard library installed?

we currently can't do that, until python3.6 gets removed. I assume it's still on
some images?



Re: Bug#916428: autopkgtest-virt-qemu: Fails to set up test environment when run with python3.7

2018-12-14 Thread Matthias Klose
On 14.12.18 12:48, Simon McVittie wrote:
> Control: forwarded -1 
> https://salsa.debian.org/ci-team/autopkgtest/merge_requests/42
> Control: tags -1 + patch
> 
> On Fri, 14 Dec 2018 at 11:31:02 +, Simon McVittie wrote:
>> tl;dr: autopkgtest-virt-qemu doesn't work with python3.7.
> 
> This seems to be caused by using socket.send() (and assuming the entire
> buffer will be sent in one transaction) instead of socket.sendall().
> This was always a bug, at least in theory. I don't know why Python 3.7
> makes it significant in practice when it wasn't previously.

if you already ran autopkg using 3.7, then that might point out to the recent
3.7.2 release candidate 1. changes. At least the timing of the report suggests 
this.

https://docs.python.org/3.7/whatsnew/changelog.html#python-3-7-2-release-candidate-1



Re: Python 3.7 or 3.6 in Buster

2018-11-21 Thread Matthias Klose
On 05.11.18 10:32, Matthias Klose wrote:
> On 05.11.18 09:17, Michael Hudson-Doyle wrote:
>> On Mon, 5 Nov 2018 at 21:09, Thomas Goirand  wrote:
>>
>>> Hi there!
>>>
>>> During Debconf, we decided we would not decide yet, and see in November
>>> if we think it's reasonable to allow Python 3.7 to reach Buster. Time
>>> has passed, RC bugs have been fixed, and now is probably the time to
>>> open this thread.
>>>
>>> To me, as much as the OpenStack packages are concerned, I'm kind of fine
>>> with 3.7, modulo fixing this bug in Glance:
>>>
>>> https://bugs.debian.org/911947
>>>
>>> which I forwarded here:
>>> https://bugs.launchpad.net/glance/+bug/1800601
>>>
>>> It looks like this bug is 3.7.1 specific (ie: there's no bug if using
>>> Python 3.7.0). Upstream had a look, and didn't find a way to fix (yet).
>>> Help appreciated.
>>>
>>
>> I commented on the bug.
>>
>>
>>> It looks like everything else in OpenStack works.
>>>
>>> What is the situation for other packages? Do we have lots of Python 3.7
>>> problems remaining? If so, it'd be nice to have them listed somewhere,
>>> so we have a clear picture of what's going on.
>>>
>>
>> We're going through the process of switching to 3.7 as default in Ubuntu
>> right now, so I think it makes sense to switch Debian too fairly soon. I'm
>> trying to remember to attach all my patches to Debian bug reports...
> 
> in Ubuntu, we were able to migrate it (equivalent to the unstable -> testing
> migration) with 3.7 as the default.  We are in progress removing 3.6 as
> supported, which exposed some autopkg test regressions which are currently
> worked on. See
> http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html#python3-defaults
> 
>> Cheers,
>> mwh
>>
>> Do other fellow DD also think it's kind of ok to keep 3.7 in Buster?
>>>
>>> Also, Doko, do you plan on upgrading to newer point release? Python
>>> 3.7.1-rc1 and 3.7.1 both were the cause of troubles for me. I just hope
>>> this wont happen again (though I'd understand if you wished to upgrade
>>> once more before the freeze). In any ways, it'd be nice to communicate
>>> what you're planning on doing.
> 
> I'm in favor of going with 3.7, and would like to update to the current 3.7
> release branch at freeze, and then to 3.7.2 later, expected in January.

fyi, the release team accepted the transition, and 3.7 should arrive as the
default in unstable very soonish.



Re: [SUSPECTED SPAM] Re: Python 3.7 or 3.6 in Buster

2018-11-06 Thread Matthias Klose
On 05.11.18 20:31, Jerome Kieffer wrote:
> On Mon, 05 Nov 2018 12:26:41 -0500
> Scott Kitterman  wrote:
> 
>> I only found out about it due to an upstream bug report on OS X.  No one in 
>> Debian (or Ubuntu) reported it.
> 
> I got one also in fabio:
> https://github.com/silx-kit/fabio/pull/243
> Actually there has been a depreciation warning for ages (back to
> python 3.4) but this has been silenced since then.
> There may be many such bugs around in many packages!

I'm seeing a handful more, in failing autopkg tests which run without the
allow-stderr restriction.  But not a lot of people care yet about succeeding
autopkg tests ...

Filing bug reports about those.



Re: Python 3.7 or 3.6 in Buster

2018-11-05 Thread Matthias Klose
On 05.11.18 09:17, Michael Hudson-Doyle wrote:
> On Mon, 5 Nov 2018 at 21:09, Thomas Goirand  wrote:
> 
>> Hi there!
>>
>> During Debconf, we decided we would not decide yet, and see in November
>> if we think it's reasonable to allow Python 3.7 to reach Buster. Time
>> has passed, RC bugs have been fixed, and now is probably the time to
>> open this thread.
>>
>> To me, as much as the OpenStack packages are concerned, I'm kind of fine
>> with 3.7, modulo fixing this bug in Glance:
>>
>> https://bugs.debian.org/911947
>>
>> which I forwarded here:
>> https://bugs.launchpad.net/glance/+bug/1800601
>>
>> It looks like this bug is 3.7.1 specific (ie: there's no bug if using
>> Python 3.7.0). Upstream had a look, and didn't find a way to fix (yet).
>> Help appreciated.
>>
> 
> I commented on the bug.
> 
> 
>> It looks like everything else in OpenStack works.
>>
>> What is the situation for other packages? Do we have lots of Python 3.7
>> problems remaining? If so, it'd be nice to have them listed somewhere,
>> so we have a clear picture of what's going on.
>>
> 
> We're going through the process of switching to 3.7 as default in Ubuntu
> right now, so I think it makes sense to switch Debian too fairly soon. I'm
> trying to remember to attach all my patches to Debian bug reports...

in Ubuntu, we were able to migrate it (equivalent to the unstable -> testing
migration) with 3.7 as the default.  We are in progress removing 3.6 as
supported, which exposed some autopkg test regressions which are currently
worked on. See
http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html#python3-defaults

> Cheers,
> mwh
> 
> Do other fellow DD also think it's kind of ok to keep 3.7 in Buster?
>>
>> Also, Doko, do you plan on upgrading to newer point release? Python
>> 3.7.1-rc1 and 3.7.1 both were the cause of troubles for me. I just hope
>> this wont happen again (though I'd understand if you wished to upgrade
>> once more before the freeze). In any ways, it'd be nice to communicate
>> what you're planning on doing.

I'm in favor of going with 3.7, and would like to update to the current 3.7
release branch at freeze, and then to 3.7.2 later, expected in January.

Matthias



Re: Updated PEP 394 (python and python2 commands)

2018-09-25 Thread Matthias Klose
sorry, I never replied to that email.

On 10.06.2018 13:59, Piotr Ożarowski wrote:
> [Matthias Klose, 2018-06-08]
>> from my point of view this doesn't address:
>>
>>  - being able to de-install the python command for buster, if
>>people don't use it.  Most dependencies are auto-generated, so
>>these could be replaced by dependencies on python2. I would
>>assume that the majority of users won't have to install the
>>python package anymore, unless they are doing python2
>>development.
> 
> how about generating dependencies on python2.7 and moving
> pycompile/pyclean there as well (this also solves pre-dependency issue)
> 
>>  - Not having a python package in bullseye (buster+1), but a
>>python2 package doesn't point to any "default" anymore. Many
>>users are installing just python, because it's the unversioned
>>package.  So make it clear that this is another version, and
>>with having the choice of python2 and python3, most users will
>>install python3.
> 
> same here
> 
> additional question: why do you need for all that python2-doc,
> python2-minimal, python2-dev, libpython2-dev, libpython2-stdlib,
> python2-dbg and libpython2-dbg binary packages?

yes, moving the symlinks to the 2.7 packages would be an option as well.  But I
don't see any difference why one or the other should be the preferred option.



Re: Policy missing from the python* packages?

2018-09-14 Thread Matthias Klose
On 14.09.2018 05:30, Joseph Herlant wrote:
> Hi guys,
> 
> It's probably a mistake on my side but I'm looking for the package
> that provides the python policy locally.
> 
> In https://packages.debian.org/testing/amd64/python/filelist I can see
> that the python package provides the files in the
> '/usr/share/doc/python/python-policy.html/' directory.
> 
> But locally the package doesn't seem to have it when looking at dpkg
> -L of python, python-doc, python3 or python3-doc.
> I do see the rule for building the policy in python3-defaults in
> https://salsa.debian.org/python-team/defaults/python3-defaults/blob/master/debian/rules#L54
>  but amm I wrong to think that it could be due to the predepend on the
> next rule to be commented? (stamp-doc: #stamp-doc-policy)

This is fixed in 3.6.6-1 in unstable.



Re: Safest way to set python3 as default for `python`

2018-07-16 Thread Matthias Klose

On 16.07.2018 09:44, Bastian Venthur wrote:

Hi,

sorry if this question has been asked before. What is the currently recommended 
way to make `python` point to `python3`? I'd like to have it set on a system 
default level if possible.


There is none.  Maybe in the far future when Debian doesn't ship a python 
command anymore, and doesn't rely on a python binary anymore, you can screw up 
your system as you like.  But until that, don't mess around with the python symlink.




Python 3.7 in testing/experimental

2018-06-08 Thread Matthias Klose
The Python 3.7 beta 5 packages are now in testing, and experimental has 
python3-defaults packages which add 3.7 as a supported version.  The release 
candidate is expected next week, only adding Unicode 11 support, and the final 
release is expected at the end of June.


I would appreciate it, if somebody could run a test rebuild using the 
python3-defaults from experimental.  Else, please test your packages using 
python3.7.


Matthias



Re: Updated PEP 394 (python and python2 commands)

2018-06-08 Thread Matthias Klose

On 19.05.2018 07:24, Stuart Prescott wrote:

Matthias Klose wrote:

The distro should get
out of the way of using the python symlink, and giving users the freedom /
choice what to do about the link.


I think I understand your rationale to stop shipping /usr/bin/python and
once the unversioned symlink disappears from use in Debian then at least
that particular avenue to breaking one's system disappears.

What I don't understand is why any changes to package names or dependencies
are required to achieve that goal.

It sounds like a reasonable amount of work in your proposal, but once we no
longer have any Python 2 applications left at some stage in the bullseye
cycle, isn't the following sufficient?

--- a/debian/rules
+++ b/debian/rules
@@ -247,12 +247,9 @@ binary-arch: build install stamp-doc
  
 : # provide the python and python.1 defaults

 mkdir -p debian/python-minimal/usr/bin
-   ln -sf python$(VER) debian/python-minimal/usr/bin/python
 ln -sf python$(VER) debian/python-minimal/usr/bin/python2
 mkdir -p debian/python-minimal/usr/share/man/man1
-   ln -sf python$(VER).1.gz \
-   debian/python-minimal/usr/share/man/man1/python.1.gz
 ln -sf python$(VER).1.gz \
 debian/python-minimal/usr/share/man/man1/python2.1.gz

and then either later in the bullseye or bookworm cycles, those python-
defaults simply go away along with all the other 'unversioned' python module
and interpreter packages.

What have I (and others!) missed that would make a rather elaborate
packaging dance preferable to this?


from my point of view this doesn't address:

 - being able to de-install the python command for buster, if
   people don't use it.  Most dependencies are auto-generated, so
   these could be replaced by dependencies on python2. I would
   assume that the majority of users won't have to install the
   python package anymore, unless they are doing python2
   development.

 - Not having a python package in bullseye (buster+1), but a
   python2 package doesn't point to any "default" anymore. Many
   users are installing just python, because it's the unversioned
   package.  So make it clear that this is another version, and
   with having the choice of python2 and python3, most users will
   install python3.

Note that with the command-not-found package installed (and using python3), and 
not having python installed, you get now a message


   $ python

   Command 'python' not found, but can be installed with:

   sudo apt install python3
   sudo apt install python

   You also have python3 installed, you can run 'python3' instead.

Assuming that you only have python2 installed in buster, you get this hint 
already in buster, pointing to python3. You can't do this without a separate 
python2 package.


Matthias



Re: Broken dbgsym packages for Python 3

2018-06-04 Thread Matthias Klose

On 04.06.2018 06:34, Scott Kitterman wrote:

On Sunday, June 03, 2018 09:26:58 PM Scott Talbert wrote:

Hi,

I've got a package (wxpython4.0) that builds modules for both Python 2.7
and Python 3.  When I rebuilt the package in early May, I started getting
the lintian warning debug-file-with-no-debug-symbols for the Python 3
dbgsym packages only.  Sure enough, looking at those files, there is not
much there.  The Python 2.7 dbgsym files are fine.  Given that I haven't
changed anything in how the Python 3 modules are compiled, it seems that
some outside change in the distribution has caused this.  Has anyone else
noticed this, or have any idea what might have caused these dbgsym
packages to stop working?


The -dbgsym packages don't work for Python anyway because you need to call the
debug interpreter, so I don't think it matters much either way.


Usually the python*-*-dbg packages contain two things:

 - the unstripped extensions for the dbg interpreter
 - the debug symbols for the python*-* packages.

If you don't have both of these, you should investigate why.



Re: Updated PEP 394 (python and python2 commands)

2018-05-18 Thread Matthias Klose
On 18.05.2018 19:24, Piotr Ożarowski wrote:
>> A bit disappointed about this style of communication, Matthias
> 
> same here (you want us to do something without explaining reasons)
> EOT for me.

well, I thought I explained in the first message.  The distro should get out of
the way of using the python symlink, and giving users the freedom / choice what
to do about the link.

But that's no reason to presume unfounded behavior by either upstream or by 
myself.

Matthias



Re: Updated PEP 394 (python and python2 commands)

2018-05-18 Thread Matthias Klose
On 18.05.2018 18:14, Scott Kitterman wrote:
> On Friday, May 18, 2018 11:31:37 AM Matthias Klose wrote:
>> On 18.05.2018 05:19, Piotr Ożarowski wrote:
>>> [Matthias Klose, 2018-05-17]
>>>
>>>> PEP 394 [1] saw an update in April 2018 [2], the diffs at [3].
>>>>
>>>> The most important change from my point of view is
>>>>
>>>> -* It is suggested that even distribution-specific packages follow the
>>>> -  ``python2``/``python3`` convention, even in code that is not intended
>>>> to
>>>> +* It is strongly encouraged that distribution-specific packages use
>>>> ``python2`` +  or ``python3`` rather than ``python``, even in code that
>>>> is not intended to>> 
>>>>operate on other distributions.
>>>
>>> FTR: the day this PEP asks us to point /usr/bin/python to python3 is the
>>> day I start ignoring it, to say the least.
>>>
>>>> I don't think there is enough time to replace all python shebangs to
>>>> python2 in time for the buster release, however there is no harm in
>>>> starting this process now.
>>>
>>> too late, this process has already started (since dh_python2 v3.20180313)
>>> ;-P> 
>>>> But I'd like to get this done for buster+1, in the case we still need to
>>>> ship a Python2/2.7, so that buster+1 doesn't ship with a python command,
>>>> but maybe with a python2 command.
>>>
>>> we already ship /usr/bin/python2. Removing /usr/bin/python makes sense
>>> as well (administrators can symlink it to whatever they want once it's
>>> gone from Debian), but...
>>>
>>>> The first step is to create a set of python2* packages in
>>>> python-defaults, with contain all the python2* symlinks, and having the
>>>> python* packages depend on those python2 packages.  This change itself
>>>> is a no-op and shouldn't affect anything.
>>>>
>>>> As a second step change the dh_python2 (in python-defaults), and
>>>> dh-python to generate dependencies on python2 instead of python, and
>>>> replacing the shebang from python to python2.
>>>>
>>>> This should cover the majority of packages to replace dependencies on
>>>> python with dependencies on python2.  There are packages which don't
>>>> check for python2, so these probably need adjustments.  But again, the
>>>> goal for buster+1 is to ship as few Python2 dependent packages as
>>>> possible, if any.
>>>
>>> this is useless. What will we gain by renaming packages?
>>
>> who said, that we should rename packages? The only packages being dropped
>> are the python defaults packages.
>>
>>> I refuse to do that work!
>>
>> There is no work in renaming the packages. It's about the dependency
>> generation and the shebang.
>>
>>> The only message it sends is that we don't think /usr/bin/python or
>>> python package is Python 2.7 anymore and that's definitely not the
>>> message I want to send.
>>
>> No, that's not what the PEP says.
> 
> Upstream is free to follow Arch in their insanity (even if more slowly) and 
> suggest pointing /usr/bin/python at a python3 version is a reasonable thing 
> to 
> do (eventually).  It's not (and they even explain why in the PEP).  Debian 
> doesn't have to follow.  We can have higher standards.

I don't see how "having higher standards" and being explicit about using
"python2" are exclusive.

And I'm asking you how you come to your statement about "Upstream insanity", and
to your statement that upstream suggests of pointing python to python3.

> I don't see any reason to be able to "apt install python2" instead of "apt 
> install python".  I think it's perfectly fine the way it is now where the 
> same 
> package provides /usr/bin/oython and /usr/bin/python2.  If you exclude 
> eventually pointint /usr/bin/python at a python3 version (and we should), 
> then 
> there's no value in doing it.
> 
> I agree with Piotr.  I don't think we need to "create a set of python2* 
> packages".

no, it gives users the freedom what to do with the python link, with the distro
explicitly going away of using the unversioned python.  This doesn't hurt the
distro.  And the amount of work to do that should not be excessive for the set
of packages targeted to buster+1, if there are any.

Matthias



Re: Updated PEP 394 (python and python2 commands)

2018-05-18 Thread Matthias Klose
On 18.05.2018 19:02, Piotr Ożarowski wrote:
>> who said, that we should rename packages? The only packages being dropped are
>> the python defaults packages.
>>>
>>> I refuse to do that work!
>>
>> There is no work in renaming the packages. It's about the dependency 
>> generation
>> and the shebang.
> 
> the work in dh-python is not trivial. The work needed in thousands of
> packages (all Python related packages build depend on python{,-dev} or
> python-all{,-dev} is not as well. Rebuilding all binary packages means
> even more work (${python:Depends} is not always the only dependency)…
> and that's only a work in Debian. There are lots of packages outside
> Debian that will need to be updated… for what purpose exactly?

buster+1 is not supposed to ship with "thousands" of Python2 packages, so you
are exaggerating this a lot. For what reason?

> To make it easier to propose later python2-foo binary package rename,
> like Fedora did?

What are you doing here? Accusing me of wrong-doing and lying?  I explicitly
said that no other packages should be renamed.

>>> The only message it sends is that we don't think /usr/bin/python or
>>> python package is Python 2.7 anymore and that's definitely not the
>>> message I want to send.
>>
>> No, that's not what the PEP says.
> 
> not yet. We all know that it will happen sooner or later, though.

Again, you are accusing upstream about not being honest about their intentions.
Please could you give any rationale for doing so?
https://github.com/python/peps/pull/630 makes it quiet explicit that this is not
the case.

A bit disappointed about this style of communication, Matthias



Re: Updated PEP 394 (python and python2 commands)

2018-05-18 Thread Matthias Klose
On 18.05.2018 05:19, Piotr Ożarowski wrote:
> [Matthias Klose, 2018-05-17]
>> PEP 394 [1] saw an update in April 2018 [2], the diffs at [3].
>>
>> The most important change from my point of view is
>>
>> -* It is suggested that even distribution-specific packages follow the
>> -  ``python2``/``python3`` convention, even in code that is not intended to
>> +* It is strongly encouraged that distribution-specific packages use 
>> ``python2``
>> +  or ``python3`` rather than ``python``, even in code that is not intended 
>> to
>>operate on other distributions.
> 
> FTR: the day this PEP asks us to point /usr/bin/python to python3 is the
> day I start ignoring it, to say the least.
> 
>> I don't think there is enough time to replace all python shebangs to python2 
>> in
>> time for the buster release, however there is no harm in starting this 
>> process
>> now.
> 
> too late, this process has already started (since dh_python2 v3.20180313) ;-P
> 
>> But I'd like to get this done for buster+1, in the case we still need to
>> ship a Python2/2.7, so that buster+1 doesn't ship with a python command, but
>> maybe with a python2 command.
> 
> we already ship /usr/bin/python2. Removing /usr/bin/python makes sense
> as well (administrators can symlink it to whatever they want once it's
> gone from Debian), but...
> 
>> The first step is to create a set of python2* packages in python-defaults, 
>> with
>> contain all the python2* symlinks, and having the python* packages depend on
>> those python2 packages.  This change itself is a no-op and shouldn't affect
>> anything.
>>
>> As a second step change the dh_python2 (in python-defaults), and dh-python to
>> generate dependencies on python2 instead of python, and replacing the shebang
>> from python to python2.
>>
>> This should cover the majority of packages to replace dependencies on python
>> with dependencies on python2.  There are packages which don't check for 
>> python2,
>> so these probably need adjustments.  But again, the goal for buster+1 is to 
>> ship
>> as few Python2 dependent packages as possible, if any.
> 
> this is useless. What will we gain by renaming packages?

who said, that we should rename packages? The only packages being dropped are
the python defaults packages.
> 
> I refuse to do that work!

There is no work in renaming the packages. It's about the dependency generation
and the shebang.

> The only message it sends is that we don't think /usr/bin/python or
> python package is Python 2.7 anymore and that's definitely not the
> message I want to send.

No, that's not what the PEP says.

Matthias



Updated PEP 394 (python and python2 commands)

2018-05-17 Thread Matthias Klose
PEP 394 [1] saw an update in April 2018 [2], the diffs at [3].

The most important change from my point of view is

-* It is suggested that even distribution-specific packages follow the
-  ``python2``/``python3`` convention, even in code that is not intended to
+* It is strongly encouraged that distribution-specific packages use ``python2``
+  or ``python3`` rather than ``python``, even in code that is not intended to
   operate on other distributions.

I don't think there is enough time to replace all python shebangs to python2 in
time for the buster release, however there is no harm in starting this process
now.  But I'd like to get this done for buster+1, in the case we still need to
ship a Python2/2.7, so that buster+1 doesn't ship with a python command, but
maybe with a python2 command.

The first step is to create a set of python2* packages in python-defaults, with
contain all the python2* symlinks, and having the python* packages depend on
those python2 packages.  This change itself is a no-op and shouldn't affect
anything.

As a second step change the dh_python2 (in python-defaults), and dh-python to
generate dependencies on python2 instead of python, and replacing the shebang
from python to python2.

This should cover the majority of packages to replace dependencies on python
with dependencies on python2.  There are packages which don't check for python2,
so these probably need adjustments.  But again, the goal for buster+1 is to ship
as few Python2 dependent packages as possible, if any.

The third step for buster+1 would be to drop the set of python* packages from
python-defaults.

Both Debian 8 and Debian 9 already have the python2 symlink, so you could even
reuse binary packages in the older releases.  If needed, python-defaults could
be packported to Debian 9 using versioned provides, and to Debian 8 introducing
the same set of python2 packages.

Matthias

[1] https://www.python.org/dev/peps/pep-0394/
[2] https://github.com/python/peps/pull/630
[3] https://github.com/python/peps/pull/630.diff



Re: Removal of easy_install

2018-05-09 Thread Matthias Klose
On 07.05.2018 03:39, Scott Talbert wrote:
> About a month ago, Matthias removed easy_install from setuptools.  I have sent
> him a few mails and bugs asking about it, but I haven't heard anything back. 
> Anyone know why he did it?  I have a package that currently FTBFS because of
> it.  :(
> 
> python-setuptools (39.0.1-2) unstable; urgency=medium
> 
>   * Make the PKG-INFO output reproducible (Chris Lamb). Closes: #894215.
>   * Stop shipping the easy_install scripts.
> 
>  -- Matthias Klose <d...@debian.org>  Mon, 02 Apr 2018 11:46:01 +0200

easy_install isn't the preferred method to install things anymore, but you you
still can use python3 -m easy_install if you still need to use it.



Re: Profile-guided optimisation / link-time optimisation in Buster

2018-04-23 Thread Matthias Klose
On 22.04.2018 21:31, Matthew Woodcraft wrote:
> I noticed that profile-guided optimisation and link-time optimisation
> have both been unconditionally disabled in Buster's Python 3.6 and 3.7
> packages, though 3.5 in Stretch had them both enabled (on common
> platforms).
> 
> Can anyone tell me why this was done?
> 
> 
> I see nothing about this in debian/changelog and the indentation in
> debian/rules looks odd, so I'm wondering if it might even have been an
> unintentional change.
> 
> 
> It's these extra lines in debian/rules:
> 
>    with_pgo =
>    with_lto =
> 
> just before the
>  ifeq ($(with_lto),yes)
> line.

yep, will turn it on again.



Re: packages that use dh_python{2,3} but don't depend on dh-python

2018-03-27 Thread Matthias Klose
On 27.03.2018 18:39, Piotr Ożarowski wrote:
> [Matthias Klose, 2018-03-27]
>> On 26.03.2018 19:32, Piotr Ożarowski wrote:
>>> Hi,
>>>
>>> Here's a list of packages that will FTBFS soon if dh-python will not be
>>> added to Build-Depends (it's time to drop dh-python from python3's
>>> Depends and old version of dh_python2 from python package).
>>>
>>> http://people.debian.org/~piotr/dh_python3_without_dh-python.list
>>> http://people.debian.org/~piotr/dh_python3_without_dh-python.ddlist
>>> http://people.debian.org/~piotr/dh_python2_without_dh-python.list
>>> http://people.debian.org/~piotr/dh_python2_without_dh-python.ddlist
>>>
>>> The plan is to report bugs first and follow up with changes in -defaults
>>> packages in April or May.
>>
>> These lists have way to many false positives. So please, don't file bug 
>> reports
>> before checking these manually.
> 
> can you name some so that I can try to figure out what went wrong?

yes. please see below for a description of such packages. Or you might share the
approach how you did construct those lists you posted.

>> You have to exclude all these source packages using python/python3 as a b-d 
>> only
>> and which don't build any binary packages without any python/python3 
>> dependency
>> in the packages, e.g. gcc-defaults.
> 
> gcc-defaults is not a false positive, it calls `dh_python2 -plibgcj-common` 
> but doesn't
> depend on dh-python

well, this one not anymore.



Re: packages that use dh_python{2,3} but don't depend on dh-python

2018-03-27 Thread Matthias Klose
On 26.03.2018 19:32, Piotr Ożarowski wrote:
> Hi,
> 
> Here's a list of packages that will FTBFS soon if dh-python will not be
> added to Build-Depends (it's time to drop dh-python from python3's
> Depends and old version of dh_python2 from python package).
> 
> http://people.debian.org/~piotr/dh_python3_without_dh-python.list
> http://people.debian.org/~piotr/dh_python3_without_dh-python.ddlist
> http://people.debian.org/~piotr/dh_python2_without_dh-python.list
> http://people.debian.org/~piotr/dh_python2_without_dh-python.ddlist
> 
> The plan is to report bugs first and follow up with changes in -defaults
> packages in April or May.

These lists have way to many false positives. So please, don't file bug reports
before checking these manually.

You have to exclude all these source packages using python/python3 as a b-d only
and which don't build any binary packages without any python/python3 dependency
in the packages, e.g. gcc-defaults.

Matthias



Re: Move to salsa? Merge modules and apps team?

2018-02-07 Thread Matthias Klose
On 07.02.2018 10:12, Ghislain Vaillant wrote:
> 2018-02-07 8:58 GMT+00:00 Matthias Klose <d...@debian.org>:
>> On 07.02.2018 08:37, W. Martin Borgert wrote:
>>> Hi,
>>>
>>> how about moving the Python team(s) to salsa?
>>> And how about merging the modules and apps teams into one?
>>>
>>> Moving git packages (modules team) is very easy using
>>> import.sh from https://salsa.debian.org/mehdi/salsa-scripts.git
>>>
>>> Moving svn packages (apps team) is probably more work.
>>>
>>> Any opinions? Any doubts?
>>
>> I don't think that is a good idea.  Both teams are not very active when it 
>> comes
>> to address RC issues and updating to new upstream versions.
> 
> How does that affect moving our hosting to salsa? Or is your point
> against merging both teams? I am confused.
> 
>>  From my point of
>> view the apps team is worse than the modules team in this regard.  Currently
>> both teams really don't care about packages which are uploaded by other team
>> members.  Maybe it's time to clean up the teams before any considerations to
>> merge them?  For single maintainers we do have process for MIA, and for QA
>> uploads, however both python teams are escaping such efforts.
> 
> Still unclear what all of this has to do with the salsa migration. I
> understand your concerns, but imo they are orthogonal to the
> transition.

no, it's about merging the teams, not the move to the new infrastructure.



Re: Move to salsa? Merge modules and apps team?

2018-02-07 Thread Matthias Klose
On 07.02.2018 08:37, W. Martin Borgert wrote:
> Hi,
> 
> how about moving the Python team(s) to salsa?
> And how about merging the modules and apps teams into one?
> 
> Moving git packages (modules team) is very easy using
> import.sh from https://salsa.debian.org/mehdi/salsa-scripts.git
> 
> Moving svn packages (apps team) is probably more work.
> 
> Any opinions? Any doubts?

I don't think that is a good idea.  Both teams are not very active when it comes
to address RC issues and updating to new upstream versions.  From my point of
view the apps team is worse than the modules team in this regard.  Currently
both teams really don't care about packages which are uploaded by other team
members.  Maybe it's time to clean up the teams before any considerations to
merge them?  For single maintainers we do have process for MIA, and for QA
uploads, however both python teams are escaping such efforts.

Matthias



Re: Python 2 removal strategy

2018-01-10 Thread Matthias Klose
On 10.01.2018 17:06, Ole Streicher wrote:
> Hi,
> 
> I am the maintainer of the "python-astropy" package, that currently
> creates packages for both Python 2 and Python 3. Both packages have a
> number of reverse dependencies.
> 
> Recently, upstream announced a new version 3.0 of astropy, which
> supports Python 3 only, and I think of the best mid-term strategy: The
> old version 2.0 is supported upstream for ~2 years, and I want to have a
> smooth migration path. I checked the wiki, but could not find good
> information about migration.

Currently discussed. See "Python2 EOL and moving towards Python3" on this ML.

> I thought of a temporary package split: create a new source package
> "astropy" that inherits of the current python-astropy package, but only
> builds python3-astropy (and the utils + doc, which depend on
> python3-astropy), and update this to version 3.0. Then I would remove
> these binary packages from the python-astropy package. In parallel, I
> would file bugs (severity: important) to remove the reverse dependencies
> of the Python 2 packages (many of them are mine, but also may have
> reverse dependencies).
> 
> As long as there are substantial problems with the removal of the Python
> 2 support, I then keep the "old" python-astropy package updated. Once
> everything is figured out and we decide to finally kick out Python 2
> support (from Debian-Astro, or from Debian), I would set the remaining
> bugs as RC, and (once they are solved) remove the "python-astropy"
> package.
> 
> Does this sound reasonable? And how should I do this technically?

well, astropy is not such a mainstream package that I would mind removing some
of it's reverse dependencies.  If you want to add the additional pain having a
separate Python2 source stack, go for it. I wouldn't want to do that myself.  If
not, just go ahead with Python3 after having identified the reverse dependencies
which are not maintained by yourself.

Matthias



Re: /usr/bin/python2 in shebangs?

2017-12-14 Thread Matthias Klose
On 14.12.2017 14:38, Piotr Ożarowski wrote:
> Hi,
> 
> I plan to prepare new dh-python upload soon and hence a question about
> a bit controversial change: should dh_python2 (as we discussed during
> last DebConf's Python BoF) replace /usr/bin/python shebangs with
> /usr/bin/python2?

+1



Python2 EOL and moving towards Python3

2017-12-13 Thread Matthias Klose
With the Python2 EOL finalized (https://pythonclock.org/) for 2020 it's again
time to ping/pester about getting packages ready for Python3.  The Python2 EOL
doesn't mean that Python2 will go away from the archive any time soon, however
even third party projects already have announced an EOL for supporting Python2
(like NumPy).  At DebCamp 2017, Stefano Rivera, Chris Lamb and I sat together to
identifiy packages which need work, and Chris implemented some new lintian flags
which are now all emitted as warnings.  The numbers are huge, however I plan to
file bug reports (one for earch source package) at some time, because bug
reports are much easier to have a status or a discussion, than just having
lintian tags.  It would be nice if people could identify false positives which
maybe can be fixed in lintian before filing bug reports for the packages.

The tags are:

https://lintian.debian.org/tags/build-depends-on-python-sphinx-only.html

546 tags, should be easily fixable, mostly packaging issues.  Are there any apps
like sphinx which should use Python3 instead of Python2?

https://lintian.debian.org/tags/alternatively-build-depends-on-python-sphinx-and-python3-sphinx.html

17 tags

https://lintian.debian.org/tags/new-package-should-not-package-python2-module.html

287 tags, at some point we maybe should make that an (overridable) lintian -F
flag, however there are applications packaged which are Python2 only.

https://lintian.debian.org/tags/python-foo-but-no-python3-foo.html

1212 tags. Should only be overridden if the package has Python3 functionality
under another name (like the gtk2 bindings stack).

https://lintian.debian.org/tags/dependency-on-python-version-marked-for-end-of-life.html

2212 tags (and lintian lab still running to add more ...)


Looks overwhelming, and people maybe should break down any planned work to some
subtask (like making some blends Python2 free), or getting Python2 off CD 
images.

Ideally we should end up with a list of packages which are not ported to Python3
upstream, and then see what should be done about a Python2 removal in 202x ...

Matthias



Re: Python3.6 plans​ for Buster

2017-10-22 Thread Matthias Klose
On 29.06.2017 05:19, Scott Kitterman wrote:
> On Friday, June 23, 2017 02:09:34 PM Scott Kitterman wrote:
>> On Saturday, June 17, 2017 04:20:27 AM Scott Kitterman wrote:
>>> Python3.6 is already in Unstable and I expect to see it in Testing soon
>>> after Stretch is released.
>>>
>>> I've just now uploaded a version of python3-defaults to Experimental that
>>> adds python3.6 as as supported (but not default) python3.  If you have
>>> binary extensions packaged, please start testing with this version to make
>>> sure your package builds correctly with both python3.5 and python3.6 as
>>> supported.
>>>
>>> As a reminder (and for anyone new) we'll do the transition to python3.6 in
>>>
>>> three stages:
>>>  - Add python3.6 as supported and rebuild all binary extensions to support
>>>
>>> both python3.5 and python3.6
>>>
>>>  - Switch to python3.6 as default
>>>  
>>>  - Drop python3.5 from supported and rebuild again to remove python3.5
>>>
>>> support.
>>
>> Unless I've missed something, I've not heard back from anyone indicating
>> significant issues that would warrant not taking the first step in this plan
>> (I'm aware astropy may have some problems, but that's it).
>>
>> My plan is to ask the release team for a transition tracker and start the
>> first part of this in Unstable once they've agreed.  If anyone thinks this
>> is premature, please let me know.
> 
> Unless I brown bagged the upload somehow, this is started now.
> 
> https://release.debian.org/transitions/html/python3.6.html

The defaults change to 3.6 is now available in unstable, pending issues can be
seen here:

https://release.debian.org/transitions/html/python3.6-default.html

 - cantor: fixed in experimental
 - libkolab: fixed in experimental
 - python-escript: #878496, test failures on 32bit
 - pyside: multiple RC issues
 - s3ql: #877204, test failures
 - yt: #873511, fixed in new upstream

Matthias



Re: Python 3 Statsmodels & Pandas

2017-09-18 Thread Matthias Klose
On 18.09.2017 19:24, Gordon Ball wrote:
> On 18/09/17 09:48, Stuart Prescott wrote:
>> Hi Diane,
>>
>> On Sunday, 17 September 2017 22:14:18 AEST Diane Trout wrote:
>>> I just did it that way because it was the least disruptive change I
>>> could make that would let me build and test the package.
>>
>> Sure, that's entirely sensible.
>>
>>> In my experience I'm much more likely to to notice a build time test
>>> failure than one from the CI system. 
>>
>> Absolutely agreed. I'm thinking that this is a rather exceptional situation 
>> because of the circular dependency and the fall-out that we regularly see 
>> from 
>> that.
>>
>>> What do other people think? If there are autopkgtests, should you still
>>> let dh_auto_test run tests?
>>
>> (I know I'm not the 'other people' from whom you solicit replies, I was just 
>> perhaps unclear in what I was musing on before so I want to be more clear 
>> now)
>>
>> Like you, I would *normally* run all tests in both places: buildd tests 
>> catch 
>> severe problems right now; ci.d.n reruns the tests each time new versions of 
>> dependencies are uploaded too and later breakage is caught.
>>
>> Perhaps this is not a normal situation, however. Either one of "circular 
>> dependencies" or "packages that often FTBFS on one or more architectures" 
>> would be unusual enough to justify doing something different. All I am 
>> wondering (from my position of ignorance!) if in this case, perhaps the 
>> tests 
>> that cause the circular dependency can be disabled or xfailed, with the 
>> remaining tests run as normal.
> 
> I now prefer to use autopkgtest in place of build-time tests; the former
> has the advantages:
> 
>  * testing the installed package rather than the source tree, which
> ensures that the binary package has sufficient dependencies and the
> correct (or at least sufficient) contents
>  * avoids dependency loops, as noted
>  * avoids the need for workarounds required because of the location of
> the files being tested at build time (see nbconvert issues with
> entry_points not being found at build time)
>  * can detect external breakage between uploads via ci.d.n
> 
> The downsides are that:
> 
>  * boilerplate is required (copying test files out of the source tree to
> $AUTOPKGTEST_TMP to avoid testing against the source tree, instead of
> the installed package)
>  * extended build times if using autopkgtest as a post-build hook
> (installing a chroot multiple times, if using sbuild/similar), and other
> devs aren't guaranteed to run such tests at build/upload time
> 
> I wonder if it is possible (or a worthwhile use of time) to try and
> provide an extended autodep8 python mode which runs the equivalent of
> dh_auto_test as an autopkgtest, for packages which would opt-in with, eg
> "Testsuite: autopkgtest-pkg-python-pytest|nose", rather than the basic
> namespace-can-be-imported test.

the biggest downside with this approach is that you *completely* skip any
testing on other architectures than amd64.  Is that what you really want? Dear
porters, have fun where to search for bugs in packages without testsuites!

autopkg testing is not a replacement for runnning tests at build time from my
point of view, but to run a different set of tests.

It's a waste of build time on the autopkg testers if you need to build the
package again and then run against the tests in the build directory.  Many
testsuites are not made for autpkg testing from the installed place from what I
can see.

Matthias



Re: Python 3 Statsmodels & Pandas

2017-09-17 Thread Matthias Klose
On 16.09.2017 22:59, Yuri D'Elia wrote:
> On Sat, Sep 16 2017, Diane Trout wrote:
>> python3-pandas: Pandas is not installable
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=875723
> 
> I would have expected the rebuild of python packages affected by the
> fpectl extension with a transition, but it doesn't seem to be the case?

no, we don't want to rename the python2.7 package.

> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=874253
> 
> More Breaks where added to {python,python3}-stdlib itself, but there are
> still packages which didn't rebuild.

fix them, they fail for unrelated reasons ;)




Re: Bug#729956: Forwarded upstream

2017-09-06 Thread Matthias Klose
On 06.09.2017 23:45, Diane Trout wrote:
> I was trying to build it right now but I'm getting a dependency error.
> 
>  libpython2.7-stdlib : Breaks: python-pandas-lib (<= 0.20.3-1) but
> 0.20.3-1 is to be installed

this is unrelated, and waiting for #874413.



pythonX.Y packages stopped building the _fpectl extension

2017-08-31 Thread Matthias Klose
The pythonX.Y packages stopped building the _fpectl extension.  Upstream stopped
building this extension a long time ago, however I failed to disable it after a
last release ...  It's now disabled, and causing some build failures because
third party extensions still reference it.  The most prominent is
python{,3}-numpy which I just NMUed to rebuild against the current pythonX.Y
packages (already having a break on the current python{,3}-numpy).  If breaks
for other packages are needed, please file bug reports for the pythonX.Y 
packages.

Matthias



Re: MBF for deprecating Python2 usage

2017-08-04 Thread Matthias Klose
On 03.08.2017 21:08, ba...@debian.org wrote:
> On Aug 3, 2017, at 17:57, Matthias Klose <d...@debian.org> wrote:
>>
>> While at DebCamp, Stefano Rivera and I sat down to analyze what needs to be 
>> done
>> to deprecate Python2 usage within the distribution.  It might not be 
>> possible to
>> drop Python2 for the next release, but there are still too many issues with
>> packages.  For now we identified some categories which need fixing. These are
>> documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, 
>> resulting in
>> more than 3000 bug reports.  That's a high number, on the other hand we won't
>> make much progress if the issues are not named.  My intent is to bring that 
>> up
>> in the Python BoF next week at DebConf and then filing these bug reports with
>> the user tags mentiond on the wiki page.
> 
> Great to hear that you guys talked about it.
> 
> Just a quick note that PEP 394 discussions have revived, lead by the Fedora 
> folks.  Please do check out the new thread, especially if you have opinions 
> about what /usr/bin/python should do once Python 2.7 is EOL.
> 
> https://mail.python.org/pipermail/linux-sig/2017-August/thread.html

I replied to this thread.  I think there should be one release which is not
shipping /usr/bin/python before /usr/bin/python should be reused and pointed at
python (>> 2). This should be good enough to get all scripts actively converted
which are not part of the distribution.  If that release is buster, we should
require the use of python2 instead of python now, document that in policy and
provide a lintian check for that.

Matthias



MBF for deprecating Python2 usage

2017-08-03 Thread Matthias Klose
While at DebCamp, Stefano Rivera and I sat down to analyze what needs to be done
to deprecate Python2 usage within the distribution.  It might not be possible to
drop Python2 for the next release, but there are still too many issues with
packages.  For now we identified some categories which need fixing. These are
documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, resulting in
more than 3000 bug reports.  That's a high number, on the other hand we won't
make much progress if the issues are not named.  My intent is to bring that up
in the Python BoF next week at DebConf and then filing these bug reports with
the user tags mentiond on the wiki page.

Matthias



Re: Bug#866668: src:python-cryptography: Misbuild with more than one supported python3 version

2017-07-01 Thread Matthias Klose
On 30.06.2017 22:00, Tristan Seligmann wrote:
> Control: severity -1 important
> 
> On Fri, 30 Jun 2017 at 19:33 Scott Kitterman  wrote:
> 
>> Technically, it builds, but in a way that's not useful.  It would actually
>> be
>> better if it had failed (I noticed this from reviewing build logs after the
>> python3 interpreter depends weren't generated correctly).
>>
> 
> In fact, the package works fine on python3.5 as well as python3.6 (the
> module imports, and the full test suite passes against the installed
> modules), thus I'm downgrading the severity of this bug.
> 
> During the build, the extension modules are first built on python3.5, and
> it is these modules that land in /usr/lib/python3, with the "abi3" ABI tag.
> The extension modules are then built again on python3.6, but these don't
> get moved into /usr/lib/python3 due to the files from the python3.5 build
> having the same name, thus landing in an incorrect directory in the  final
> package and serving no useful purpose.
> 
> The reason the package still works is the "abi3" ABI tag; these modules
> (like all CFFI-built modules, by default) are using the Python 3 "stable
> ABI"[1] so the modules built on python3.5 work fine when imported on
> python3.6, and vice-versa. I think what this means is that we only need to
> run the Python 3 build once (using the default version?); also, I think the
> interpreter depends are in fact correct in light of this. I tested
> downgrading to 1.7.1-3 and the module still imports and works on python3.6,
> even though 1.7.1-3 was only ever built against python3.5.
> 
> However, testing against all supported versions at build time is probably
> still appropriate.
> 
> I'm looping in debian-python here as others may be surprised by this
> combination of effects as well, and we should probably be handling the
> issue of modules using the stable ABI consistently. Also, I'm not 100%
> certain what the best way forward is here; teach pybuild/dh_python3 about
> the stable ABI?

I think testing the build against all supported python versions makes sense. But
how do we make sure that a package built against the new supported version end
up in the archive for a release?

Matthias



Re: Ad-hoc Debian Python BoF at PyCon US 2017

2017-06-20 Thread Matthias Klose
On 10.06.2017 05:32, Barry Warsaw wrote:
> On Jun 06, 2017, at 10:57 AM, Sandro Tosi wrote:
> 
>> if we plan (and it looks like we do) to support and distribute 2.7
>> with buster, why not support it *properly*? what's the point of
>> deprecating python2.7? either we ship it or not, but if we do then
>> let's not cripple it by removing python2 modules packages. do yo think
>> that just because the module i want to use is not available will make
>> realize "oh sh*t, let's migrate this 50k lines of code application to
>> py3k so that i can implement this 5-minutes-of-work-funcionality if i
>> had the module on py2"?
> 
> So what's the plan for when upstream stops supporting Python 2 in 2020?  Given
> the pronouncement at Pycon 2017 that maintenance will end at Pycon 2020, we
> really need to decide what Debian's official policy will be, and what the
> timeline will be to get there.
> 
> If Buster is 2 years in development, that means it will be the last Debian
> release before Python 2.7 is EOL'd.  Yes, I know it's possible that 2.7 will
> get security releases for some time after that, but that's a much reduced
> commitment from upstream.
> 
> Once upstream stops supporting 2.7, should we also stop supporting it?  That
> wouldn't mean that developers on Debian can't use Python 2.7, just that they
> will be on their own.  I know it sucks for people who can't port to Python 3,
> but if a decade or more isn't enough time to switch, then that's really saying
> they'll never switch, and how much responsibility does Debian have at that
> point?
> 
> Python 2.7 isn't going away today, but 3 years goes by quickly and we need to
> decide what our policy will be when the day arrives.

There's a big chunk of work getting the python2 dependencies replaced by python3
dependencies.  I think we should track these packages with bug reports, so that
every source package depending on python2 only, and not providing python3 binary
packages has it's own bug report. For now, one big cluster might be packages b-d
on python-sphinx instead of python3-sphinx, another one many openstack packages.

We can only speculate on the amount of work until we have such a list ...

Matthias



Re: Bug#850789: Patch upload not showing up in deferred queue

2017-01-10 Thread Matthias Klose
On 11.01.2017 00:37, Taylor Kline wrote:
> I uploaded a patch for Python3 about ~15 hours ago, but it's not
> showing up on https://ftp-master.debian.org/deferred.html
> 
> Attempting to upload again does indicate that it should have
> successfully uploaded:
> 
> $ dput -e 10 python3-defaults_3.5.1-4.1_amd64.changes
> Trying to upload package to ftp-master (ftp.upload.debian.org)
> Package has already been uploaded to ftp-master on ftp.upload.debian.org
> Nothing more to do for python3-defaults_3.5.1-4.1_amd64.changes
> 
> 
> Does it take > 15 hours for an upload to show up? Is there something I
> am missing?

please don't upload this package.  You filed a bug report with severity wishlist
today, so you should give maintainers a chance to react to it.

Thanks, Matthias



pypi source packages: changes regarding PEP-0527

2017-01-08 Thread Matthias Klose
Hi,

PEP 527 [1] asks for exactly one source "tarball" for uploads to pypi, so you
may expect that tarballs are dropped and just zip files are uploaded to pypi.
So please be aware to look for both zip files and tarballs in watch files and
anything else.  I became aware of this for the setuptools package [2].

Matthias

[1] https://www.python.org/dev/peps/pep-0527/
[2] https://github.com/pypa/setuptools/issues/902



Re: Python 2 in the default installation -- progress made!

2016-12-29 Thread Matthias Klose
On 30.12.2016 05:41, Stuart Prescott wrote:
> 
> Hi everyone,
> 
> one of the objectives for stretch was to reduce the number of Python 2 
> packages that are installed in common scenarios, instead having the Python 3 
> stack take over. The "standard task" within tasksel is a reasonable place to 
> look.¹
> 
> Thanks to the recent work on reportbug and debianbts, this is now largely 
> complete.
> 
> In a stretch chroot, install the standard task:²
> 
> # aptitude install ~prequired ~pimportant ~pstandard
> 
> and the only python packages installed are:
> 
> # aptitude search ~npython~i
> i A dh-python
> i A libpython-stdlib
> i A libpython2.7
> i A libpython2.7-minimal
> i A libpython2.7-stdlib
> i A libpython3-stdlib
> i A libpython3.5-minimal
> i A libpython3.5-stdlib
> i   python
> i   python-apt
> i A python-apt-common
> i   python-gpg
> i   python-minimal
> i   python2.7
> i A python2.7-minimal
> i A python3
> i A python3-apt
> i A python3-chardet
> i A python3-debian
> i A python3-debianbts
> i   python3-gpg
> i A python3-httplib2
> i A python3-minimal
> i A python3-pkg-resources
> i A python3-pycurl
> i A python3-pysimplesoap
> i   python3-reportbug
> i A python3-requests
> i A python3-six
> i A python3-urllib3
> i A python3.5
> i A python3.5-minimal
> 
> where the packages marked "A" (Automatically Installed) are there to satisfy 
> dependencies only.
> 
> We see that:
> 
> python
> python-apt
> python-gpg
> python-minimal
> python2.7
> 
> are now only installed because they are marked as Priority: standard.
> 
> 
> Question for discussion: Is Python 2 a tool that would be expected to be 
> found 
> on any linux machine and so should it be Priority: standard? Is Python 3? Is 
> now the time to ask ftp-masters to change the overrides to move them to 
> Priority: optional?

I think it's time to move these to optional, and bump the priority of the
equivalent python3 packages. Should be done in the packages as well.

> (My personal feeling is that none of the above packages should be Priority: 
> standard, including Python 3)

I don't know the rationale for having the apt and gpg bindings as Priority
standard, but if these can be made optional, then the depending packages can be
made optional as well.

> The libpython2.7* packages are present because:
> 
> i   logrotate Recommends mailx
> i A mailutils Provides   mailx
> i A mailutils Dependslibpython2.7 (>= 2.7)
> 
> I don't know if mailutils can already use Python 3 instead of Python 2 or how 
> it is actually using Python at all -- the build system will certainly need 
> some work. Perhaps a keen Python 3 porter can investigate that as an opening 
> move for stretch+1 (it's too late for stretch for that transition).

while the code has some comments about python 3.0 it fails in the configury and
later building the extensions, so apparently it's not used, and not tested with
recent python3 versions.  I filed #849721 for that.

Otoh, logrotate could explicitly recommend bsd-mailx | mailx to avoid that
dependency.

I would prefer having stretch shipping with only one Python version in the base
task.

Matthias



Re: Issue on setup.py with imports

2016-09-06 Thread Matthias Klose
On 06.09.2016 09:36, IOhannes m zmölnig (Debian/GNU) wrote:
> 
> 
> On 2016-09-03 10:18, Marcos wrote:
>> I'm migrating Gufw [1] from python2 to python3 and honestly I'm stuck
>> with an error from the setup.py: Python module controller not found:
>>
>> costales@dev:~/Desktop/16.10$ sudo python3 setup.py install --prefix=/usr
>> ERROR: Python module controller not found
>> running install
>> running build
>> running build_py
>> running build_scripts
>> running build_i18n
>> intltool-update -p -g gufw
>> running build_icons
>> running build_help
>> running install_lib
>> running install_scripts
>> changing mode of /usr/bin/gufw to 755
>> changing mode of /usr/bin/gufw-pkexec to 755
>> running install_data
>> running install_egg_info
>> Removing /usr/lib/python3.5/site-packages/gufw-16.10.0-py3.5.egg-info
>> Writing /usr/lib/python3.5/site-packages/gufw-16.10.0-py3.5.egg-info
>> costales@dev:~/Desktop/16.10$
> 
> so the "error" is non-fatal and the installation proceeds.
> i fail to see the problem.

the path looks suspicious, it should be /usr/lib/python3/dist-packages



Python 3.6

2016-05-25 Thread Matthias Klose
Python 3.6.0 alpha 1 is now available in experimental.  The upstream release 
date is scheduled for December 2016, about two months before the stretch freeze. 
 The question is, if 3.6 should be targeted for stretch, as a default, or 
non-default version.  That would of course require some ahead-of-upstream work 
for some packages, and maybe freeze exceptions for some upstream packages 
releasing a first 3.6 support after the stretch freeze.


In any case, 2.7 should be at 2.7.13 and 3.5 should be at 3.5.3 for the stretch 
release.


Matthias



Re: python debug packages

2016-05-17 Thread Matthias Klose

On 14.05.2016 23:26, Iustin Pop wrote:

On 2016-04-22 19:36:12, Matthias Klose wrote:

On 22.04.2016 16:58, Jean-Michel Vourgère wrote:

Hi

Now that debug symbols are automatically generated in -dbgsym packages,
how do you handle the debug
/usr/lib/python2.7/dist-packages/.x86_64-linux-gnu_d.so files?

They used to go in a generic -dbg package.


[…]


- Do not migrate to new style -dbgsym packages and keep everything in
rrtool-dbg, like it is now.


that would be my preferred solution.


Reading
https://wiki.debian.org/Python/LibraryStyleGuide#Building_python_-dbg_packages,
there is some hints to this, but it's not clear that only automatic
debug packages work for Python packages. Would it make sense to update
the wiki page and say "don't migrate to dbgsym packages as Python needs
debug extensions and not only debug symbols"?


sounds fine.



Re: python debug packages

2016-04-22 Thread Matthias Klose

On 22.04.2016 16:58, Jean-Michel Vourgère wrote:

Hi

Now that debug symbols are automatically generated in -dbgsym packages,
how do you handle the debug
/usr/lib/python2.7/dist-packages/.x86_64-linux-gnu_d.so files?

They used to go in a generic -dbg package.

I'm thinking about rrdtool, and it already has a lot of packages:
https://tracker.debian.org/pkg/rrdtool

I'm considering creating a specific python-rrdtool-dbg package.

Other options I can think of are:
- Put the debug .so file into the main python-rrdtool package


no, that would add dependencies on the python-dbg packages by default.


- Do not migrate to new style -dbgsym packages and keep everything in
rrtool-dbg, like it is now.


that would be my preferred solution.


- Stop bothering about this debug .so file, and trash it.


please don't.



Re: Test suite in github but missing from pypi tarballs

2016-04-21 Thread Matthias Klose

On 21.04.2016 16:52, Thomas Goirand wrote:

On 04/21/2016 04:10 PM, Edward Betts wrote:

Recently I've come across some Python libraries that have a test suite in
their github repo but don't include it in the tarball they upload to pypi.

Debian binary packages don't normally include the test suite.


Why? It's my view that it's a good idea to include it, if it is located
within the lib itself.


there is no benefit to include the testsuite if it cannot be run in the install 
location.  So why include that for these cases?  Write an autopkg test and see 
if it runs *and* succeeds in the installed location.  Usually testsuites tend to 
need a lot of patches for doing that.


Also for large testsuites a separate binary package would be preferred, see e.g. 
the size of python-cryptography and python-cryptography-vectors.


Matthias



Use Python3 where possible

2016-03-15 Thread Matthias Klose
The recent update of pep8 to use Python3 by default, and the regressions 
mentioned in #807409 reminded me, that we probably should address the use of 
Python3 more pro-actively.


The pep8 binary package included the Python2 modules, plus the scripts, the 
python3-pep8 package only included the Python3 modules.  The recent upload added 
a python-pep8 package, and let the pep8 package depend on python3-pep8, breaking 
packages which use the module, but not the binary package.  I still think this 
way forward is the correct way to go (maybe adding some Breaks if these are 
known in advance).


There are probably more packages like these, which should be converted this way. 
 Maybe a popular candidate is sphinx, where almost always python-sphinx is used 
as a b-d instead of python3-sphinx.


I would like to come up with a recommendation that if a python module ships 
scripts, Python3 is used for these scripts, and the Python2 version of these 
scripts should be dropped (and python -m ...) should be used instead.  An 
alternative might be to use separate names for the scripts (e.g. ending with 2, 
or like in pillow one set without a suffix (for Python3), and one set with a .py 
suffix (Python2)).  The most conforming name for the scripts should always use 
Python3.


Having a lintian warning that a package still uses Python2 instead of Python3 
might help as well, however maybe it should start as an "information" instead of 
a warning.


Matthias



Bug#815720: transition: python3.5-only

2016-02-23 Thread Matthias Klose

Package: release.debian.org
Severity: normal
User: release.debian@packages.debian.org
Usertags: transition

python3-defaults now no longer depends on python3.4, making python3.5 the only 
supported python3 version.  python3.4 should be removed before stretch is 
released (mostly by binNMUing, and then removing python3.4).  not urgent, just 
adding this in the bug tracker.




Re: Switching Default Python3 To Python3.5

2016-01-01 Thread Matthias Klose

On 31.12.2015 10:44, Scott Kitterman wrote:

I went through the bad/unknown packages in the python3.5 transition tracker
[1] and the remainder seems reasonable for doing the transition.

Many of them only build support for the default python3 and so they will be
bad until after they are rebuilt following the switch.  I filed bugs with
patches for packages that seemed to be ~easy to make build for multiple
versions.

Some of the remainder should not be build-depending on python3-dev at all
since they don't have any arch specific content.  I filed a stack of bugs on
those too (as well as doing a team upload or two).

That should leave in the neighborhood of 45 - 50 binNMUs, which isn't so many.

Does anyone object if I go ahead and ask the release team for a transition
slot?


No.  However there is a real issue with ipython. For Ubuntu xenial I just 
disabled the tests.  It would be nice if ipython would have an update strategy, 
whether this is the 2.x, 3.x or the 4.x branch.


Matthias




  1   2   3   4   >