Re: Bug#1063957: python-pytest-lazy-fixture: autopkgtest regression with pytest 8
(Trimming To/Cc list to debian-python and the maintainers involved) Ileana - this email relates to python-limits, for which you made the last change Sandro - this email relates to your package prettytable On Tue, May 14, 2024 at 08:47:24AM +0200, Alexandre Detiste wrote: > [...] > I made a mistake in my attempt too..., here's the real list: > > > Package: prettytable > > Maintainer: Debian Python Team > Package: python-limits Alexandre has kindly fixed some more of these packages, so there are only two packages still (build-)depending on python3-pytest-lazy-fixture: python-limits: Maintainer: Debian Python Team Last upload: Ileana Dumitrescu (signed by Jeroen Ploemen ) on 19 Oct 2023; upstream is still using pytest-lazy-fixture prettytable: Maintainer: Sandro Tosi Last upload: 2024-03-02 (version 3.6.0-2): build failed because it depends on python3-lazy-fixture (which crashes with pytest 8) Ileana - would you be able to look at python-limits? Sandro - would you be able to look at prettytable? Best wishes, Julian
Re: Bug#1063957: python-pytest-lazy-fixture: autopkgtest regression with pytest 8
On Thu, May 16, 2024 at 06:55:50AM +0200, Antonio Valentino wrote: > Dear Alexandre > [...] > > > > I made a mistake in my attempt too..., here's the real list: > > > > Maintainer: Sandro Tosi > > Package: prettytable > > > > Maintainer: Debian GIS Project > > -> CC'ing Antonio > > Package: pycoast > > Package: pyresample > > > Thanks a lot. > Yesterday I have uploaded the new version of pycoast and pyresample Thanks Antonio! Best wishes, Julian
Re: Bug#1063957: python-pytest-lazy-fixture: autopkgtest regression with pytest 8
On Mon, May 13, 2024 at 11:07:54PM +0200, Alexandre Detiste wrote: > Le lun. 13 mai 2024 à 22:59, Scott Kitterman a écrit : > > >I suggest that we soon ask ftpmaster to drop pytest-lazy-fixture from > > >Debian unstable. > > > > Please transition all the rdepends first. Asking before that's done just > > creates more work for everyone. > > > > Scott K > > It looks like for this one package it's already clear. > > @Julian: here it looks you forgot to check build-depends: > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067200 Oh, gosh, I thought I had done so (this is cython3-legacy), but I clearly made a serious mistake in my attempt! > I don't know what's the best way to check this, I use this template hereunder Thanks Alexandre! > #!/bin/sh > Sources=/var/lib/apt/lists/ftp.*.debian.org_debian_dists_unstable_*_source_Sources > Packages=/var/lib/apt/lists/ftp.*.debian.org_debian_dists_unstable_*_binary-amd64_Packages > > ben query '.build-depends ~ "python3-lazy-fixture"' $Sources -s > Package,Maintainer > ben query '.build-depends-indep ~ "python3-lazy-fixture"' $Sources -s > Package,Maintainer > ben query '.depends ~ "python3-lazy-fixture"' $Packages -s Package,Maintaine As you say, though, in this case, pytest-lazy-fixture itself has an unfixable RC bug, so all of the rdepends need to be fixed (and by "soon", I was thinking "once all these packages no longer depend on it, but I should have said that explicitly ;-) Best wishes, Julian
Advice about pydevd and debugpy
Hi! Background: debugpy vendors pydevd (which in turn vendors an old version of bytecode). Until version 1.8.0 of debugpy (currently in testing), the vendored copy of pydevd was (almost) identical to the separately published pydevd, so I had replaced the vendored version of pydevd with the Debian-packaged version. pydevd now has an FTBFS, so needs fixing or updating. But now I've found that the version of pydevd published separately (now bumped from version 2.10.0 to 3.0.x) has started diverging fairly significantly from the vendored version in debugpy (as it moved from version 1.8.0 to 1.8.1): both are being modified, but along different paths. Trying to use pydevd 3.0.3 with debugpy 1.8.1 leads to multiple test failures. My thought is that at this point, seeing that these two packages are now diverging sufficiently that tests break, that for debugpy, I should stop trying to use a separately-packaged version of pydevd and instead switch to using the vendored version shipped with debugpy. If we do this, then at some point (probably after trixie, if nothing changes with the forked debugpy - pydevd relationship), I will suggest that we remove the pydevd package from Debian, as debugpy is the only package that depends on it, and I doubt that anyone else is using pydevd directly. (A separate issue is that the chances are that the pydevd tests that have started failing with the separately packaged pydevd will fail with the vendored version in debugpy as well, but they are not being run in the upstream test suite, and I'm not sure they would be able to run in this vendored environment anyway.) Does anyone have any comments on my plan? Best wishes, Julian
Re: Bug#1063957: python-pytest-lazy-fixture: autopkgtest regression with pytest 8
On Wed, Apr 03, 2024 at 03:58:21PM -0400, Jeremy Bícha wrote: > I noticed one package affected by this issue, prettytable, has > switched to a fork, pytest-lazy-fixtures (note the s at the end of the > name). > > Would someone like to package this for Debian to fix several packages > failing to build? > > https://github.com/dev-petrov/pytest-lazy-fixtures > > Thank you, > Jeremy Bícha Dear all affected by pytest-lazy-fixture: pytest-lazy-fixtures is now in testing and unstable; in many cases, it can be used as a drop-in replacement for pytest-lazy-fixture (though not all, it turns out). I suggest that we soon ask ftpmaster to drop pytest-lazy-fixture from Debian unstable. Best wishes, Julian
Bug#1070452: ITP: pytest-lazy-fixtures -- pytest plugin to use fixtures in @pytest.mark.parametrize
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org, 1063...@bugs.debian.org * Package name: pytest-lazy-fixtures Version : 1.0.7 Upstream Contact: Petrov Anton * URL : https://github.com/dev-petrov/pytest-lazy-fixtures * License : Expat Programming Lang: Python Description : pytest plugin to use fixtures in @pytest.mark.parametrize This plugin was inspired by pytest-lazy-fixture; that plugin is incompatible with pytest 8.x, so this can be used as a replacement. . Improvements that have been made in this project: . * You can use fixtures in any data structures * You can access the attributes of fixtures * You can use functions in fixtures This will allow those packages using pytest-lazy-fixture (which is now unusable in debian/testing) in their tests to migrate to this maintained alternative. (See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1063957) It will be maintained within the Debian Python Team, and I will be listed as an Uploader.
Bug#1068557: ITP: sphinxcontrib-openapi -- Sphinx extension to generate API docs from OpenAPI spec
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: sphinxcontrib-openapi Version : 0.8.4 Upstream Author : Ihor Kalnytskyi * URL : https://github.com/sphinx-contrib/openapi * License : BSD-2-clause Programming Lang: Python Description : Sphinx extension to generate API docs from OpenAPI spec This extension generates API documentation for reStructuredText documentation using the Sphinx system. It also depends on `sphinxcontrib.httpdomain` that provides an HTTP domain for describing RESTful HTTP APIs. This package is a (recursive) dependency of the new version of jupyter-server. It will be team-maintained within the Debian Python Team. The Debian packaging is on salsa at https://salsa.debian.org/python-team/packages/sphinxcontrib-openapi
Bug#1068556: ITP: sphinxcontrib-emojicodes -- Sphinx extension to use emoji codes in Sphinx documentation
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: sphinxcontrib-emojicodes Version : 0.3.1 Upstream Author : Copyright: The Sphinx Emoji Codes contributors * URL : https://github.com/sphinx-contrib/emojicodes * License : BSD-3-clause Programming Lang: Python Description : Sphinx extension to use emoji codes in Sphinx documentation This extension allows one to write things like |:smile:| in Sphinx documentation. This package is a (recursive) dependency of the new version of jupyter-server. It will be team-maintained within the Debian Python Team. The Debian packaging is on salsa at https://salsa.debian.org/python-team/packages/sphinxcontrib-emojicodes
Bug#1068555: ITP: rfc3986-validator -- RFC 3986 validator (Python library)
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: rfc3986-validator Version : 0.1.1 Upstream Author : Nicolas Aimetti * URL : https://github.com/naimetti/rfc3986-validator * License : Expat Programming Lang: Python Description : RFC 3986 validator (Python library) This package provides a validator for URIs to determine whether they conform to the Internet Engineering Task Force (IETF) specification (RFC 3986). This package is a (recursive) dependency of the new version of jupyter-server. It will be team-maintained within the Debian Python Team. The Debian packaging is on salsa at https://salsa.debian.org/python-team/packages/rfc3986-validator
Bug#1068554: ITP: rfc3339-validator -- RFC 3339 validator (Python library)
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: rfc3339-validator Version : 0.1.4 Upstream Author : Nicolas Aimetti * URL : https://github.com/naimetti/rfc3339-validator * License : Expat Programming Lang: Python Description : RFC 3339 validator (Python library) This package provides a validator for date-time strings to determine whether they conform to the Internet Engineering Task Force (IETF) Internet Date/Time Format (RFC 3339). This package is a (recursive) dependency of the new version of jupyter-server. It will be team-maintained within the Debian Python Team. The Debian packaging is on salsa at https://salsa.debian.org/python-team/packages/rfc3339-validator
Bug#1068553: ITP: python-overrides -- Python decorator to verify that expected overrides are maintained
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: python-overrides Version : 7.7.0 Upstream Author : Copyright: Mikko Korpela * URL : https://github.com/mkorpela/overrides * License : Apache-2.0 Programming Lang: Python Description : Python decorator to verify that expected overrides are maintained Provides a decorator @override that verifies that a method that should override an inherited method actually does it. Python has no standard mechanism by which to guarantee that (1) a method that previously overrode an inherited method continues to do so, and (2) a method that previously did not override an inherited will not override now. This package allows this to be addressed in an automated manner. This package is a (recursive) dependency of the new version of jupyter-server. It will be team-maintained within the Debian Python Team. The Debian packaging is on salsa at https://salsa.debian.org/python-team/packages/python-overrides
Bug#1068552: ITP: python-isoduration -- Operations with ISO 8601 durations (Python 3 package)
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: python-isoduration Version : 20.11.0+git20211126.ae0bd61 Upstream Author : Víctor Muñoz * URL : https://github.com/bolsote/isoduration * License : ISC Programming Lang: Python Description : Operations with ISO 8601 durations (Python 3 package) ISO 8601 supports time durations in string format, for example "P3Y6M4DT12H30M5S" represents a duration of 3 years, 6 months, 4 days, 12 hours, 30 minutes, and 5 seconds. This package supports working with such durations, addressing the shortcomings of the isodate package. This package is a (recursive) dependency of the new version of jupyter-server. It will be team-maintained within the Debian Python Team. The Debian packaging is on salsa at https://salsa.debian.org/python-team/packages/python-isoduration
Bug#1068550: ITP: pytest-mypy-testing -- Plugin to test mypy output with pytest
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: pytest-mypy-testing Version : 0.1.3 Upstream Author : Copyright: David Fritzsche * URL : https://github.com/davidfritzsche/pytest-mypy-testing * License : CC0-1.0 Programming Lang: Python Description : Plugin to test mypy output with pytest This plugin provides a way to test that mypy produces a given output. As mypy can be told to display the type of an expression, this allows one to check mypy's type interference. This package is a test-time dependency of the new version of traitlets; the current version in unstable has the relevant tests ignored at present, but it would be good to be able to include these tests in the autopkgtest suite. It will be team-maintained within the Debian Python Team. The Debian packaging is on salsa at https://salsa.debian.org/python-team/packages/pytest-mypy-testing
Bug#1068551: ITP: python-fqdn -- Python library to validate fully qualified domain names (FQDNs)
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: python-fqdn Version : 1.5.1 Upstream Author : ypcrts * URL : https://github.com/ypcrts/fqdn * License : MPL-2.0 Programming Lang: Python Description : Python library to validate fully qualified domain names (FQDNs) This package validates FQDNs conforming to the Internet Engineering Task Force (IETF) specification (RFC 1123 in particular). The design intent is to validate that a string would be traditionally acceptable as a public Internet hostname to RFC-conforming software. Configuration options can be used to obtain more relaxed checks. This package is a (recursive) dependency of the new version of jupyter-server. It will be team-maintained within the Debian Python Team. The Debian packaging is on salsa at https://salsa.debian.org/python-team/packages/python-fqdn
Bug#1068549: ITP: pytest-console-scripts -- Pytest plugin for running Python scripts from within tests
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: pytest-console-scripts Version : 1.4.1 Upstream Author : Vasily Kuznetsov * URL : https://github.com/kvas-it/pytest-console-scripts * License : Expat Programming Lang: Python Description : Pytest plugin for running Python scripts from within tests This plugin is quite similar to `subprocess.run()`, but it also has an in-process mode, where the scripts are executed by the interpreter that's running `pytest` (using some amount of sandboxing). . In-process mode significantly reduces the run time of the test suites that run many external scripts. This is speeds up development. In a CI environment subprocess mode can be used to make sure the scripts also work (and behave the same) when run by a fresh interpreter. This package is a (recursive) dependency of the new version of jupyter-server. It will be team-maintained within the Debian Python Team. The Debian packaging is on salsa at https://salsa.debian.org/python-team/packages/pytest-console-scripts
Bug#1068547: ITP: jupyter-server-terminals -- Jupyter server extension providing support for terminals
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: jupyter-server-terminals Version : 0.5.3 Upstream Author : Jupyter Development Team * URL : https://github.com/jupyter-server/jupyter_server_terminals * License : BSD-3-clause Programming Lang: Python Description : Jupyter server extension providing support for terminals This package allows Jupyter servers to interface to terminals via Tornado websockets. This package is a (recursive) dependency of the new version of jupyter-server. It will be team-maintained within the Debian Python Team. The Debian packaging is on salsa at https://salsa.debian.org/python-team/packages/jupyter-server-terminals
Bug#1068548: ITP: picobox -- Opinionated Python dependency injection framework
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: picobox Version : 4.0.0 Upstream Author : Ihor Kalnytskyi * URL : https://github.com/ikalnytskyi/picobox * License : Expat Programming Lang: Python Description : Opinionated Python dependency injection framework Picobox is designed to be clean, pragmatic and with Python in mind. No complex graphs, no implicit injections, no type bindings - just picoboxes and explicit demands. . It is a small, thread-safe, pure Python library with no dependencies. This package is a (recursive) dependency of the new version of jupyter-server. It will be team-maintained within the Debian Python Team. The Debian packaging is on salsa at https://salsa.debian.org/python-team/packages/picobox
Bug#1068546: ITP: async-asgi-testclient -- Python async client for testing ASGI web applications
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: async-asgi-testclient Version : 1.4.11 Upstream Author : Jordi Masip * URL : https://github.com/vinissimus/async-asgi-testclient * License : Expat Programming Lang: Python Description : Python async client for testing ASGI web applications This library is used for testing ASGI (Asynchronous Server Gateway Interface) applications. It is designed to be a common testing library for such applications that does not depend on the specific web framework being used. . It works by calling the ASGI app directly rather than via a web server. This package is a (recursive) dependency of the new version of jupyter-server. (More precisely, it is a test-time dependency of a currently unreleased version of picobox.) It will be team-maintained within the Debian Python Team. The Debian packaging is on salsa at https://salsa.debian.org/python-team/packages/async-asgi-testclient
Re: morph's abandoned packages (list)
On Sun, Mar 31, 2024 at 02:16:39PM +0200, tho...@goirand.fr wrote: > The bug is about the --pristine-tar option of bgp... >> It turns out that doing pristine-tar by hand often gives different >> results, as I discovered: >> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1065445 Yes, indeed; the original poster was talking about using pristine-tar by hand rather than using the --pristine-tar option of gbp. Best wishes, Julian
Re: morph's abandoned packages (list)
On Sat, Mar 30, 2024 at 10:21:06PM +0100, Thomas Goirand wrote: > [...] > > [0]: https://wiki.debian.org/Python/GitPackaging#Creating_a_new_package > > [1]: https://wiki.debian.org/Python/GitPackaging#New_upstream_release > I would not do this way, but use gbp import-orig instead. I'm not sure why > the wiki recommends, IMO wrongly, to do things by hand. Indeed, all of this: It turns out that doing pristine-tar by hand often gives different results, as I discovered: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1065445 So I don't know what best to recommend, personally. Anyway, once this bug is fixed, definitely using gbp import-orig is the way to go (and probably even before it is). Best wishes, Julian
Re: Seeking a small group to package Apache Arrow (was: Bug#970021: RFP: apache-arrow -- cross-language development platform for in-memory analytics)
Hi Diane, On Sat, Mar 30, 2024 at 08:59:39PM -0700, Diane Trout wrote: > Hi Julian, > > On Sat, 2024-03-30 at 20:22 +0000, Julian Gilbey wrote: > > Lovely to hear from you, and oh wow, that's amazing, thank you! > > > > I can't speak for anyone else, but I suggest that pushing your > > updates > > to the science-team package would be very sensible; it would be silly > > for someone else to have to redo your work. > > > > What more is needed for it to be ready for unstable? > > > The things I think are kind of broken are: > > We've got 7.0.0 and upstreams current version is 15.0.2. Yes, that does seem a little less than ideal! > the pyarrow 7.0.0 tests fail because it depends on a python test > library that breaks with pytest 8.0. Either I need to disable the > python tests or upgrade to a newer version. It may well be that newer versions would work with pytest 8.x. I don't think it's worth spending time trying to patch such a relatively old version. > My upgrade didn't go smoothly because uscan found also upstreams debian > watch file which is too loose and matches some other tar balls on their > distribution site. > > (Though I don't know why uscan keeps looking for watch files after > finding one in debian/watch) Oh dear. uscan(1) does say: Unless --watchfile is given, uscan looks recursively for valid source trees starting from the current directory (see the below section "Directory name checking" for details). and then: For each valid source tree found, typically the following happens: [...] so yes, it will look at more than one location. > And you were probably right in that arrow needs to be a team, because I > have no idea how to get other the other languages interfaces packaged. I suggest that without anyone else volunteering to do those other language interfaces (perhaps it's not a pressing need for people working with language X), I wonder whether it's worth just packaging the Python (and presumably C++) interfaces for now, and then if others want to join the effort to support language X later on, a new version of the Debian package can be uploaded with a new binary package for language X. It does mean more trips through the NEW queue if and when that happens, but given that no-one's shown interest in language X for the last several years, this is unlikely to be much of an issue. Version 7.0 provided support (it seems) for: GLib (seems that a draft framework for building this is already in the Debian package, and it can then be used in lots of languages), C++ (this is the core libraries), C# (not of interest to us), Go, Java, JavaScript, Julia, Matlab (not of interest to us), Python, R, Ruby. > Oh and I probably need to get the pyarrow installed somewhere, since it > was stopping at the tests I hadn't run into dh_missing errors yet. Oh. Would pybuild do that automatically (perhaps specifying PYBUILD_PACKAGE)? Best wishes, Julian
Re: Seeking a small group to package Apache Arrow (was: Bug#970021: RFP: apache-arrow -- cross-language development platform for in-memory analytics)
Hi Diane, On Fri, Mar 29, 2024 at 11:49:07AM -0700, Diane Trout wrote: > On Mon, 2024-03-25 at 18:17 +0000, Julian Gilbey wrote: > > > > > > So this is a plea for anyone looking for something really helpful to > > do: it would be great to have a group of developers finally package > > this! There was some initial work done (see the RFP bug report for > > details: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021), > > but that is fairly old now. As Apache Arrow supports numerous > > languages, it may well benefit from having a group of developers with > > different areas of expertise to build it. (Or perhaps it would make > > more sense to split the upstream source into a collection of > > different > > Debian source packages for the different supported languages. I > > don't > > know.) Unfortunately I don't have the capacity to devote any time to > > it myself. > > > > Thanks in advance for anyone who can step forward for this! > > I've been maintain dask and anndata and saw that apache arrow was > getting increasingly popular. > > I took the current science-team preliminary packaging 7.0.0 packaging > and managed to get it to build through a combination of patches and > turning off features. > > I even mostly managed to get pyarrow to build. (Though some tests fail > due to pytest lazy-fixture being abandoned). > > I pushed my current work in progress to. > > https://salsa.debian.org/diane/arrow.git > > Was anyone else planning on working on it or should I push my updates > to the science-team package? Lovely to hear from you, and oh wow, that's amazing, thank you! I can't speak for anyone else, but I suggest that pushing your updates to the science-team package would be very sensible; it would be silly for someone else to have to redo your work. What more is needed for it to be ready for unstable? Best wishes, Julian
Re: Pytest 8.x bug? "Permission denied: '/tmp/systemd-private-...-systemd-logind.service-.../__init__.py'"
On Wed, Mar 27, 2024 at 11:36:55AM +0100, Timo Röhling wrote:
> Hi,
>
> Am 27. März 2024 10:59:46 MEZ schrieb Julian Gilbey :
> >I'm stymied by a pytest-related bug. I thought it was a bug in a
> >particular pytest plugin (pytest-order), but it's now shown up in
> >another pytest plugin as well, so I wonder if either there's a bug in
> >pytest 8.x or something subtle has changed that requires a
> >modification to the plugins. I couldn't see anything obvious on the
> >pytest changelog page, and the error message itself is mysterious to
> >me. The bug does not show with pytest 7.4.4, so it is certainly
> >related to the new pytest version.
> I wasn't able to pinpoint the cause yet, but I noticed that the failing
> sessions have "rootdir: /tmp" instead of the usual
> /tmp/autopkgtest-lxc.*/downtmp/autopkgtest_tmp/build
Ah, I think you've hit the nail on the head!!
https://github.com/pytest-dev/pytest/issues/11781
And then pytest looks for any __init__.py file it can find, including
in unreadable directories...
Unfortunately, changing the argument from str(tmpdir) to
f"--rootdir={tmpdir}" caused my computer to crash (CPU usage went
through the roof until the computer became unresponsive) - clearly
there's something not quite right here yet!!
Best wishes,
Julian
Re: Pytest 8.x bug? "Permission denied: '/tmp/systemd-private-...-systemd-logind.service-.../__init__.py'"
On Wed, Mar 27, 2024 at 11:36:55AM +0100, Timo Röhling wrote: > Hi, > > Am 27. März 2024 10:59:46 MEZ schrieb Julian Gilbey : > >I'm stymied by a pytest-related bug. I thought it was a bug in a > >particular pytest plugin (pytest-order), but it's now shown up in > >another pytest plugin as well, so I wonder if either there's a bug in > >pytest 8.x or something subtle has changed that requires a > >modification to the plugins. I couldn't see anything obvious on the > >pytest changelog page, and the error message itself is mysterious to > >me. The bug does not show with pytest 7.4.4, so it is certainly > >related to the new pytest version. > I wasn't able to pinpoint the cause yet, but I noticed that the failing > sessions have "rootdir: /tmp" instead of the usual > /tmp/autopkgtest-lxc.*/downtmp/autopkgtest_tmp/build > > > Cheers > Timo Hi Timo, Oh, that is interesting! Good spot, thank you! I wonder whether tmpdir is creating a weird location? BTW, this problem appears when running autopkgtest under lxc. In my other package, the tests passed when running under sbuild but not with autopkgtest under lxc, which suggests that there's something weird about the lxc setup. Best wishes, Julian
Pytest 8.x bug? "Permission denied: '/tmp/systemd-private-...-systemd-logind.service-.../__init__.py'"
I'm stymied by a pytest-related bug. I thought it was a bug in a particular pytest plugin (pytest-order), but it's now shown up in another pytest plugin as well, so I wonder if either there's a bug in pytest 8.x or something subtle has changed that requires a modification to the plugins. I couldn't see anything obvious on the pytest changelog page, and the error message itself is mysterious to me. The bug does not show with pytest 7.4.4, so it is certainly related to the new pytest version. It occurs when running a script from within a test, and the error is when collecting the tests. Here is an example, taking from the report I filed at https://github.com/pytest-dev/pytest-order/issues/110 __ ERROR collecting . __ /usr/lib/python3/dist-packages/pluggy/_hooks.py:501: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) /usr/lib/python3/dist-packages/pluggy/_manager.py:119: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) /usr/lib/python3/dist-packages/_pytest/python.py:211: in pytest_collect_directory if pkginit.is_file(): /usr/lib/python3.12/pathlib.py:894: in is_file return S_ISREG(self.stat().st_mode) /usr/lib/python3.12/pathlib.py:842: in stat return os.stat(self, follow_symlinks=follow_symlinks) E PermissionError: [Errno 13] Permission denied: '/tmp/systemd-private-2dd08cf217854d6285343c302b696375-systemd-logind.service-qOgrXp/__init__.py' (That page also has the full autopkgtest log.) I have no idea why it is looking at this private systemd directory; it's running under autopkgtest so HOME should be sane. Any ideas would be much appreciated, as I would be able to use it to fix the tests on the other plugin I'm working on as well. Best wishes, Julian
Seeking a small group to package Apache Arrow (was: Bug#970021: RFP: apache-arrow -- cross-language development platform for in-memory analytics)
Hi all, [NB: sent to d-science, d-python, d-devel and the RFP bug; reply-to set to d-science and the RFP bug only] An update on Apache Arrow, and in particular the Python library PyArrow. For those who don't know: Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enable big data systems to process and move data fast. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. The project is developing a multi-language collection of libraries for solving systems problems related to in-memory analytical data processing. This includes such topics as: * Zero-copy shared memory and RPC-based data movement * Reading and writing file formats (like CSV, Apache ORC, and Apache Parquet) * In-memory analytics and query processing (from: https://arrow.apache.org/docs/index.html) Pandas has announced that Pandas 3.x will depend on PyArrow in a critical way (it will back the "string" datatype), and it is due to be released imminently. So this is a plea for anyone looking for something really helpful to do: it would be great to have a group of developers finally package this! There was some initial work done (see the RFP bug report for details: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021), but that is fairly old now. As Apache Arrow supports numerous languages, it may well benefit from having a group of developers with different areas of expertise to build it. (Or perhaps it would make more sense to split the upstream source into a collection of different Debian source packages for the different supported languages. I don't know.) Unfortunately I don't have the capacity to devote any time to it myself. Thanks in advance for anyone who can step forward for this! Best wishes, Julian
Changing hatchling shared-data installation directory: /usr/etc -> /etc
I'm trying to package jupyter-server-terminals, and have hit a snag. The pyproject.toml file includes the lines: [tool.hatch.build.targets.wheel.shared-data] "jupyter-config" = "etc/jupyter/jupyter_server_config.d" but the resulting file is installed at /usr/etc/jupyter/jupyter_server_config.d Now, I can obviously move this to its correct location in debian/rules (either directly or using an appropriate dh-recognised file in debian/), but I wonder whether there is a "better" way to do this. Is there a way to tell hatchling that shared-data should be installed in / rather than in /usr? Or if I tell hatchling to use / as the base directory, would that mess everything else up? I couldn't figure it out from a quick skim of the hatchling docs, so any thoughts or pointers would be welcome. Thanks! Julian
Bug#1067248: ITP: pytest-jupyter -- Pytest plugins for Jupyter libraries and extensions
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: pytest-jupyter Version : 0.9.1 * URL : https://github.com/jupyter-server/pytest-jupyter * License : BSD-3-clause and Expat Programming Lang: Python Description : Pytest plugins for Jupyter libraries and extensions This set of pytest plugins are used for testing Jupyter. It includes an echo kernel to speed up testing. It uses pytest-tornasync internally for async test suite running, and may not be compatible with pytest-asyncio. This package is a new requirement for the package tests for jupyter-client and jupyter-server (newer versions of these packages that have not yet been packaged). It will be maintained within the Debian Python Team.
Bug#1067200: RM: cython-legacy -- ROM; Legacy transition package no longer needed
Package: ftp.debian.org Severity: normal User: ftp.debian@packages.debian.org Usertags: remove X-Debbugs-Cc: debian-python@lists.debian.org cython-legacy was introduced as a transitional measure until cython version 3.x was supported by all source packages using cython. There are now no source packages in unstable or testing (main) depending on cython3-legacy and no binary packages either, so this legacy package can be removed. (It also has an RC bug against it, and doesn't really work with Python 3.12.) Best wishes, Julian
Re: Salsa: Setup tokens
Hi Christian, On Tue, Mar 19, 2024 at 01:29:55PM +0100, Joost van Baal-Ilić wrote: > Hi Christian, > > Thanks for working on the documentation! > > Op Tue, Mar 19, 2024 at 10:54:43AM + schreef c.bu...@posteo.jp: > > Hello, > > [...] > > I can answer some of your questions, but not all of them. What I usually do: > > Get a personal access token "somesecret" from the salsa web interface. Then: > > $ touch ~/.salsa-token > $ chmod og-r ~/.salsa-token > $ echo somesecret > ~/.salsa-token > $ SALSA_TOKEN=`cat ~/.salsa-token` > > For some actions, the salsa(1) tool (and/or gitlab on salsa.d.o) seems to need > such a token. An SSH key is not always enough. > > HTH, Bye, > > Joost Might also be worth taking a look at https://wiki.debian.org/Javascript/Tutorial which has a whole discussion about using salsa and tokens. Best wishes, Julian
Re: morph's abandoned packages (list)
On Fri, Mar 15, 2024 at 10:04:42AM +, Jelmer Vernooij wrote: > On Thu, Mar 14, 2024 at 06:20:11AM +0000, Julian Gilbey wrote: > > [...] > Thanks for collecting the list of packages. I'm planning to adopt these: > > > #1065327 O: python-levenshtein -- extension for computing string > > similarities and edit distances > > #1065223 O: pysimplesoap -- simple and lightweight SOAP Library Hi Jelmer, I've just taken a look at python-levenshtein, as I remember the name now: it might make more sense for me to take it as it depends on rapidfuzz and rapidfuzz-cpp, which I've just packaged and are sitting in NEW. But if you want to take it, please feel free to do so! (Once rapidfuzz makes it into unstable, a lot of debian/rules could probably also be simplified.) Best wishes, Julian
morph's abandoned packages (list)
Dear all (and Bcc-ing the RM bugs), For information, here is a list of packages that morph has either requested removal of or orphaned. If you are interested in taking one or more of them on, that would be great! Removal requested: #1066146 RM: flask-basicauth -- ROM; RC buggy, dead upstream, leaf package #1065141 RM: gmplot -- ROM; leaf package #1064947 RM: nb2plots -- ROM; leaf package #1065200 RM: overpass -- ROM; leaf package #1065199 RM: pprintpp -- ROM; leaf package #1065045 RM: pyannotate -- ROM; leaf package #1065201 RM: python-overpy -- ROM; leaf package #1065202 RM: python-ppmd -- ROM; leaf package #1064946 RM: sphinx-a4doc -- ROM; leaf package Recently-orphaned packages (removing those in wnpp which have been retitled "ITA") sorted alphabetically; these could, of course, be brought into team maintenance. #1065235 O: basemap -- matplotlib toolkit to plot on map projections #1065243 O: colorspacious -- library for doing colorspace conversions #1065151 O: commonmark -- Python parser for the CommonMark Markdown spec #1065246 O: contourpy -- Python library for calculating contours of 2D quadrilateral grids #1065248 O: cppy -- C++ headers for (Python) C extension development #1065139 O: dot2tex -- Graphviz to LaTeX converter #1065140 O: fastkml -- fast KML processing #1065142 O: html5lib -- HTML parser/tokenizer based on the WHATWG HTML5 specification #1065244 O: kiwisolver -- fast implementation of the Cassowary constraint solver #1065238 O: lazy-object-proxy -- Python 3 fast and thorough lazy object proxy #1065037 O: m2crypto -- Python wrapper for the OpenSSL library #1065325 O: matplotlib -- Python based plotting system #1065143 O: mkautodoc -- AutoDoc for MarkDown #1065042 O: mpl-sphinx-theme -- documentation for the mpl-sphinx-theme Python library #1065220 O: mpmath -- library for arbitrary-precision floating-point arithmetic #1065224 O: mysql-connector-python -- pure Python implementation of MySQL Client/Server protocol #1065198 O: networkx -- tool to create, manipulate and study complex networks #1065329 O: numpy -- Fast array facility to the Python 3 language #1065221 O: py7zr -- pure Python 7-zip library #1065222 O: pychm -- Python binding for CHMLIB #1065231 O: pydot -- Python interface to Graphviz's dot #1065152 O: pygeoif -- basic implementation of the __geo_interface__ #1065036 O: pyopenssl -- Python wrapper around the OpenSSL library #1065149 O: pyproject-metadata -- Dataclass for PEP 621 metadata with support for [core metadata] generation #1065223 O: pysimplesoap -- simple and lightweight SOAP Library #1064977 O: python-cryptography-vectors -- Test vectors for python-cryptography #1065327 O: python-levenshtein -- extension for computing string similarities and edit distances #1065025 O: sphinx-book-theme -- clean book theme for scientific explanations and documentation with Sphinx #1065026 O: sphinx-bootstrap-theme -- bootstrap theme for Sphinx #1065030 O: sphinxcontrib-log-cabinet -- Organize changelog directives in Sphinx docs #1065027 O: sphinx-copybutton -- sphinx extension to add a "copy" button to code blocks #1065028 O: sphinx-gallery -- extension that builds an HTML gallery of examples from Python scripts #1065029 O: sphinx-panels -- documentation for the sphinx-panels Python library #1065043 O: sphinxtesters -- utilities for testing Sphinx extensions #1064948 O: texext -- sphinx extensions for working with LaTeX math There's also an old ITP that was closed: #1015231 ITP: sphinx-theme-builder -- tool for authoring Sphinx themes with a simple (opinionated) workflow Best wishes, Julian
Re: "Fatal Python error: Aborted" errors for Python/Qt packages
On Tue, Mar 12, 2024 at 07:55:55AM +0100, Roland Mas wrote: > Hi Julian, Ghislain, list, > > I'm working on various Qt-related Python packages, and I'm seeing strange > errors when building in cowbuilder chroots (with git-buildpackage). They > don't seem to happen when building out-of-chroot. So far I managed to track > them down to qtpy, but I'm stumped as to the why and how to fix. For > instance, when building from commit > b360a9defbb470fe6ab1793371d16487e52b548b, I get the following output during > the testsuite: Hi Roland, I wonder whether it's related to the current t64 library changes in unstable? These are causing all sorts of issues during the transition period. Do the packages build OK in a testing chroot? Best wishes, Julian
Re: Suggesting change in DPT Policy
On Sat, Mar 09, 2024 at 06:46:52PM +0100, Anton Gladky wrote: > Same for me. Thanks for proposal. +1 > Anton > Am Sa., 9. März 2024 um 17:51 Uhr schrieb Nilesh Patra : > > I am late to the party but I agree with the policy change. Following on from some earlier discussions, I've been thinking about the relationship between the DPT (presumably a group of developers who work together) and salsa (could there be packages in the python-team/packages area which are not team maintained?). I reread much of the policy at https://salsa.debian.org/python-team/tools/python-modules/blob/master/policy.rst and discovered something quite strange. The introduction begins: --- Introduction: The Debian Python Team (DPT) aims to improve the Python packages situation in Debian, by packaging available modules and applications that may be useful and providing a central location for packages maintained by a team, hence improving responsiveness, integration, and standardization. The DPT is hosted at salsa.debian.org, the Debian GitLab installation. We currently have a Git repository and a mailing list whose email address can be used in the Maintainer or Uploaders fields on co-maintained packages. --- If the DPT is a team (a group of maintainers/developers/helpful others), what does "The DPT is hosted at salsa" mean - how can a "team" be hosted? (And in the first paragraph, "maintained by a team" seems a little strange too.) Perhaps something like the following would be better (shifting the focus from the tools to the people), and would also separate concerns more clearly: --- Introduction: The Debian Python Team (DPT) is a group of maintainers who are jointly responsible for a large number of Python packages in Debian. They package and support available Python modules and applications that may be useful. By using a central location on salsa.debian.org, the Debian GitLab instance, for these team-maintained packages, the DPT are able to improve responsiveness, integration, and standardization. --- Then the details of how to mark a package as being team-maintained can be left to the Maintainership section. We could then include a statement along the lines of the following (though I'm not sure where would be best): --- Python module packages which are not team-maintained by the DPT can also be stored in the python-team/packages namespace on salsa in order to benefit from the integration and standardization tools such as Janitor. Manual changes to these packages by someone other than the package's maintainer should be proposed via salsa merge requests or comments in the BTS (or using NMUs if appropriate) as for any other individually-maintained package. --- It would be good to say something about Uploaders in the Maintainership section. Perhaps something like this: --- A package maintained within the team must have the name of the team in the Maintainer field: Maintainer: Debian Python Team This enables the team to have an overview of its packages on the DDPO_website. If a particular developer wishes to take primary responsibility for a package, they should put their name in the Uploaders field. [*** What does this mean though? Maybe something like: In this case, any DPT member is still welcome to make changes to the package, though it is polite to contact the developer(s) named in the Uploaders field first. ***] If there are complications in the packaging of the module, for example, if certain modules are interdependent and need to be updated together, this should be documented in debian/README.source [*** or somewhere else ***] --- Best wishes, Julian
Please delete two empty salsa repositories
Hello, Please could someone with the required permissions delete the following two empty salsa repositories; due to an upstream reorganisation of the package, they will no longer be needed, and will instead be part of the rapidfuzz package: https://salsa.debian.org/python-team/packages/jarowinkler https://salsa.debian.org/python-team/packages/rapidfuzz-capi Thanks, Julian
Re: Suggesting change in DPT policy
Hi Jeroen, On Thu, Feb 29, 2024 at 08:48:33PM +0100, Jeroen Ploemen wrote: > On Tue, 27 Feb 2024 18:32:54 + > Scott Kitterman wrote: > > > While I do take advantage of this for a few packages, I don't > > personally care much either way. I suspect that packages will be > > removed from team maintenance as a result though and I think that's > > a bad idea. > > > > I'd prefer the current approach over people removing packages from > > the team, so I think it's important that anyone who feels strongly > > enough about this to do so, speak up. > > For me, the weaker collaboration option that the DPT provides is key > to putting my packages under a team umbrella. > > As I find myself completely AFK for up to a month at a time for both > work and private reasons, having a knowledgeable bunch of developers > around with full access to my packages (VCS and CI included) is a > definite plus, if only out of a strong sense of responsibility. The > same goes for benefiting from the shared knowledge of the team, > including routine fixes and similar minor changes being rolled out > across all packages in the team. These are really interesting points. Under the proposed system, I presume that one could leave "privately maintained" packages within the python-team area of salsa and still benefit from these automatic changes without giving automatic permission to other developers to make manual changes. Being part of the team is a relationship between developers; it surely doesn't say anything about a specific package's maintenance, just as one can ask questions on debian-devel without saying "anyone can do anything to my packages without asking me". > That said, I'm very careful not to spend more time on volunteer > efforts than I can afford to, and not looking to offload the full > maintenance of any of my packages. Upstream involvement often gives > me advance knowledge of upcoming releases, their compatibility issues > and other quirks, which in turn helps avoid problems on the Debian > end. I do not share the optimism that documenting such details > somewhere in individual packages - as suggested elsewhere in this > thread - would be effective to avoid trouble; more so considering > that the inability to stick to a single, concise, and rather clear > team policy is ultimately what triggered this whole discussion in the > first place. I don't think it's a binary choice: "offload the full maintenance" of a package versus "keep the full maintenance". As far as I understand it, a package maintained by the team will typically have a main person who does the day-to-day work on it (few people have the time to start doing serious work on lots of other packages), but anyone on the team could work on it. In the cases mentioned, there are RC bugs in packages which have not been addressed in a timely fashion and are holding up transitions or similar. In some of "my" packages, other developers on the team have uploaded more regular updates (thanks!). In most cases, updates are routine and I'm very appreciative of it. While documenting quirks might not fully avoid trouble, it's much better than not documenting them. After all, this is detailed knowledge of a package or collection of packages that has been gained over time, and in addition to helping anyone stepping in to do a team upload, documenting it will help whoever ends up taking over the package one day (as well as helping the maintainer themselves!). The question for your quirky packages then becomes: what does the current team maintenance position offer that having the package solely maintained by yourself would not provide? I can see very little; anyone wanting to help with a package outside of the team can still offer to do an NMU (and push their changes to salsa). > As for the inclusion of codes of conduct or similar wording, > documenting common sense just feels unnecessary. While being on the > receiving end of a compliment for bug-squashing work is certainly > nice, the lack thereof isn't a measure of disrespect. I cannot recall > any discussion on the team's IRC channel or mailing list crossing > that line. As far as I understand, this thread was started not because Andreas did not receive a compliment, but because he received quite unpleasant emails "telling him off" for doing it. These were not public communications, so you would not have seen them. I don't think anyone is suggesting that every team upload requires a compliment (though such things are, of course, nice and appreciated!). Best wishes, Julian
Re: Suggesting change in DPT policy
On Tue, Feb 27, 2024 at 09:05:44AM +0100, Andreas Tille wrote: > Hi, > > [...] +1 from me, too. I had completely forgotten this paragraph in the Python policy and it doesn't make that much sense. I personally generally do send an email to anyone in the "Maintainers" field or the last uploader just as a matter of courtesy before working on something, and then wait for a day before doing anything; for all I know, they may already be working on a patch, especially if it's a very new bug. But if the model of team maintainership is made much stronger and clearer, though I would still encourage sending an email to the "main maintainer" for anything non-trivial, even if just to let them know the situation. In response to other comments, I suggest that those maintainers who do not wish other team members to help work on their packages under this new approach should just remove DPT from the Maintainer and Uploader fields, and regard any offers of help as an NMU or merge request. One tweak to Andreas's patch: > diff --git a/policy.rst b/policy.rst > index 27bf6f3..7185d6d 100644 > --- a/policy.rst > +++ b/policy.rst > @@ -48,20 +48,14 @@ Maintainership > == > > A package maintained within the team should have the name of the team either this should be: - A package maintained within the team should have the name of the team either + A package maintained within the team should have the name of the team > -in the ``Maintainer`` field or in the ``Uploaders`` field. > +in the ``Maintainer``. Best wishes, Julian
Re: Breaking changes in pytest 8
Hi Timo, And the transition is now complete :-) Best wishes, Julian On Tue, Jan 09, 2024 at 05:45:04PM +, Julian Gilbey wrote: > Hi Timo, > > Please can we hold back on uploading pytest 8 to unstable until the > current Python 3.12 transition is complete? It is entirely possible > that several of the packages that currently break with pytest 8 > already have newer upstream versions that address these issues, but > it's probably not wise to mix that in with the current transition. > > Best wishes, > >Julian > > On Mon, Jan 08, 2024 at 11:32:29PM +0100, Timo Röhling wrote: > > Hi, > > > > I recently uploaded the pre-release pytest 8.0.0~rc1 to experimental as an > > early warning for the breaking changes which typically happen on major > > version bumps. > > > > I've attached a dd-list of packages which exhibit autopkgtest regressions > > [1], with the intent of MBF'ing (with separate announcement) once pytest 8 > > is released. > > > > Typically, packages will fail if they > > - have deprecation warnings of type PytestRemovedIn8Warning, or > > - assume a particular pytest stdout/stderr output which might have > > changed, or > > - rely on the precise order in which pytest collects tests, especially the > > behavior of the pytest.Package collector. > > > > Please refer to the upstream changelog [2] for a complete list of breaking > > changes. > > > > > > Cheers > > Timo > > > > [1] https://qa.debian.org/excuses.php?experimental=1=pytest > > [2] https://docs.pytest.org/en/latest/changelog.html
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
On Fri, Jan 26, 2024 at 08:43:03AM +0200, Graham Inggs wrote: > Hi > > On Tue, 23 Jan 2024 at 14:38, Julian Gilbey wrote: > > We're nearly there (the transition page says it's 99% done), and when > > this transition is complete, then python3-defaults 3.11.6+ will be > > able to migrate to testing. > > python3-defaults/3.11.6-1 with Python 3.12 as a supported version is > now in testing [1]. Wonderful news! Congratulations to everyone who helped to make this happen! Best wishes, Julian
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
On Mon, Jan 22, 2024 at 08:50:55PM +, Rebecca N. Palmer wrote: > On 22/01/2024 11:51, Julian Gilbey wrote: > > Please could we wait until the "Python 3.12 is a supported version" > > transition is completed? > > How are you defining that? python3-defaults 3.11.6+ in testing? (I was > previously told 3.12-supporting pandas and numpy in testing, which has > happened. I don't think any of these 25 packages are on > https://qa.debian.org/excuses.php?package=python3-defaults , but I haven't > checked carefully, and at least influxdb-python and tqdm do have what I > suspect are Python 3.12 related issues.) https://release.debian.org/transitions/html/python3.12-add.html We're nearly there (the transition page says it's 99% done), and when this transition is complete, then python3-defaults 3.11.6+ will be able to migrate to testing. I don't fully understand the problem with transitions, but there was a request to hold back with significant upgrades until a python3.12-supporting python3-defaults has reached testing. > > Adding another 25 or so RC bugs at this > > point will just slow down that transition. > > What exactly do you want not done until then? Just not uploading pandas > 2.x to unstable, or is it also a problem to have these bugs marked as RC in > the BTS? (In all 22 cases that are in testing at all, the bug is also > present in the version in testing, so it being RC shouldn't block > migration.) Yes - please don't upload it to unstable yet. Uploading to experimental is fine. > > (Unless pandas 1.5 is > > preventing the transition, that is.) > > It isn't. Good! Julian
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
On Sun, Jan 21, 2024 at 03:29:21PM +, Rebecca N. Palmer wrote: > Control: severity 1053943 1053939 1053942 1044053 1044056 serious > Control: severity 1044074 1053946 1044078 1044079 1044077 serious > Control: severity 1044071 1044067 1044068 1044055 1044060 serious > Control: severity 1044072 1044073 1044064 1053945 1044054 serious > Control: severity 1044076 1053940 1044057 1053944 1050144 serious > > As previously discussed in this bug, I'd like to move pandas 2.x into > unstable reasonably soon. I'm aiming to get it in before the Ubuntu 24.04 > freeze (in about a month), but I am open to disagreement on whether this is > a good idea. Please could we wait until the "Python 3.12 is a supported version" transition is completed? Adding another 25 or so RC bugs at this point will just slow down that transition. (Unless pandas 1.5 is preventing the transition, that is.) Best wishes, Julian
Re: Breaking changes in pytest 8
Hi Timo, Please can we hold back on uploading pytest 8 to unstable until the current Python 3.12 transition is complete? It is entirely possible that several of the packages that currently break with pytest 8 already have newer upstream versions that address these issues, but it's probably not wise to mix that in with the current transition. Best wishes, Julian On Mon, Jan 08, 2024 at 11:32:29PM +0100, Timo Röhling wrote: > Hi, > > I recently uploaded the pre-release pytest 8.0.0~rc1 to experimental as an > early warning for the breaking changes which typically happen on major > version bumps. > > I've attached a dd-list of packages which exhibit autopkgtest regressions > [1], with the intent of MBF'ing (with separate announcement) once pytest 8 > is released. > > Typically, packages will fail if they > - have deprecation warnings of type PytestRemovedIn8Warning, or > - assume a particular pytest stdout/stderr output which might have > changed, or > - rely on the precise order in which pytest collects tests, especially the > behavior of the pytest.Package collector. > > Please refer to the upstream changelog [2] for a complete list of breaking > changes. > > > Cheers > Timo > > [1] https://qa.debian.org/excuses.php?experimental=1=pytest > [2] https://docs.pytest.org/en/latest/changelog.html
Re: sphinx-build module not found in sbuild
On Mon, Jan 08, 2024 at 01:22:54PM +0100, Jérémy Lal wrote: > Hi, > I'm stuck at this odd behavior: > when I build a package in my current environment (debian/testing), > sphinx-build ... works correctly. > when building in sbuild, sphinx-build doesn't find current module: > ModuleNotFoundError: No module named 'xxx' > in docs/conf.py at > from xxx import __version__ > I don't understand which other debian package is modifying that outcome. > Any idea ? Perhaps your local build is importing from a globally-installed version of the package? (For example, you're building version 1.3 of python3-foo, but you already have version 1.2 of python3-foo installed.) Best wishes, Julian
Working on dask and dask.distributed update
FYI: I'm most of the way through updating dask and dask.distributed to fix their FTBFS bug reports. I hope to upload either tomorrow or Wednesday. I haven't pushed anything to salsa yet. Happy new year! Julian
Re: cython 3.x (for Python 3.12)
On Sat, Nov 25, 2023 at 04:23:46PM +, Stefano Rivera wrote: > As part of preparing for Python 3.12 in Debian, I've uploaded cython 3 > to experimental. > [...] > > So, that's 71 regressions with cython3. dd-list below. Please help us > port to cython 3. If this isn't possible, Graham is preparing a > cython-legacy package, to help the stragglers. But we're expecting that > this won't have great Python 3.12 support... > https://ftp-master.debian.org/new/cython-legacy_0.29.36-1~exp1.html Here's an update on cython-legacy. I just tried running autopkgtest on cython-legacy with Python 3.12, and unfortunately it fails. I can't quite work out how many tests fail (I don't understand the format of the test output), but there are 25 lines containing the string "FAIL" in the output of autopkgtest with the latest version of cython-legacy (0.29.36-2, uploaded today). We could just skip these tests - some of them are probably harmless - but they probably indicate incompatibilty (unsurprisingly) with Python 3.12, though some of these incompatibilties may be restricted to the tests themselves rather than the underlying Cython 0.x code. I'd be happy to try uploading a version with these tests skipped - I just wanted to check here first what the thoughts on this are. (Note that it is much better to modify other packages to work with Cython 3.x!) Best wishes, Julian
Re: cython 3.x (for Python 3.12)
On Mon, Dec 11, 2023 at 08:06:41PM +0100, Matthias Klose wrote: > > [...] > > Excellent - I didn't know about that. Are you OK for me to upload > > versions of cython and cython-legacy which depend on this to fix the > > Python 3.12 breakage? > > not for cython, which won't need that soonish for 3.0.x. and if you have to > update the b-d to cython3-legacy, why not add the zombie-imp dependency as > well manually for the few packages that need it? The problem is not with other packages importing imp (which need to be fixed in such a case), rather that pyximport (in cython/cython-legacy) imports imp. So it's cython that needs to be patched. I'm wondering what the provisional timescale for cython 3.0.x is? Should I just leave my package with an autopkgtest failure until cython 3.0.x is in unstable or ideally testing? That's why I was thinking of also patching cython in the short term until we are ready for cython 3.0.x to enter unstable. Best wishes, Julian
Re: cython 3.x (for Python 3.12)
On Mon, Dec 11, 2023 at 04:34:17PM +0100, Matthias Klose wrote: > On 11.12.23 16:19, Julian Gilbey wrote: > > On Mon, Dec 11, 2023 at 08:09:31AM +0100, Matthias Klose wrote: > > > [...] > > > You could package a non-conflicting cython-legacy, however that would > > > require more changes, also how to build it. > > > > Hi Matthias, > > > > Unfortunately, at least some of cython3-legacy doesn't currently work > > with Python 3.12, and is the primary cause of (at least) #1056531. > > cython3 provides the pyximport module, and that uses the imp module > > which has been removed from Python 3.12. > > > > Two possible ways forward on this particular bug: > > > > - Disable all of the cython tests for this package for the time being, > >until cython 3.x migrates to testing - this is simple and effective. > > > > - Patch cython3-legacy to use importlib rather than imp. This is > >probably a good thing to do anyway. (It may also be good to do this > >with cython3 version 0.x currently in testing/unstable until cython > >3.x is able to be uploaded to unstable.) Then have my package's > >autopkgtest depend on cython3-legacy (unless cython3 0.x is also > >patched). > > I won't working on this. Have you tried to depend on the python3-zombie-imp > instead? Excellent - I didn't know about that. Are you OK for me to upload versions of cython and cython-legacy which depend on this to fix the Python 3.12 breakage? Best wishes, Julian
Re: cython 3.x (for Python 3.12)
On Mon, Dec 11, 2023 at 08:09:31AM +0100, Matthias Klose wrote: > [...] > You could package a non-conflicting cython-legacy, however that would > require more changes, also how to build it. Hi Matthias, Unfortunately, at least some of cython3-legacy doesn't currently work with Python 3.12, and is the primary cause of (at least) #1056531. cython3 provides the pyximport module, and that uses the imp module which has been removed from Python 3.12. Two possible ways forward on this particular bug: - Disable all of the cython tests for this package for the time being, until cython 3.x migrates to testing - this is simple and effective. - Patch cython3-legacy to use importlib rather than imp. This is probably a good thing to do anyway. (It may also be good to do this with cython3 version 0.x currently in testing/unstable until cython 3.x is able to be uploaded to unstable.) Then have my package's autopkgtest depend on cython3-legacy (unless cython3 0.x is also patched). Thoughts? Julian
Re: cython 3.x (for Python 3.12)
On Mon, Dec 11, 2023 at 08:09:31AM +0100, Matthias Klose wrote: > [...] > > > > I find that there's also a significant issue with relying on > > > > cython3-legacy: it conflicts with cython3, meaning that it will be > > > > impossible to simultaneously install packages depending on cython3 and > > > > cython3-legacy. Once cython 3.x moves from experimental to unstable > > > > to testing and packages start depending on it, this will become a > > > > significant issue. I assume that the aim will be for everything to be > > > > ported to cython 3.x and for cython3-legacy to be dropped from testing > > > > before the trixie freeze? > [...] > You could package a non-conflicting cython-legacy, however that would > require more changes, also how to build it. I had a very quick look at the packaged cython3-legacy. One could change the name of the module to Cython_legacy or probably better, Cython0, and cython.py to cython0.py, pyximport to pyximport0. Just a handful of changes would be needed, I'm guessing. Then it would be co-installable with Cython 3.x, and any packages that actually need cython3-legacy rather than Cython 3.x would have to be patched to call the legacy version. It's probably easier to just patch them for Cython 3.x ;-) Best wishes, Julian
Re: pandas 1.5 -> 2.1?
Hi Kingsley, On Sun, Dec 10, 2023 at 12:55:43PM -0800, Kingsley G. Morse Jr. wrote: > Hi Rebecca, Julian and all science minded pythonistas of debian, great and > small! > > I like your correspondence about upgrading from > version 1.5 of pandas to 2.1. > > It's open, scientific and explores the ideal of > proceeding wisely in a matter of public interest. > > My humble thoughts are: > > 1.) Rebecca: *Why* did you write that you'd like > to move forward with the pandas 1.5 -> 2.1 > transition? What's your reason? A thought from me on this: pandas 2.1 has many improvements over pandas 1.5. And increasingly, other packages will be requiring these new features. So why would one not want to move forward with it? > 2.) What may be the advantage of migrating to > version 3.0 of Cython? It is compatible with Python 3.12, whereas the current version of Cython in Debian (0.29.x) is not really. (For example, it has an "import imp" in it, and this breaks with Python 3.12, which has removed this deprecated module.) As Cython 0.29.x is no longer maintained upstream, having been superseded by Cython 3.x after many years of development, our options are to either continue to patch Cython 0.29.x within Debian to keep it working with Python 3.12 or to upgrade to Cython 3.x. As there is also software which now depends on Cython 3.x to build, the former option seems unappealing. (At best, we might wish to keep the cython-legacy package around for building packages which can't yet use Cython 3.x, but that should be a short-term thing, not a long-term one.) > 3.) The following one-liner suggests 44 debian > packages might be affected by the breaks > Rebecca said would be caused by pandas 2.x: > > $ for s in augur cnvkit dyda emperor esda mirtop pymatgen pyranges > python-anndata python-biom-format python-cooler python-nanoget python-skbio > python-ulmo q2-quality-control q2-demux q2-taxa q2-types q2templates > sklearn-pandas ; do apt-cache search "$s" ; done | less This does not seem like a particularly helpful one-liner; it picks up packages such as python3-dyda-pipeline-config which are not in the original list. Instead, you perhaps want to count the number of packages depending on these packages. But what Rebecca is looking at (I think) is how many packages would need fixing by the pandas upgrade. (But it is probably worse than this: I'm guessing these are only the packages which fail to build with pandas 2.x or whose autopkgtest fails with pandas 2.x. But there may well be other breakage caused by the upgrade which is not detectable in this way. That is an issue which will have to be handled by individual packages as they are discovered, and the timing of the pandas upgrade is not related to this problem.) > 4.) The break that worries me the most is > sklearn-pandas, because it seems to me that > sklearn is > > popular and > > fundamental. It seems that sklearn-pandas is abandoned; there were just two commits in 2022, and prior to that was May 2021. There has been no activity since. If someone is willing to patch it for Pandas 2.x, great (perhaps you might help the maintainer to do this?), otherwise it might have to drop out of Debian. Best wishes, Julian
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
On Sun, Dec 10, 2023 at 01:06:01PM +, Rebecca N. Palmer wrote: > I'd like to move forward with the pandas 1.5 -> 2.1 transition reasonably > soon. > > Given that pandas 2.x is *not* required for Python 3.12 (but is required for > Cython 3.0), should we wait for the Python 3.12 transition to be done first? Well, I have seen at least one package that has an RC bug for the Python 3.12 transition that might be because it's still using an old version of cython3 :( So it's a bit of chicken-and-egg - having Cython 3.0 might be very helpful. But then there is this list of 28 packages broken by pandas 2.x. On the other hand, these will need fixing at some point soon anyway, so I'd be in favour of doing the pandas transition now, which will allow Cython 3.0 to move into unstable. Just my 2 cents' worth... Best wishes, Julian
Re: cython 3.x (for Python 3.12)
On Sat, Nov 25, 2023 at 04:23:46PM +, Stefano Rivera wrote: > As part of preparing for Python 3.12 in Debian, I've uploaded cython 3 > to experimental. > [...] > > So, that's 71 regressions with cython3. dd-list below. Please help us > port to cython 3. If this isn't possible, Graham is preparing a > cython-legacy package, to help the stragglers. But we're expecting that > this won't have great Python 3.12 support... > https://ftp-master.debian.org/new/cython-legacy_0.29.36-1~exp1.html I find that there's also a significant issue with relying on cython3-legacy: it conflicts with cython3, meaning that it will be impossible to simultaneously install packages depending on cython3 and cython3-legacy. Once cython 3.x moves from experimental to unstable to testing and packages start depending on it, this will become a significant issue. I assume that the aim will be for everything to be ported to cython 3.x and for cython3-legacy to be dropped from testing before the trixie freeze? Best wishes, Julian
Re: cython 3.x (for Python 3.12)
On Sat, Nov 25, 2023 at 04:23:46PM +, Stefano Rivera wrote: > [...] > So, that's 71 regressions with cython3. dd-list below. Please help us > port to cython 3. If this isn't possible, Graham is preparing a > cython-legacy package, to help the stragglers. But we're expecting that > this won't have great Python 3.12 support... > https://ftp-master.debian.org/new/cython-legacy_0.29.36-1~exp1.html Indeed; there's already one Python 3.12 related bug in the legacy version: it tries to import imp in pyximport.py. Best wishes, Julian
Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build
On Wed, Sep 06, 2023 at 08:05:45AM -0700, Soren Stoutner wrote: > As a followup question, I have noticed that a lot of packages (including > electrum, which I have recently started maintaining) ship the egg-info > directory. Looking through /usr/lib/python3/dist-packages/, this is common > but > not universal. Is there any reason to ship this directory or should it be > removed from the binary packages? Lots of packages depend on the egg-info directory being present. $ grep EASY-INSTALL-ENTRY-SCRIPT /usr/bin/* will give a (probably very long) list of executables that depend on an egg-info (or equivalent) directory being present. Best wishes, Julian
Re: Bug#1042443: ITP: pathos -- Framework for heterogeneous parallel computing
Hi Agathe, On Fri, Sep 01, 2023 at 09:46:00AM +0200, Agathe Porte wrote: > Hi Julian, > > 2023-07-28 10:59 CEST, Julian Gilbey: > > Package: wnpp > > Severity: wishlist > > Owner: Julian Gilbey > > X-Debbugs-Cc: debian-de...@lists.debian.org, Debian Python Team > > > > > > * Package name: pathos > > Version : 0.3.1 > > Upstream Contact: Mike McKerns > > * URL : https://github.com/uqfoundation/pathos > > * License : BSD-3-clause > > Programming Lang: Python > > Description : Framework for heterogeneous parallel computing > > > > […] > > > > > > This is a package I've started using; it provides a very effective > > framework for parallel computing, allowing for constructs that the > > standard Python library does not support. > > > > I will maintain it within the Debian Python Team. > > Like python-ppft, this was already packaged in the Python team but not > uploaded: https://salsa.debian.org/python-team/packages/python-pathos > > Maybe you can find inspiration in it. I think we should only keep one of > the two repos in the DPT because it can be confusing to have the same > package twice. Oh! I had not realised. I didn't see an ITP for this package, and so I went ahead and did it myself. So yes, let's delete or archive the duplicate repos for this and ppft; would you be able to do that? Now I'm just waiting on upgrades to python3-multiprocessing and python3-dill before I can upload a source-only version of these new packages. Best wishes, Julian
Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build
On Fri, Aug 18, 2023 at 09:23:17AM -0400, Scott Talbert wrote: > On Fri, 18 Aug 2023, Andreas Tille wrote: > > > Am Fri, Aug 18, 2023 at 01:42:53PM +0100 schrieb Julian Gilbey: > > > I'm sure I'm not the only one who received a whole bunch of bugs > > > entitled "Fails to build source after successful build" last weekend. > > > There was one theme common to most of them: the presence of a > > > *.egg-info directory which was not cleaned by debian/rules clean. > > > [...] > > It is being worked on: > https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/46 Amazing! Thanks, Julian
Uncleaned egg-info directory giving lots of bugs about failing to build after successful build
I'm sure I'm not the only one who received a whole bunch of bugs entitled "Fails to build source after successful build" last weekend. There was one theme common to most of them: the presence of a *.egg-info directory which was not cleaned by debian/rules clean. I know the bug report said that this policy is currently under discussion, but I did get thinking about it. I imagine that this particular directory should be the responsibility of dh-python to clean up, but it may not be sensible to always delete *.egg-info directories, as they may be present in the orig.tar.gz file. One could handle it by manually adding this directory to debian/clean in each package, but perhaps this should be the default behaviour of dh-python? Any thoughts? Best wishes, Julian
Re: dask.distributed RC bug #1042135
Hi Diane, On Fri, Aug 11, 2023 at 10:05:53AM -0700, Diane Trout wrote: > > > > > > > Thanks so much! I see you've already started on dask :) > > > > I took at quick look at arrow - yikes! There is potentially work > > afoot on this though: > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021 > > > > Dask & dask.distributed 2023.8.0 was easier to update than some of the > other versions they had between 2022.12 and now. > > Dask would still benifit from pyarrow, by I added enough > pytest.importorskip to avoid triggering the tests that depend on > pyarrow. > > It also looks like it builds for me and the debian builder so I closed > 1042135. > > Hopefully that helps. (And it looks like it's got some code for pandas > 2.0 so hopefully that'll help Rebecca Palmer. That's fantastic - thank you so much! :-) Julian
Re: dask.distributed RC bug #1042135
Hi Diane (and cc'ing the correct mailing list - oops!), On Fri, Aug 04, 2023 at 09:37:48AM -0700, Diane Trout wrote: > On Fri, 2023-08-04 at 14:02 +0100, Julian Gilbey wrote: > > Hi Diane, > > > > I just got emails telling me that some of my packages would be > > removed > > from testing because they (recursively) (build-)depend on > > dask.distributed: https://bugs.debian.org/1042135 > > > > Are you able to take a look at this soon, or would you like someone > > to > > do a team upload? > > > > There is also a newer version available: 2023.7.1, which may well > > resolve this. > > Last time I tried updating pyarrow wasn't properly optional and we > don't have it packaged. (And it's actually in a single repository that > builds multiple language modules) > > Dask has updated their build system some, and let me see if it'll > update. > > If I don't manage to get to it, make sure that if you try you have to > update both dask and dask.distributed together, and it's best to makes > sure that you run the dask.distributed tests using the updated dask. > > That's where why it's usually been difficult to update. Thanks so much! I see you've already started on dask :) I took at quick look at arrow - yikes! There is potentially work afoot on this though: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970021 Best wishes, Julian
Re: #!/usr/bin/python3 vs virtualenv
On Fri, Mar 03, 2023 at 04:22:11PM -0500, Jorge Moraleda wrote: > Jeremy, Thank you for your quick reply! > > I did not know about `sudo pip install --break-system-packages foo` or `sudo > rm > /usr/lib/python3.11/EXTERNALLY-MANAGED` (Frankly I only knew about this issue > what I have read on this discussion). This is very helpful and it really > changes > my outlook on this topic. The --break-system-packages option is noted in /usr/share/doc/python3.11/README.venv, and this file is mentioned in the NEWS file for python3.11. The /usr/lib/python3.11/EXTERNALLY-MANAGED file is not mentioned there; I personally think that having to type --break-system-packages every time one installs a package via pip globally or on a per-user basis is safer, as it reminds you that you run risks doing so. Best wishes, Julian
Re: Python 3.10 in bookworm
On Tue, Feb 07, 2023 at 12:31:23PM +0100, Joost van Baal-Ilić wrote: > Op Tue, Feb 07, 2023 at 05:52:21AM + schreef Danial Behzadi دانیال بهزادی: > > Does it worth trying to package pyenv for Debian? Ain't it against any > > rules? > > See "ITP pyenv" @ http://bugs.debian.org/978149 . Oh, how embarrassing - I already knew about this software a year ago! Best wishes, Julian
Re: Python 3.10 in bookworm
Hi Andrey, On Mon, Feb 06, 2023 at 11:53:33AM +0100, Andrey Rakhmatullin wrote: > On Sun, Feb 05, 2023 at 01:50:34PM +0000, Julian Gilbey wrote: > > Our social contract #4 says "Our priorities are our users and free > > software". What benefits would having the python3.10 base packages in > > bookworm bring for our users (as I point out, for some users, this is > > a necessity) and what disadvantages would it bring (none that I can > > think of)? Why would we tell a whole bunch of our users: "Don't > > upgrade to Debian 12 until all of the critical packages you use from > > PyPI are upgraded to support Python 3.11, or fix those packages > > yourself"? > The next obvious step for these use cases is to just install 3.10 with > pyenv. Oh, that's brilliant! I didn't know about this tool (and I note it's not currently in Debian). This solution for users who need Python 3.10, together with the support burden of having python3.10 in bookworm (which I hadn't appreciated), I withdraw my objection to the removal of python3.10. Best wishes, Julian
Re: Python 3.10 in bookworm
Hi Michael, On Sun, Feb 05, 2023 at 02:29:10PM +0100, Michael Kesper wrote: > Hi Julian, > > Am 05.02.23 um 11:38 schrieb Julian Gilbey: > > Why is the current intention not to ship the python3.10 package in > > bookworm? > > Because it would amount to about double the work for all those involved. I doubt it would be double the work, but as Scott points out in his email, it would require paying attention to security issues in the Python interpreter for both the 3.10 and 3.11 interpreters. I had not considered that. > Besides, Python 3.11 has some points for it: > - Real performance gains for real workloads > - It will be supported one year longer (so EOL is expected to be around the > time bookworm will be out of stable, too). I'm not proposing that we revert to Python 3.10 as default for bookworm, only that we have the python3.10 package itself in bookworm. > > I was trying to run some experiments in a virtual environment a few > > days ago, and it turns out that several of the Python packages I > > needed do not yet run on Python 3.11. I was saved by being able to > > run in a Python 3.10 venv and download all the required packages from > > PyPI. If bookworm shipped without python3.10, I would not have been > > able to do my work. Removing python3.10 from bookworm will seriously > > affect many of our users in a similar situation to me. > ... > > P.S. We should also fix #1036268 if we do keep python3.10 in bookworm; > > I'm happy to do an NMU if needed. > > Maybe you could sponsor a "backport" of Python3.11? I don't understand this suggestion. #1036268 says that running "python3.10 -m venv envname" if the python3.10-venv package is not installed should output a meaningful error message rather than crash with an "undefined variable" error. Best wishes, Julian
Re: Python 3.10 in bookworm
On Sun, Feb 05, 2023 at 02:41:08PM +, Stefano Rivera wrote: > Hi Julian (2023.02.05_10:38:23_+) > > > Why is the current intention not to ship the python3.10 package in > > bookworm? > > Because we aim to have a single Python release supported in every stable > release. I am not suggesting that we revert to having Python 3.10 as a "supported version" (that would be a whole separate discussion); I am suggesting that we keep just the Python 3.10 interpreter and python3.10-venv in bookworm, so that users can use it to run a virtual environment if they need to do so. > > I was trying to run some experiments in a virtual environment a few > > days ago, and it turns out that several of the Python packages I > > needed do not yet run on Python 3.11. I was saved by being able to > > run in a Python 3.10 venv and download all the required packages from > > PyPI. If bookworm shipped without python3.10, I would not have been > > able to do my work. Removing python3.10 from bookworm will seriously > > affect many of our users in a similar situation to me. > > By the time bookworm releases, that probably won't be the case any more. I honestly don't know if that will be the case or not; some packages will be much slower to adapt than others. That's why I'm suggesting we leave the python3.10 and python3.10-venv packages in bookworm. > But anything that gets removed from Debian, because it isn't ready yet > obviously gets hurt in the process... I'm not sure what you mean here? Best wishes, Julian
Re: Python 3.10 in bookworm
Our social contract #4 says "Our priorities are our users and free software". What benefits would having the python3.10 base packages in bookworm bring for our users (as I point out, for some users, this is a necessity) and what disadvantages would it bring (none that I can think of)? Why would we tell a whole bunch of our users: "Don't upgrade to Debian 12 until all of the critical packages you use from PyPI are upgraded to support Python 3.11, or fix those packages yourself"? And may I politely remind you, Thomas, that you are very concerned about breaking things for people: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=973617#40 This is likely a far greater impact than the discussion there on many more people. Best wishes, Julian On Sun, Feb 05, 2023 at 12:25:18PM +0100, Thomas Goirand wrote: > How about fixing the 3.11 issues if you hit them ? How about using Buster and > 3.9 if 3.11 doesn't work (yet) for you ? > > Thomas Goirand (zigo) > On Feb 5, 2023 11:38, Julian Gilbey wrote: > > > > Why is the current intention not to ship the python3.10 package in > > bookworm? > > > > I was trying to run some experiments in a virtual environment a few > > days ago, and it turns out that several of the Python packages I > > needed do not yet run on Python 3.11. I was saved by being able to > > run in a Python 3.10 venv and download all the required packages from > > PyPI. If bookworm shipped without python3.10, I would not have been > > able to do my work. Removing python3.10 from bookworm will seriously > > affect many of our users in a similar situation to me. > > > > Best wishes, > > > > Julian > > > > P.S. We should also fix #1036268 if we do keep python3.10 in bookworm; > > I'm happy to do an NMU if needed.
Python 3.10 in bookworm
Why is the current intention not to ship the python3.10 package in bookworm? I was trying to run some experiments in a virtual environment a few days ago, and it turns out that several of the Python packages I needed do not yet run on Python 3.11. I was saved by being able to run in a Python 3.10 venv and download all the required packages from PyPI. If bookworm shipped without python3.10, I would not have been able to do my work. Removing python3.10 from bookworm will seriously affect many of our users in a similar situation to me. Best wishes, Julian P.S. We should also fix #1036268 if we do keep python3.10 in bookworm; I'm happy to do an NMU if needed.
Re: Old generated binary dependencies after renaming psycopg
On Tue, Jan 17, 2023 at 09:08:01AM +0100, Tomasz Rybak wrote: > Hello. > After fixing #1016031 "psycopg3: binary package name should be python3- > psycopg" > (I renamed package names, full changes: > * python3-psycopg3 -> python3-psycopg > * python3-psycopg3-pool -> python3-psycopg-pool > * python-psycopg3-doc -> python3-psycopg-doc) > I tried to rebuild reverse dependencies, > i.e. pgcli and python3-pgspecial. > Rebuild went without problems, new packages are the same > as old ones, but their binary packages still depend on python3- > psycopg3, > even though they build-depend on python3-psycopg. Nope, pgcli does not build-depend on it, rather it explicitly specifies Depends: python3-psycopg3. Likewise, python-pgspecial specifies the same Depends (though it also has a Build-Depends: python3-psycopg3). These packages will need their dependencies updating. (You might also consider making python3-psycopg Provides: python3-psycopg3 and likewise for the other two binary packages for bookworm.) Best wishes, Julian
Re: How to create multi-source tarball with different submodules for scipy
On Mon, Jan 16, 2023 at 05:05:39PM +0100, Andreas Tille wrote: > Hi, > > I tried to create a multi-source tarball for scipy in its experimental > branch[1]. Upstream includes a set of git submodules in its build > process. I intended to merge all these submodules in a single > scipy_1.10.0.orig-submodules.tar.gz. This tarball is created with a > script[2] which makes sure that the exact directory structure as it is > used by upstream is conserved. This directory layout is needed in the > build process. Unfortunately `dpkg-source -x` extracts the content of > the submodules tarball into a subdirectory submodules/. > > Is there any trick to unpack this tarball right into the root? > Otherwise I need to do some symlinks workaround in d/rules to provide > all files where these are needed. Not that I know of; this is the design of the multi-source tarball setup: each component tarball is extracted into a directory with the name of the component. Best wishes, Julian
Re: Policy Proposal python3-supported-(min|max) virtual packages
On Sat, Jan 14, 2023 at 07:34:59PM +, Scott Kitterman wrote: > Typically though doesn't the python interpreter package provide modules that > are now incorporated? If python3.11 provides python3-tomli, won't that mess > this up? In this case, it doesn't; the Python 3.11 standard library module is called tomllib, presumably to avoid conflicting with the toml or tomli library. (And I'm guessing the package in question imports tomllib if using 3.11 or higher and tomli otherwise.) Best wishes, Julian
Re: eric and jquery.js to a symbolic link
On Thu, Jan 05, 2023 at 07:14:40PM +, Guðjón Guðjónsson wrote:
> Hi list
> I am working on eric and I do have a problem with the lintian requirement to
> replace the jquery.js file with the debian provided jquery.js file.
> The upstream author pointed out that it doesn't work as well and I have
> verified
> the behavior.
> If you run eric7_browser and press ==->Bookmarks->Speed Dial it doesn't show
> the
> links when using the debian jquery.js file.
> Is it ok to keep the original jquery files and add a lintian-override in this
> case?
> Regards
> Gudjon
Hi Gudjon,
Looking at it, the problem seems to be that eric is using an ancient
version of jQuery (1.7.1, released Nov 22, 2011; 1.7.2 was released on
Mar 21, 2012). jQuery-UI 1.8.16 is similarly old (released Aug 15,
2011, 1.8.17 was released on Nov 29, 2011). You could embed it (but
not the minimised version in the upstream eric sources, but rather the
original source from github.com/jquery/{jquery,jquery-ui}, but it
would be far from ideal - who knows how many bugs or possible security
issues there are in such an old version? A much better solution, if
feasible, is to ask the eric upstream to switch to a recent version of
jQuery and jQuery UI, and to update the code depending on it
accordingly. If upstream won't do that, then we should.
Best wishes,
Julian
Re: Python 3.11 for bookworm?
Hi Jochen, On Mon, Dec 19, 2022 at 04:53:58PM +0100, Jochen Sprickerhof wrote: > Hi Julian, > > * Julian Gilbey [2022-12-19 09:41]: > > Quick update: with the updating of python3-bytecode from 0.13 to 0.14 > > in unstable/testing, which allows it to handle Python 3.11, something > > has changed and now pydevd doesn't even pass the tests on Python > > 3.10. The python3-bytecode underwent a major restructuring, so it is > > entirely possible that something has changed that wasn't part of the > > advertised API or something like that. But that's for upstream pydevd > > developers to deal with. > > I've uploaded 0.14.0-2 that should fix this. As far as I've found that was > only a minor fix in the Debian specific offset patch, sorry for the trouble. Phew! I didn't think to check that. Unfortunately, though, there are still numerous pydevd test errors even with 0.14.0-2, so I think something has changed in bytecode that the pydevd maintainers will have to adapt to. So either I skip 14 newly failing tests on Python 3.10 (they're mostly skipped on 3.11 as the current pydevd version skips bytecode tests on Python 3.11) or wait for a new version of pydevd. Hmmm. Anyway, thanks so much for all your work updating this package - it's been really helpful, as I've been a bit overloaded and Spyder 5.4.0 together with the Python 3.11 transition has been a lot to handle. I also learnt a lot from your changes! Best wishes, Julian
Re: Python 3.11 for bookworm?
On Sun, Dec 18, 2022 at 06:02:58PM +, Julian Gilbey wrote: > On Thu, Dec 15, 2022 at 04:10:05PM +0100, Thomas Goirand wrote: > > On 12/13/22 13:34, Julian Gilbey wrote: > > > If Python 3.11 is the default, then it is highly likely that Spyder > > > will not be included: debugpy, which is a dependency of Spyder and > > > python3-ipykernel (and lots of things that depend on that) seems to > > > require major work upstream to make it fully compatible with Python > > > 3.11. This is work in progress, but I don't know whether it will be > > > ready in time for the freeze. At the moment, I have worked around > > > this problem by just skipping the failing tests, but that is far from > > > an ideal solution. > > > > It's probably ok if it's a *TEMPORARY* solution until upstream fixes > > everything in time for the release (which is months after the freeze). The > > question is: do you believe this may happen for let's say next March? > > The truth is that I don't know. Upstream is very good, but the Python > 3.11 internal changes are very significant, and debugpy (along with > pydevd, which is part of debugpy) are deeply affected by this, as they > work at the level of Python's internals. So I don't know how long it > will take them to make the required changes (and it's far beyond my > capability or capacity to do myself). I can hope they'll do it in > time for the freeze, but I wouldn't like to place a bet on it. Quick update: with the updating of python3-bytecode from 0.13 to 0.14 in unstable/testing, which allows it to handle Python 3.11, something has changed and now pydevd doesn't even pass the tests on Python 3.10. The python3-bytecode underwent a major restructuring, so it is entirely possible that something has changed that wasn't part of the advertised API or something like that. But that's for upstream pydevd developers to deal with. One possibility is that we revert to the situation in bullseye if pydevd is not ready in time for the freeze. Bullseye didn't have bytecode/pydevd/debugpy, and it meant that debugging was limited in Spyder: we remove python3-debugpy from the dependencies of python3-ipykernel and then the rest of the dependency chain will be OK. It's certainly not an ideal solution, but it may be the best we can do if we're sticking with Python 3.11. It may actually be worth doing that at this point so that Spyder can stay in testing for the time being, even though pydevd and debugpy won't. Best wishes, Julian
Re: Python 3.11 for bookworm?
On Thu, Dec 15, 2022 at 04:10:05PM +0100, Thomas Goirand wrote: > On 12/13/22 13:34, Julian Gilbey wrote: > > If Python 3.11 is the default, then it is highly likely that Spyder > > will not be included: debugpy, which is a dependency of Spyder and > > python3-ipykernel (and lots of things that depend on that) seems to > > require major work upstream to make it fully compatible with Python > > 3.11. This is work in progress, but I don't know whether it will be > > ready in time for the freeze. At the moment, I have worked around > > this problem by just skipping the failing tests, but that is far from > > an ideal solution. > > It's probably ok if it's a *TEMPORARY* solution until upstream fixes > everything in time for the release (which is months after the freeze). The > question is: do you believe this may happen for let's say next March? The truth is that I don't know. Upstream is very good, but the Python 3.11 internal changes are very significant, and debugpy (along with pydevd, which is part of debugpy) are deeply affected by this, as they work at the level of Python's internals. So I don't know how long it will take them to make the required changes (and it's far beyond my capability or capacity to do myself). I can hope they'll do it in time for the freeze, but I wouldn't like to place a bet on it. Best wishes, Julian
Re: Python 3.11 for bookworm?
Hi Graham, On Mon, Dec 12, 2022 at 11:51:11PM +, Graham Inggs wrote: > Dear Python Team > > Looking at the current state of the 'adding Python 3.11 as a supported > version' transition [1], the tracker [2] shows only 12 red packages > (excluding unknowns and packages not in testing) remaining, copied > below for reference. > [...] If Python 3.11 is the default, then it is highly likely that Spyder will not be included: debugpy, which is a dependency of Spyder and python3-ipykernel (and lots of things that depend on that) seems to require major work upstream to make it fully compatible with Python 3.11. This is work in progress, but I don't know whether it will be ready in time for the freeze. At the moment, I have worked around this problem by just skipping the failing tests, but that is far from an ideal solution. Best wishes, Julian
Re: Fixing upstream branch after tagging
On Mon, Dec 05, 2022 at 06:24:48AM +, Guðjón Guðjónsson wrote: > Hi list > I am working on eric and I made a mistake while updating the git repository. > Some paths have changed so files were not excluded correctly and now upstream > and pristine-tar contain jquery*.js files. > How can I remove the files after having tagged? > I read that the pristine-tar branch should be removed [1]. Is that correct? > Regards > Gudjon Hi Gudjon, It depends on whether you have pushed to a remote repository yet, or whether it's still only on your local machine. If you've already pushed, then you'll have to update your local versions and give it a different version number (for example, +ds2 rather than +ds1), doing a fresh gbp import-orig on the repacked source package. If you're still only on your local machine, this is an error I have made a number of times, only noticing after doing gpb import-orig. To fix it, you can roll back the gbp import-orig. With care, do the following (where git co is shorthand for git checkout): git co debian/unstable [or whatever your branch is] git log git reset --hard git co upstream git log git reset --hard git co pristine-tar git log git reset --hard git tag -d upstream/ There is probably a better way to do it, but this has worked for me. Good luck! Julian
Re: Python 3.11, bytecode and new internals
On Wed, Nov 23, 2022 at 12:21:38AM +0100, Thomas Goirand wrote: > On 11/22/22 17:59, Julian Gilbey wrote: > > > > Or should we mark them as X-Python3-Version: << 3.11 so they can stay in > > > > testing as long as Python 3.10 is the default? > > > > > > I don't think this is the way. > > > > I'm sorry, I don't understand - which is not the way? > > I don't think you should "mark them as X-Python3-Version: << 3.11", because > either we use 3.10 or 3.11 in Bookworm, I don't think that there's a plan > for having both interpreters available. It's more about the present time: if I mark them as << 3.11, then they will build fine and be able to stay in testing. If I don't, they'll be pulled from testing in the next few weeks because of FTBFS bugs. Obviously, once they do build OK with 3.11, then I can remove that clause. Best wishes, Julian
Re: Python 3.11, bytecode and new internals
On Tue, Nov 22, 2022 at 05:01:03PM +0100, Thomas Goirand wrote: > > If there are people with the expertise to help upstream update > > bytecode and parso (and probably several other low-level packages) for > > 3.11 so that the software that depends on them works with 3.11, then > > fine. (And it is a non-trivial update, AFAICT.) But until then, I'd > > be very reluctant to make 3.11 the default. > > Have you tried this PR? > https://github.com/MatthieuDartiailh/bytecode/pull/107 As you can see by reading it, there is still at least one blocking point (to use Matthieu's language), and Matthieu is the core bytecode developer. Once that is sorted, then pydevd needs to be updated to use it. But this is far outside my expertise, and I'm not going to apply a huge patch that I don't understand and that is known to still be buggy. So I'm just going to have to wait. I don't know the current state of parso; it would be interesting to see whether parso 0.8.3 successfully works under 3.11, but that is Piotr Ożarowski's package (not under the Python team); he is active, but has not yet uploaded this newer version. > > I haven't decided what to do with packages which now FTBFS under 3.11 > > because of this. Should we just let them fall out of testing (that > > certainly includes spyder, and quite possibly ipython as well)? > > Or should we mark them as X-Python3-Version: << 3.11 so they can stay in > > testing as long as Python 3.10 is the default? > > I don't think this is the way. I'm sorry, I don't understand - which is not the way? > > If we make 3.11 the > > default, these packages will likely not be in bookworm, which might > > upset our users even more. > > We're not there yet. We have until January to fix, and we haven't decided > yet if 3.11 will be the default. Fair enough. But I still don't know what to do in the meantime with the spyder ecosystem besides either wait for upstream to sort bytecode and pydevd and Piotr (and possibly upstream) to sort parso, or to mark them as Python 3.10 only. Best wishes, Julian
Re: Python 3.11, bytecode and new internals
On Tue, Nov 22, 2022 at 09:22:05AM +0100, Thomas Goirand wrote: > > this, 100 times > > I very much don't agree. I think it's going pretty well, and the number of > breakage isn't high. We just need a little bit of effort to make it in good > enough shape. > [...] > Now, out of *many* of my packages, only a very few broke. Complicated > packages like Eventlet for example, just worked. Others had upstream patches > I applied. And I am in the opinion that we should go ahead and make 3.11 the > default. If there are people with the expertise to help upstream update bytecode and parso (and probably several other low-level packages) for 3.11 so that the software that depends on them works with 3.11, then fine. (And it is a non-trivial update, AFAICT.) But until then, I'd be very reluctant to make 3.11 the default. I haven't decided what to do with packages which now FTBFS under 3.11 because of this. Should we just let them fall out of testing (that certainly includes spyder, and quite possibly ipython as well)? Or should we mark them as X-Python3-Version: << 3.11 so they can stay in testing as long as Python 3.10 is the default? If we make 3.11 the default, these packages will likely not be in bookworm, which might upset our users even more. Best wishes, Julian
Help with pydevd tests breaking with 3.11 on s390x
Dear all, A strange one. I've uploaded pydevd version 2.9.2+ds-4 to unstable. It turned out that many of the package tests fail on s390x with Python 3.11 but pass on all of the other architectures. I really don't understand the internals of the package (it's pretty low-level stuff) and I have no idea what's different about s390x that might cause these test failures on Python 3.11 (but they pass fine on Python 3.10). If anyone has the time, inclination and expertise to help me with this, please do drop me a line! In the meantime, I'm just skipping those tests on s390x. (And upstream only support the package for amd64 and i386, so I'm unlikely to get support from them.) Best wishes, Julian
Python 3.11, bytecode and new internals
I've been having a somewhat interesting time with the python3.11-add transition. Python 3.11 has made some significant changes to its bytecode representation, and also changed some of it's internal data structures related to frames quite significantly. In my corner of the Python world, several important packages use this information, including Spyder, IPython and Jupyter (which all (recursively) depend on python3-parso and/or python3-bytecode), but these low-level packages are not really ready for 3.11 yet (parso claims "basic support for Python 3.11 and 3.12", so it's unlikely it will actually yet work with 3.11, though I haven't tested this, and bytecode makes no claims to yet be ready for 3.11). Several packages have tests which now fail on 3.11, preventing migration to testing (spyder-kernels is one). In pydevd (which does not yet support 3.11, because it depends on python3-bytecode), I've just disabled all the failing tests and put a note in README.Debian, but this is not a great solution. I could add X-Python3-Version: << 3.11 to debian/control, but that is surely not that desirable either. I'm just flagging this up here, with a question about how we should proceed. Certainly we are not ready to make Python 3.11 the default Python version!! Best wishes, Julian
Bug#1023796: ITP: pylint-venv -- Pylint hook to use same pylint with different virtual envs
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: pylint-venv Version : 2.3.0 Upstream Author : Jan Gosmann , Federico Jaramillo * URL : https://github.com/jgosmann/pylint-venv * License : MIT (Expat) Programming Lang: Python Description : Pylint hook to use same pylint with different virtualenvs Pylint does not respect the currently activated virtualenv if it is not installed in every virtual environment individually. This module provides a Pylint init-hook to use the same Pylint installation with different virtual environments. This package is a new dependency of spyder (version 5.4.0) which I'm currently preparing to upload. It is a single short Python module (about 90 lines excluding initial comments). It will be maintained within the Debian Python team, with me as the primary maintainer.
Re: [Debian-salsa-ci] Enabling salsa-ci on all Debian Python Team repos
On Mon, Sep 19, 2022 at 01:52:09PM +0200, Iñaki Malerba wrote: > [...] > > Perhaps there's an opportunity to automate and getting wider CI usage. > > One of the biggest issues we had when a team adopted the pipeline was > DDOSing of the instance because of the multiple pipelines generated when > pusing the .gitlab-ci.yml file to all the projects. > > If you're planning to do this, please: > > - Use the API and configure the 'CI/CD configuration file' project > field, as you mentioned in the email. This won't generate a pipeline > when configured but only on the next push. Indeed; setting the configuration file to recipes/debian.yml@salsa-ci-team/pipeline will avoid any need to touch the actual repository. > - If you need create the .gitlab-ci.yml file, please use the > `ci.skip`[1] push option. And that should only be needed if the configuration is non-standard. > Thanks, and good luck :) Best wishes, Julian
Bug#1019436: ITP: jarowinkler -- fast approximate string matching using Jaro(-Winkler) similarity
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: jarowinkler Version : 1.2.1 Upstream Author : Max Bachmann * URL : https://github.com/maxbachmann/JaroWinkler * License : MIT Programming Lang: Python Description : Fast approximate string matching using Jaro(-Winkler) similarity JaroWinkler is a Python library to calculate the Jaro and Jaro-Winkler similarity. It is easy to use, is much faster than other comparable libraries, and is designed to integrate seemingless with RapidFuzz. There is also a C++ library contained within this package, managed as a separate GitHub subrepository. This is jarowinkler-cpp, and I am not sure whether to bundle this as a single package or whether to package this independently. This is a dependency of python3-rapidfuzz, which is a new dependency of the latest upstream release of python3-textdistance. This package will be maintained within the Python Packaging Team.
Re: Bug#1019435: ITP: rapidfuzz -- rapid fuzzy string matching
On Fri, Sep 09, 2022 at 10:00:49AM +0100, Julian Gilbey wrote: > Package: wnpp > Severity: wishlist > Owner: Julian Gilbey > X-Debbugs-Cc: debian-de...@lists.debian.org, debian-de...@lists.debian.org, > debian-python@lists.debian.org > > * Package name: rapidfuzz > Version : 2.6.1 > Upstream Author : Max Bachmann > * URL : https://github.com/maxbachmann/RapidFuzz > * License : MIT > Programming Lang: Python > Description : rapid fuzzy string matching > > RapidFuzz is a fast string matching library for Python and C++, which > uses the string similarity calculations from > [FuzzyWuzzy](https://github.com/seatgeek/fuzzywuzzy). However there > are a couple of aspects that set RapidFuzz apart from FuzzyWuzzy: > 1) It is MIT licensed so it can be used whichever License you might want to > choose for your project, while you're forced to adopt the GPL license when > using FuzzyWuzzy > 2) It provides many string_metrics like hamming or jaro_winkler, which are > not included in FuzzyWuzzy > 3) It is mostly written in C++ and on top of this comes with a lot of > Algorithmic improvements to make string matching even faster, while still > providing the same results. For detailed benchmarks check the > [documentation](https://maxbachmann.github.io/RapidFuzz/fuzz.html) > 4) Fixes multiple bugs in the `partial_ratio` implementation > > This is a dependency of the latest upstream release of > python3-textdistance. > > There are also two C++ libraries contained within this package, > managed as separate GitHub subrepositories. One is rapidfuzz-cpp, and > I am not sure whether to bundle this as a single package or whether to > package this independently. The other is taskflow > (https://github.com/taskflow/taskflow)), a C++ header-only package; I > think this should probably be packaged separately. > > This package will be maintained within the Python Packaging Team. I should have added: this package depends on Cython >= 3.0.0a7, so this cannot be packaged until we have the new version of Cython available. The same applies for the JaroWinkler package. Julian
Bug#1019435: ITP: rapidfuzz -- rapid fuzzy string matching
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: rapidfuzz Version : 2.6.1 Upstream Author : Max Bachmann * URL : https://github.com/maxbachmann/RapidFuzz * License : MIT Programming Lang: Python Description : rapid fuzzy string matching RapidFuzz is a fast string matching library for Python and C++, which uses the string similarity calculations from [FuzzyWuzzy](https://github.com/seatgeek/fuzzywuzzy). However there are a couple of aspects that set RapidFuzz apart from FuzzyWuzzy: 1) It is MIT licensed so it can be used whichever License you might want to choose for your project, while you're forced to adopt the GPL license when using FuzzyWuzzy 2) It provides many string_metrics like hamming or jaro_winkler, which are not included in FuzzyWuzzy 3) It is mostly written in C++ and on top of this comes with a lot of Algorithmic improvements to make string matching even faster, while still providing the same results. For detailed benchmarks check the [documentation](https://maxbachmann.github.io/RapidFuzz/fuzz.html) 4) Fixes multiple bugs in the `partial_ratio` implementation This is a dependency of the latest upstream release of python3-textdistance. There are also two C++ libraries contained within this package, managed as separate GitHub subrepositories. One is rapidfuzz-cpp, and I am not sure whether to bundle this as a single package or whether to package this independently. The other is taskflow (https://github.com/taskflow/taskflow)), a C++ header-only package; I think this should probably be packaged separately. This package will be maintained within the Python Packaging Team.
Bug#1019431: ITP: rapidfuzz-capi -- C-API of the Python RapidFuzz package
Package: wnpp Severity: wishlist Owner: Julian Gilbey X-Debbugs-Cc: debian-de...@lists.debian.org, debian-python@lists.debian.org * Package name: rapidfuzz-capi Version : 1.0.5 Upstream Author : Max Bachmann * URL : https://github.com/maxbachmann/rapidfuzz_capi * License : MIT Programming Lang: Python Description : C-API of the Python RapidFuzz package, used to build rapidfuzz This package provides the C API of RapidFuzz. It can be used inside `pyproject.toml` to compile an extension module extending RapidFuzz. Providing this C API in a separate package simplifies the build process. . This package is only needed for building packages such as python3-rapidfuzz and python3-jarowinkler; it is not needed at runtime. This is a dependency of python3-rapidfuzz and python3-jarowinkler, string-matching packages which are new (recursive) dependencies of the latest upstream release of python3-textdistance. This package will be maintained within the Python Packaging Team.
Re: Lintian info message "hardening-no-bindnow" with vanilla debian/rules
On Tue, Aug 30, 2022 at 07:33:07PM +0200, Gregor Riepl wrote: > > I: python3-pyxdameraulevenshtein: hardening-no-bindnow > > [usr/lib/python3/dist-packages/pyxdameraulevenshtein.cpython-310-x86_64-linux-gnu.so] > > > > and there is nothing about CFLAGS or the like in the setup.py file. > > So if having this hardening flag enabled is a good thing, it should > > probably be enabled somewhere within the pybuild system, rather than > > every individual package with an extension file doing it. > > Hardening is generally a good thing, but can break code in subtle ways. > I suppose that's why it was decided that enabling it by default in Debian > was deemed too risky. > > Enabling it is quite easy, though: Just add > > export DEB_BUILD_MAINT_OPTIONS = hardening=+all > [...] Thanks Gregor, I'll try that! > Also, note that hardening-no-bindnow is an Informational message, so not > strictly something that needs to be acted upon: > https://lintian.debian.org/tags/hardening-no-bindnow Indeed, hence the title of this message :-) Best wishes, Julian
Lintian info message "hardening-no-bindnow" with vanilla debian/rules
Hi! A package I maintain within the team (python3-pyxdameraulevenshtein) gives the following lintian message: I: python3-pyxdameraulevenshtein: hardening-no-bindnow [usr/lib/python3/dist-packages/pyxdameraulevenshtein.cpython-310-x86_64-linux-gnu.so] The debian/rules file is very bland, essentially: %: dh $@ --buildsystem=pybuild and there is nothing about CFLAGS or the like in the setup.py file. So if having this hardening flag enabled is a good thing, it should probably be enabled somewhere within the pybuild system, rather than every individual package with an extension file doing it. Or have I missed something? Best wishes, Julian
Re: Auto-handling of closed bugs - how does it work?
On Sun, Aug 14, 2022 at 11:38:10AM -0400, Sandro Tosi wrote: > > It's a salsa webhook: > > https://wiki.debian.org/Salsa/Doc#Dealing_with_Debian_BTS_from_commit_messages > > > > We don't have tooling that automatically configures all the repos, but > > when we migrated to salsa, we set them all up for tagpending, and > > posting to #debian-python-changes on IRC > > shameless plug, i fixed most of the packages in our repo to have the > proper wehbooks using > https://github.com/sandrotosi/dpt-repos-check/blob/main/dpt-fix-integrations-webhooks.py > (and now automation is available in pypi2deb, when you let it create > the repo on salsa) -- consider new packages wont get the right setup > old webhooks, etc > > I should probably run it more periodically Oh wow, that's awesome, thanks!! Best wishes, Julian
Re: Cython 3.0.0
On Sun, Aug 14, 2022 at 08:49:06AM +, Stefano Rivera wrote: > Hi Julian (2022.08.14_07:41:26_+) > > I don't know how many packages in Debian would be broken by the move > > to 3.0.0; that may be something worth exploring. It may well be that > > approach (2) makes most sense for the short term. > > I think that's the first question to answer. Once we know how bad > the incompatibilities are, we can decide on the best approach. > > So, first step is probably to package the new cython version (locally), > and try to rebuild everything against it. That sounds sensible, indeed, once the beta is released. As cython is used quite widely (240 packages or so in testing), I wonder whether it would be appropriate to upload it to experimental and ask Lucas to run the test builds across the archive? Best wishes, Julian
Re: Auto-handling of closed bugs - how does it work?
On Sun, Aug 14, 2022 at 08:45:32AM +, Stefano Rivera wrote: > Hi Julian (2022.08.14_07:18:49_+) > > A question of curiosity: when I push a commit to salsa with a "Closes: > > #n" in the changelog, the BTS gets a "tag: pending" notification. > > I looked and looked, and could not find out how salsa does this? > > Could anyone enlighten me? (The standard debian-ci scripts, which the > > repositories use for their CI, appear to only do something with RC > > bugs.) > > It's a salsa webhook: > https://wiki.debian.org/Salsa/Doc#Dealing_with_Debian_BTS_from_commit_messages > > We don't have tooling that automatically configures all the repos, but > when we migrated to salsa, we set them all up for tagpending, and > posting to #debian-python-changes on IRC Ah, super, thanks! I'll start adding those to the projects I've created. Best wishes, Julian
Cython 3.0.0
Dear all, I am intending to package a new dependency of textdistance called rapizfuzz (along with its dependencies jarowinkler and rapizfuzz-capi, and including rapizfuzz-cpp and jarowinkler-cpp within the packages). It's relatively low priority though (and I haven't filed ITPs yet). But it needs cython 3.0.0alpha7 or later to be able to compile. There is talk of moving cython 3.0.0 into beta in the not-too-distant future: https://github.com/cython/cython/issues/4022 It does have some breaking changes in comparison to cython 0.29.x. I wonder what our strategy should be? Here are three reasonable approaches: (1) Keep the existing cython package (source: cython, binaries: cython3, cython-doc, cython3-dbg) and have a new package for the 3.* releases. Advantages: * won't break lots of existing packages Disadvantages: * no obvious name for new package * will end up with an old cython package over time that cannot be easily dropped * will lead to confusion - what is the cython3 package, is it the new or old version of cython? (2) Create a new cython0.29 package (source: cython0.29, binaries: cython3-0.29, cython0.29-doc, cython3-0.29-dbg for example) to house the "old" version, and the cython package becomes cython 3.0.0 Advantages: * clear naming scheme * those packages which "just work" with the new version of cython will not need to do anything to migrate * allows the cython0.29 package to be dropped in time without needing lots of renaming once no packages still rely on it Disadvantages: * there are two packages to maintain instead of just one (cython0.29 and cython) * those packages which don't work with 3.0.0 will either need patching or their dependency will need to be changed to cython3-0.29 (3) Let the cython package become cython 3.0.0 once it is released. Advantages: * only one package to maintain * keep at the cutting edge of cython development Disadvantages * may break lots of packages, requiring a lot of effort to patch them I don't know how many packages in Debian would be broken by the move to 3.0.0; that may be something worth exploring. It may well be that approach (2) makes most sense for the short term. I imagine that this is unlikely to hit before the bookworm freeze, but I wanted to flag it up now. Best wishes, Julian
Auto-handling of closed bugs - how does it work?
A question of curiosity: when I push a commit to salsa with a "Closes: #n" in the changelog, the BTS gets a "tag: pending" notification. I looked and looked, and could not find out how salsa does this? Could anyone enlighten me? (The standard debian-ci scripts, which the repositories use for their CI, appear to only do something with RC bugs.) Best wishes, Julian
Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF)
On Wed, Jul 27, 2022 at 09:32:19PM +0100, Julian Gilbey wrote: > [...] > > > > I'd be wary about 2.2 and 2.3. I have several packages where I know > > that an automated test will fail; there are all sorts of weird cases > > [...] > > I'd be wary about adding lintian tags for this, though: with so many > packages not being able to use the autodep8 system (I vaguely recall > someone suggesting that a third of Python packages would not be able > [...] I realise that I may have come across as quite negative. Apologies if that is the case - it was not my intention. I think the autopkgtest-pkg-pybuild/pybuild-autodep8 work is very helpful and a positive addition to our infrastructure, and I'm very grateful to those who've made it happen. My only concern is with how it is introduced; because of the wide variety of Python packages and the hugely varying nature of the testsuites present in them (or not), I think that trying to force this into packages is a poor idea. (I wish that most of my packages could simply use this pybuild-autodep8 tool. I fear that this won't be the case, but I will certainly adopt it for those which can.) Best wishes, Julian
Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF)
On Wed, Jul 27, 2022 at 07:45:12PM +0100, Julian Gilbey wrote: > On Wed, Jul 27, 2022 at 10:26:33AM -0400, Louis-Philippe Véronneau wrote: > > The way I see it: > > > > 1. We should have a Lintian tag for packages not using the new > > pybuild-autodep8 autopkgtest. It would be even better if this tag would be a > > pointed hint that identified 'manually' written unit test autopkgtests that > > could be replaced. > > > > This way, you get something like: > > > > python-foo source: not-using-pybuil-autodep8 [debian/tests/unittests] > > > > for python packages that have old 'manually' written unit test autopkgtests > > and: > > > > python-foo source: not-using-pybuild-autodep8 [no-autopkgtest] > > > > for python packages without any autopkgtest. > > > > 2. lintian-brush (or something else, but I think lintian-brush is the right > > tool) would go over these packages to: > > > > 2.1 Add the new autodep8 autopkgtests and build the package to see if they > > pass > > 2.2 Remove the "manual" unit test autopkgtests if 2.1 succeeds > > 2.3 Open a bug report if 2.1 fails > > I'd be wary about 2.2 and 2.3. I have several packages where I know > that an automated test will fail; there are all sorts of weird cases > where I've had to write tests manually. I would also be quite cross > if manually crafted tests were automatically removed, especially in > cases such as Simon mentioned where they do things that that > automatically generated test does not do. Another thing I could > imagine happening is that the automated test succeeds in a trivial way > - it succeeds but doesn't actually test much because of the nature of > the package. > > On the other hand, a bug report saying something like the following > seems much more reasonable: "We've tested this package using the > automated autopkgtest system and it seems to work by adding the line > 'Testsuite: autopkgtest-pkg-pybuild'; please check that the automated > tests cover all of the tests of your manually written debian/tests/* > and if so, then please remove them. The autopkgtest-pkg-pybuild logs > are attached." This would give the maintainer the chance to decide > how best to proceed. Here's another alternative to steps 2.1-2.3 based on this: For packages which currently have manually-written autopkgtests: 2.A Try removing debian/tests and adding Testsuite: autopkgtest-pkg-pybuild to debian/control, then building the package and running autopkgtest. 2.B If this works, then submit a bug report to the BTS as I suggested above. 2.C If this does not work, don't do anything more; trust that the maintainer knew what they were doing when they wrote the manual autopkgtests. For packages which don't currently have manually-written autopkgtests: 2.A' Try adding Testsuite: autopkgtest-pkg-pybuild to debian/control, then building the package and running autopkgtest. 2.B' If this works, then either Janitor adds this line to debian/control or submit a bug report to the BTS to recommend this. (But we would not expect Janitor to do step 2.1', so this might have to be a different setup, or maybe the script doing 2.A' could leave a list of packages for Janitor, or something like that.) 2.C' If this does not work, submit a wishlist bug to the BTS to recommend that the maintainer adds some autopkgtest tests, either by making the autodep8 system work or by writing some manual tests. I'd be wary about adding lintian tags for this, though: with so many packages not being able to use the autodep8 system (I vaguely recall someone suggesting that a third of Python packages would not be able to use the system), that's a lot of packages getting false positives. In particular, for the suggested first version of not-using-pybuild-autodep8 tag (which would probably be better named manual-autopkgtest-could-be-pybuild-autodep8), how would lintian go about identifying which packages fall into category 2.B and which into 2.C? The second version of the tag (better named something like no-autopkgtest-could-use-pybuild-autodep8, but that's still not very good) is less problematic. Best wishes, Julian
Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF)
On Wed, Jul 27, 2022 at 10:26:33AM -0400, Louis-Philippe Véronneau wrote: > The way I see it: > > 1. We should have a Lintian tag for packages not using the new > pybuild-autodep8 autopkgtest. It would be even better if this tag would be a > pointed hint that identified 'manually' written unit test autopkgtests that > could be replaced. > > This way, you get something like: > > python-foo source: not-using-pybuil-autodep8 [debian/tests/unittests] > > for python packages that have old 'manually' written unit test autopkgtests > and: > > python-foo source: not-using-pybuild-autodep8 [no-autopkgtest] > > for python packages without any autopkgtest. > > 2. lintian-brush (or something else, but I think lintian-brush is the right > tool) would go over these packages to: > > 2.1 Add the new autodep8 autopkgtests and build the package to see if they > pass > 2.2 Remove the "manual" unit test autopkgtests if 2.1 succeeds > 2.3 Open a bug report if 2.1 fails I'd be wary about 2.2 and 2.3. I have several packages where I know that an automated test will fail; there are all sorts of weird cases where I've had to write tests manually. I would also be quite cross if manually crafted tests were automatically removed, especially in cases such as Simon mentioned where they do things that that automatically generated test does not do. Another thing I could imagine happening is that the automated test succeeds in a trivial way - it succeeds but doesn't actually test much because of the nature of the package. On the other hand, a bug report saying something like the following seems much more reasonable: "We've tested this package using the automated autopkgtest system and it seems to work by adding the line 'Testsuite: autopkgtest-pkg-pybuild'; please check that the automated tests cover all of the tests of your manually written debian/tests/* and if so, then please remove them. The autopkgtest-pkg-pybuild logs are attached." This would give the maintainer the chance to decide how best to proceed. Best wishes, Julian
Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF)
On Tue, Jul 26, 2022 at 11:50:19AM -0300, Antonio Terceiro wrote: > I think the notes did not capture the consensus correctly. The point was > that it should be possible to automate updating the `Testsuite:` field > to run tests with pybuild-autopkgtest, and that we should probably do > that across team packages with the help of some scripting. This makes more sense, thanks. There seems to be little point running both pybuild-autopkgtest and a manually written debian/tests/* test suite. So would the script only add pybuild-autopkgtest to packages which don't have a manually written debian/tests/* suite? Best wishes, Julian
pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF)
On Sat, Jul 23, 2022 at 07:52:19PM +0200, Louis-Philippe Véronneau wrote: > == pybuild improvements == > > getting the autopkgtest MR in would be great > > https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/27 > > We need people to test this MR some more, although it seems fairly mature. > > It might be a good idea to have a line in d/control to let us migrate from > the existing autopkgtests running unit tests to the new automated MR. I've been looking at this a bit more. I'm not sure what this last paragraph means: the new "automated" autopkgtest will only be run if the maintainer explicitly adds: Testsuite: autopkgtest-pkg-pybuild to debian/control (see the autodep8 MR at https://salsa.debian.org/ci-team/autodep8/-/merge_requests/27/diffs - it will never automatically detect a pybuild package). And a maintainer would presumably only add that if they are also removing their existing debian/tests/control (or want to run the pybuild tests in addition). An alternative would be for the autodep8 patch to try to determine whether to run pybuild-autopkgtest. One approach could be: if the package would run autopkgtest-pkg-python: if debian/control does not contain an override_dh_auto_test stanza: run pybuild-autopkgtest Note, though, that if autodep8 is called, it will run all of the detected tests. (At least that is what I believe happens from reading /usr/bin/autodep8; I haven't double-checked this.) So, for example, if a package specifies Testsuite: autopkgtest-pkg-python it will also run the autopkgtest-pkg-pybuild suite as it will be detected as being a Python package, and vice versa. That is a possible reason *not* to use the above suggestion, as it would potentially run pybuild-autopkgtest even if not desired. Best wishes, Julian
Re: Notes from the DC22 Python Team BoF
On Sat, Jul 23, 2022 at 07:52:19PM +0200, Louis-Philippe Véronneau wrote: > Hey folks, > > We had a Python Team BoF at DC22 earlier today and I thought relaying the > notes we took in gobby here would be a good idea. Thanks for the notes, Louis-Philippe, and sorry I couldn't join you! A few comments > -- > == python3.11 == > > python3.11 release has been delayed, from october 2022 to december 2022. > [...] My 2 cents' worth is as the 3.9->3.10 transition took several months, and was quite complicated, it is not wise to attempt the 3.10->3.11 before the freeze. We could then potentially go straight to 3.12 a few months after the bookworm freeze rather than going to 3.11 first. And that will probably be quite painful. > == pybuild improvements == > > getting the autopkgtest MR in would be great > > https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/27 > > We need people to test this MR some more, although it seems fairly mature. > > It might be a good idea to have a line in d/control to let us migrate from > the existing autopkgtests running unit tests to the new automated MR. I'll take this to a separate email. > == lintian tags requests for the team == > > pollo will write you Python-related lintian tags. Ask him to. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1004746 :-) Julian
Re: Build and run-time triplets
On Mon, Jul 25, 2022 at 12:41:16AM +0500, Andrey Rahmatullin wrote:
> On Sun, Jul 24, 2022 at 08:30:42PM +0100, Julian Gilbey wrote:
> [...]
> > > > are they all effectively Multi-Arch: no? Is this worth thinking about
> > > > in the longer term?
> > > What do you propose?
> >
> > I think the fix to bug #812228 might have done the job nicely ;-)
> If it actually ships extensions, the "it should usually get a dependency
> on the Python interpreter for the same architecture" part should still
> apply as far as I understand it.
Thanks Andrey!
OK. So let's dissect this tag info and see where we're currently at.
Explanation: This Multi-Arch: same package uses
pycompile or
py3compile in the specified maintainer script.
.
py{,3}compile are tools used to byte-compile Python source
files. It is typically run on installation of Debian packages that ship
Python modules. However, they do not support installing several
architectures of the same package and this is not Multi-Arch: safe.
This is now out-of-date: firstly, we can presumably get rid of the
pycompile mention, as there are only a tiny handful of Python 2
packages still around, and we're trying to get rid of them.
Secondly, py3compile now supports installing several architectures of
the same package; see the closing changelog message on bug 812228:
Architecture-qualify py*compile and py*clean calls in maintainer scripts,
for architecture-specific Python packages. This allows co-installation
(and even concurrent unpacking) of different architectures of a package.
So the rest of the paragraph is also out of date.
If the contents of the package is not architecture dependent, it should
usually be made binary-all.
That is still certainly true.
If the contents of the package is architecture dependent, it should
usually get a dependency on the Python interpreter for the same
architecture. This is a dependency in the form of python3, not
an architecture-qualified dependency such as python3:any (which
can be fulfilled by the Python interpreter for any architecture).
This is interesting; dh-python gives the dependency:
python3 (<< 3.11), python3 (>= 3~), python3:any
which has both same-architecture and qualified architecture
dependencies; obviously the same-architecture one "wins". But this
paragraph is probably unnecessary for most dh-python-using packages
(though it doesn't seem to harm).
If a dependency on the Python interpreter for the same architecture
exists (usually generated by dh-python), the
Multi-Arch: same has no effect and should be dropped.
Ah. I see the point. Because python3 and python3-minimal are
Multi-Arch: allowed, the different arches of python3 are not
co-installable, and so there is no point in labelling the
arch-dependent module packages as Multi-Arch: same; they still could
not be co-installed.
See-Also: pycompile(1), py3compile(1), Bug#812228
This list can probably be trimmed down to py3compile.
I hope this reasoning is useful; shall I pass it on to the lintian
folk?
Best wishes,
Julian
