Bug#1011937: python-debian: Please consider moving deb822 to a standalone distro-independent Python package
Control: retitle -1 Make python-debian (more) portable Control: tags -1 + pending There's a lot of work in git that should make python-debian portable across platforms (operating systems, distributions, etc). The next release will address this bug, so tagging as 'pending'. -- Stuart Prescotthttp://www.nanonanonano.net/ stu...@nanonanonano.net Debian Developer http://www.debian.org/ stu...@debian.org GPG fingerprint90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7
Bug#1011937: python-debian: Please consider moving deb822 to a standalone distro-independent Python package
On Sun, 29 May 2022 00:51:10 +1000 Stuart Prescott wrote: > Hi Stephan, > > [...] > > There's a few options here: > > [...] > > * use something else to make the .deb files for the test suite. I could > imagine > a fallback for missing ar(1) that uses a pure python ar implementation that > can be installed via pip on these other platforms. The arpy module can't make > ar files at present which is a shame; the `unix_ar` or `ar` modules look > promising for that, there might be others. > > https://github.com/getninjas/unix_ar > https://github.com/vidstige/ar > [...] > > cheers > Stuart > > For reference, I had a look at https://github.com/vidstige/ar and I suspect it does not support writing based on: * Its definition of valid open modes are `MODES = 'rbt'` * There are no obvious "write"/"add member" functions. I can only find read/load methods. The `unix_ar` module looks more solid in that regard, so that is still an option. Thanks, ~Niels
Bug#1011937: python-debian: Please consider moving deb822 to a standalone distro-independent Python package
Hi Stephan,
On Friday, 27 May 2022 20:09:29 AEST Jelmer Vernooij wrote:
> On Fri, May 27, 2022 at 12:01:17PM +0200, Stephan Lachnit wrote:
> > I'm not an expert on python-debian and I don't use other distros than
> > Debian, so I can only forward you some bug reports from
Thanks! I'd spotted one of these in the past but not others.
> > https://github.com/fsfe/reuse-tool/issues/466:
I'd definitely rather make python-debian more portable than have projects
forking bits of it into local vendored versions. Vendoring code creates
technical debt and is somewhat antithetical to the idea of making code more
reusable.
> > - https://github.com/fsfe/reuse-tool/issues/425: `apt_pkg.Error:
> > W:Unable to read /etc/apt/apt.conf.d/ - DirectoryExists (2: No such
> > file or directory), E:Unable to determine a suitable packaging system
> > type`
As already noted, python-debian works fine without python-apt installed; the
unexpected situation here is that python-apt is installed but non-functional
in some way. I'm not sure whether the bug is really that a non-functional
python-apt is installed, but if we can work around it, then even better.
I have a feeling that we can change the way we use apt_pkg these days and that
we can avoid generating that error. I need to talk to the python-apt people
about that, but also I also need a way of reproducing an environment where a
non-functional python-apt is installed so that I can test this out.
From https://salsa.debian.org/python-debian-team/python-debian/-/
merge_requests/85
> > Since Alpine really doesn't offer anything, a lot of fail because of
> > missing `bin/ar`, which is an excellent test for a non-Debian-standard
> > environment. Not sure how this should be handled best though: maybe
> > something similar to the `have_apt_pkg` variable?
This is only the tests and in particular, it's making the test data -- ArFile
and DebFile are actually much more portable (except for the zstd compressed
.debs where there remain portability problems because of the requirement for a
zstd binary). It's only the tests that need the 'ar' binary and nothing in the
module code itself.
There's a few options here:
* require that ar be installed for the purposes of testing just as we do in
Debian; this is an explicit dependency in Debian, not something that happens
to be there because more batteries are included. The binutils package gives us
ar and is listed in debian/control, it's just that Python hasn't come up with
a way of expressing such dependencies in setup.py or similar. For alpine, ar
is in the binutils package and there are a few different versions available
for windows, such as via a gcc package from choco.
* use something else to make the .deb files for the test suite. I could imagine
a fallback for missing ar(1) that uses a pure python ar implementation that
can be installed via pip on these other platforms. The arpy module can't make
ar files at present which is a shame; the `unix_ar` or `ar` modules look
promising for that, there might be others.
https://github.com/getninjas/unix_ar
https://github.com/vidstige/ar
* skip the tests that need ar; in practice, that's probably all the tests from
debian/tests/test_debfile.py, and so a module level skip might be appropriate,
with something like
```
pytestmark = pytest.mark.skipif(not shutil.which("ar"), reason="...")
```
I'm cautious about automatically skipping tests because that's a route to non-
determinism, accidentally skipping tests, and therefore missing problems.
However, we do that for python-apt already, so there's precedent already in
the code; I'd be fine with that as a short term solution in advance of
something better.
Thanks in advance for your contribution to python-debian :)
cheers
Stuart
--
Stuart Prescotthttp://www.nanonanonano.net/ stu...@nanonanonano.net
Debian Developer http://www.debian.org/ stu...@debian.org
GPG fingerprint90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7
Bug#1011937: python-debian: Please consider moving deb822 to a standalone distro-independent Python package
On Fri, May 27, 2022 at 12:01:17PM +0200, Stephan Lachnit wrote: > I'm not an expert on python-debian and I don't use other distros than > Debian, so I can only forward you some bug reports from > https://github.com/fsfe/reuse-tool/issues/466: > - https://github.com/fsfe/reuse-tool/issues/311 (fixed now) > - https://github.com/fsfe/reuse-tool/issues/425: `apt_pkg.Error: > W:Unable to read /etc/apt/apt.conf.d/ - DirectoryExists (2: No such > file or directory), E:Unable to determine a suitable packaging system > type` > This one is probably tricky to solve, maybe python-debian can check > if it is in a Debian env by checking /etc/os-release? > > Ofc if python-debian would ensure cross-platform operability, that > would be great, then there would be no need for a package split. I > guess it depends a bit on the policy: is it tested on non-Debian > platforms or just an afterthought where bugs get fixed when reported > in version? If it is the latter, REUSE will probably still try to > replace it. If it is the former, I think there is little reason to > move away from python-debian. cc/ Max Mehl? It looks like reuse doesn't juse use deb822, but the copyright module as well - so just extracting deb822 would not be sufficient: https://github.com/fsfe/reuse-tool/issues/454 > Looking quickly at the CI though, it does not seem that this is > currently the case. Maybe you could add a `unit-tests-alpine` and > `unit-tests-windows` job based on the official Python Docker images > [1] (w/ dependencies pulled via pip)? IMHO this should give enough > confidence in the cross-platform operability of this package (again > cc/ Max Mehl, I'm not a developer of REUSE). I'd prefer that avenue; we could potentially disable some of the functionality on windows (e.g. stuff that relies on python3-apt, which AFAIK is already optional). >From what I can tell, the testsuite runs fine on other Linux platforms - we haven't had many bugreports from packagers on those platforms. I wouldn't be surprised if people came along in the future that e.g. needed watch file parsing on Windows or copyright file parsing, and I wouldn't want to have to split out all of those as separate modules as well. Cheers, Jelmer > On Fri, May 27, 2022 at 11:15 AM Jelmer Vernooij wrote: > > > > Hi Stephan, > > > > On Fri, May 27, 2022 at 10:05:47AM +0200, Stephan Lachnit wrote: > > > I'm working on a project that aims to use REUSE [1] to automate the > > > process of creating and maintaing the d/copyright file. > > > > > > tl;dr: REUSE is a specification to annote license and copyright holder > > > in a machine-readable way in the source files itself. Since Debian has a > > > similar per-file copyright concept, using information from source > > > packages that follow this specification can in principle be automated to > > > reduce the work maintainers have to do. > > > > > > > > > For several reasons it would be nice to have a d/copyright parser at > > > hand, and since python-debian provides this with the deb822 module, it > > > would be unneccessary work to write this again. REUSE already uses > > > python-debian for this, but wants to get rit of it because there are > > > some issue with this module on non-Debian OSes [2]. > > > > > > > > > To come to the topic of this wishlist-bugreport: would it be possible to > > > factor the deb822 parser out in a completetly separate Python package > > > (on PyPi), e.g. python3-deb822, that explicitly does not depend on a > > > Debian environment? > > > > What are the portability issues that exist in non-Debian environments? I'd > > prefer to address those if possible rather than factoring out the deb822 > > module, since that complicates on an ongoing basis in the future. > > > > FWIW python-debian is packaged in other Linux distributions, so my guess > > is that the issue is non-Linux rather than non-Debian environments? > > https://repology.org/project/python:debian/versions > > > > Jelmer > > -- > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-python-debian-maint
Bug#1011937: python-debian: Please consider moving deb822 to a standalone distro-independent Python package
Hi Jelmer, Thanks for the quick response! I'm not an expert on python-debian and I don't use other distros than Debian, so I can only forward you some bug reports from https://github.com/fsfe/reuse-tool/issues/466: - https://github.com/fsfe/reuse-tool/issues/311 (fixed now) - https://github.com/fsfe/reuse-tool/issues/425: `apt_pkg.Error: W:Unable to read /etc/apt/apt.conf.d/ - DirectoryExists (2: No such file or directory), E:Unable to determine a suitable packaging system type` This one is probably tricky to solve, maybe python-debian can check if it is in a Debian env by checking /etc/os-release? Ofc if python-debian would ensure cross-platform operability, that would be great, then there would be no need for a package split. I guess it depends a bit on the policy: is it tested on non-Debian platforms or just an afterthought where bugs get fixed when reported in version? If it is the latter, REUSE will probably still try to replace it. If it is the former, I think there is little reason to move away from python-debian. cc/ Max Mehl? Looking quickly at the CI though, it does not seem that this is currently the case. Maybe you could add a `unit-tests-alpine` and `unit-tests-windows` job based on the official Python Docker images [1] (w/ dependencies pulled via pip)? IMHO this should give enough confidence in the cross-platform operability of this package (again cc/ Max Mehl, I'm not a developer of REUSE). Best, Stephan [1]: https://hub.docker.com/_/python On Fri, May 27, 2022 at 11:15 AM Jelmer Vernooij wrote: > > Hi Stephan, > > On Fri, May 27, 2022 at 10:05:47AM +0200, Stephan Lachnit wrote: > > I'm working on a project that aims to use REUSE [1] to automate the > > process of creating and maintaing the d/copyright file. > > > > tl;dr: REUSE is a specification to annote license and copyright holder > > in a machine-readable way in the source files itself. Since Debian has a > > similar per-file copyright concept, using information from source > > packages that follow this specification can in principle be automated to > > reduce the work maintainers have to do. > > > > > > For several reasons it would be nice to have a d/copyright parser at > > hand, and since python-debian provides this with the deb822 module, it > > would be unneccessary work to write this again. REUSE already uses > > python-debian for this, but wants to get rit of it because there are > > some issue with this module on non-Debian OSes [2]. > > > > > > To come to the topic of this wishlist-bugreport: would it be possible to > > factor the deb822 parser out in a completetly separate Python package > > (on PyPi), e.g. python3-deb822, that explicitly does not depend on a > > Debian environment? > > What are the portability issues that exist in non-Debian environments? I'd > prefer to address those if possible rather than factoring out the deb822 > module, since that complicates on an ongoing basis in the future. > > FWIW python-debian is packaged in other Linux distributions, so my guess > is that the issue is non-Linux rather than non-Debian environments? > https://repology.org/project/python:debian/versions > > Jelmer
Bug#1011937: python-debian: Please consider moving deb822 to a standalone distro-independent Python package
Hi Stephan, On Fri, May 27, 2022 at 10:05:47AM +0200, Stephan Lachnit wrote: > I'm working on a project that aims to use REUSE [1] to automate the > process of creating and maintaing the d/copyright file. > > tl;dr: REUSE is a specification to annote license and copyright holder > in a machine-readable way in the source files itself. Since Debian has a > similar per-file copyright concept, using information from source > packages that follow this specification can in principle be automated to > reduce the work maintainers have to do. > > > For several reasons it would be nice to have a d/copyright parser at > hand, and since python-debian provides this with the deb822 module, it > would be unneccessary work to write this again. REUSE already uses > python-debian for this, but wants to get rit of it because there are > some issue with this module on non-Debian OSes [2]. > > > To come to the topic of this wishlist-bugreport: would it be possible to > factor the deb822 parser out in a completetly separate Python package > (on PyPi), e.g. python3-deb822, that explicitly does not depend on a > Debian environment? What are the portability issues that exist in non-Debian environments? I'd prefer to address those if possible rather than factoring out the deb822 module, since that complicates on an ongoing basis in the future. FWIW python-debian is packaged in other Linux distributions, so my guess is that the issue is non-Linux rather than non-Debian environments? https://repology.org/project/python:debian/versions Jelmer
Bug#1011937: python-debian: Please consider moving deb822 to a standalone distro-independent Python package
Source: python-debian Version: 0.1.43 Severity: wishlist X-Debbugs-Cc: stephanlach...@debian.org, max.m...@fsfe.org I'm working on a project that aims to use REUSE [1] to automate the process of creating and maintaing the d/copyright file. tl;dr: REUSE is a specification to annote license and copyright holder in a machine-readable way in the source files itself. Since Debian has a similar per-file copyright concept, using information from source packages that follow this specification can in principle be automated to reduce the work maintainers have to do. For several reasons it would be nice to have a d/copyright parser at hand, and since python-debian provides this with the deb822 module, it would be unneccessary work to write this again. REUSE already uses python-debian for this, but wants to get rit of it because there are some issue with this module on non-Debian OSes [2]. To come to the topic of this wishlist-bugreport: would it be possible to factor the deb822 parser out in a completetly separate Python package (on PyPi), e.g. python3-deb822, that explicitly does not depend on a Debian environment? Cheers, Stephan [1]: https://reuse.software/ [2]: https://github.com/fsfe/reuse-tool/issues/466 -- System Information: Debian Release: bookworm/sid APT prefers unstable APT policy: (500, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 5.17.0-2-amd64 (SMP w/8 CPU threads; PREEMPT) Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=en_US.utf8, LC_CTYPE=en_US.utf8 (charmap=UTF-8), LANGUAGE=en_US:en Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled
