Mitchell's Debian Python Team join request
Hello! My name is Mitchell and I would like to join the Debian Python Team. I would like to be a maintainer of https://tracker.debian.org/pkg/pyparted. My Salsa login is mitchdz. I have read https://salsa.debian.org/python-team/tools/python-modules/blob/master/policy.rst and accept it. Thanks, -Mitch
Re: Package testing with Salsa CI for new packages
Hi, On 18 Aug 2023 at 19:03:48, Paul Boddie wrote: > On Friday, 18 August 2023 16:12:19 CEST Carles Pina i Estany wrote: > > > > For the jobs it is happening via > > https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built- > apt-repository > > Reviewing this documentation is actually more helpful than I thought it would > be. I had noticed the "aptly" task in the YAML files, and I had started to > wonder if that meant that some kind of repository publishing was occurring > somewhere. Ha! I wasn't aware of the aptly option (https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built-apt-repository and SALSA_CI_DISABLE_APTLY=0). I think that I might have re-invented the wheel in a tiny part of Debusine CI/CD. I will point out at some things that might safe some time to you (great) or not (ignore! :-) ): debusine's .gitlab-ci.yml: https://salsa.debian.org/freexian-team/debusine/-/blob/devel/.gitlab-ci.yml Latest CI build: https://salsa.debian.org/freexian-team/debusine/-/pipelines/566796 In the "build" job: https://salsa.debian.org/freexian-team/debusine/-/jobs/4574451 You can see what's doing and it's using salsa-ci/pipeline's defaults in the build. All comes from https://salsa.debian.org/salsa-ci-team/pipeline/ , specifically from: https://salsa.debian.org/salsa-ci-team/pipeline/-/blob/master/salsa-ci.yml#L201 (I sometimes grep salsa-ci-team/pipeline for commands that I see in debusine's CI output to understand the flow...) In the autopkgtest job it sets things up and invokes autopkgtest: https://salsa.debian.org/freexian-team/debusine/-/jobs/4574458 The *.deb from build are made available via "artifacts" (and that autopkgtest needs the artifact of build). The job autopkgtest, via debci command, runs autopkgtest: https://salsa.debian.org/freexian-team/debusine/-/jobs/4574458#L65 And, to be honest, I didn't look into these details for long time. From my point of view I just write debian/tests/control with the tests (commands) to run: https://salsa.debian.org/freexian-team/debusine/-/blob/devel/debian/tests/control And that things run in a container/chroot (depends), etc. You can invoke autopkgtest locally as well... > > > In the Salsa CI environment, I would need to have the built packages > > > (found in the artefacts for each package's build job) copied somewhere > > > that can then be found by the Moin package's pipeline jobs and the > > > scripts creating a special repository of new packages. > > > > Archiving artifacts should happen automatically on the "build" step of > > Salsa CI (salsa-ci/pipeline). If I understand correctly what you > > wanted... > > I think you have a good understanding of what I am trying to achieve. If I > can > get the new package dependencies (emeraldtree, feedgen, and so on) to yield > installable packages when built that can then be referenced by Salsa CI as it > runs the build jobs for Moin, I have a chance of running the test suite. yep! > I'm currently persuading the CI system to run the "aptly" task and to publish > package repositories. I will then augment a customised YAML file for the Moin I might look more about this on Monday (I will write it down now) because I am using "dpkg-scanpackages" and I might be able to avoid it via SALSA_CI_DISABLE_APTLY: 0 I see the salsa-CI like a fancy environment where to run scripts, generate files, pass files between scripts, refactor scripts (in YAML in my .gitlab-ci.yml or to upstream salsa-ci/pipeline), scheduled jobs, etc. so almost anything that you can think of you can do it. Either with artifacts to keep and retrieve files between "jobs" or via GitHub Pages. If you need help creating a new job (a new bubble in the pipeline) let me know (even if you want off-list, if this is getting off-topic of debian-python). Or look at the https://salsa.debian.org/freexian-team/debusine/-/blob/devel/.gitlab-ci.yml where were created the stage "upstream-tests" with things like "code-linting". So You could even create dependencies of your task before the tests are executed and pass the .deb files via artifacts (or even repositories). > package with references to these repositories and see if the test > suite can be invoked somewhat more successfully as a consequence. It's possible for sure. Other people in this list might come up with a different idea. I don't have almost any experience with Debian packaging, but I have some experience on the salsa CI. So I might be giving you solutions that might be sub-optimal! :-) Cheers, -- Carles Pina i Estany https://carles.pina.cat || Wiktionary translations: https://kamus.pina.cat
Re: Package testing with Salsa CI for new packages
On Fri, Aug 18, 2023 at 07:19:18PM +0200, Paul Boddie wrote: > > > Here, it would seem that the most prudent approach is to use the Salsa CI > > > service instead of trying to get the test suite to run during the package > > > build process. > > > > You should do both if possible, assuming that by "Salsa CI service" you > > mean autopkgtests which you can, and IMO should, also run locally. > > I'm not really clear on what autopkgtest really is, other than a tool that > uses some kind of test suite description that may reside in debian/test. I'm > also not completely clear on what is supposed to invoke it, other than either > the Salsa CI mechanism or dh_auto_test. The maintainer is supposed to invoke it before uploading the package, and the Debian infra invokes it on all packages that are uploaded. Notably, dh_auto_test is unrelated. > In the Debian Wiki documentation... > > https://wiki.debian.org/Python/LibraryStyleGuide > > ...it mentions a field in debian/control: > > Testsuite: autopkgtest-pkg-python > > My impression is that this calls autodep8 to generate some kind of test suite Yes. Though note that it generates a trivial test that only checks a top-level import (just like the wiki page says). > description which is then invoked by dh_auto_test. It's not invoked by dh_auto_test. autopkgtests are not a part of the package build process and they operate on built binary packages. > It doesn't help that there > is an alternative to this that resembles it but behaves differently: > > Testsuite: autopkgtest-pkg-pybuild It just generates a different (better) test. > > > I have also found it difficult to persuade the tests to run successfully > > > during the build process. A few of these attempt to invoke the moin > > > program, but this cannot be located since it is not installed in the > > > build environment. > > > > This should also be fine, unless it's completely impossible to run it > > without installing into /. > > The moin program is made available in setup.py using an entry point. Maybe if > there were some kind of script instead, it would work. AFAIK there should be ways to work with this. Ideally, of course, the upstream test suite should support this, but I understand that not all upstreams use common practices. > > > However, one conclusion is that testing a system, as some of the > > > test cases appear to do, and as opposed to testing library functionality, > > > is not necessarily appropriate when directed by something like > > > dh_auto_test. > > > > If there are tests that can't be run at build time you can skip those. You > > can even ask the upstream to provide tool arguments to simplify that. > > I may well discuss such matters with them. One challenge that is relevant in > this situation is that upstream have been working in their own virtualenv (or > venv, or whatever it is now) world for a few years, using plenty of > dependencies that were not packaged in Debian. Which is by itself not a problem from the technical side, as you could just package those (I understand that this requires effort and may have other problems, but by itself it's normal).
Re: Package testing with Salsa CI for new packages
On Friday, 18 August 2023 18:21:00 CEST Andrey Rakhmatullin wrote: > On Thu, Aug 17, 2023 at 05:10:08PM +0200, Paul Boddie wrote: > > Here, it would seem that the most prudent approach is to use the Salsa CI > > service instead of trying to get the test suite to run during the package > > build process. > > You should do both if possible, assuming that by "Salsa CI service" you > mean autopkgtests which you can, and IMO should, also run locally. I'm not really clear on what autopkgtest really is, other than a tool that uses some kind of test suite description that may reside in debian/test. I'm also not completely clear on what is supposed to invoke it, other than either the Salsa CI mechanism or dh_auto_test. In the Debian Wiki documentation... https://wiki.debian.org/Python/LibraryStyleGuide ...it mentions a field in debian/control: Testsuite: autopkgtest-pkg-python My impression is that this calls autodep8 to generate some kind of test suite description which is then invoked by dh_auto_test. It doesn't help that there is an alternative to this that resembles it but behaves differently: Testsuite: autopkgtest-pkg-pybuild What I have previously managed to do with the Salsa CI mechanism is to run tests specified in debian/test, disabling these tests at build time using the following in debian/rules: export PYBUILD_DISABLE=test For another package which has limited dependencies, I do this because the tests take a long time to run, and I even need to increase the timeout for the Salsa CI jobs. For this package, the tests don't take too long to run (those that do run), but some tests fail for reasons previously described, and this is due to some configuration issue that is unpleasant to debug. Success with testing using Salsa CI steers me away from using the Testsuite field in the normal build environment because the resulting test suite invocation failures suggest that some customisation is needed or that the environment is inappropriate in some way (even if I specify numerous build dependencies). > > One motivation for doing so involves not having to specify > > build dependencies for packages only needed for test suite execution, > > which itself requires the invocation of gbp buildpackage with > > --extra-package arguments since some packages are completely new to > > Debian. > > This is fine. Not to mention that the same problem exists for > autopkgtests, as you say below. > > > I have also found it difficult to persuade the tests to run successfully > > during the build process. A few of these attempt to invoke the moin > > program, but this cannot be located since it is not installed in the > > build environment. > > This should also be fine, unless it's completely impossible to run it > without installing into /. The moin program is made available in setup.py using an entry point. Maybe if there were some kind of script instead, it would work. > > However, one conclusion is that testing a system, as some of the > > test cases appear to do, and as opposed to testing library functionality, > > is not necessarily appropriate when directed by something like > > dh_auto_test. > > If there are tests that can't be run at build time you can skip those. You > can even ask the upstream to provide tool arguments to simplify that. I may well discuss such matters with them. One challenge that is relevant in this situation is that upstream have been working in their own virtualenv (or venv, or whatever it is now) world for a few years, using plenty of dependencies that were not packaged in Debian. That hasn't been entirely conducive to adoption of Moin 2 which has in turn isolated its developers further. Fortunately, the gulf between that world and Debian can be bridged, and some of the more tedious packaging seems to have been done for many of the dependencies involved. It just needs someone to shepherd this effort to an acceptable conclusion, which is of concern to Debian itself since Moin runs the Debian Wiki. Thanks for the encouragement and advice. Paul
Re: Package testing with Salsa CI for new packages
On Friday, 18 August 2023 16:12:19 CEST Carles Pina i Estany wrote: > > For the jobs it is happening via > https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built- apt-repository Reviewing this documentation is actually more helpful than I thought it would be. I had noticed the "aptly" task in the YAML files, and I had started to wonder if that meant that some kind of repository publishing was occurring somewhere. > > In the Salsa CI environment, I would need to have the built packages > > (found in the artefacts for each package's build job) copied somewhere > > that can then be found by the Moin package's pipeline jobs and the > > scripts creating a special repository of new packages. > > Archiving artifacts should happen automatically on the "build" step of > Salsa CI (salsa-ci/pipeline). If I understand correctly what you > wanted... I think you have a good understanding of what I am trying to achieve. If I can get the new package dependencies (emeraldtree, feedgen, and so on) to yield installable packages when built that can then be referenced by Salsa CI as it runs the build jobs for Moin, I have a chance of running the test suite. I'm currently persuading the CI system to run the "aptly" task and to publish package repositories. I will then augment a customised YAML file for the Moin package with references to these repositories and see if the test suite can be invoked somewhat more successfully as a consequence. Thanks once again for the guidance! Paul
Re: Package testing with Salsa CI for new packages
On Thu, Aug 17, 2023 at 05:10:08PM +0200, Paul Boddie wrote: > Here, it would seem that the most prudent approach is to use the Salsa CI > service instead of trying to get the test suite to run during the package > build process. You should do both if possible, assuming that by "Salsa CI service" you mean autopkgtests which you can, and IMO should, also run locally. > One motivation for doing so involves not having to specify > build dependencies for packages only needed for test suite execution, which > itself requires the invocation of gbp buildpackage with --extra-package > arguments since some packages are completely new to Debian. This is fine. Not to mention that the same problem exists for autopkgtests, as you say below. > I have also found it difficult to persuade the tests to run successfully > during the build process. A few of these attempt to invoke the moin program, > but this cannot be located since it is not installed in the build > environment. This should also be fine, unless it's completely impossible to run it without installing into /. > However, one conclusion is that testing a system, as some of the > test cases appear to do, and as opposed to testing library functionality, is > not necessarily appropriate when directed by something like dh_auto_test. If there are tests that can't be run at build time you can skip those. You can even ask the upstream to provide tool arguments to simplify that.
Re: pdm: Please replace python3-pep517 with python3-pyproject-hooks in Depends/Build-Depends
On Friday, August 18, 2023 9:33:48 AM EDT Andreas Tille wrote: > Hi Scott, > > Am Fri, Aug 18, 2023 at 01:15:18PM + schrieb Scott Kitterman: > > On August 18, 2023 1:04:26 PM UTC, Andreas Tille wrote: > > >> In Debian terms, it's not the preferred form for modification, so it's > > >> not source. In this regard DFSG goes farther than some software > > >> licenses.> > > > >I think the point Jeroen wanted to make is that these are actually > > >source file archives which "by chance" are featuring a .whl extension > > >rather than a .zip extension. > > > > A wheel is not the preferred form for modification. They're not wheels by > > chance at all. > Yes, thanks to Jeroen's hint I realised this as well and I agree that > this is a nasty way to hide the fact that the files are actually source > archives. They aren't. > However, you confirmed yourself that future_fstrings is an exception and > comes with source and thus does not violate DFSG. The only difference > I personally can see is that the archives are just hiding what they are. > We could simply add do some >for whl in *.whl ; do ln -s $whl $(basename .whl).zip ; done > and we have source archives that are obviously what they are. > > > From a DFSG perspective, > > Hmmm, the only thing where I can draw a violation of the DFSG is that > there are no d/copyright entries for the source code that is hidden > inside these *.whl files. Otherwise its "just" duplicated code (in most > cases) which is definitely not nice but IMHO not a violation of DFSG. > The disagreement here is that Python wheels aren't source. DFSG #2 requires the source be present and these aren't it. If you look at the WAF entry in the FTP team reject FAQ, this is similar. The FTP team view has long been that DFSG #2 means the actual preferred form for modification. > > the most straightforward approach is to build-depend on the relevant > > Debian packages and build any needed wheels from that. > Do avoid source code duplication I'm willing to do that. Yes, I > perfectly agree that its pretty ugly (I'm just a bit unsure about > the DFSG violation). I'm wondering whether a simple > >zip whl.zip /path/to/python/files ; mv whl.zip whl.whl > > will be something that can replace the current packages. I think > we also need to patch the tests to fit the version numbers (if > we do not want to cheat and simply use the version numbers of the > original whl files to avoid patching). Perhaps, but there are nuances to the wheel format. Please use Wheel to generate them. > > It won't necessarily get you the same version as upstream uses, but it's > > definitely DFSG compliant. > We also might symlink our resulting whl files with the version number > pdm upstream might expect in their tests. The question is, whether all > this effort might break the tests in some way and we might end up with > endless patching by at the same time loosing the chance to discuss with > upstream. But it might be time to discuss the issue with upstream > anyway. Perhaps. If it were me, what I'd do is locate the missing tarballs and stash them in debian/missing-sources/ and worry about more complex solutions later. Once you're done that, you've met DSFG #2 and there's no need to strip the wheels from the binary. It's not super maintainable, but it will allow you to focos on getting the tests working as upstream ships them without any Debian customizations that might cause Debian specific failures. Scott K signature.asc Description: This is a digitally signed message part.
Re: Package testing with Salsa CI for new packages
Hi, On 18 Aug 2023 at 13:16:15, Paul Boddie wrote: > On Friday, 18 August 2023 09:51:29 CEST Carles Pina i Estany wrote: > > > > I'm not a Debian developer but I have some experience on Salsa CI, so I > > thought that I might be able to help... but then I was confused by a > > specific part of the message: > > > > On 17 Aug 2023 at 17:10:08, Paul Boddie wrote: > > > > [...] > > > > > For another package I have been working on, the Salsa CI facility has > > > proven to be usable, configured using files in debian/test, particularly > > > as it allows test-related dependencies to be specified independently. > > > However, this other package has no dependencies that are currently > > > unpackaged in Debian. Meanwhile, the testing of this new Moin package > > > depends on brand new packages somehow being made available. > > > > If this dependencies are available on the "build" step: could they be > > made available on the autopkgtest? I didn't quite understand why this is > > not possible. I've found the autopkgtest quite flexible (since the tests > > are scripts that could prepare some environment) > > The package has dependencies on installation but these dependencies > are not strictly necessary when building. However, if I wanted to run I have the same situation on this side. Two solutions for two places. Debusine uses https://salsa.debian.org/salsa-ci-team/pipeline approach (extending it) > the test suite when building, I would indeed need to pull in these > dependencies as build dependencies so that the software being tested > can run without import errors. In our repo I have two solutions (or hacks, or workarounds) for this situation. AUTOPKGTEST TIME (after build) == In the autopkgtest job (from salsa-ci/pipeline), autopkgtest runs a shell script (below integration-tests-generic). Some of them set things up: -- Tests: integration-tests-generic Depends: debusine-client, debusine-server, debusine-worker, postgresql, redis-server, sudo, nginx Restrictions: allow-stderr, needs-root -- It could pull things from the internet (Pypi, git...) if it also has, in Restrictions: "needs-internet" Alternatively (and perhaps better, to keep things separated?), you could use salsa CI variable: SALSA_CI_AUTOPKGTEST_ARGS with this format: SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/pin-django-from-backports.sh Instead of pinning you could install packages. I don't know if it needs "needs-internet" in Restrictions. Note that the debusine-* packages are made available automatically thanks to Salsa CI pipeline and that tests-autopkgtest has: -- needs: - job: build artifacts: true -- So I don't need to do any repository. The build job just creates the *.deb packages (in a specific directory?), they are saved as artifacts and made available to "autopkgtest" job. UNIT-TESTS (before build) = We have a job before build doing (I'm simplifying): - unit-tests: stage: upstream-tests dependencies: [] script: - NON_INTERACTIVE=1 bin/quick-setup.sh install_packages - make coverage # to run the tests with coverage - In "bin/quick-setup.sh" it installs the required packages (some from Debian, some from Pypi if need to be) In here we execute tests before packaging them (because we are upstream as well). > I have to add that the other package I refer to has a test suite that takes a > long time to run, so that is another reason why I chose Salsa CI for that > package instead of letting autopkgtest do its work: Note that when I say "autopkgtest" above I always think of autopkgtest ran by Salsa CI. > One can imagine having a common storage area holding these newly Until now I see "the whole internet" as a "common storage area" (I know, not ideal! could be closer to the process, more robust, faster, etc.) > introduced packages that the CI scripts could access in preference to > the usual archives. In fact, this would be something that might also > affect existing packages. Consider the situation where fixes to a > dependency are required to fix functionality in a particular package. > One would have to wait for the fixed dependency to become integrated > into the unstable archive before the principal package's tests would > start to work again. Semi-offtopic: MMmmm... Debusine is the project that I mentioned before where I implemented different things in .gitlab-ci.yml. What you mentioned here is a thing that debusine, in the future (not impelmented yet) might help. But other people can talk better about this than me... and it's not available right now. > > I also created a repo and hosted it on a Salsa CI page for internal > > testing but this is a bit of a workaround. This is in a new job but just > > download the artifacts (via a Salsa CI dependency) and run > > dpkg-scanpackages and copy the files to the right place). > > This sounds like something related to what might be required. In
Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build
On Fri, Aug 18, 2023 at 09:23:17AM -0400, Scott Talbert wrote: > On Fri, 18 Aug 2023, Andreas Tille wrote: > > > Am Fri, Aug 18, 2023 at 01:42:53PM +0100 schrieb Julian Gilbey: > > > I'm sure I'm not the only one who received a whole bunch of bugs > > > entitled "Fails to build source after successful build" last weekend. > > > There was one theme common to most of them: the presence of a > > > *.egg-info directory which was not cleaned by debian/rules clean. > > > [...] > > It is being worked on: > https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/46 Amazing! Thanks, Julian
Re: pdm: Please replace python3-pep517 with python3-pyproject-hooks in Depends/Build-Depends
Hi Scott, Am Fri, Aug 18, 2023 at 01:15:18PM + schrieb Scott Kitterman: > On August 18, 2023 1:04:26 PM UTC, Andreas Tille wrote: > >> In Debian terms, it's not the preferred form for modification, so it's not > >> source. In this regard DFSG goes farther than some software licenses. > > > >I think the point Jeroen wanted to make is that these are actually > >source file archives which "by chance" are featuring a .whl extension > >rather than a .zip extension. > > A wheel is not the preferred form for modification. They're not wheels by > chance at all. Yes, thanks to Jeroen's hint I realised this as well and I agree that this is a nasty way to hide the fact that the files are actually source archives. However, you confirmed yourself that future_fstrings is an exception and comes with source and thus does not violate DFSG. The only difference I personally can see is that the archives are just hiding what they are. We could simply add do some for whl in *.whl ; do ln -s $whl $(basename .whl).zip ; done and we have source archives that are obviously what they are. > From a DFSG perspective, Hmmm, the only thing where I can draw a violation of the DFSG is that there are no d/copyright entries for the source code that is hidden inside these *.whl files. Otherwise its "just" duplicated code (in most cases) which is definitely not nice but IMHO not a violation of DFSG. > the most straightforward approach is to build-depend on the relevant Debian > packages and build any needed wheels from that. Do avoid source code duplication I'm willing to do that. Yes, I perfectly agree that its pretty ugly (I'm just a bit unsure about the DFSG violation). I'm wondering whether a simple zip whl.zip /path/to/python/files ; mv whl.zip whl.whl will be something that can replace the current packages. I think we also need to patch the tests to fit the version numbers (if we do not want to cheat and simply use the version numbers of the original whl files to avoid patching). > It won't necessarily get you the same version as upstream uses, but it's > definitely DFSG compliant. We also might symlink our resulting whl files with the version number pdm upstream might expect in their tests. The question is, whether all this effort might break the tests in some way and we might end up with endless patching by at the same time loosing the chance to discuss with upstream. But it might be time to discuss the issue with upstream anyway. Kind regards Andreas. -- http://fam-tille.de
Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build
* Julian Gilbey [2023-08-18 13:42]: One could handle it by manually adding this directory to debian/clean in each package, but perhaps this should be the default behaviour of dh-python? See https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/46 Cheers Timo -- ⢀⣴⠾⠻⢶⣦⠀ ╭╮ ⣾⠁⢠⠒⠀⣿⡁ │ Timo Röhling │ ⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │ ⠈⠳⣄ ╰╯ signature.asc Description: PGP signature
Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build
On Fri, 18 Aug 2023, Andreas Tille wrote: Am Fri, Aug 18, 2023 at 01:42:53PM +0100 schrieb Julian Gilbey: I'm sure I'm not the only one who received a whole bunch of bugs entitled "Fails to build source after successful build" last weekend. There was one theme common to most of them: the presence of a *.egg-info directory which was not cleaned by debian/rules clean. I know the bug report said that this policy is currently under discussion, but I did get thinking about it. I imagine that this particular directory should be the responsibility of dh-python to clean up, but it may not be sensible to always delete *.egg-info directories, as they may be present in the orig.tar.gz file. One could handle it by manually adding this directory to debian/clean in each package, but perhaps this should be the default behaviour of dh-python? Any thoughts? I agree that having this a no-brainer and getting all this *.egg-info caused bugs solved by a simple dh-python update without changing packaging code would be extremely convenient. I could imagine creating a backup of the affected *.egg-info files if existent and copy these back in clean target could solve this. It is being worked on: https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/46 Regards, Scott
Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build
Hi, Am Fri, Aug 18, 2023 at 01:42:53PM +0100 schrieb Julian Gilbey: > I'm sure I'm not the only one who received a whole bunch of bugs > entitled "Fails to build source after successful build" last weekend. > There was one theme common to most of them: the presence of a > *.egg-info directory which was not cleaned by debian/rules clean. > > I know the bug report said that this policy is currently under > discussion, but I did get thinking about it. I imagine that this > particular directory should be the responsibility of dh-python to > clean up, but it may not be sensible to always delete *.egg-info > directories, as they may be present in the orig.tar.gz file. One > could handle it by manually adding this directory to debian/clean in > each package, but perhaps this should be the default behaviour of > dh-python? > > Any thoughts? I agree that having this a no-brainer and getting all this *.egg-info caused bugs solved by a simple dh-python update without changing packaging code would be extremely convenient. I could imagine creating a backup of the affected *.egg-info files if existent and copy these back in clean target could solve this. Kind regards Andreas. -- http://fam-tille.de
Re: pdm: Please replace python3-pep517 with python3-pyproject-hooks in Depends/Build-Depends
On August 18, 2023 1:04:26 PM UTC, Andreas Tille wrote: >Hi Scott, > >Am Tue, Aug 15, 2023 at 02:18:35PM + schrieb Scott Kitterman: >> >They are zip files containing python source code. It is possible to include >> >compiled C extensions in wheels, but I checked and these wheels are all pure >> >python, so no binary blobs are included. >> >> In Debian terms, it's not the preferred form for modification, so it's not >> source. In this regard DFSG goes farther than some software licenses. > >I think the point Jeroen wanted to make is that these are actually >source file archives which "by chance" are featuring a .whl extension >rather than a .zip extension. A wheel is not the preferred form for modification. They're not wheels by chance at all. From a DFSG perspective, the most straightforward approach is to build-depend on the relevant Debian packages and build any needed wheels from that. It won't necessarily get you the same version as upstream uses, but it's definitely DFSG compliant. Scott K
Re: pdm: Please replace python3-pep517 with python3-pyproject-hooks in Depends/Build-Depends
Hi Scott, Am Tue, Aug 15, 2023 at 02:18:35PM + schrieb Scott Kitterman: > >They are zip files containing python source code. It is possible to include > >compiled C extensions in wheels, but I checked and these wheels are all pure > >python, so no binary blobs are included. > > In Debian terms, it's not the preferred form for modification, so it's not > source. In this regard DFSG goes farther than some software licenses. I think the point Jeroen wanted to make is that these are actually source file archives which "by chance" are featuring a .whl extension rather than a .zip extension. Kind regards Andreas. -- http://fam-tille.de
Uncleaned egg-info directory giving lots of bugs about failing to build after successful build
I'm sure I'm not the only one who received a whole bunch of bugs entitled "Fails to build source after successful build" last weekend. There was one theme common to most of them: the presence of a *.egg-info directory which was not cleaned by debian/rules clean. I know the bug report said that this policy is currently under discussion, but I did get thinking about it. I imagine that this particular directory should be the responsibility of dh-python to clean up, but it may not be sensible to always delete *.egg-info directories, as they may be present in the orig.tar.gz file. One could handle it by manually adding this directory to debian/clean in each package, but perhaps this should be the default behaviour of dh-python? Any thoughts? Best wishes, Julian
Re: Package testing with Salsa CI for new packages
On Friday, 18 August 2023 09:51:29 CEST Carles Pina i Estany wrote: > > I'm not a Debian developer but I have some experience on Salsa CI, so I > thought that I might be able to help... but then I was confused by a > specific part of the message: > > On 17 Aug 2023 at 17:10:08, Paul Boddie wrote: > > [...] > > > For another package I have been working on, the Salsa CI facility has > > proven to be usable, configured using files in debian/test, particularly > > as it allows test-related dependencies to be specified independently. > > However, this other package has no dependencies that are currently > > unpackaged in Debian. Meanwhile, the testing of this new Moin package > > depends on brand new packages somehow being made available. > > If this dependencies are available on the "build" step: could they be > made available on the autopkgtest? I didn't quite understand why this is > not possible. I've found the autopkgtest quite flexible (since the tests > are scripts that could prepare some environment) The package has dependencies on installation but these dependencies are not strictly necessary when building. However, if I wanted to run the test suite when building, I would indeed need to pull in these dependencies as build dependencies so that the software being tested can run without import errors. Since some of these dependencies are new packages, I had been specifying them using the --extra-package option of gbp buildpackage in order to satisfy such build dependencies, but these were really "test dependencies". I have since decided that I would rather distinguish between build and test dependencies, as previously noted. I have to add that the other package I refer to has a test suite that takes a long time to run, so that is another reason why I chose Salsa CI for that package instead of letting autopkgtest do its work: https://salsa.debian.org/pboddie/shedskin When I get the courage to do so, I will probably submit an intention-to- package for that package. [...] > For me, the packages on the build job are made available via a Salsa > artifact automatically (and easy to download as a .zip of the *.deb). I > think that this happens on all the "build" jobs on Salsa CI. They can be > downloaded as a .zip file > (e.g. https://salsa.debian.org/freexian-team/debusine/-/jobs/4564890, > right hand side "Job artifacts" and click "Download"). Would that be > enough? I think the problem is that when running the test suite, the Moin package would need to obtain the test dependencies specified in debian/test/control. These include various new packages that cannot be obtained from the usual Debian archives, and so some kind of equivalent to gbp's --extra-package would be needed in the Salsa CI environment. One can imagine having a common storage area holding these newly introduced packages that the CI scripts could access in preference to the usual archives. In fact, this would be something that might also affect existing packages. Consider the situation where fixes to a dependency are required to fix functionality in a particular package. One would have to wait for the fixed dependency to become integrated into the unstable archive before the principal package's tests would start to work again. > I also created a repo and hosted it on a Salsa CI page for internal > testing but this is a bit of a workaround. This is in a new job but just > download the artifacts (via a Salsa CI dependency) and run > dpkg-scanpackages and copy the files to the right place). This sounds like something related to what might be required. In effect, you seem to be doing what I am doing when I actually install my built packages in a chroot. I run apt-ftparchive (which runs dpkg-scanpackages) to generate Packages, Sources and Release files that tells apt about the new packages when added as another package source. In the Salsa CI environment, I would need to have the built packages (found in the artefacts for each package's build job) copied somewhere that can then be found by the Moin package's pipeline jobs and the scripts creating a special repository of new packages. Thanks for replying to my enquiry! Regards, Paul
Re: Package testing with Salsa CI for new packages
Hi, I'm not a Debian developer but I have some experience on Salsa CI, so I thought that I might be able to help... but then I was confused by a specific part of the message: On 17 Aug 2023 at 17:10:08, Paul Boddie wrote: [...] > For another package I have been working on, the Salsa CI facility has proven > to be usable, configured using files in debian/test, particularly as it > allows > test-related dependencies to be specified independently. However, this other > package has no dependencies that are currently unpackaged in Debian. > Meanwhile, the testing of this new Moin package depends on brand new packages > somehow being made available. If this dependencies are available on the "build" step: could they be made available on the autopkgtest? I didn't quite understand why this is not possible. I've found the autopkgtest quite flexible (since the tests are scripts that could prepare some environment) > So, I would appreciate guidance on how I might enable testing of this new > Moin > package, along with its new dependencies, in the Salsa environment or any > other appropriate environment that would satisfy Debian packaging needs and > policies. It would also be quite helpful if built packages might be published > somewhere for people to test, but this is a separate consideration even if > such packages would obviously need to be generated as part of the testing > regime. For me, the packages on the build job are made available via a Salsa artifact automatically (and easy to download as a .zip of the *.deb). I think that this happens on all the "build" jobs on Salsa CI. They can be downloaded as a .zip file (e.g. https://salsa.debian.org/freexian-team/debusine/-/jobs/4564890, right hand side "Job artifacts" and click "Download"). Would that be enough? I also created a repo and hosted it on a Salsa CI page for internal testing but this is a bit of a workaround. This is in a new job but just download the artifacts (via a Salsa CI dependency) and run dpkg-scanpackages and copy the files to the right place). Cheers, -- Carles Pina i Estany https://carles.pina.cat || Wiktionary translations: https://kamus.pina.cat
