Re: How to watch pypi.org
On 2020-11-01 20:23:20 +0100 (+0100), Thomas Goirand wrote: [...] > However, if I am to put more efforts on stuff like that, my priority > would be first on getting the reno release notes published in the Debian > package. I've been thinking about this for a long time, and haven't > figured out yet what would be the best way, with a reasonable workflow. > > From the Debian perspective, I'd have to: > - generate the release notes from the last version in Debian Stable, up > to what's in Sid. For example, I would include all the OpenStack release > notes for Stein, Train, Ussuri and Victoria in all packages uploaded for > Debian Bullseye, as this would concern anyone upgrading from Buster. > - make it so that release notes would be generated from Git, and maybe > stored in a debian/release-notes folder, so that it wouldn't generate a > diff with the original upstream tag. > > The drawback would be that, on each upload of a new Debian revision, the > debian.tar.xz will contain the release notes, which could be of > significant size (especially if they include static objects like CSS, > JS, and all what makes a theme). > > If you have any other suggestion concerning how to handle these release > notes, please let me know. It's likely I'm missing some subtle reason for the complexities you outline above, but if you install python3-reno and then run `reno report` in the upstream Git repository for any project with a releasenotes tree (or pass the path to said Git repository in the command line) it will generate a reStructuredText compilation of the release notes contained therein. Very lightweight, no need for extra files or anything. I'd think you could just dump that output into a NEWS file or similar at binary package build time. This is basically the same thing reno's Sphinx extension does under the covers. Check out `reno report --help` for a number of flags you might want to pass it to make the results more readable like omitting the source filename comments, skipping notes earlier than a certain version, collapsing pre-release notes, and so on. A quick test of Nova's release notes indicates that even if you don't truncate them though and include everything back to when the project started using reno 5 years ago, that NEWS file would only increase the compressed size of the nova-doc package by 1%. -- Jeremy Stanley signature.asc Description: PGP signature
Re: How to watch pypi.org
On 10/31/20 1:10 PM, Jeremy Stanley wrote: > On 2020-10-31 12:03:50 +0100 (+0100), Thomas Goirand wrote: > [...] >> On 10/31/20 3:07 AM, Jeremy Stanley wrote: >>> I have to agree, though in the upstream projects with which I'm >>> involved, those generated files are basically a lossy re-encoding of >>> metadata from the Git repositories themselves: AUTHORS file >>> generated from committer headers, ChangeLog files from commit >>> subjects, version information from tag names, and so on. Some of >>> this information may be referenced from copyright licenses, so it's >>> important in those cases for package maintainers to generate it when >>> making their source packages if not using the sdist tarballs >>> published by the project. >> >> Unfortunately, the FTP masters do not agree with you. I've been told >> that the OpenStack changelog is a way too big, and it's preferable to >> not have it in the binary packages. > > PBR started creating much smaller changelogs years ago, after you > asked ftpmaster. I get that you see no value in changelog files, but > it seems like it would be worth revisiting. Probably it'd be worth revisiting indeed, especially by finding a way to push the logs only in the relevant -doc packages. Maybe by using the same kind of trick which I want to do with the release notes (see below), ie storing these files in the debian folder. However, if I am to put more efforts on stuff like that, my priority would be first on getting the reno release notes published in the Debian package. I've been thinking about this for a long time, and haven't figured out yet what would be the best way, with a reasonable workflow. >From the Debian perspective, I'd have to: - generate the release notes from the last version in Debian Stable, up to what's in Sid. For example, I would include all the OpenStack release notes for Stein, Train, Ussuri and Victoria in all packages uploaded for Debian Bullseye, as this would concern anyone upgrading from Buster. - make it so that release notes would be generated from Git, and maybe stored in a debian/release-notes folder, so that it wouldn't generate a diff with the original upstream tag. The drawback would be that, on each upload of a new Debian revision, the debian.tar.xz will contain the release notes, which could be of significant size (especially if they include static objects like CSS, JS, and all what makes a theme). If you have any other suggestion concerning how to handle these release notes, please let me know. Cheers, Thomas Goirand (zigo)
Re: How to watch pypi.org
On Sat, Oct 31, 2020 at 12:03:50PM +0100, Thomas Goirand wrote: > Pypi is often thought as a Python module source repository. It is *NOT*. > It is a repository for binaries to be consumed by pip. Oooh, that's a very interesting thought I never considered. -- WBR, wRAR signature.asc Description: PGP signature
Re: How to watch pypi.org
On 2020-10-31 12:03:50 +0100 (+0100), Thomas Goirand wrote: [...] > On 10/31/20 3:07 AM, Jeremy Stanley wrote: > > I have to agree, though in the upstream projects with which I'm > > involved, those generated files are basically a lossy re-encoding of > > metadata from the Git repositories themselves: AUTHORS file > > generated from committer headers, ChangeLog files from commit > > subjects, version information from tag names, and so on. Some of > > this information may be referenced from copyright licenses, so it's > > important in those cases for package maintainers to generate it when > > making their source packages if not using the sdist tarballs > > published by the project. > > Unfortunately, the FTP masters do not agree with you. I've been told > that the OpenStack changelog is a way too big, and it's preferable to > not have it in the binary packages. PBR started creating much smaller changelogs years ago, after you asked ftpmaster. I get that you see no value in changelog files, but it seems like it would be worth revisiting. > Also, there's nothing in the Apache license that mandates having > an AUTHORS list as per what PBR builds. If we are to care that > much in OpenStack, then the license must be changed. [...] I agree, it's not commonplace in OpenStack other than this possible exception: https://opendev.org/openstack/python-openstackclient/src/branch/master/doc/source/cli/man/openstack.rst#user-content-copyright You do tend to find it in other Python projects however, for example: https://github.com/pygments/pygments/blob/master/LICENSE#L1 My point was that, in general, some Python projects do autogenerate an AUTHORS file from commit metadata at dist time rather than storing it directly in a file within their Git repositories, and some projects (including Python projects) refer to AUTHORS from copyright statements, so it's a good idea to build/keep it. -- Jeremy Stanley signature.asc Description: PGP signature
Re: How to watch pypi.org
On 10/30/20 3:18 PM, Fioddor Superconcentrado wrote: > As I said I'm very new to this and all (python) packages I'm using > lately use the usual python tools (pipy, setup.py, etc) and my first > approach has been to stick as close as possible to the upstream > procedures. But I might very likely be taking a wrong decision. What are > the reasons to go for git instead of pypi? I see that it is 'more > upstream' but it seems that everyone else is pointing to pypi as a > distro-agnostic solution. Pypi is often thought as a Python module source repository. It is *NOT*. It is a repository for binaries to be consumed by pip. Therefore, the tarballs may omit some artifacts, and add some more. For example, it's preferable that the sources do not include the egg-info folder, or pre-generated documentation. These must be built from source. On 10/31/20 3:07 AM, Jeremy Stanley wrote: > I have to agree, though in the upstream projects with which I'm > involved, those generated files are basically a lossy re-encoding of > metadata from the Git repositories themselves: AUTHORS file > generated from committer headers, ChangeLog files from commit > subjects, version information from tag names, and so on. Some of > this information may be referenced from copyright licenses, so it's > important in those cases for package maintainers to generate it when > making their source packages if not using the sdist tarballs > published by the project. Unfortunately, the FTP masters do not agree with you. I've been told that the OpenStack changelog is a way too big, and it's preferable to not have it in the binary packages. Also, there's nothing in the Apache license that mandates having an AUTHORS list as per what PBR builds. If we are to care that much in OpenStack, then the license must be changed. In Debian, quite the opposite, and like it or not (I personally don't really feel the policy is right, but that's how it is), what Debian cares is what's in the source code marked with "copyright (c) yeah, copyright-holder", and this must already be in debian/copyright. Cheers, Thomas Goirand (zigo)
Re: How to watch pypi.org
On Sat, Oct 31, 2020 at 2:31 AM Jeremy Stanley wrote: > I have to agree, though in the upstream projects with which I'm > involved, those generated files are basically a lossy re-encoding of > metadata from the Git repositories themselves: AUTHORS file > generated from committer headers, ChangeLog files from commit > subjects, version information from tag names, and so on. Some of > this information may be referenced from copyright licenses, so it's > important in those cases for package maintainers to generate it when > making their source packages if not using the sdist tarballs > published by the project. As the maintainer of the autorevision package (which aims at dumping a cache of VCS version metadata, for when exporting a tarball from a VCS), I've been thinking about this for a while now. I've been thinking about modifying automake (and other build tools) to have a mode that basically does `git archive` instead of just calling tar and also creates separate tarballs for all the other generated or embedded files. One for the autorevision metadata, one for the autotools cruft, one for a cache of the data needed for AUTHORS/ChangeLog/NEWS etc and possibly other ones. Then Debian can use our multi-component quilt 3.0 format to take the git archive, the autorevision metadata, and the cache of the VCS data, but leave the autotools cruft behind. Then we can audit the git repo for generated files and embedded data/code copies and when there are none, we can confidently build the configure script from source knowing that everything required is available in the build-deps. The same could apply to projects uploading to NPM or PyPi, although I'm not sure if those support this sort of thing though. -- bye, pabs https://wiki.debian.org/PaulWise
Re: How to watch pypi.org
On 2020-10-31 01:33:36 + (+), Paul Wise wrote: > On Fri, Oct 30, 2020 at 2:19 PM Fioddor Superconcentrado wrote: > > As I said I'm very new to this and all (python) packages I'm > > using lately use the usual python tools (pipy, setup.py, etc) > > and my first approach has been to stick as close as possible to > > the upstream procedures. But I might very likely be taking a > > wrong decision. What are the reasons to go for git instead of > > pypi? I see that it is 'more upstream' but it seems that > > everyone else is pointing to pypi as a distro-agnostic solution. > > As Andrey says, missing files is one issue, another is that tarballs > often contain extra generated files that should be built from source, > but if you use the tarball then they quite likely will not be built > from source. I have to agree, though in the upstream projects with which I'm involved, those generated files are basically a lossy re-encoding of metadata from the Git repositories themselves: AUTHORS file generated from committer headers, ChangeLog files from commit subjects, version information from tag names, and so on. Some of this information may be referenced from copyright licenses, so it's important in those cases for package maintainers to generate it when making their source packages if not using the sdist tarballs published by the project. -- Jeremy Stanley signature.asc Description: PGP signature
Re: How to watch pypi.org
On Fri, Oct 30, 2020 at 2:19 PM Fioddor Superconcentrado wrote: > As I said I'm very new to this and all (python) packages I'm using lately use > the usual python tools (pipy, setup.py, etc) and my first approach has been > to stick as close as possible to the upstream procedures. But I might very > likely be taking a wrong decision. What are the reasons to go for git instead > of pypi? I see that it is 'more upstream' but it seems that everyone else is > pointing to pypi as a distro-agnostic solution. As Andrey says, missing files is one issue, another is that tarballs often contain extra generated files that should be built from source, but if you use the tarball then they quite likely will not be built from source. -- bye, pabs https://wiki.debian.org/PaulWise
Re: How to watch pypi.org
On Fri, Oct 30, 2020 at 03:18:58PM +0100, Fioddor Superconcentrado wrote: > As I said I'm very new to this and all (python) packages I'm using lately > use the usual python tools (pipy, setup.py, etc) and my first approach has > been to stick as close as possible to the upstream procedures. But I might > very likely be taking a wrong decision. What are the reasons to go for git > instead of pypi? I see that it is 'more upstream' but it seems that > everyone else is pointing to pypi as a distro-agnostic solution. One reason is tarballs may be missing some files, including tests. > El lun., 5 oct. 2020 a las 4:24, Paul Wise () escribió: > > > On Sun, Oct 4, 2020 at 3:29 PM Fioddor Superconcentrado wrote: > > > > > I've packaged a project provided via https://pipi.org and I wanted to > > create a debian/watch file but pipi.org publishes the tarball behind a > > strange url like > > > > I would suggest using the upstream git repo instead of the PyPi tarballs. > > -- WBR, wRAR signature.asc Description: PGP signature
Re: How to watch pypi.org
As I said I'm very new to this and all (python) packages I'm using lately use the usual python tools (pipy, setup.py, etc) and my first approach has been to stick as close as possible to the upstream procedures. But I might very likely be taking a wrong decision. What are the reasons to go for git instead of pypi? I see that it is 'more upstream' but it seems that everyone else is pointing to pypi as a distro-agnostic solution. El lun., 5 oct. 2020 a las 4:24, Paul Wise () escribió: > On Sun, Oct 4, 2020 at 3:29 PM Fioddor Superconcentrado wrote: > > > I've packaged a project provided via https://pipi.org and I wanted to > create a debian/watch file but pipi.org publishes the tarball behind a > strange url like > > I would suggest using the upstream git repo instead of the PyPi tarballs. > > -- > bye, > pabs > > https://wiki.debian.org/PaulWise > >
Re: How to watch pypi.org
Thank you as well, Geert. And yes. It has taken me some time to come back to this issue but I've tested and seems to work fine. El dom., 4 oct. 2020 a las 17:44, Geert Stappers () escribió: > On Sun, Oct 04, 2020 at 05:28:57PM +0200, Fioddor Superconcentrado wrote: > > I've packaged a project provided via https://pipi.org and I wanted to > > create a debian/watch file but pipi.org publishes the tarball behind a > > strange url like > > > > > https://files.pythonhosted.org/packages/d6/72/9848a2d631dad70d7ea582540f0619e1a7ecf31b3a117de9d9f2b6b28029/django-settings-export-1.2.1.tar.gz > > > > I guess pypi.org is a common source of python code so perhaps there's > > already a solution for this. > > Quoting the manual page of uscan. > ( uscan is the utility reads debian/watch for watching. ) > > >PyPI >For PyPI based projects, pypi.debian.net runs a redirector >which allows a simpler form of URL. The format below will >automatically be rewritten to use the redirector with the watch >file: > > version=4 > https://pypi.python.org/packages/source/// \ > -(.+)\.tar\.gz debian uupdate > >For cfn-sphere, set the watch file as: > > version=4 > https://pypi.python.org/packages/source/c/cfn-sphere/ \ > cfn-sphere-([\d\.]+).tar.gz debian uupdate > >Please note, you can still use normal functionalities of uscan >to set up a watch file for this site without using the >redirector. > > version=4 > opts="pgpmode=none" \ > https://pypi.python.org/pypi/cfn-sphere/ \ > https://pypi.python.org/packages/.*/.*/.*/\ > cfn-sphere-([\d\.]+).tar.gz#.* debian uupdate > > > > > Thanks in advanced. > > Looking forward to something like "OK, that works" > > > Regards > Geert Stappers > -- > Silence is hard to parse > >
Re: How to watch pypi.org
Thank you very much, Andrey. El dom., 4 oct. 2020 a las 17:42, Andrey Rahmatullin () escribió: > On Sun, Oct 04, 2020 at 05:28:57PM +0200, Fioddor Superconcentrado wrote: > > I've packaged a project provided via https://pipi.org and I wanted to > > create a debian/watch file but pipi.org publishes the tarball behind a > > strange url like > > > > > https://files.pythonhosted.org/packages/d6/72/9848a2d631dad70d7ea582540f0619e1a7ecf31b3a117de9d9f2b6b28029/django-settings-export-1.2.1.tar.gz > > > > I guess pypi.org is a common source of python code so perhaps there's > > already a solution for this. > The solution (that transparently uses pypi.debian.net) is described in > uscan(1). > > -- > WBR, wRAR >
Re: How to watch pypi.org, example
On Sun, Oct 04, 2020 at 10:53:47PM -0400, Scott Kitterman wrote: > On Sunday, October 4, 2020 10:24:22 PM EDT Paul Wise wrote: > > On Sun, Oct 4, 2020 at 3:29 PM Fioddor Superconcentrado wrote: > > > I've packaged a project provided via https://pipi.org and I wanted to > > > create a debian/watch file but pipi.org publishes the tarball behind a > > > strange url like > > I would suggest using the upstream git repo instead of the PyPi tarballs. > > I think that's a different argument. > > There is a pypi redirector for Debian watch files. Something like this works > (this is from the pyspf package): > > https://pypi.debian.net/pyspf/pyspf-([0-9][0-9t\.\-]*).tar.gz > > This currently works, but no guarantee for how long: > > https://pypi.python.org/packages/source/x/@PACKAGE@/@PACKAGE@@ANY_VERSION@@ARCHIVE_EXT@ https://pypi.python.org/packages/source/x/@PACKAGE@/@PACKAGE@-@ANY_VERSION@@ARCHIVE_EXT@ And learning is also learning from examples. stappers@paddy:~ $ cd packaging_dir/ stappers@paddy:packaging_dir $ debcheckout pyspf declared git repository at https://salsa.debian.org/python-team/modules/pyspf.git git clone https://salsa.debian.org/python-team/modules/pyspf.git pyspf ... Cloning into 'pyspf'... remote: Enumerating objects: 1039, done. remote: Counting objects: 100% (1039/1039), done. remote: Compressing objects: 100% (368/368), done. remote: Total 1039 (delta 612), reused 943 (delta 551), pack-reused 0 Receiving objects: 100% (1039/1039), 237.12 KiB | 415.00 KiB/s, done. Resolving deltas: 100% (612/612), done. stappers@paddy:packaging_dir $ cd pyspf stappers@paddy:packaging_dir/pyspf $ cat debian/watch version=3 https://pypi.debian.net/pyspf/pyspf-([0-9][0-9t\.\-]*).tar.gz stappers@paddy:packaging_dir/pyspf $ Regards Geert Stappers -- Silence is hard to parse
Re: How to watch pypi.org
On Sunday, October 4, 2020 10:24:22 PM EDT Paul Wise wrote: > On Sun, Oct 4, 2020 at 3:29 PM Fioddor Superconcentrado wrote: > > I've packaged a project provided via https://pipi.org and I wanted to > > create a debian/watch file but pipi.org publishes the tarball behind a > > strange url like > I would suggest using the upstream git repo instead of the PyPi tarballs. I think that's a different argument. There is a pypi redirector for Debian watch files. Something like this works (this is from the pyspf package): https://pypi.debian.net/pyspf/pyspf-([0-9][0-9t\.\-]*).tar.gz This currently works, but no guarantee for how long: https://pypi.python.org/packages/source/x/@PACKAGE@/ @PACKAGE@@ANY_VERSION@@ARCHIVE_EXT@ Scott K
Re: How to watch pypi.org
On Sun, Oct 4, 2020 at 3:29 PM Fioddor Superconcentrado wrote: > I've packaged a project provided via https://pipi.org and I wanted to create > a debian/watch file but pipi.org publishes the tarball behind a strange url > like I would suggest using the upstream git repo instead of the PyPi tarballs. -- bye, pabs https://wiki.debian.org/PaulWise
Re: How to watch pypi.org
On Sun, Oct 04, 2020 at 05:28:57PM +0200, Fioddor Superconcentrado wrote: > I've packaged a project provided via https://pipi.org and I wanted to > create a debian/watch file but pipi.org publishes the tarball behind a > strange url like > > https://files.pythonhosted.org/packages/d6/72/9848a2d631dad70d7ea582540f0619e1a7ecf31b3a117de9d9f2b6b28029/django-settings-export-1.2.1.tar.gz > > I guess pypi.org is a common source of python code so perhaps there's > already a solution for this. Quoting the manual page of uscan. ( uscan is the utility reads debian/watch for watching. ) PyPI For PyPI based projects, pypi.debian.net runs a redirector which allows a simpler form of URL. The format below will automatically be rewritten to use the redirector with the watch file: version=4 https://pypi.python.org/packages/source/// \ -(.+)\.tar\.gz debian uupdate For cfn-sphere, set the watch file as: version=4 https://pypi.python.org/packages/source/c/cfn-sphere/ \ cfn-sphere-([\d\.]+).tar.gz debian uupdate Please note, you can still use normal functionalities of uscan to set up a watch file for this site without using the redirector. version=4 opts="pgpmode=none" \ https://pypi.python.org/pypi/cfn-sphere/ \ https://pypi.python.org/packages/.*/.*/.*/\ cfn-sphere-([\d\.]+).tar.gz#.* debian uupdate > Thanks in advanced. Looking forward to something like "OK, that works" Regards Geert Stappers -- Silence is hard to parse
Re: How to watch pypi.org
On Sun, Oct 04, 2020 at 05:28:57PM +0200, Fioddor Superconcentrado wrote: > I've packaged a project provided via https://pipi.org and I wanted to > create a debian/watch file but pipi.org publishes the tarball behind a > strange url like > > https://files.pythonhosted.org/packages/d6/72/9848a2d631dad70d7ea582540f0619e1a7ecf31b3a117de9d9f2b6b28029/django-settings-export-1.2.1.tar.gz > > I guess pypi.org is a common source of python code so perhaps there's > already a solution for this. The solution (that transparently uses pypi.debian.net) is described in uscan(1). -- WBR, wRAR signature.asc Description: PGP signature
