Re: a few quick questions on gbp pq workflow

[Thomas Goirand]

On 08/07/2017 12:20 AM, Jeremy Stanley wrote:
> Thomas references the AUTHORS and ChangeLog files (which embed
> important metadata from the revision control system into the release
> tarball, far from useless in my opinion); but taking the
> nova-15.0.0.tar.gz release for example, those two files account for
> a total of 5% of the unpacked tree according to du and probably
> compress fairly well. By comparison, the unit tests and fixtures
> make up 42% of the size of the tree on their own. He's also
> referring to a situation from _years_ ago, which was subsequently
> changed after he asked... those current tarballs only include author
> names and very abbreviated information parsed out of the git log and
> have been that way since 2013 (pbr commit 94a6bb9 released in 0.6),
> but mentioning that would likely have undermined his argument.

Jeremy, you still don't get it, sorry, it probably is my fault.


Using upstream sdist tarballs, we get a ChangeLog file. Due to all the
automation inside dh_* helpers, if nobody takes care of it, then
ChangeLog files automatically gets pushed into each and every individual
.deb files. This mean that even a tiny metapackage will get it, even if
it doesn't carry anything else but dependency information.

Multiply this by so many packages that the OpenStack package maintainer
have to deal with, adding such manual removal of the ChangeLog file in
each package is too much of a pain, when it's not needed at all if we're
using the upstream git as a source.

That, plus generated docs, and so many things that the sdist is
attempting to deal with, which we don't really want. It may even forget
to package some files for example (yes, I saw this a few time...).

PyPi's format is *not* designed as a mean to ship source code, but as a
mean to ship *binary* (ie: food for pip). The fact that you're shipping
built docs shows exactly that.

BTW, pristine-tar is a broken concept. Anyone that pretends otherwise
really has no clue about how tarballs work, their internal timestamps
that needs to be removed, and the fact that one has to order files in a
certain way when adding them to the archive, otherwise everything is
completely broken. Did I mention it also depends on the implementation
of tar itself, that BSD people have a different one, and that Debian has
to carry patches to fix upstream issues that generate different tarballs
depending on the tar utility version? Not to mention also that the
original author of pristine-tar (Joey Hess) agrees with me...

So why should we even attempt to bother? That's additional pain that
you're asking for, when there's not even enough man power, and we're
probably even on the way to get OpenStack removed form Debian if the
situation doesn't change.

Re: a few quick questions on gbp pq workflow

[Thomas Goirand]

On 08/06/2017 09:15 PM, Jeremy Stanley wrote:
> On 2017-08-06 20:00:59 +0100 (+0100), Ghislain Vaillant wrote:
> [...]
>> You'd still have to clean the pre-built files, since they would be
>> overwritten by the build system and therefore dpkg-buildpackage
>> would complain if you run the build twice.
>>
>> So, you might as well just exclude them from the source straight
>> away, no?
> 
> Repacking an upstream tarball just to avoid needing to tell
> dh_install not to copy files from a particular path into the binary
> package seems the wrong way around to me

What's wrong is for upstream to pretend that one tarball / archive is
its released source, when in fact it contains binary / generated files.

A source tarball / archive from upstream must contain *only* source
code, nothing else. If it contains anything that comes from the original
source, then it's additional pain for the package maintainer.

> but maybe I'm missing
> something which makes that particularly complicated? This comes up
> on debian-mentors all the time, and the general advice is to avoid
> repacking tarballs unless there's a policy violation or you can get
> substantial (like in the >50% range) reduction in size on especially
> huge upstream tarballs.

That's one view, probably motivated by the fact it's probably easier to
deal with in the long run. However convenient it may be, I don't think
it feels "clean".

And by the way, when it comes to the OpenStack stuff, FTP masters have
already expressed their dislike of the upstream ChangeLog: it is a *WAY*
to big, at the level of megabytes sometimes, and it may appear in .deb
files that would otherwise be a few kilobytes. All this isn't new...

Cheers,

Thomas Goirand (zigo)

Re: a few quick questions on gbp pq workflow

[Thomas Goirand]

On 08/06/2017 05:37 PM, Jeremy Stanley wrote:
> On 2017-08-06 10:44:36 -0400 (-0400), Allison Randal wrote:
>> The OpenStack packaging team has been sprinting at DebCamp, and
>> we're finally ready to move all general Python dependencies for
>> OpenStack over to DPMT. (We'll keep maintaining them, just within
>> DPMT using the DPMT workflow.)
>>
>> After chatting with tumbleweed, the current suggestion is that we
>> should migrate the packages straight into gbp pq instead of making
>> an intermediate stop with git-dpm.
> [...]
> 
> More a personal curiosity on my part (I'm now a little disappointed
> that I didn't make time to attend), but are you planning to leverage
> pristine tarballs as part of this workflow shift so you can take
> advantage of the version details set in the sdist metadata and the
> detached OpenPGP signatures provided upstream? Or are you sticking
> with operating on a local fork of upstream Git repositories (and
> generating intermediate sdists on the fly or supplying version data
> directly from the environment via debian/rules)?
> 
> I'm eager to see what upstream release management features you're
> taking advantage of so we can better know which of those efforts are
> valuable to distro package maintainers

Jeremy,

If you think what Alison described includes artifacts from upstream
OpenStack, you're mistaking. She was talking about moving *general
purpose* Python libraries only. The rest of will continue to use the
workflow of generating orig files using "git archive".

At least, that's what I understand from our sprint meetings, and that is
if anyone (but me) in the team dares to start doing a little bit of
packaging, which I haven't seen happening so far...

Cheers,

Thomas Goirand (zigo)

Re: a few quick questions on gbp pq workflow

[Scott Kitterman]

On August 7, 2017 5:37:34 PM EDT, Diane Trout  wrote:
>
>> 
>> Why would you need to repack a tarball just because it contains
>> prebuilt docs (non-DFSG-free licensed documentation aside)? I'm all
>
>I've occasionally repacked a tarball because upstream included minified
>jquery or mathjax.
>
>Diane

Those are sourceless binary blobs, so repacking for that is exactly correct.

Scott K

Re: a few quick questions on gbp pq workflow

[Diane Trout]

> 
> Why would you need to repack a tarball just because it contains
> prebuilt docs (non-DFSG-free licensed documentation aside)? I'm all

I've occasionally repacked a tarball because upstream included minified
jquery or mathjax.

Diane

signature.asc
Description: This is a digitally signed message part

Re: a few quick questions on gbp pq workflow

[Barry Warsaw]

On Aug 6, 2017, at 11:11, Ondrej Novy  wrote:

> It's not always possible/simple/nice to use sdist, because it contains 
> prebuild docs. And I don't like to do +dfsg rebuild just for removing docs. 
> Sometimes sdists doesn't contain tests.

I will also generally prefer the PyPI tarballs for the packages I help 
maintain.  There’s good tooling for that and the tarballs are usually fine.  If 
not, and it’s fairly easy to DFSG them (e.g. via d/copyrights or some such), 
then I’ll go that route.  So far I haven’t had to pull things in from upstream 
git although once in a while I’ll just grab a file if needed.

-Barry



signature.asc
Description: Message signed with OpenPGP

Re: a few quick questions on gbp pq workflow

[Paul Wise]

On Sun, Aug 6, 2017 at 2:53 PM, Jeremy Stanley wrote:

> Why would you need to repack a tarball just because it contains
> prebuilt docs (non-DFSG-free licensed documentation aside)?

I'd suggest removing prebuilt files should happen in both the upstream
VCS and tarballs, failing that then at `debian/rules clean` and
`debian/rules build` time should be enough for Debian's purposes.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: a few quick questions on gbp pq workflow

[Scott Kitterman]

On August 6, 2017 5:24:59 PM EDT, Ondrej Novy  wrote:
>Hi,
>
>2017-08-06 14:53 GMT-04:00 Jeremy Stanley :
>
>> Why would you need to repack a tarball just because it contains
>> prebuilt docs (non-DFSG-free licensed documentation aside)? I'm all
>>
>
>Lintian considers this as "bug": https://lintian.debian.
>org/tags/source-contains-prebuilt-sphinx-documentation.html
>(pedantic, I know).

Not all things linitian tags are bugs.  This is only a bug if the documentation 
can't be rebuilt from Debian Main.

Linitian is a tool to help maintainers.  Don't just blindly "fix" things on 
linitian's say so.  At most the is a hint to make sure that the documentation 
can be rebuilt.

Scott K

Re: a few quick questions on gbp pq workflow

[Brian May]

Allison Randal  writes:

> - Will DPMT be following DEP-14 and using the branch name debian/master
> instead of master?

I believe so. Although from memory the conversion instructions are
incomplete on how to change the default branch to debian/master, which
is required before you can delete the old master branch.

Having a master branch that is no longer useful may have confused me
with some packages that other people have converted, but haven't had
time to check.

> - The gbp pq workflow page still points at the general Debian git-dpm
> instructions in the links for "import an existing .dsc file" and "start
> from scratch". Are there alternate versions of those pages for gbp pq
> somewhere?
>
> https://wiki.debian.org/PackagingWithGit/GitDpm/ImportDsc
> https://wiki.debian.org/PackagingWithGit/GitDpm/Initialize

Whoops. Possibly these links should be removed, all the information
required should be on this page. If not, feel free to add it...

> - Does DPMT prefer signed tags or unsigned tags, the wiki page is pretty
> agnostic on the topic.

Personally I use signed tags. gbp has a --git-sign-tags option.

> - Are there other lingering workflow questions for gbp pq not yet
> recorded on the wiki page?

The only major issue is the lack of proper procedure on how do the
master --> debian/master branch rename. As in I think the procedures
leaves behind a master branch, and keeps it as the default.

I have a script on git.debian.org - hopefully everyone should be able to
read this:

/srv/home/users/bam/convert

Which will handle the entire conversion. Must be run on git.debian.org,
and for best results git should have your username and email setup on
git.debian.org. Works for me for every case, except for the one package
where I hadn't pushed upstream changes correctly beforehand.

The script also includes the correct procedure for converting from
master to debian/master.
-- 
Brian May 

Re: a few quick questions on gbp pq workflow

[Ondrej Novy]

Hi,

2017-08-06 14:53 GMT-04:00 Jeremy Stanley :

> Why would you need to repack a tarball just because it contains
> prebuilt docs (non-DFSG-free licensed documentation aside)? I'm all
>

Lintian considers this as "bug": https://lintian.debian.
org/tags/source-contains-prebuilt-sphinx-documentation.html
(pedantic, I know).

Pretty sure OpenStack at least would consider any content which
> requires Debian package maintainers to alter tarballs prior to
> including them in the archive as fairly serious bug in its software.
>

But they want to keep generated (read: useless for our orig tarballs)
things inside release tarballs.

And sdist orig tarballs are too big, because they contains prebuild things,
FTP masters are not happy. (Thomas words).

Context: http://lists.alioth.debian.org/pipermail/openstack-devel/2017-July/
021447.html
... I've been already flamed by the FTP masters because small
packages like nova-compute-* became insanely huge (ie: megs instead of
kbytes). ...

I consider git repository to be better source of "source code" than sdist,
which is built from that same git. But it's just about POV :).

-- 
Best regards
 Ondřej Nový

Email: n...@ondrej.org
PGP: 3D98 3C52 EB85 980C 46A5  6090 3573 1255 9D1E 064B

Re: a few quick questions on gbp pq workflow

[Jeremy Stanley]

On 2017-08-06 20:00:59 +0100 (+0100), Ghislain Vaillant wrote:
[...]
> You'd still have to clean the pre-built files, since they would be
> overwritten by the build system and therefore dpkg-buildpackage
> would complain if you run the build twice.
> 
> So, you might as well just exclude them from the source straight
> away, no?

Repacking an upstream tarball just to avoid needing to tell
dh_install not to copy files from a particular path into the binary
package seems the wrong way around to me, but maybe I'm missing
something which makes that particularly complicated? This comes up
on debian-mentors all the time, and the general advice is to avoid
repacking tarballs unless there's a policy violation or you can get
substantial (like in the >50% range) reduction in size on especially
huge upstream tarballs. Otherwise the ability to compare the
upstream tarball from the source package to upstream release
announcements/checksums/signatures is a pretty large benefit you're
robbing from downstream recipients who might wish to take advantage
it.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature

Re: a few quick questions on gbp pq workflow

[Ghislain Vaillant]

On 06/08/17 19:56, Scott Kitterman wrote:

Generally when I find shortcomings in the tarball, I file bugs upstream.  In 
general, I've found upstream developers to be accepting of such changes.


Same here.


There's no need to DFSG the tarball if you can rebuild the docs.  The best way 
to ensure that is to rebuild them during the build process.


Also, the HTML docs are sometimes included by mistake because the docs 
folder was grafted and the Sphinx build directory not pruned in the 
corresponding MANIFEST.in.

Re: a few quick questions on gbp pq workflow

[Ghislain Vaillant]

On 06/08/17 19:53, Jeremy Stanley wrote:

Why would you need to repack a tarball just because it contains
prebuilt docs (non-DFSG-free licensed documentation aside)? I'm all
for rebuilding those at deb build time just to be sure you have the
right deps packaged too, but if the ones in the tarball are built
from DFSG-compliant upstream source, included in the archive for
that matter, then leaving the tarball pristine shouldn't be a policy
violation, right? That's like repacking a tarball for an
autotools-using project because upstream is shipping a configure
script built from an included configure.in file.


You'd still have to clean the pre-built files, since they would be 
overwritten by the build system and therefore dpkg-buildpackage would 
complain if you run the build twice.


So, you might as well just exclude them from the source straight away, no?

Ghis

Re: a few quick questions on gbp pq workflow

[Scott Kitterman]

On August 6, 2017 2:11:13 PM EDT, Ondrej Novy  wrote:
>Hi,
>
>2017-08-06 12:26 GMT-04:00 Scott Kitterman :
>>
>> I don't work on the OpenStack packages, but I do maintain a
>reasonable
>> number of Python packages.  I always work from the released tarball. 
>I
>> haven't added the keys yet to verify all my packages, but am
>gradually
>> doing so as I have time.
>>
>> I know some people like working from the upstream git (and the team
>> maintenance workflow allows for either), but I definitely prefer
>working
>> from the upstream tarballs and appreciate the effort that goes into
>> producing them.
>>
>
>It's not always possible/simple/nice to use sdist, because it contains
>prebuild docs. And I don't like to do +dfsg rebuild just for removing
>docs.
>Sometimes sdists doesn't contain tests.
>
>So my preference is:
>
>   - use sdist if it's possible (have tests, don't have prebuilds, ...)
>   - use git tag tarballs (https://github.com///tags)
>
>I already migrated few packages OS->DPMT so far.

Great.

Generally when I find shortcomings in the tarball, I file bugs upstream.  In 
general, I've found upstream developers to be accepting of such changes.

There's no need to DFSG the tarball if you can rebuild the docs.  The best way 
to ensure that is to rebuild them during the build process.

Scott K

Re: a few quick questions on gbp pq workflow

[Jeremy Stanley]

On 2017-08-06 14:11:13 -0400 (-0400), Ondrej Novy wrote:
> It's not always possible/simple/nice to use sdist, because it contains
> prebuild docs. And I don't like to do +dfsg rebuild just for removing docs.
> Sometimes sdists doesn't contain tests.
> 
> So my preference is:
> 
>- use sdist if it's possible (have tests, don't have prebuilds, ...)
>- use git tag tarballs (https://github.com///tags)
> 
> I already migrated few packages OS->DPMT so far.

Why would you need to repack a tarball just because it contains
prebuilt docs (non-DFSG-free licensed documentation aside)? I'm all
for rebuilding those at deb build time just to be sure you have the
right deps packaged too, but if the ones in the tarball are built
from DFSG-compliant upstream source, included in the archive for
that matter, then leaving the tarball pristine shouldn't be a policy
violation, right? That's like repacking a tarball for an
autotools-using project because upstream is shipping a configure
script built from an included configure.in file.

Pretty sure OpenStack at least would consider any content which
requires Debian package maintainers to alter tarballs prior to
including them in the archive as fairly serious bug in its software.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature

Re: a few quick questions on gbp pq workflow

[Ondrej Novy]

Hi,

2017-08-06 12:26 GMT-04:00 Scott Kitterman :
>
> I don't work on the OpenStack packages, but I do maintain a reasonable
> number of Python packages.  I always work from the released tarball.  I
> haven't added the keys yet to verify all my packages, but am gradually
> doing so as I have time.
>
> I know some people like working from the upstream git (and the team
> maintenance workflow allows for either), but I definitely prefer working
> from the upstream tarballs and appreciate the effort that goes into
> producing them.
>

It's not always possible/simple/nice to use sdist, because it contains
prebuild docs. And I don't like to do +dfsg rebuild just for removing docs.
Sometimes sdists doesn't contain tests.

So my preference is:

   - use sdist if it's possible (have tests, don't have prebuilds, ...)
   - use git tag tarballs (https://github.com///tags)

I already migrated few packages OS->DPMT so far.

-- 
Best regards
 Ondřej Nový

Email: n...@ondrej.org
PGP: 3D98 3C52 EB85 980C 46A5  6090 3573 1255 9D1E 064B

Re: a few quick questions on gbp pq workflow

[Scott Kitterman]

On August 6, 2017 11:37:51 AM EDT, Jeremy Stanley  wrote:
>On 2017-08-06 10:44:36 -0400 (-0400), Allison Randal wrote:
>> The OpenStack packaging team has been sprinting at DebCamp, and
>> we're finally ready to move all general Python dependencies for
>> OpenStack over to DPMT. (We'll keep maintaining them, just within
>> DPMT using the DPMT workflow.)
>> 
>> After chatting with tumbleweed, the current suggestion is that we
>> should migrate the packages straight into gbp pq instead of making
>> an intermediate stop with git-dpm.
>[...]
>
>More a personal curiosity on my part (I'm now a little disappointed
>that I didn't make time to attend), but are you planning to leverage
>pristine tarballs as part of this workflow shift so you can take
>advantage of the version details set in the sdist metadata and the
>detached OpenPGP signatures provided upstream? Or are you sticking
>with operating on a local fork of upstream Git repositories (and
>generating intermediate sdists on the fly or supplying version data
>directly from the environment via debian/rules)?
>
>I'm eager to see what upstream release management features you're
>taking advantage of so we can better know which of those efforts are
>valuable to distro package maintainers.

I don't work on the OpenStack packages, but I do maintain a reasonable number 
of Python packages.  I always work from the released tarball.  I haven't added 
the keys yet to verify all my packages, but am gradually doing so as I have 
time.

I know some people like working from the upstream git (and the team maintenance 
workflow allows for either), but I definitely prefer working from the upstream 
tarballs and appreciate the effort that goes into producing them.

Scott K

Re: a few quick questions on gbp pq workflow

[Jeremy Stanley]

On 2017-08-06 10:44:36 -0400 (-0400), Allison Randal wrote:
> The OpenStack packaging team has been sprinting at DebCamp, and
> we're finally ready to move all general Python dependencies for
> OpenStack over to DPMT. (We'll keep maintaining them, just within
> DPMT using the DPMT workflow.)
> 
> After chatting with tumbleweed, the current suggestion is that we
> should migrate the packages straight into gbp pq instead of making
> an intermediate stop with git-dpm.
[...]

More a personal curiosity on my part (I'm now a little disappointed
that I didn't make time to attend), but are you planning to leverage
pristine tarballs as part of this workflow shift so you can take
advantage of the version details set in the sdist metadata and the
detached OpenPGP signatures provided upstream? Or are you sticking
with operating on a local fork of upstream Git repositories (and
generating intermediate sdists on the fly or supplying version data
directly from the environment via debian/rules)?

I'm eager to see what upstream release management features you're
taking advantage of so we can better know which of those efforts are
valuable to distro package maintainers.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature

a few quick questions on gbp pq workflow

[Allison Randal]

The OpenStack packaging team has been sprinting at DebCamp, and we're
finally ready to move all general Python dependencies for OpenStack over
to DPMT. (We'll keep maintaining them, just within DPMT using the DPMT
workflow.)

After chatting with tumbleweed, the current suggestion is that we should
migrate the packages straight into gbp pq instead of making an
intermediate stop with git-dpm. We'll try to catch up with Piotr
sometime this week too.

The alternate wiki page for the gbp pq workflow seems to have been
mostly updated, but leaves a few questions:

https://wiki.debian.org/Python/GitPackagingPQ

- Will DPMT be following DEP-14 and using the branch name debian/master
instead of master?

- The gbp pq workflow page still points at the general Debian git-dpm
instructions in the links for "import an existing .dsc file" and "start
from scratch". Are there alternate versions of those pages for gbp pq
somewhere?

https://wiki.debian.org/PackagingWithGit/GitDpm/ImportDsc
https://wiki.debian.org/PackagingWithGit/GitDpm/Initialize

- Does DPMT prefer signed tags or unsigned tags, the wiki page is pretty
agnostic on the topic.

- Are there other lingering workflow questions for gbp pq not yet
recorded on the wiki page?

Thanks,
Allison