Bug#907727: Empty directory is already present in orig tarball

[Russ Allbery]

Felix Lechner  writes:

> It is actionable in that we can contact upstream (if the project is
> alive), but it will not improve the relationship. The tag is a
> widespread problem in the archive and a nuisance to many people. The tag
> should be removed. May I please retitle this bug?

Sure, yes, please go ahead.  After thinking about this some more, I agree
with your reasoning.

-- 
Russ Allbery (r...@debian.org)  

Bug#907727: Empty directory is already present in orig tarball

[Felix Lechner]

Hi Russ,

On Fri, Oct 25, 2019 at 1:40 PM Russ Allbery  wrote:
>
> To me, an override implies that Lintian is wrong, and I don't think it
> is.

Why did you file a bug report? Please use an override. :)

Joking aside, I do not think you are right. An override indicates the
maintainer will not address something. When Lintian is wrong, people
file bug reports.

> (Whether the tag should exist is a different question; not all
> problems are worth fixing.)

The tag is not well-named, but its assertion is undisputed: The
upstream sources contain an empty directory.

Since your upstream is gone there is nothing you can do about it. You
should override it (caveat below).

> It's otherwise unactionable by the maintainer.

It is actionable in that we can contact upstream (if the project is
alive), but it will not improve the relationship. The tag is a
widespread problem in the archive and a nuisance to many people. The
tag should be removed. May I please retitle this bug?

> It's bad practice to ship empty directories
> in tarballs precisely because they're not representable in Git (and
> generally have a tendency to get lost in various ways).

I do not think it is necessarily bad practice to include empty
directories in tarballs, although I would not do so personally. They
are superfluous and should be ignored. If git, pristine-tar or other
tools cannot deal with them in ways that work for their users, those
are bugs over there.

> It means that if anything relied on the directory existing, the directory
> would be recreated by unpacking the source package and whatever relied on
> that would succeed.

Lintian can technically suppress the tag when a file is added the way
you describe (by comparing the patched index against the unpatched
index), but it would go against the spirit of the tag: Upstream
shipped an empty folder. No maintainer addition can change that.

As a side note, I disagree on the use of a placeholder. If the tag is
not removed, I would use an override instead.

Kind regards
Felix Lechner

Bug#907727: Empty directory is already present in orig tarball

[Felix Lechner]

Hi Russ,

On Wed, Oct 23, 2019 at 1:08 PM Russ Allbery  wrote:
>
> the original request was to
> suppress the tag source-contains-empty-directory if the Debian patch set
> explicitly adds a file to that directory

An override is more explicit, and also more self-explanatory, when
compared to a '.placeholder'. Could dpkg perhaps add such a
placeholder automatically if the directory mentioned in the override
is still empty after all the patches have been applied?

> The result is that the patches-applied Debian
> packaging tree is then representable in Git

I re-read our prior messages, but still do not see why the ability to
round-trip through Git is so important. Either way, the issue should
IMHO be fixed in the tool chain that processes such round trips. Then
we can remove the tag from Lintian.

> which did seem mildly
> superior to recreating the directory in debian/rules

How would a placeholder created in debian/rules (or dh) help with the
round-tripping issue? Does anyone create source packages from built
source trees?

Upon further reflection, how about we record a manifest of the
upstream sources in the debian repo? Such a manifest could also
contain checksums for individual files, which would relieve us from
all problems synchronizing signed git tags and tarball signatures. (I
once wrote such a tool but it has seen no adoption.) Empty directories
could be easily re-created from the manifest when needed.

Kind regards
Felix

Bug#907727: Empty directory is already present in orig tarball

[Russ Allbery]

Felix Lechner  writes:

> An override is more explicit, and also more self-explanatory, when
> compared to a '.placeholder'.

To me, an override implies that Lintian is wrong, and I don't think it
is.  (Whether the tag should exist is a different question; not all
problems are worth fixing.)  It's bad practice to ship empty directories
in tarballs precisely because they're not representable in Git (and
generally have a tendency to get lost in various ways).

> I re-read our prior messages, but still do not see why the ability to
> round-trip through Git is so important. Either way, the issue should
> IMHO be fixed in the tool chain that processes such round trips. Then we
> can remove the tag from Lintian.

I don't know that it's so important.  It feels like minor to pedantic.
It's just an avoidable minor issue.

> How would a placeholder created in debian/rules (or dh) help with the
> round-tripping issue? Does anyone create source packages from built
> source trees?

It means that if anything relied on the directory existing, the directory
would be recreated by unpacking the source package and whatever relied on
that would succeed.

You're correct that it doesn't help with keeping the directory in the
source package; it just makes sure it's created by the build step before
anything might use it.

-- 
Russ Allbery (r...@debian.org)  

Bug#907727: Empty directory is already present in orig tarball

[Russ Allbery]

Apologies for the long delay in getting back to this bug.  To recap
history briefly since it's been a few months, the original request was to
suppress the tag source-contains-empty-directory if the Debian patch set
explicitly adds a file to that directory, and the example package affected
by this is xfonts-jmk.

Felix Lechner  writes:
> On Sat, Aug 10, 2019 at 4:51 PM Russ Allbery  wrote:

>> I don't think there was anything specific to git-buildpackage there.

> Isn't the whole problem specific to git-buildpackage?

No, the root problem, and the reason why the Lintian tag exists at all, is
that Git cannot represent empty directories and therefore will drop them
if a tree of files that contains an empty directory is imported into Git
and then exported out of Git again.

git-buildpackage is one tool that imports trees of files into Git and
later exports them as part of packaging, but just about any Git-based
packaging tool would the same problem.

See https://git.wiki.kernel.org/index.php/GitFaq#Can_I_add_empty_directories.3F

>> The result is that the patches-applied Debian packaging tree is then
>> representable in Git, which did seem mildly superior to recreating the
>> directory in debian/rules if it had disappeared due to a round trip
>> through Git.

> I would prefer not to add such an obscure patch to 11,000 packages. Is
> it really sufficient to create the directories in d/rules?

Yes, it should be.  I suspect that 99% of the time this causes no
practical issues since the directory isn't used by the build anyway, or
upstream recreates the directory as needed.  That does imply that removing
the tag may make sense.

I do think it's unfortunate that a round trip through Git results in a
different source package than the canonical archive representation, since
it could lead to some weird and obscure failures, but like you I'm dubious
that problem is worth asking people to fix 11,000 packages.

> That is a bug in our tools, not in upstream's delivery. Besides,
> couldn't you patch the upstream build system?

Yes, there are a lot of simple solutions to recreate the directory.

-- 
Russ Allbery (r...@debian.org)  

Bug#907727: Empty directory is already present in orig tarball

[Felix Lechner]

On Sat, Aug 10, 2019 at 4:51 PM Russ Allbery  wrote:
>
> I don't think there was anything specific to
> git-buildpackage there.

Isn't the whole problem specific to git-buildpackage?

> The result is that the patches-applied Debian
> packaging tree is then representable in Git, which did seem mildly
> superior to recreating the directory in debian/rules if it had disappeared
> due to a round trip through Git.

I would prefer not to add such an obscure patch to 11,000 packages. Is
it really sufficient to create the directories in d/rules?

> I can think of a situation where it
> could mask an obscure bug: if the upstream build system expects to be able
> to write to that directory without creating it first, this will work
> correctly if the upstream tarball is unpacked, the Debian unpacked over
> it, and then debian/rules run.  But if a Git import happens in the middle
> of that process, the package will then fail to build.

That is a bug in our tools, not in upstream's delivery. Besides,
couldn't you patch the upstream build system?

> Given that, I can see some merit to adding a file to that directory in the
> Debian package or otherwise arranging for that directory to be recreated.

Why not let 'dh' take care of it via a newly added d/empty-dirs? It
would be more explicit and self-explanatory.

> It's a kind of annoying problem otherwise since whether or not it's
> reproducible depends on whether one checks the package into (or out of)
> Git or just unpacks it directly, something that most people wouldn't
> expect to make a difference.

Exactly, it's a bug in our tools. Perhaps even in git. I am sure it's
been discussed and discarded.

> If [the tag] had gone away once I added a file to
> that directory in the packaging, I would have been quite happy with the
> tag and its recommendation.

Let's not jump to conclusions, then. We will find a good solution.

Kind regards,
Felix

Bug#907727: Empty directory is already present in orig tarball

[Russ Allbery]

Felix Lechner  writes:

> Please forgive me. I misunderstood your original filing.

Oh, it's no problem!  Apologies if I came across as upset.  I think I
didn't phrase my reply very well.

> Well, I do not use git-buildpackage, and such an intricate and obscure
> solution does nothing for me.

To be clear, all I did was add a placeholder file to that directory in the
Debian packaging (which because I'm using 3.0 quilt means that it gets
included in the diff).  I don't think there was anything specific to
git-buildpackage there.  The result is that the patches-applied Debian
packaging tree is then representable in Git, which did seem mildly
superior to recreating the directory in debian/rules if it had disappeared
due to a round trip through Git.

> At the same time, I will continue to forward many items, such as
> spelling errors in manpages and binaries, to upstream. :)

Oh, sure, absolutely.

> I think the tag is wrong with or without more data. It appears in the
> archive 11,482 times, with 302 overrides. If there are no objections
> within a reasonable time frame (perhaps a month), I will propose to
> remove the tag from Lintian.

That also works for me.  That said, I can think of a situation where it
could mask an obscure bug: if the upstream build system expects to be able
to write to that directory without creating it first, this will work
correctly if the upstream tarball is unpacked, the Debian unpacked over
it, and then debian/rules run.  But if a Git import happens in the middle
of that process, the package will then fail to build.

Given that, I can see some merit to adding a file to that directory in the
Debian package or otherwise arranging for that directory to be recreated.
It's a kind of annoying problem otherwise since whether or not it's
reproducible depends on whether one checks the package into (or out of)
Git or just unpacks it directly, something that most people wouldn't
expect to make a difference.

So I'm not sure the Lintian tag is completely wrong (although pedantic is
probably the right severity).  If it had gone away once I added a file to
that directory in the packaging, I would have been quite happy with the
tag and its recommendation.

-- 
Russ Allbery (r...@debian.org)   

Bug#907727: Empty directory is already present in orig tarball

[Felix Lechner]

Hi Russ,

On Sat, Aug 10, 2019 at 3:36 PM Russ Allbery  wrote:
>
> I would like Lintian to stop complaining about this when a file is
> explicitly added to that directory by the packaging.  It's otherwise
> unactionable by the maintainer.

Please forgive me. I misunderstood your original filing.

I agree with you, and I have the same problem with my packages. The
burden of dealing with empty directories in upstream sources should be
placed on our tools. The recommendation, in the tag description, that
upstream should remove the directory is awful. I tried that once; it
was embarrassing. That part should be erased from the tag description.

> > Adding placeholder solely to your debianized sources
> > will not help.
>
> Why not?

I was just stating fact.

> why is that not a reasonable solution to
> the underlying problem ...
> The directory is now representable in Git and other
> directory-unaware views of the Debian package, since it now has a file in
> it.

Well, I do not use git-buildpackage, and such an intricate and obscure
solution does nothing for me. Perhaps the tag should be removed from
Lintian. A warning in git-buildpackage, if possible, would be more
appropriate.

> Maybe I should say explicitly that I think of Lintian as a linter for the
> *Debian packaging*, not a linter for the upstream package, so I'm a little
> dubious of tags that report purely upstream problems the Debian packager
> cannot fix without changing upstream.

I like that perspective very much. Thank you for sharing. Your comment
shall help guide me in the future.

At the same time, I will continue to forward many items, such as
spelling errors in manpages and binaries, to upstream. :)

> I can add an override if I need to, but I do feel like this tag is wrong
> in a way that Lintian *could* get right with more data.

I think the tag is wrong with or without more data. It appears in the
archive 11,482 times, with 302 overrides. If there are no objections
within a reasonable time frame (perhaps a month), I will propose to
remove the tag from Lintian.

> I totally understand if you don't think it's worth fixing, but I'm not
> sure what Lintian could reasonably expect me to do about it other than
> what I already did.

Sorry, my previous response was based on a misunderstanding. Your
concern is important, and you are right.

Kind regards,
Felix

Bug#907727: Empty directory is already present in orig tarball

[Russ Allbery]

Felix Lechner  writes:

> I don't think this is a bug in Lintian.

> The source tarball xfonts-jmk_3.0.orig.tar.gz contains an empty
> directory 'neep/ascii/':

> $ dget 
> http://deb.debian.org/debian/pool/main/x/xfonts-jmk/xfonts-jmk_3.0-22.dsc
> $ tar tf xfonts-jmk_3.0.orig.tar.gz
> . . .
> jmk-x11-fonts-3.0/neep/
> jmk-x11-fonts-3.0/neep/Makefile
> jmk-x11-fonts-3.0/neep/Makefile.chardesc
> jmk-x11-fonts-3.0/neep/Makefile.fonts
> jmk-x11-fonts-3.0/neep/Makefile.fonts.in
> jmk-x11-fonts-3.0/neep/Makefile.neep
> jmk-x11-fonts-3.0/neep/ascii/ <<
> jmk-x11-fonts-3.0/neep/fragments/
> jmk-x11-fonts-3.0/neep/fragments/Makefile
> . . .

> That triggers the tag when the file index for the orig tarball is examined:

> 
> https://salsa.debian.org/lintian/lintian/blob/master/checks/cruft.pm#L450-451

Yes.  That's the bug report.  :)

I would like Lintian to stop complaining about this when a file is
explicitly added to that directory by the packaging.  It's otherwise
unactionable by the maintainer.

> The comment in the tag description [1] about adding a placeholder to
> the directory "as needed" applies to upstream before they ship the
> orig tarball. Adding placeholder solely to your debianized sources
> will not help.

Why not?

Or, to be more precise, I understand that it doesn't suppress the Lintian
tag (hence the bug report), but why is that not a reasonable solution to
the underlying problem that prompted Lintian to complain about this in the
first place?  The directory is now representable in Git and other
directory-unaware views of the Debian package, since it now has a file in
it.

Maybe I should say explicitly that I think of Lintian as a linter for the
*Debian packaging*, not a linter for the upstream package, so I'm a little
dubious of tags that report purely upstream problems the Debian packager
cannot fix without changing upstream.

> With upstream defunct, wouldn't a Lintian override, or perhaps even a
> repacking of the source, be good options for you?

I can add an override if I need to, but I do feel like this tag is wrong
in a way that Lintian *could* get right with more data.  I tend to report
those as bugs rather than adding overrides (as opposed to cases where I
can't imagine a way for Lintian to be able to do better).

The tag is only pedantic, so it's not any sort of urgent or important bug
and I totally understand if you don't think it's worth fixing, but I'm not
sure what Lintian could reasonably expect me to do about it other than
what I already did.  Surely this isn't important enough to warrant
repacking the upstream tarball and thus breaking verifiable provenance?

-- 
Russ Allbery (r...@debian.org)   

Bug#907727: Empty directory is already present in orig tarball

[Felix Lechner]

Hi Russ,

I don't think this is a bug in Lintian.

The source tarball xfonts-jmk_3.0.orig.tar.gz contains an empty
directory 'neep/ascii/':

$ dget 
http://deb.debian.org/debian/pool/main/x/xfonts-jmk/xfonts-jmk_3.0-22.dsc
$ tar tf xfonts-jmk_3.0.orig.tar.gz
. . .
jmk-x11-fonts-3.0/neep/
jmk-x11-fonts-3.0/neep/Makefile
jmk-x11-fonts-3.0/neep/Makefile.chardesc
jmk-x11-fonts-3.0/neep/Makefile.fonts
jmk-x11-fonts-3.0/neep/Makefile.fonts.in
jmk-x11-fonts-3.0/neep/Makefile.neep
jmk-x11-fonts-3.0/neep/ascii/ <<
jmk-x11-fonts-3.0/neep/fragments/
jmk-x11-fonts-3.0/neep/fragments/Makefile
. . .

That triggers the tag when the file index for the orig tarball is examined:


https://salsa.debian.org/lintian/lintian/blob/master/checks/cruft.pm#L450-451

> but a patch in the quilt series adds a file to that directory.  I believe
> this addresses the concerns described in that tag, and will ensure
> that any view of the source package will contain that directory.

The comment in the tag description [1] about adding a placeholder to
the directory "as needed" applies to upstream before they ship the
orig tarball. Adding placeholder solely to your debianized sources
will not help.

With upstream defunct, wouldn't a Lintian override, or perhaps even a
repacking of the source, be good options for you?

Please feel free to close the bug if you agree with the beginning of
this letter. :)

Kind regards,
Felix

[1] https://lintian.debian.org/tags/source-contains-empty-directory.html