Re: [Summary]: RFC: Standardizing source package artifacts build paths

[Calum McConnell]

> > > I kinda like the idea of prefixing *all* temporary directory with
> > > a '_'
> > Completely reasonable and almost auto-explaining, I'd say. +1 for
> > Michael's suggestion.
> 
> What about placing temporaries below debian/build/? Anything where
> names
> aren't fixed in Policy, i.e. which could be renamed with an
> underscore at
> the beginning, could also be moved.

I agree with the idea of prefixing build and other temporary
directories with an underscore, but I don't think we should move all
files that aren't explicitly defined in policy to be under
debian/build.  For instance, the git-buildpackage configurations aren't
fixed in policy, but they definitely shouldn't go in a build directory.

Additionally, I think it is sometimes useful to have multiple temporary
directories.  Some teams might use a cache folder: and if all
temporaries had to be under debian/build/, then they'd likely need to
put their caches there, moving their actual build folder to
debian/build/build/.  Which is all well and good, but then you get
packages who create a debian/build/ folder containing just one
subfolder named build/, and so on and so forth. The point is that I
think there is a real benefit to saying "Okay, put all your temporary
directories under the debian/ folder, and prefix their names with an
underscore".  Clean operations are still simple: but teams have
flexibility in how they run their builds.


signature.asc
Description: This is a digitally signed message part

Re: [Summary]: RFC: Standardizing source package artifacts build paths

[Simon Richter]

Hi,

> > If you want to avoid name collisions, you could also use
> > debian/_build

> > I kinda like the idea of prefixing *all* temporary directory with a '_'

> Completely reasonable and almost auto-explaining, I'd say. +1 for
> Michael's suggestion.

What about placing temporaries below debian/build/? Anything where names
aren't fixed in Policy, i.e. which could be renamed with an underscore at
the beginning, could also be moved.

   Simon

Re: [Summary]: RFC: Standardizing source package artifacts build paths

[Gunnar Wolf]

Michael Biebl dijo [Mon, May 04, 2020 at 11:51:05AM +0200]:
> >> Personally, I don't see any real benefit of standardizing on (making up an 
> >> example here) debian/.build over debian/build.
> > 
> > Same here. The arguments against debian/build are very weak. If we care
> > about a source package building a binary package named "build" or
> > whatever, it's really a unique case and surely it can be built with
> > some overrides and passing a different path where needed.
> 
> If you want to avoid name collisions, you could also use
> debian/_build
> 
> I kinda like the idea of prefixing *all* temporary directory with a '_'

Completely reasonable and almost auto-explaining, I'd say. +1 for
Michael's suggestion.


signature.asc
Description: PGP signature

Re: [Summary]: RFC: Standardizing source package artifacts build paths

[Michael Biebl]

Am 04.05.20 um 09:55 schrieb Raphael Hertzog:
> On Sat, 02 May 2020, Scott Kitterman wrote:
>> Personally, I don't see any real benefit of standardizing on (making up an 
>> example here) debian/.build over debian/build.
> 
> Same here. The arguments against debian/build are very weak. If we care
> about a source package building a binary package named "build" or
> whatever, it's really a unique case and surely it can be built with
> some overrides and passing a different path where needed.

If you want to avoid name collisions, you could also use
debian/_build

I kinda like the idea of prefixing *all* temporary directory with a '_'





signature.asc
Description: OpenPGP digital signature

Re: [Summary]: RFC: Standardizing source package artifacts build paths

[Raphael Hertzog]

On Sat, 02 May 2020, Scott Kitterman wrote:
> Personally, I don't see any real benefit of standardizing on (making up an 
> example here) debian/.build over debian/build.

Same here. The arguments against debian/build are very weak. If we care
about a source package building a binary package named "build" or
whatever, it's really a unique case and surely it can be built with
some overrides and passing a different path where needed.

There's also the precedence of debian/source that has taken over a part of
the namespace below the debian directory.

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS


signature.asc
Description: PGP signature

Re: +1 (no hidden directory, please) Re: [Summary]: RFC: Standardizing source package artifacts build paths

[Richard Laager]

On 5/3/20 5:54 PM, Holger Levsen wrote:
> On Sun, May 03, 2020 at 10:56:32PM +0200, Simon Richter wrote:
>> Frankly, I don't see the point in hiding the directory. The only person
>> who'd ever look into that directory would be someone inspecting what
>> happened during a build process, and all that hiding directories
>> achieves is adding more mental load to them where they have to remember
>> to use ls -a, and type a dot before trying to tab-complete anything.
>>
>> Basically, it would just be annoying with no good reason.
> 
> my thoughts exactly.

Agreed. Nothing else in the debian directory is hidden. I think the
burden should be on justifying why this should be the one hidden directory.

-- 
Richard



signature.asc
Description: OpenPGP digital signature

+1 (no hidden directory, please) Re: [Summary]: RFC: Standardizing source package artifacts build paths

[Holger Levsen]

On Sun, May 03, 2020 at 10:56:32PM +0200, Simon Richter wrote:
> Frankly, I don't see the point in hiding the directory. The only person
> who'd ever look into that directory would be someone inspecting what
> happened during a build process, and all that hiding directories
> achieves is adding more mental load to them where they have to remember
> to use ls -a, and type a dot before trying to tab-complete anything.
> 
> Basically, it would just be annoying with no good reason.

my thoughts exactly.


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: [Summary]: RFC: Standardizing source package artifacts build paths

[Simon Richter]

Hi,

On 02.05.20 17:53, Andreas Metzler wrote:

> I do think it is a splendid idea to separate generated stuff from
> everything else, I think there is no real good reason for using a
> hidden toplevel directory.

Frankly, I don't see the point in hiding the directory. The only person
who'd ever look into that directory would be someone inspecting what
happened during a build process, and all that hiding directories
achieves is adding more mental load to them where they have to remember
to use ls -a, and type a dot before trying to tab-complete anything.

Basically, it would just be annoying with no good reason.

I could see some potential in a debian/build/ directory, and moving
everything that is generated under it, but that would first have to be
implemented in debhelper, then we could start moving non-debhelper
packages over, and only then it could be turned into policy.

   Simon



signature.asc
Description: OpenPGP digital signature

Re: RFC: Standardizing source package artifacts build paths

[Sean Whitton]

Hello Niels,

On Tue 14 Apr 2020 at 10:54AM +02, Niels Thykier wrote:

> Guillem and I have debated this and come to a solution, which we believe
> will be able to address the concerns about the path being "hidden". It
> also enables us to simplify parts of the migration in some cases.
>
> The short version of that solution is to enable debhelper to expose the
> relevant directories where you want it and under debian/.build at the
> same time (the latter being a symlink to the former)[1].
>   This can be made to cover the directories you mention above, but also
> e.g. the build directory for upstream builds, which was mentioned
> elsewhere in this thread.
>
> Based on the feedback so far, I will go with the following defaults in
> debhelper:
>
>  * build directories are exposed if -B/--builddirectory is used to
>request it.  If there is no explicit -B/--builddirectory, it will
>(in a new compat level) be hidden by default.

So, if you want the build directories to be exposed, you have to pass
--builddirectory to debhelper, so it becomes a choice made by the
maintainer of each package?

I'm concerned that this doesn't help address the issue of new
contributors struggling to find .build when trying to figure out where
their build artifacts have gone -- it's not going to be any easier for
them to find the --builddirectory option.

As has been said, a non-hidden 'build' is easier to notice if you're
looking around.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: [Summary]: RFC: Standardizing source package artifacts build paths

[Scott Kitterman]

On Saturday, May 2, 2020 11:53:26 AM EDT Andreas Metzler wrote:
> In gmane.linux.debian.devel.general Niels Thykier  wrote:
> [...]
> 
> >  3) We followed up with an [update to the proposal] were debhelper would
> >  
> > optionally expose some of the relevant directories (some by default,
> > others on request) with symlinks while still supporting the new
> > layout. It did not attempt to change the debian/.build directory.
> >  
> >  4) There is not been any visible feedback on our updated proposal from
> >  
> > people, who raised concerns about the path, on whether this
> > alleviated their concern.  Nor any visible feedback on the choice of
> > paths being exposed by default.
> 
> [...]
> 
> FWIW I did not followup there because symlimking a hidden directory
> just complicates things without addressing arguments against using a
> hidden directory at all. I have not received a real answer to
> 20200331172540.ga1...@argenau.bebt.de. - But please bear with me ...
> 
> I do think it is a splendid idea to separate generated stuff from
> everything else, I think there is no real good reason for using a
> hidden toplevel directory. There is not a *strong* reason for not
> doing so either. This subquestion is mainly a bikeshed-type question,
> a matter of personal preference, so there is no consensus to be found.
> Please if you use a hidden toplevel dir just do it, do not complicate
> thing by symlinking outside the directory, which would sabotage the
> original aim of clearly separating generated stuff without increasing
> consensus. TIA.

We have some experience with this kind of question with Debian Python 
packaging.  The standard build system plugin, pybuild, use .pybuild for its 
build files.  This has good and bad points in my experience:

Good:
Usually I don't have to think about it at all.

Bad:
Relative to upstream expectations, the build directory is non-standard so 
sometimes extra effort is needed to run tests in the proper location.

When there are problems, people have to know to look for hidden directories to 
troubleshoot.

When this was first introduced, there was a fair amount of work to adapt to it 
(IIRC, it's been a few years), but since then it seems like tools and people 
have adapted.  I agree that symlinks only complicate the picture.

Personally, I don't see any real benefit of standardizing on (making up an 
example here) debian/.build over debian/build.

Scott K

signature.asc
Description: This is a digitally signed message part.

Re: [Summary]: RFC: Standardizing source package artifacts build paths

[Andreas Metzler]

In gmane.linux.debian.devel.general Niels Thykier  wrote:
[...]
>  3) We followed up with an [update to the proposal] were debhelper would
> optionally expose some of the relevant directories (some by default,
> others on request) with symlinks while still supporting the new
> layout. It did not attempt to change the debian/.build directory.


>  4) There is not been any visible feedback on our updated proposal from
> people, who raised concerns about the path, on whether this
> alleviated their concern.  Nor any visible feedback on the choice of
> paths being exposed by default.
[...]

FWIW I did not followup there because symlimking a hidden directory
just complicates things without addressing arguments against using a
hidden directory at all. I have not received a real answer to
20200331172540.ga1...@argenau.bebt.de. - But please bear with me ...

I do think it is a splendid idea to separate generated stuff from
everything else, I think there is no real good reason for using a
hidden toplevel directory. There is not a *strong* reason for not
doing so either. This subquestion is mainly a bikeshed-type question,
a matter of personal preference, so there is no consensus to be found.
Please if you use a hidden toplevel dir just do it, do not complicate
thing by symlinking outside the directory, which would sabotage the
original aim of clearly separating generated stuff without increasing
consensus. TIA.

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

[Summary]: RFC: Standardizing source package artifacts build paths

[Niels Thykier]

Guillem Jover:
> Hi!
> 
> We currently have many built artifacts being dropped directly under
> debian/ or under tool specific directories such as debian/.debhelper/.
> These have at least the following problems:
> 
>   - Make cleaning, an operation that requires executing code from the
> source package itself.
>   - Require keeping up updating things like .gitignore or artifacts
> might end up being committed into a VCS.
>   - Can stomp over existing files or other tools files, as these are
> generally not namespaced.
>   - Clutter the source tree, with debian/ mixing generated and manual
> files.
>   - dpkg-dev tools cannot assume the location of the binary package
> to be built, as that depends on the packaging logic and how that
> might drive the various commands.
> 
> [...]
> 
> Thanks,
> Niels and Guillem
> 

Hi,

It seems that the discussion on this topic has stopped and I will now
attempt to summarize the discussion and related consensus as I
understand it.

 1) No one seem to have voiced concerns about the general concept of the
proposal - only particular choices for path names and their
default visibility.

 2) The primary concern seem to be that the directory debian/.build is
hidden.  For some, this was a concern in general, for others the
concern appeared to be that paths they thought useful were being
"hidden by default" (concrete voiced examples include debian/tmp,
debian/ and upstream build directories).

 3) We followed up with an [update to the proposal] were debhelper would
optionally expose some of the relevant directories (some by default,
others on request) with symlinks while still supporting the new
layout. It did not attempt to change the debian/.build directory.


 4) There is not been any visible feedback on our updated proposal from
people, who raised concerns about the path, on whether this
alleviated their concern.  Nor any visible feedback on the choice of
paths being exposed by default.


*My interpretation of where we are*


>From what I can gather:

 * The proposal in concept has consensus.

 * Lack of visible feedback on the amendment make it hard to tell if the
   amendment alleviates the concern of people wanting certain
   directories to remain visible.

The lack of feedback on our amendment can sadly mean anything from
people accepting the amendment, people not being invested in the topic
or even people silently giving up on the discussion - neither documents
consensus in a good way, which is unfortunate as it leaves us in a
stalemate.


*What happens now*
Guillem and I will give another week for follow ups in the hope that
people will provide feedback on the amendment - either that the
amendment alleviates their concern or they would like a different
trade-off on the proposal.

In the absence of any update in the next week, Guillem and I will have
to consider an alternative way of resolving the stalemate.  Should that
happen, my personal take would be to apply the "Do'ers decide"-rule from
the constitution and move forward.

Thanks,
~Niels

[update to the proposal]:
  https://lists.debian.org/debian-dpkg/2020/04/msg5.html

Re: RFC: Standardizing source package artifacts build paths

[Niels Thykier]

Sean Whitton:
> Hello,
> 
> On Tue 14 Apr 2020 at 10:54AM +02, Niels Thykier wrote:
> 
>> Guillem and I have debated this and come to a solution, which we believe
>> will be able to address the concerns about the path being "hidden". It
>> also enables us to simplify parts of the migration in some cases.
> 
> Thank you for your continued efforts!
> 

Thanks. :)

>> The short version of that solution is to enable debhelper to expose the
>> relevant directories where you want it and under debian/.build at the
>> same time (the latter being a symlink to the former)[1].
> 
> Sorry, but don't you mean the former (non-dotdirs) being a symlink to
> the latter (.build)?
> 

Originally, no.  Though I think I will leave it unspecified while we
experiment to see if I become wiser on the topic than I am today.


My thought was this would enable debhelper to see that the directory was
exposed and look it up without additional state tracking (which would in
turn enable debhelper to ensure that we clean up properly since both
"ends" must be removed during clean).

But worst case, it will cost me a compat level to flip the order of the
two if the epiphany comes to me after initial deployment.

~Niels

Re: RFC: Standardizing source package artifacts build paths

[Sean Whitton]

Hello,

On Tue 14 Apr 2020 at 10:54AM +02, Niels Thykier wrote:

> Guillem and I have debated this and come to a solution, which we believe
> will be able to address the concerns about the path being "hidden". It
> also enables us to simplify parts of the migration in some cases.

Thank you for your continued efforts!

> The short version of that solution is to enable debhelper to expose the
> relevant directories where you want it and under debian/.build at the
> same time (the latter being a symlink to the former)[1].

Sorry, but don't you mean the former (non-dotdirs) being a symlink to
the latter (.build)?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: RFC: Standardizing source package artifacts build paths

[Niels Thykier]

Sam Hartman:
> I'm concerned about a leading . at least for:
> 
> * the debian/tmp replacement
> * the replacement for the package install directories under debian.
> 
> I think that maintaining those directories such that ls shows them will
> be more friendly for new maintainers.
> So I'd prefer something like debian/build rather than debian/.build for
> at least those directories.
> 
> I don't care as much about substvars, debhelper intermediates, debhelper
> logs and the like.
> 

Hi,

Guillem and I have debated this and come to a solution, which we believe
will be able to address the concerns about the path being "hidden". It
also enables us to simplify parts of the migration in some cases.

The short version of that solution is to enable debhelper to expose the
relevant directories where you want it and under debian/.build at the
same time (the latter being a symlink to the former)[1].
  This can be made to cover the directories you mention above, but also
e.g. the build directory for upstream builds, which was mentioned
elsewhere in this thread.

Based on the feedback so far, I will go with the following defaults in
debhelper:

 * build directories are exposed if -B/--builddirectory is used to
   request it.  If there is no explicit -B/--builddirectory, it will
   (in a new compat level) be hidden by default.

 * The staging directories a la d/tmp will be exposed if --destdir is
   used with dh_auto_install.
   - I will expose d/tmp by default for now but "parallel" d/tmp-X dirs
 are not (they currently require a --destdir to work as it is now).

 * d/ are exposed by default for now.


Any other debhelper artifact will move to a new path in a future compat
level (at this stage, we are talking debhelper/14 rather than debhelper/13).

~Niels

[1] Note that dpkg will migrate to always use the paths under
debian/.build.  This should not matter in practise unless you fiddle
with the symlinks that debhelper creates.

Re: RFC: Standardizing source package artifacts build paths

[Andreas Metzler]

On 2020-03-09 Sam Hartman  wrote:
> I'm concerned about a leading . at least for:

> * the debian/tmp replacement
> * the replacement for the package install directories under debian.

> I think that maintaining those directories such that ls shows them will
> be more friendly for new maintainers.
> So I'd prefer something like debian/build rather than debian/.build for
> at least those directories.

I would like to add a strong AOL. Please do not use a hidden toplevel
directory. debian/.build does not avoid clutter - it just makes it harder
to recognize a cluttered directory. Not "stomping over any existing
directories" is also a weak argument for ".build" vs any other name,
neither has a guarantee that there are no conflicts currently.

cu Andreas

Re: RFC: Standardizing source package artifacts build paths

[Guillem Jover]

On Mon, 2020-03-30 at 13:54:27 +0200, Simon Richter wrote:
> On 30.03.20 00:52, Guillem Jover wrote:
> > And, of course there are always going to be remaining sticking points
> > not covered by features I or others have in mind, but IMO their presence
> > will still mean there's something to improve somewhere.
> 
> This is the same debate we had in a lot of other places by now: do we
> want a descriptive interface that requires explicit provisions for every
> case, or an imperative interface that requires programming skill to use?

This proposal is orthogonal to that debate. Of course some things
might need declarative interfaces, but that does not mean that
excludes or would disallow programmatic ones. The proposal mostly
deals with defaults and conventions, for already existing interfaces.
For example for dpkg-shlibdeps or dh_install to look automatically under
these directories. Some of these commands might also grow new options
to override those defaults, when they do not yet have them. So it might
actually increase the control possibilities!

(But just to appease whoever else might be worried, while at least Niels
and I have been working and have plans to further improve the declarative
packaging support, we also are extremely aware that we'll always need
to support escape hatches for programmatic interfaces.)

> The namespace below debian/ has not been tightly regulated so far. Any
> name that isn't mentioned in Policy is free for the package to use for
> anything, and access to it is brokered again through debian/rules (with
> "clean" being pretty much the only operation allowed on it).

Again the proposal is not to tighly regulate anything. It would be a
matter of convention and defaults. If people do not want or cannot use
these, then that'd be fine too. Perhaps this was not really clear from
the initial proposal, or the fact that it was proposed at all made it
sound more forceful than it was intended.

Also policy does list many of these default pathnames already, most in
its appendices. Because that's what dpkg has used as defaults for a
very long time. Of course that has not disallowed debhelper, f.ex., to
use some different pathnames over time. The purpose of the proposal was
to try to create a new convention and coordinate over it.

> In that regard, the debian/ directory is not much different from the
> rest of the source tree. We still have quite a few packages doing
> in-tree builds because the upstream build system does not work for
> out-of-tree; these would still require cleaning through "debian/rules
> clean", so we won't get by without our escape hatch for quite a while.

Nothing would force antyhing to place stuff anywhere, as long as the
convention is not used. The point as mentioned above, is that if you
need, say, an out-of-tree pathname, then there would be conventions to
use, and the tools would have these as defaults or easy support for them
(for example with options to select different build flavors). If for
whatever reason these cannot be used then overrides would still be
possible for non-internal stuff, like it is currently the case.

> Unrelated to Policy, debhelper could move all of their intermediate
> files below debian/debhelper, that would probably get us 90% of the way
> without requiring a transition period.

This would still be quite unsatisfactory and not give the important
properties proposed, and moving all intermediate debhelper stuff would
still require a transition, or it would need to leave things behind.

Thanks,
Guillem

Re: RFC: Standardizing source package artifacts build paths

[Simon Richter]

Hi,

On 30.03.20 00:52, Guillem Jover wrote:

> And, of course there are always going to be remaining sticking points
> not covered by features I or others have in mind, but IMO their presence
> will still mean there's something to improve somewhere.

This is the same debate we had in a lot of other places by now: do we
want a descriptive interface that requires explicit provisions for every
case, or an imperative interface that requires programming skill to use?

For packaging, we have an imperative interface, and a declarative
wrapper on top that handles most of the simple cases. That works well,
and I'm not sure there is much potential for improvement there, because
even if we switched to a declarative-by-default system, we'd still need
an imperative escape hatch for those cases not covered yet.

The namespace below debian/ has not been tightly regulated so far. Any
name that isn't mentioned in Policy is free for the package to use for
anything, and access to it is brokered again through debian/rules (with
"clean" being pretty much the only operation allowed on it).

In that regard, the debian/ directory is not much different from the
rest of the source tree. We still have quite a few packages doing
in-tree builds because the upstream build system does not work for
out-of-tree; these would still require cleaning through "debian/rules
clean", so we won't get by without our escape hatch for quite a while.

Unrelated to Policy, debhelper could move all of their intermediate
files below debian/debhelper, that would probably get us 90% of the way
without requiring a transition period.

   Simon



signature.asc
Description: OpenPGP digital signature

Re: RFC: Standardizing source package artifacts build paths

[Guillem Jover]

On Sun, 2020-03-29 at 22:48:04 +0200, Marco d'Itri wrote:
> On Mar 29, Guillem Jover  wrote:
> > While it's true that we might need to use such pathnames in debian/rules
> > or debhelper fragment files (which some might consider ugly), IMO that
> > has always felt like a sign that there's something missing in our
> > packaging helpers/tools.

> If you believe this then it means that you have been maintaining too 
> modern or too much simple packages.

Aww thanks, you are, as always, way too kind, and you probably even
listed too many reasons there.

> https://salsa.debian.org/md/inn/-/blob/master/debian/rules
> https://salsa.debian.org/md/inn2/-/blob/master/debian/rules
> https://salsa.debian.org/md/kmod/-/blob/master/debian/rules
> https://salsa.debian.org/md/binkd/-/blob/master/debian/rules
> https://salsa.debian.org/md/libxcrypt/-/blob/master/debian/rules

Most of these show some of the things I had in mind. Which have not
been supported (easily) in the past or yet by our packaging toolchain,
like:

  - declarative file metadata,
  - renaming files,
  - flavor builds (like udeb vs no-udeb, or w/ features disabled in
configure, etc.),

Some other usages there though, look like things that can really be
simplified in the packaging, by say, just:

  - using dh_install, dh_link and friends,
  - being explicit in things to install in the .install fragments
instead of just adding whole hierarchies,
  - using a temporary destdir (like debhelper does by default) and only
installing the desired pathnames to the final installation directory.

Other things there can be simplified by improving the upstream build
system, also by parametrizing things, and submitting these upstream or
just carrying as local patches.

So yes, thanks, to me these examples seem to still confirm that the
explicit usage of the staged directories is an indication of things that
need fixing or improving in either the packaging or the toolchain.

And, of course there are always going to be remaining sticking points
not covered by features I or others have in mind, but IMO their presence
will still mean there's something to improve somewhere.

Regards,
Guillem

Re: RFC: Standardizing source package artifacts build paths

[Marco d'Itri]

On Mar 29, Guillem Jover  wrote:

> While it's true that we might need to use such pathnames in debian/rules
> or debhelper fragment files (which some might consider ugly), IMO that
> has always felt like a sign that there's something missing in our
> packaging helpers/tools.
If you believe this then it means that you have been maintaining too 
modern or too much simple packages.

https://salsa.debian.org/md/inn/-/blob/master/debian/rules
https://salsa.debian.org/md/inn2/-/blob/master/debian/rules
https://salsa.debian.org/md/kmod/-/blob/master/debian/rules
https://salsa.debian.org/md/binkd/-/blob/master/debian/rules
https://salsa.debian.org/md/libxcrypt/-/blob/master/debian/rules

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: RFC: Standardizing source package artifacts build paths

[Guillem Jover]

On Mon, 2020-03-09 at 09:23:46 -0400, Sam Hartman wrote:
> I'm concerned about a leading . at least for:
> 
> * the debian/tmp replacement
> * the replacement for the package install directories under debian.
> 
> I think that maintaining those directories such that ls shows them will
> be more friendly for new maintainers.
> So I'd prefer something like debian/build rather than debian/.build for
> at least those directories.
> 
> I don't care as much about substvars, debhelper intermediates, debhelper
> logs and the like.

As mentioned on the proposal the use of a hidden directory is to
reduce clutter (getting the packaged bits alongside the generated
pathnames when listing the debian/ directory contents has always
seemed extremely distracting, which also makes it too easy for them
to end up on a VCS, f.ex.) and to avoid collisions with existing
packaging (due to temporary directory usage or due to staged package
directories).

There's also precedent with hidden directories with debhelper, even
for the staged directories for dbgsyms packages. Making internal stuff
non-hidden would make it more distracting, and if this would mean having
at least two hierarchies (one hidden and one non-hidden), that can be
even more confusing and is more work to cleanup and ignore, which goes
in the opposite direction of simplifying things IMO.

While it's true that we might need to use such pathnames in debian/rules
or debhelper fragment files (which some might consider ugly), IMO that
has always felt like a sign that there's something missing in our
packaging helpers/tools. To me this calls f.ex. for making dpkg-dev and
debhelper support build flavors (for multiple builds) or for debhelper
to automatically look in the relevant build directories when looking up
for files, so in addition to look for them in the installation stage
directory and the source package root directory, it would look there
too, removing the major needs for having to list the full pathnames
explicitly.

Thanks,
Guillem

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

[Mattia Rizzolo]

(note, this is a barely structured brain dump)

On Tue, Mar 10, 2020 at 08:10:55AM +0100, Niels Thykier wrote:
> >> Though, can you elaborate a bit on why the above approach would be
> >> better than a standard ENV variable a la AUTOPKGTEST_ARTIFACTS and some
> >> easy way to declare additional artifacts to be extracted?
> > 
> > Mainly, I'd prefer something declarative with glob patterns (a bit like
> > debian/clean or Gitlab-CI's artifacts:paths) rather than having to write
> > logic like these pseudo-patches:

Same.
Having such directory be hidden inside
/build/foo-1.2.3/debian/.build/artefacts/ is kind of a mouthful, but if
that gets to be standardized I think it would be awesome (builders
(sbuild, pbuilder, …) decide on the first '/build/foo-1.2.3/' part of
the path and they know of it; package building happens with CWD in that
place, so build tools should just try to stick to relative paths
'./debian/.build/artefacts/'; everything should Just Work that way).

One thing that strikes me of this proposal, is that you were trying to
"hide" that .build directory from the maintainer; doing this would be
going against that design decision.  This is the only "concern" I have
with the proposal.  Probably this can be avoided by providing a dh_
helper.

> Ack, I get the part of having a declarative mechanism for selecting files.

And then builder could just take out the whole directory.  If that gets
to be (g|x|)zipped or not would be an implementation detail of the
builders (sbuild, pbuilder, …) and of whatever frontend (launchpad,
buildd + wanna-build, …) is used.

> Just to clarify something related.  Should debhelper and other tools by
> default archive "certain files of possible interest" (e.g. config.log)?
> Or should we limit it to "on request only"?

That would be some nice automatism indeed, but I think it's something
for "later".  If you do, please consider these bits:
 * naming the files: you risk clashing with maintainer-set file names
 * deciding on whether to put those files there only on failure or all
   the time

> The former makes it simpler for people that are interested in the
> "default" parts but also bloats the archive for people that are
> interested in just one file.

"bloating" is indeed important.  If we start doing this, frontends need
to decide on a retaining policy.  Do we want maintainers to have a say
on this?  Like, adding a metadata file to the artifacts to indicate any
interest on those files (this is a successful build: keep for x
days/keep until next successful build + y days, etc etc).

> > I don't have any particular opinion on whether artifacts should be
> > collected into debian/.build/artifacts/, into $DPKG_ARTIFACTS/, or
> > directly into some sort of archive (Gitlab and Jenkins seem to use zip
> > files, which have the advantage of being seekable, so web frontends
> > can presumably read individual logs directly out of the zip file if
> > that's desirable).
> 
> Thanks for clarifying.  This answered the question I was trying to write. :)


I think I took care of those thoughts above, but to reiterate:
 * IMHO ./debian/.build/artefacts/ (or artifacts? :P) is a cool and
   accessible place for all interested software
 * perhaps, you could consider using
   ${DPKG_ARTEFACTS:-$PWD/debian/.build/artefacts} so that some builders
   can override the directory if they find it more convenient for some
   reason, but otherwise I'd rather stick to a stable, non-changable
   path.
 * I think eventual tarball/compression should be left as a matter for
   the build driver (sbuild, pbuilder, …).

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
More about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

[Mattia Rizzolo]

On Tue, Mar 10, 2020 at 09:07:57AM +, Simon McVittie wrote:
> On Tue, 10 Mar 2020 at 07:19:59 +, Paul Wise wrote:
> > On Tue, Mar 10, 2020 at 7:12 AM Niels Thykier wrote:
> > > Standardized way of extracting additional build-time artefacts
> > 
> > This reminds me of the BYHAND stuff, I forget how that works though.
[…]
> Similarly, we probably don't want to publish the build products to users
> if the build(-time tests) failed (because we can't be confident that any
> products that were already produced are good), although we might well
> want to make them available through a contributor-oriented interface to
> help to debug the failures; but we do want to publish build and test logs
> to contributors, regardless of success or failure.

And this highlights one important aspect of such interface: such
artifacts would be collected even after a build failure.
That's not possible at all now.

> The .buildinfo file is arguably already in the same category as build
> and test logs. We currently capture it in the .changes file and upload
> it to ftp-master, but it isn't reproducible, and ftp-master doesn't
> republish it through our user-facing interface (the archive). Ideally,
> failed builds would capture their .buildinfo as well as their log for
> subsequent analysis, although I don't know whether they actually do.

That's somewhat of a tough argument, that I'd try to keep separate (it
has to do with the semantic meaning of a .buildinfo (i.e., it tries to
attach a *built artifacts* to the way it was build, a .buildinfo without
any hashes would be quite meaningless when tied to its original meaning.
Also, we do want it to be published, but we are still waiting for the
ftp-masters to tell us their distribution requirements...).

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
More about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

[Simon McVittie]

On Tue, 10 Mar 2020 at 07:19:59 +, Paul Wise wrote:
> On Tue, Mar 10, 2020 at 7:12 AM Niels Thykier wrote:
> > Standardized way of extracting additional build-time artefacts
> 
> This reminds me of the BYHAND stuff, I forget how that works though.

I think how that works is that at the appropriate time during a successful
build, you run dpkg-distaddfile to insert extra entries that are not a
recognised file type (.deb, .udeb etc.) into debian/files?

The difference (as I understand it) is that BYHAND is for extra build
products that are listed in the .changes file and intended to be published
to Debian users via ftp.debian.org; whereas in this thread we're talking
about non-essential things that are produced as a side-effect of the
build, are potentially useful to Debian contributors for debugging and
analysis of the build process itself, but are not actually the "product".

Some important trade-offs are different. For example, for the build
products mentioned in the .changes file (whether .deb or BYHAND) we want
reproducible builds that don't capture unnecessary information like the
properties of the build system; whereas in build and test logs, we *do*
want to capture system-specific information in case it's relevant, for
example to help a Debian contributor to realise correlations like "this
test fails whenever we're building on a btrfs filesystem" that can help
them to find and fix bugs.

Similarly, we probably don't want to publish the build products to users
if the build(-time tests) failed (because we can't be confident that any
products that were already produced are good), although we might well
want to make them available through a contributor-oriented interface to
help to debug the failures; but we do want to publish build and test logs
to contributors, regardless of success or failure.

The .buildinfo file is arguably already in the same category as build
and test logs. We currently capture it in the .changes file and upload
it to ftp-master, but it isn't reproducible, and ftp-master doesn't
republish it through our user-facing interface (the archive). Ideally,
failed builds would capture their .buildinfo as well as their log for
subsequent analysis, although I don't know whether they actually do.

smcv

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

[Simon McVittie]

On Tue, 10 Mar 2020 at 08:10:55 +0100, Niels Thykier wrote:
> Just to clarify something related.  Should debhelper and other tools by
> default archive "certain files of possible interest" (e.g. config.log)?
> Or should we limit it to "on request only"?

I think it would probably make most sense for dpkg (which doesn't know
about specific build systems) to not archive anything by default, or to
archive only things it produced itself.

For debhelper it might make sense for build system classes to archive
well-known logs like Autotools' ${builddir}/config.log and Meson's
${builddir}/meson-logs/ by default, but probably not logs that are in
an unpredictable location like Autotools' test logs.

If there's an exclusion mechanism for packages that know a particular
artifact is not useful and monstrously large ("!meson-logs/big.log"?) then
it doesn't necessarily matter much either way. If artifacts aren't kept
forever then the damage from archiving too much will be temporary.

> The former makes it simpler for people that are interested in the
> "default" parts but also bloats the archive for people that are
> interested in just one file.

Have you seen the UI Gitlab-CI and Jenkins provide for this? If you look
at a Gitlab-CI job like ,
there's a Download link that gives you the complete bundle of
artifacts in a zip file, but there's also a Browse link like

that lets you look at individual logs online (which is often enough to
debug an issue). Jenkins has a similar system.

On Salsa, ci.debian.net (for tests with the needs-build restriction),
and similar systems, I think it would make most sense to drop the
artifacts into somewhere the "larger" CI system will pick them up, and
let the "larger" CI system handle browsing and expiry. On
buildd.debian.org, I think build logs are kept forever(?) but artifacts
should probably have some sort of expiration mechanism, similar to the
way ci.debian.net remembers test results indefinitely but discards old
logs after a while.

smcv

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

[Paul Wise]

On Tue, Mar 10, 2020 at 7:12 AM Niels Thykier wrote:

>

Standardized way of extracting additional build-time artefacts


This reminds me of the BYHAND stuff, I forget how that works though.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

[Niels Thykier]

Simon McVittie:
> On Mon, 09 Mar 2020 at 20:45:13 +0100, Niels Thykier wrote:
>> Simon McVittie:
>>> For example, dpkg-buildpackage could perhaps read one glob per
>>> line from debian/artifacts and hardlink matched files (if any) into
>>> debian/.build/artifacts for collection by a "larger" framework like
>>> sbuild or pbuilder, or individual packages could copy/link files into there
>>> as they go, or debhelper build-system classes like Autotools and Meson
>>> could know the names of common log files from their build system, or
>>> some combination of those.
>>
>> Though, can you elaborate a bit on why the above approach would be
>> better than a standard ENV variable a la AUTOPKGTEST_ARTIFACTS and some
>> easy way to declare additional artifacts to be extracted?
> 
> Mainly, I'd prefer something declarative with glob patterns (a bit like
> debian/clean or Gitlab-CI's artifacts:paths) rather than having to write
> logic like these pseudo-patches:
> 
> [...]
> 

Ack, I get the part of having a declarative mechanism for selecting files.

Just to clarify something related.  Should debhelper and other tools by
default archive "certain files of possible interest" (e.g. config.log)?
Or should we limit it to "on request only"?

The former makes it simpler for people that are interested in the
"default" parts but also bloats the archive for people that are
interested in just one file.

> I don't have any particular opinion on whether artifacts should be
> collected into debian/.build/artifacts/, into $DPKG_ARTIFACTS/, or
> directly into some sort of archive (Gitlab and Jenkins seem to use zip
> files, which have the advantage of being seekable, so web frontends
> can presumably read individual logs directly out of the zip file if
> that's desirable).
> 
> smcv
> 

Thanks for clarifying.  This answered the question I was trying to write. :)

~Niels

Re: RFC: Standardizing source package artifacts build paths

[Simon McVittie]

On Mon, 09 Mar 2020 at 20:45:13 +0100, Niels Thykier wrote:
> Simon McVittie:
> > For example, dpkg-buildpackage could perhaps read one glob per
> > line from debian/artifacts and hardlink matched files (if any) into
> > debian/.build/artifacts for collection by a "larger" framework like
> > sbuild or pbuilder, or individual packages could copy/link files into there
> > as they go, or debhelper build-system classes like Autotools and Meson
> > could know the names of common log files from their build system, or
> > some combination of those.
> 
> Though, can you elaborate a bit on why the above approach would be
> better than a standard ENV variable a la AUTOPKGTEST_ARTIFACTS and some
> easy way to declare additional artifacts to be extracted?

Mainly, I'd prefer something declarative with glob patterns (a bit like
debian/clean or Gitlab-CI's artifacts:paths) rather than having to write
logic like these pseudo-patches:

-./configure
+if ./configure; then
+: # OK
+else
+error=$?
+if [ -n "$DPKG_ARTIFACTS" ]; then
+cp config.log "$DPKG_ARTIFACTS/config.log" || :
+fi
+exit "$error"
+fi

-make check
+if make check; then
+: # OK
+else
+error=$?
+if [ -n "$DPKG_ARTIFACTS" ]; then
+find tests/ -name '*.log' | something something xargs cp
+fi
+exit "$error"
+fi

(doubly so if I have to sprinkle backslashes, tabs and double $ over
it to make it a valid Makefile), and I'm sure you think similarly from
the perspective of writing debhelper buildsystem classes.

In particular, if you want to capture Automake test logs without also
capturing unrelated files that happen to be named *.log, as far as
I'm aware there is no general way to do that without knowledge of the
specific package and how its tests are named or arranged, so individual
maintainers are likely to have to be involved in designating which logs
are of interest.

I don't have any particular opinion on whether artifacts should be
collected into debian/.build/artifacts/, into $DPKG_ARTIFACTS/, or
directly into some sort of archive (Gitlab and Jenkins seem to use zip
files, which have the advantage of being seekable, so web frontends
can presumably read individual logs directly out of the zip file if
that's desirable).

smcv

Re: RFC: Standardizing source package artifacts build paths

[Sean Whitton]

Hello Julian,

On Mon 09 Mar 2020 at 09:43PM +01, Julian Andres Klode wrote:

> On Mon, Mar 09, 2020 at 10:09:46AM +0100, Guillem Jover wrote:
>> We'd like to standardize on a new set of artifact build pathnames
>> for our deb toolchain. [...]
> [...]
>> The use of a hidden directory is to reduce clutter and stomping over any
>
> Love the hidden directory.

Fair enough, but what do you think about the counterarguments that have
been raised?  What's good about having it hidden?

Thanks.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: RFC: Standardizing source package artifacts build paths

[Julian Andres Klode]

On Mon, Mar 09, 2020 at 10:09:46AM +0100, Guillem Jover wrote: 
> We'd like to standardize on a new set of artifact build pathnames
> for our deb toolchain. [...]
[...]
> The use of a hidden directory is to reduce clutter and stomping over any

Love the hidden directory.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: RFC: Standardizing source package artifacts build paths

[Niels Thykier]

Simon McVittie:

Hi :)

> On Mon, 09 Mar 2020 at 10:09:46 +0100, Guillem Jover wrote:
>> We'd like to standardize on a new set of artifact build pathnames
>> for our deb toolchain. These would have the following form:
>>
>>   - debian/.build/upstream*
>>
>> These could be used for out-of-tree builds replacing current
>> build-tree/ and similar directories.
> 
> Presumably the '*' means it's considered OK for packages that do separate
> production/debug/udeb builds to use multiple directories matching that
> glob?
> 
> For example, a simple debhelper build might eventually (in a future compat
> level) default to "dh $@ --builddirectory=debian/.build/upstream",
> but because src:glib2.0 does separate deb and udeb builds
> (with non-essential features disabled in the udeb), it
> would use "--builddirectory=debian/.build/upstream-deb"
> and "--builddirectory=debian/.build/upstream-udeb" instead
> of its current "--builddirectory=debian/build/deb" and
> "--builddirectory=debian/build/udeb"?
> 

Yes.  :)

Among other, it will hopefully also enable debhelper to support multiple
builds natively or better, when we have a standardized path for these
things.

>>   - We could just (say) .gitignore a single pathname.
>>   - Cleaning the Debian artifacts would imply doing just
>> «rm -rf debian/.build/» (even though this could be abstracted via a
>> new dpkg-clean command).
>>   - Namespacing would avoid any collisions.
>>   - Would make parallelization way easier and faster.
>>   - dpkg-dev tools could start defaulting to use package specfific
>> directories, instead of requiring the user/tool to specify them.
> 
> This all seems good to have.
> 
> The Salsa-CI pipeline (which runs in a git clone of a package, and
> currently uses debian/output/ for intermediate build stuff) could maybe
> also move its scratch directory to debian/.build/salsa-ci.
> 
> Have you seen  and
> related discussions in the context of sbuild and pbuilder? It seems
> like a related topic. [...]
> 
> For example, dpkg-buildpackage could perhaps read one glob per
> line from debian/artifacts and hardlink matched files (if any) into
> debian/.build/artifacts for collection by a "larger" framework like
> sbuild or pbuilder, or individual packages could copy/link files into there
> as they go, or debhelper build-system classes like Autotools and Meson
> could know the names of common log files from their build system, or
> some combination of those.
> 
> smcv
> 

I could see how having a well-defined method for extracting artifacts
will be useful (also for CI purposes).

Though, can you elaborate a bit on why the above approach would be
better than a standard ENV variable a la AUTOPKGTEST_ARTIFACTS and some
easy way to declare additional artifacts to be extracted?
  I am fine either way, but I could image that the
debian/.build/artifacts will feature interact with e.g.
"dpkg-buildpackage -tc" where a AUTOPKGTEST_ARTIFACTS replacement would not.

~Niels

Re: RFC: Standardizing source package artifacts build paths

[Weatherby,Gerard]

FWIW:

We have been using debian packaging from a private repository for the last 
three years to configure virtual machines.

As a subversion shop, at least, the extraneous files generated by a build 
aren't an issue because we simply svn-clean after a build. I believe "git 
clean" does the equivalent operation. 

I concur that hiding the build directory with a leading dot is less friendly. 
It takes a while to understand how package creation / building works, and 
hiding information from users will make it more difficult. 

--
Gerard Weatherby| Application Architect
NMRbox | Department of Molecular Biology and Biophysics | UConn Health
263 Farmington Avenue, Farmington, CT 06030-6406
Phone: 860 679 8484
uchc.edu


From: Sam Hartman 
Sent: Monday, March 9, 2020 9:23 AM
To: debian-devel@lists.debian.org
Cc: debian-d...@lists.debian.org; debhel...@packages.debian.org
Subject: Re: RFC: Standardizing source package artifacts build paths

*** Attention: This is an external email. Use caution responding, opening 
attachments or clicking on links. ***

I'm concerned about a leading . at least for:

* the debian/tmp replacement
* the replacement for the package install directories under debian.

I think that maintaining those directories such that ls shows them will
be more friendly for new maintainers.
So I'd prefer something like debian/build rather than debian/.build for
at least those directories.

I don't care as much about substvars, debhelper intermediates, debhelper
logs and the like.

Re: RFC: Standardizing source package artifacts build paths

[Sam Hartman]

I'm concerned about a leading . at least for:

* the debian/tmp replacement
* the replacement for the package install directories under debian.

I think that maintaining those directories such that ls shows them will
be more friendly for new maintainers.
So I'd prefer something like debian/build rather than debian/.build for
at least those directories.

I don't care as much about substvars, debhelper intermediates, debhelper
logs and the like.

Re: RFC: Standardizing source package artifacts build paths

[Simon McVittie]

On Mon, 09 Mar 2020 at 10:09:46 +0100, Guillem Jover wrote:
> We'd like to standardize on a new set of artifact build pathnames
> for our deb toolchain. These would have the following form:
> 
>   - debian/.build/upstream*
> 
> These could be used for out-of-tree builds replacing current
> build-tree/ and similar directories.

Presumably the '*' means it's considered OK for packages that do separate
production/debug/udeb builds to use multiple directories matching that
glob?

For example, a simple debhelper build might eventually (in a future compat
level) default to "dh $@ --builddirectory=debian/.build/upstream",
but because src:glib2.0 does separate deb and udeb builds
(with non-essential features disabled in the udeb), it
would use "--builddirectory=debian/.build/upstream-deb"
and "--builddirectory=debian/.build/upstream-udeb" instead
of its current "--builddirectory=debian/build/deb" and
"--builddirectory=debian/build/udeb"?

>   - We could just (say) .gitignore a single pathname.
>   - Cleaning the Debian artifacts would imply doing just
> «rm -rf debian/.build/» (even though this could be abstracted via a
> new dpkg-clean command).
>   - Namespacing would avoid any collisions.
>   - Would make parallelization way easier and faster.
>   - dpkg-dev tools could start defaulting to use package specfific
> directories, instead of requiring the user/tool to specify them.

This all seems good to have.

The Salsa-CI pipeline (which runs in a git clone of a package, and
currently uses debian/output/ for intermediate build stuff) could maybe
also move its scratch directory to debian/.build/salsa-ci.

Have you seen  and
related discussions in the context of sbuild and pbuilder? It seems
like a related topic. Briefly, there's a desire to be able to have
"artifacts" that are not part of the main build result, are produced by
both successful and failing builds, are kept for a while by the buildd
infrastructure, and are probably not uploaded to the Debian archive,
as a generalization of the build log.

For example, Autotools packages have ${builddir}/config.log, which is not
a useful thing to publish in a .deb (not at all reproducible!), but can
be useful to diff between successful and failing builds to debug why a
build failed. At the moment there's a special case in debhelper to cat
config.log if configure fails, but it is forgotten about after the build
if configure succeeds. Meson has ${builddir}/meson-logs/meson-log.txt,
which is analogous.

Similarly, Autotools packages normally output a test summary on success
or a more detailed log on failure (${builddir}/**/test-suite.log
describes all tests that failed, were partially or entirely skipped,
or were otherwise not 100% successful, and there are also detailed logs
that capture diagnostics from even the completely successful tests),
and Meson ${builddir}/meson-logs/testlog.txt is analogous.

At the moment the only artifact, other than the .changes file and the
files it references when the build is successful, is the build log itself,
which means the build process must either cat these other artifacts to
get them into the build log (resulting in large build logs where it's
difficult to find what you're looking for), or lose them forever.

Some packages have machine-readable test output that is used to track
regressions and intermittent failures between builds, for which cat'ing
the test output into the build log is less than ideal: to compare test
results between runs it's necessary to screen-scrape the test output
from among the unstructured/human-readable messages that make up most
of the build log. A few packages work around this by packaging their
test results, which usually makes them non-reproducible.

autopkgtest's ${AUTOPKGTEST_ARTIFACTS} (a directory into which tests are
encouraged to write or copy more detailed results than the ones shown
on stdout) and Gitlab-CI's artifacts:paths: parameter (a list of globs
matching paths in the built directory) are conceptually similar ideas.

For example, dpkg-buildpackage could perhaps read one glob per
line from debian/artifacts and hardlink matched files (if any) into
debian/.build/artifacts for collection by a "larger" framework like
sbuild or pbuilder, or individual packages could copy/link files into there
as they go, or debhelper build-system classes like Autotools and Meson
could know the names of common log files from their build system, or
some combination of those.

smcv

RFC: Standardizing source package artifacts build paths

[Guillem Jover]

Hi!

We currently have many built artifacts being dropped directly under
debian/ or under tool specific directories such as debian/.debhelper/.
These have at least the following problems:

  - Make cleaning, an operation that requires executing code from the
source package itself.
  - Require keeping up updating things like .gitignore or artifacts
might end up being committed into a VCS.
  - Can stomp over existing files or other tools files, as these are
generally not namespaced.
  - Clutter the source tree, with debian/ mixing generated and manual
files.
  - dpkg-dev tools cannot assume the location of the binary package
to be built, as that depends on the packaging logic and how that
might drive the various commands.

These files include for example:

  - debian/files
  - debian/*substvars
  - debian//
  - debian/*debhelper.log
  - debian/.debhelper/
  - debian/tmp/

We'd like to standardize on a new set of artifact build pathnames
for our deb toolchain. These would have the following form:

  - debian/.build/upstream*

These could be used for out-of-tree builds replacing current
build-tree/ and similar directories.

  - debian/.build/fsys* (debian/.build/{stage,installed}* ?)

These would replace the current debian/tmp/ and variants.

  - debian/.build/binary/

These would be equivalent to the current debian//.

  - debian/.build//

These would be tool specific, for example:

debian/.build/dpkg/files
debian/.build/dpkg/files.d/
debian/.build/dpkg/substvars/

debian/.build/debhelper/<...>

The use of a hidden directory is to reduce clutter and stomping over any
existing directories (there's for example a 3D engine called «build» :).
Most of this could be transparent to most packages, as it would be hidden
behind debhelper logic on a new compat level. There are really few
packages not using debhelper (either directly or indirectly via cdbs).
dpkg-dev tools would preserve backwards compatibility, where applicable,
or could enable this by default only on hypothetical new dpkg-dev compat
levels .

This would give us at least the following properties:

  - We could just (say) .gitignore a single pathname.
  - Cleaning the Debian artifacts would imply doing just
«rm -rf debian/.build/» (even though this could be abstracted via a
new dpkg-clean command).
  - Namespacing would avoid any collisions.
  - Would make parallelization way easier and faster.
  - dpkg-dev tools could start defaulting to use package specfific
directories, instead of requiring the user/tool to specify them.
  - Could make it possible (but not necessary) to move parts of the logic
from debian/rules (such as calling dpkg-deb) into the build driver
(see the upcoming proposal), further simplifiying the build process.

Thanks,
Niels and Guillem