On Tue, 27 Aug 2024 09:42:09 +0200 Niels Thykier <[email protected]> wrote:
Package: dgit
Version: 11.10
Severity: wishlist
Tags: upstream
X-Debbugs-Cc: [email protected]
Hi
Filing this as a bug for discussion as requested in
https://salsa.debian.org/dgit-team/dgit/-/issues/7#note_520035.
I would prefer to keep the salsa issue open in parallel, since I can
update the proposal in place there to have the final form easily
discoverable in case of amendments.
I will submit the first version by email once this bug has been
accepted. My previous attempt to send (a variant of) the proposal to
mailing lists was silently dropped by debian.org (neither the List nor
Sean seems to have received it) and vocally by Ian's SMTP server as
spam. By splitting the two, I am hoping you will all get the
notification and we will have the discussion bug even if the full text
gets stuck.
Best regards,
Niels
As promised, here is the initial version of the content of the salsa issue.
# Content
I am looking into supporting features that involves generating or
enriching certain packaging files automatically. Notably
`debian/control` but in theory also `debian/tests/control`. I noted
that `dgit` generates its `debian/tests/control`, which is under some of
the same constraints as `debian/control` in practice, so I presume
support for this would also have some merit for you as well. At the
moment, my proposal is a high level idea so there is less concrete "how".
While doing the high level design, I remembered that `dgit` might be
adversely affected, so I figured I would reach out to you to hear what
your views are on it and where we can meet between what I want and what
`dgit` expects/wants. To keep things simple, I will only focus on
`debian/control` in the text below.
# Purpose
My motivation for this proposal is to enable us to enable to
tool-assisted packaging. This is not a novel idea; the `debdry`
prototype back in 2014 tried to achieve something similar (though with a
different approach than I what I think we will land on for this proposal).
My go to example is the `Section` field, for which the contributor is
expected to research which section the package should go in and then set
the `Section` field accordingly. Then you run `lintian` or `debputy
lint` and they will tell you "According to the package name, you should
have used `Section: foo` instead" and you get to correct the file.
Similar, for some language specific packaging, we have the tooling to
provide defaults for a number of things like description or
Build-Depends (which was supported by the `debdry` prototype).
I want to flip that on its head and have our tooling provide defaults
where possible. The human contributor should only spend effort on things
the tool cannot provide, or for which the tooling has a bug, or where
the package is too much of a special-case for the auto-detection to work.
That is the conceptual level of what I want to do and why I want to do
it. Note this is very much a "UX" (User eXperience) problem for me and
not a "It is not possible to do X"-problem. Our tooling could be a lot
better at assisting us which packaging (it has the knowledge/data), but
we as a project have painted ourselves into a corner where we made it
hard for us to use our tooling for it. So an alternative phrasing is
that I am looking for ways to paint ourselves out of that corner.
## Practicalities
One practicality that is relevant to understand early is that no matter
how this is sliced or diced, we end up with a (partial) generated
`debian/control` file for any package that would leverage the features
of this proposal.
Personally, I am strongly leaning towards a solution where the
`debian/control` ends up 100% generated for packages leveraging this,
since having `debian/control` being its own template gets complicated
and in all solutions I have found so far, there will be a delta between
the `git` tree and the generated `source package` (we are "just"
discussing how that delta looks and how it materializes) due to the
constraint 3 (see below). As in example, it is possible to generate an
augmented `debian/control` file (like `debian/control.full`), however
the `git` repo would not have `debian/control.full` while the source
package would have (so the delta between git and the source would remain
conceptually; we just move it to a different file).
In the text below, I will use `debian/control` as a generated file for
simplicity in wording.
# Constraints / Design criteria
I have the following constraints/design criteria in mind that I am
hoping to achieve or that I need to satisfy (in no particular order at
the moment):
1. `dpkg` will ideally provide the hook to do the generation, but I do
not think it will do the actual generation itself (that would be
dispatched some how).
Note: This is not aligned with the `dpkg` maintainers yet (I will
have that conversation separately when I am ready to discuss the "how").
Though a minimal prototype of this proposal can be done by
`dpkg-divert`'ing `dpkg-buildpackage` and `dpkg-source` with a small
script the generates `debian/control` if absent and uses `trap` to
remove the file again if generated (add some conditional logic to ensure
it only triggers for relevant sub commands, etc.). Therefore, I do not
see it as infeasible to have `dpkg` provide the conceptual feature. It
is a question of "how early" to generate/runtime patch the file rather
than "is it possible to generate/runtime patch the file?" (and do the
`dpkg` maintainers agree to the concrete way of doing it).
2. Backwards compatibility. I see three major aspects here
a. Anything working with a git checkout that does not involve
calling `dpkg-buildpackage` (or `dpkg-source`) or that might need to
read `debian/control` before invoking the tools. These are tools like
`debputy lsp/lint/reformat`, `cme`, `lintian-brush`, and `apt-get
build-dep`. I think `gbp import-dsc` and `dgit` might also end up here
to cope with the changes for different reasons.
Per definition these would have to be updated to cope. Though
what is really changing is that we formalize the ability to generate (or
runtime patch) the `debian/control`. There are already packages today
that generate their `debian/control` and most of these tools would now
be able to automatically detect this and work with such git checkouts.
Also, the roll out will de facto be done slowly over time in an
opt-in fashion, so it is not like they will break everything immediately.
b. Anything only looking at the generated package. These are tools
like `dak`, `lintian`, archive-qa tools, etc. These should work out of
the box (this is a side-effect of the requirement of contraint 3).
c. Anything that is involved during a package build (think
`debian/rules` and below). The solution here should ideally also be
backwards compatible as it is infeasible to have all those tools
changed. This is a choice of optimizing for adoption and because
`debian/control` must be materialized for `dpkg-source` anyway (see
constraint 4). Though, I am the maintainer of the primary packaging
helper tools, so if needed be, this design constraint can be dropped if
moving the full version to a separate file is necessary.
3. The `debian/control` must remain stable during build and during the
life-time of the package in the archive. This requirement comes out of
the FTP masters reject FAQ as authority but there many other reasons why
this is a desirable property (determinism, debugging, etc.)
This is one of the key reason why I want the source package to
include the full form of `debian/control`. Existing solutions involve
`debian/rules` failing if `debian/control` (which is extra work to write
and maintain that invariant). Including the full generated form and
avoiding generation for uploaded packages will both solve this
constraint but also simplify existing solutions.
4. Certain information must be available at `dpkg-source` time such as
`Priority`, `Section`, `Testsuite` (relevant if auto-generated
`debian/tests/control` is to be in scope), and the synopsis
(`Description`). Therefore, it is a non-option to move this logic / data
into build-time packaging helpers like `debhelper` that add extra fields
via `dh_gencontrol`. That would simply neuter how much assistance our
tooling can do to the point where this proposal would not change the
status quo.
5. The solution should **not** involve committing a generated or
augmented version of the `debian/control` to git.
I know it is possible to write code the validates the
`debian/control` is out of date (etc.), but I would prefer not having
people thinking about this as that becomes its own UX problem, because
then we have to introduce special-case CI/pre-commit logic that people
have to know about and validate works as intended.
**This might be the controversial part for `dgit`.**
My rationale here is that generated artifacts do not be long in
`git` and it would enable the `debian/control` to go out of sync with
the source (which is an error condition I would like to avoid).
6. When we get to the how, I would like the solution to be generic
enough to support package specific generators provided by the source
package (git checkout) itself, to support our existing `debian/control`
generators.
They would need to add themselves to the relevant hook and maybe
change how they are triggered (etc.). Obviously, the template source
(regardless of form) would belong in git in my view.
This implies that I want to support generating `debian/control` not
being its own source/template.
## Solution sketch
To avoid this being **too** abstract, here is a short conceptual
solution that demonstrate how the above solution could be implemented. A
minimal prototype could be done by dpkg-diverting dpkg-buildpackage with
the following pseudo code:
#!/bin/sh
can_generate_dctrl() {
... # Detect if the package opted in to generating `debian/control`
}
generate_dctrl() {
... # Dispatch to relevant generate
}
if [ ! -f debian/control ] && can_generate_dctrl ; then
trap 'rm -f debian/control' EXIT ERR
generate_dctrl > debian/control
fi
/usr/bin/dpkg-buildpackage.real "$@"
exit $?
A similar prototype could be done for `dpkg-source` for consumers that
run `dpkg-source` directly (with a bit more effort).
This prototype works from the assumption that `debian/control` once
generated is the "leading" form. This trivially satisfies Backwards
compats and the stable `d/control` (FTP Master auto-reject) criteria.
Though as stated, we can do alternatives as well if that would enable
`dgit` to support this work flow.
# Your views on this ...?
Thanks for reading through this and I hope you can help me with the
following:
1. Is this problematic for `dgit` to support?
2. Assuming it is problematic, where can we meet to have it palatable
for `dgit` while still achieving the principle goal that I want to solve?
a. The "no generated artifacts in git" is one I feel strongly for.
While I assume it would trivially resolve any concerns from the `dgit`
side, I am hoping we can come to a better solution without that one.
Like, what would it take for `dgit` to support this without waiving that
constraint?
b. The "debian/control must be stable" (FTP-master auto-reject)
constraint is not in my control to change and does not seem feasible to
change either, since the consequences would be unclear for the archive
at large.
