Re: Another take on package relationship substvars
> Can we also consider ${*:Built-Using} as typically seen in
> ${sphinxdoc:Built-Using}?
> This is another field that people keep forget adding. While missing
> this field is not severely harmful, having it automatically handled
> would be beneficial.
Automatic expansion of ${*:(Static-)Built-Using} would improve the
situation for variables managed by packaging helpers.
But for the record, dh-builtusing [1] already provides an
auto-definition of ${dh-builtusing:[DPSa-z]+} variables expanded by
the maintainer in the (Static-)Built-Using fields.
The use cases should rarely overlap, but the mechanisms seem
compatible. For example, dh_builtusing parses debian/*.substvars so
that dh_sphinxdoc may one day set
sphinxdoc:Built-Using=${dh-builtusing:libjs-sphinxdoc}.
[1] https://manpages.debian.org/testing/dh-builtusing/dh_builtusing.1.en.html
Splitting collectd dependencies [Was: Re: Another take on package relationship substvars]
None of this is relevant to the substvars discussion, but the collectd side is worth looking at on its own. On Sat, Feb 24, 2024 at 01:36:33PM +0100, Gioele Barabucci wrote: > On 24/02/24 11:26, Bernd Zeimetz wrote: > > Absolutely. collectd for example - otherwise you would install *all* > > plugin dependencies with collectd, which would be a big waste of space. > > > > The other option would be to make one packe per plugin as redhat does, > > but do we really want 20 packages with a single file? > > Yes, please. So that installing package collectd-foo ensures that all the > required dependencies are installed, instead of having to hunt them down (a > task better left to the package maintainers rather than the end users). There is a balance to be struck here. Adding one package per plugin is a lot of plugins and you often install multiple plugins together. It is not obvious that the benefit of splitting is worth the associated cost. I think there is a middle ground here. Having one package per plugin definitely does have advantage. However, consider the option of turning those packages virtual. So you'd have tons of collect-plugin-foo packages provided from collectd initially. Then, multiple plugins tend to use the same dependencies and some plugins tend to use no additional dependencies. The latter can just be left in the main package together with their provides. The former can be grouped together to say collectd-plugins-hardware or something that you wouldn't want on a virtual machine. Together with the plugins, you'd also move the provides and recommends (maybe upgraded to depends then). In particular, you can later restructure the plugins provided that downstreams only depend on your virtual collect-plugin-* packages rather than the underlying physical packages. An example where this has successfully been implemented with Depends and a small installation footprint is lighttpd. Helmut
Re: Another take on package relationship substvars
Hi!
On Fri, 2024-02-23 at 17:59:14 -0800, Steve Langasek wrote:
> One generic case that this doesn't handle is Essential: yes packages. For
> many of these, the ${shlibs:Depends} gets promoted in debian/control to
> Pre-Depends, not to Depends.
Ah! Good point.
I think the particular case of the Essential: yes and Pre-Depends
should really be handled by dpkg-shlibdeps. So I've now implemented a
new --package option for it, that will grab the package information
from debian/control, and if it is «Essential: yes» it will change the
default from Depends to Pre-Depends (in addition to also setting the
default Package-Type, so no need for explicit -t anymore), and the
package name will be added to -x. If -p is not passed, then it will
behave as now.
This, combined with the upcoming implicit substvar support should then
by default give the expected behavior (which can always be overridden).
Thanks,
Guillem
Re: Another take on package relationship substvars
Am 24. Februar 2024 11:26:56 MEZ schrieb Bernd Zeimetz :
>On Thu, 2024-02-22 at 20:52 +0100, IOhannes m zmölnig wrote:
>>
>> While I like the idea in general, I wonder how I could override these
>> automatic additions.
>> I think there are some packages that even demote `${shlibs:Depends}`
>> to Recommends.
>>
>
>Absolutely. collectd for example
Yes, that's what I had in mind.
mfh.her.fsr
IOhannes
Re: Another take on package relationship substvars
On Feb 22, IOhannes m zmölnig wrote:
> While I like the idea in general, I wonder how I could override these
> automatic additions.
> I think there are some packages that even demote `${shlibs:Depends}` to
> Recommends.
E.g. inn2 does:
override_dh_shlibdeps:
dh_shlibdeps --exclude=/usr/lib/news/bin/auth/passwd/auth_krb5 -- \
-dSuggests $D/usr/lib/news/bin/auth/passwd/auth_krb5 \
-dDepends
--
ciao,
Marco
signature.asc
Description: PGP signature
Re: Another take on package relationship substvars
Le Thu, Feb 22, 2024 at 08:52:36PM +0100, IOhannes m zmölnig a écrit :
> Am 22. Februar 2024 20:25:32 MEZ schrieb Boyuan Yang :
> >在 2024-02-22星期四的 19:32 +0100，Niels Thykier写道：
> >> I think our package helper tooling should just automatically aggregate
> >> all provided substvars of the format ${*:Depends} and append it the
> >> Depends field. Rinse and repeat for other relationship fields.
> >>
> >> The list of fields where this is applied would be curated, so it only
> >> applies to known relationship fields where we feel it makes sense. My
> >> starting list would be:
> >>
> >> * Any dependency field, that is: Pre-Depends, Depends, Recommends, and
> >> Suggests
> >>
> >> * The Provides field.
> >>
> >> I am omitting Breaks, Conflicts, and Replaces because I am not aware of
> >> any users of these at the moment. I am open to adding them, if there is
> >> a strong use-case.
> >
> >Can we also consider ${*:Built-Using} as typically seen in
> >${sphinxdoc:Built-Using}?
>
> While I like the idea in general, I wonder how I could override these
> automatic additions.
> I think there are some packages that even demote `${shlibs:Depends}` to
> Recommends.
This is sometime used to avoid circular dependencies between libxxx and
libxxx-bin packages, when a library can exec a binary in /usr/lib linked with
itself.
Since libxxx-bin binaries are always exec-ed by libxxx, there is no
need for them to depend on libxxx, while libxxx needs to depend on
libxxx-bin.
Demoting the shlibs:Depends prevent the otherwise circular dependency.
Cheers,
--
Bill.
Imagine a large red swirl here.
Re: Another take on package relationship substvars
Gioele Barabucci:
On 24/02/24 11:26, Bernd Zeimetz wrote:
On Thu, 2024-02-22 at 20:52 +0100, IOhannes m zmölnig wrote:
I think there are some packages that even demote `${shlibs:Depends}`
to Recommends.
Absolutely. collectd for example - otherwise you would install *all*
plugin dependencies with collectd, which would be a big waste of space.
The other option would be to make one packe per plugin as redhat does,
but do we really want 20 packages with a single file?
Yes, please. So that installing package collectd-foo ensures that all
the required dependencies are installed, instead of having to hunt them
down (a task better left to the package maintainers rather than the end
users).
PS: I volunteer to do the splitting.
Regards,
For the people following from home, collectd is a non-issue for this
proposal since it uses dpkg-shlibdeps to split the substvars into the
buckets it wants and therefore everything would "just work"(tm).
Whether collectd should be split might be a relevant conversation but
please take it in a separate thread or in the BTS. Thank you! :)
Best regards,
Niels
Re: Another take on package relationship substvars
On 24/02/24 11:26, Bernd Zeimetz wrote:
On Thu, 2024-02-22 at 20:52 +0100, IOhannes m zmölnig wrote:
I think there are some packages that even demote `${shlibs:Depends}`
to Recommends.
Absolutely. collectd for example - otherwise you would install *all*
plugin dependencies with collectd, which would be a big waste of space.
The other option would be to make one packe per plugin as redhat does,
but do we really want 20 packages with a single file?
Yes, please. So that installing package collectd-foo ensures that all
the required dependencies are installed, instead of having to hunt them
down (a task better left to the package maintainers rather than the end
users).
PS: I volunteer to do the splitting.
Regards,
--
Gioele Barabucci
Re: Another take on package relationship substvars
On Thu, 2024-02-22 at 20:52 +0100, IOhannes m zmölnig wrote:
>
> While I like the idea in general, I wonder how I could override these
> automatic additions.
> I think there are some packages that even demote `${shlibs:Depends}`
> to Recommends.
>
Absolutely. collectd for example - otherwise you would install *all*
plugin dependencies with collectd, which would be a big waste of space.
The other option would be to make one packe per plugin as redhat does,
but do we really want 20 packages with a single file?
Bernd
--
Bernd ZeimetzDebian GNU/Linux Developer
http://bzed.dehttp://www.debian.org
GPG Fingerprint: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F
Re: Another take on package relationship substvars
Steve Langasek:
Hi Niels,
On Thu, Feb 22, 2024 at 07:32:21PM +0100, Niels Thykier wrote:
[...]
I am omitting Breaks, Conflicts, and Replaces because I am not aware of any
users of these at the moment. I am open to adding them, if there is a strong
use-case.
One generic case that this doesn't handle is Essential: yes packages. For
many of these, the ${shlibs:Depends} gets promoted in debian/control to
Pre-Depends, not to Depends.
Maybe it would make sense to auto-aggregate these substvars, *IFF* there is
not already a reference to the substvar in question in the package stanza in
debian/control? This would provide adequate flexibility for any other
exceptions that might be out there, beyond the Pre-Depends case.
Cheers,
In case of a promotion, it does not matter if ${shlibs:Depends} is
applied twice (once in Depends and in Pre-Depends}. You get the
dependencies twice and then dpkg will clean up the duplicates in favor
of the "strongest" dependency[1]. The hard part is a demotion because
this duplication works against you.
Accordingly, the Essential: yes packages that leverage this technique
can just keep doing it without any changes or consequences as far as I
can tell.
Best regards,
Niels
[1] man:dpkg-gencontrol(1):
dpkg-gencontrol reads information from an unpacked Debian source tree
and generates a binary
package control file (which defaults to debian/tmp/DEBIAN/control);
during this process it will
simplify the relation fields.
Thus Pre-Depends, Depends, Recommends and Suggests are simplified in
this order by removing
dependencies which are known to be true according to the stronger
dependencies already parsed.
It will also remove any self-dependency (in fact it will remove any
dependency which evaluates
to true given the current version of the package as installed).
Logically it keeps the
intersection of multiple dependencies on the same package. The order of
dependencies is
preserved as best as possible: if any dependency must be discarded due
to another dependency
appearing further in the field, the superseding dependency will take the
place of the discarded
one.
Re: Another take on package relationship substvars
Hi Niels,
On Thu, Feb 22, 2024 at 07:32:21PM +0100, Niels Thykier wrote:
> When I am talking about package relationship substvars, I mean basically any
> substvar of the format ${*:} where Field is a relationship field such
> as Depends, Pre-Depends, etc.
[...]
> I think our package helper tooling should just automatically aggregate all
> provided substvars of the format ${*:Depends} and append it the Depends
> field. Rinse and repeat for other relationship fields.
> The list of fields where this is applied would be curated, so it only
> applies to known relationship fields where we feel it makes sense. My
> starting list would be:
> * Any dependency field, that is: Pre-Depends, Depends, Recommends, and
>Suggests
> * The Provides field.
> I am omitting Breaks, Conflicts, and Replaces because I am not aware of any
> users of these at the moment. I am open to adding them, if there is a strong
> use-case.
One generic case that this doesn't handle is Essential: yes packages. For
many of these, the ${shlibs:Depends} gets promoted in debian/control to
Pre-Depends, not to Depends.
Maybe it would make sense to auto-aggregate these substvars, *IFF* there is
not already a reference to the substvar in question in the package stanza in
debian/control? This would provide adequate flexibility for any other
exceptions that might be out there, beyond the Pre-Depends case.
Cheers,
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org
signature.asc
Description: PGP signature
Re: Another take on package relationship substvars
On Thu, Feb 22, 2024 at 09:40:22PM +0100, Matthias Klose wrote:
> Package: libgcc-13-dev
> Recommends: ${dep:libcdev}
> Depends: gcc-13-base (= ${gcc:Version}), ${dep:libgcc}, ${dep:libssp},
> ${dep:libgomp}, ${dep:libitm},
> ${dep:libatomic}, ${dep:libbtrace}, ${dep:libasan}, ${dep:liblsan},
> ${dep:libtsan}, ${dep:libubsan}, ${dep:libhwasan}, ${dep:libvtv},
> ${dep:libqmath}, ${dep:libunwinddev}, ${shlibs:Depends}, ${misc:Depends}
>
> Some of those are undefined depending on the architecture. And most of
> these are unused in other packages, however passing every macro into a
> package, independent if it's used or not.
>
> Not seeing any benefit in this feature (hard failure).
Your examples aren't what Niels refers to as "relationship substvars",
so aren't affected either way by this proposal.
--
Colin Watson (he/him) [cjwat...@debian.org]
Re: Another take on package relationship substvars
On 2024-02-23, Niels Thykier wrote:
> If it was to happen, I suspect that ${shlibs:Depends} would not be the
> best argument. First off, dpkg-shlibdeps has infrastructure to
> selectively demote scanned elf binaries to a different substvar.
> Secondly, I struggle to think of a real world case, where demoting
> ${shlibs:Depends} would matter a lot. That is, a case where the right
> answer is not just splitting these binaries into a separate package if
> they are that consuming in dependencies.
I have seen it as real life usecases where there is a main binary that
has plugins for integrations with , and just having those
plugins work when the integration point is needed.
Usually though, they are using dpkg-shlibdeps to selectively demote
scanned elfs to a different substvar, given that the 'main binary' is
already a elf binary with some dependencies, so doing it with 'all of
them' is likely unlikely for that case.
For example:
dpkg-shlibdeps -Tdebian/dirmngr.substvars -dRecommends
debian/dirmngr/usr/lib/gnupg/dirmngr_ldap -dDepends
debian/dirmngr/usr/bin/dirmngr
http://codesearch.debian.net/search?q=-dRecommends=1
http://codesearch.debian.net/search?q=-dSuggests=1
/Sune
Re: Another take on package relationship substvars
Guillem Jover: Hi! On Thu, 2024-02-22 at 19:32:21 +0100, Niels Thykier wrote: [...] Right, this is annoying. This was actually brought up some time ago (2010) in debian-devel as part of #597340. There was not much reaction at the time (one good, a couple bad). I reinvented a decade old idea that I forgotten existed, but very much remember having read in the past now that I see it again. # How do we get here? High level, any helper framework above dpkg can provide this feature if it wants to. It is basically a question of parsing d/*.substvars to figure out which substvars are relevant, compute a new field from substvars file + the static provided data from d/control, and then passing -DField=<...> to dpkg-gencontrol. There are probably a bunch of nitty gritty implementation details in practice, but the overall approach stands. I think each helper stack (debhelper, cdbs, debputy, etc.) would define their own rule set for how to get there. As an example, I could see this be the new default in the next debhelper compat level, possibly with a way to be early adopter on existing compat levels. If we agree we want this implicit mechanism of adding some substvars to fields, then doing it in helpers seems like a suboptimal path, as each one would need to replicate the logic, and implement substvar substitution logic. I think if this is to be done, it should be done at the dpkg-dev level. [...] I am happy to look into moving parts of the responsibility of this into dpkg. Though, I would like to move that part of the discussion to debian-dpkg as for the vast majority this is implementation detail. For the hand-full of people that wants to be in the discussion, I don't think it is a major ask to have them track debian-dpkg for a short while. ## Alternative solutions We could also make unused substvars a hard failure (FTBFS). Personally, I feel auto-managing them will be less painful for users. But if the consensus goes down this direction, then I would be behind it as it is still better than the status quo in my book. I also implemented in that branch another operator !=, which marks a substvar as required, so failure to use it will generate an error. I might have to move the error emission into some other function that is not called warn_about_unused() though… :) Ok. Based on Simon's follow up, I suspect we will not be going down this route. Obviously, dpkg can still support the feature either way. :) That is my proposal for how Debian contributors can spend less mental effort tracking relationship substvars and use their new spare mental capacity providing value for our users. Thanks for reviewing the proposal and any feedback you might have for making relationship substvars be less annoying for users and Debian as a whole. Personally I think I agree with this being a problem and with a potential solution for this based on implicit substvars. [...] Thanks, Guillem Thanks for the feedback. (The snipped part I believe I already answered above) Best regards, Niels
Re: Another take on package relationship substvars
IOhannes m zmölnig:
[...]
While I like the idea in general, I wonder how I could override these automatic
additions.
I think there are some packages that even demote `${shlibs:Depends}` to
Recommends.
mfh.her.fsr
IOhannes
I had the same conceptual concern when I originally thought about this
proposal. However, first off, I am not aware of wide-spread usage of
people demoting entire substvars.
If it was to happen, I suspect that ${shlibs:Depends} would not be the
best argument. First off, dpkg-shlibdeps has infrastructure to
selectively demote scanned elf binaries to a different substvar.
Secondly, I struggle to think of a real world case, where demoting
${shlibs:Depends} would matter a lot. That is, a case where the right
answer is not just splitting these binaries into a separate package if
they are that consuming in dependencies.
Additionally, Colin replied with the solution of "tweaking" the
substvars file. I was aware of that solution, but I did not want to
promote it as a general solution, so I omitted it from my ideal email.
If we find ourselves regularly "correcting" substvars in every package,
then I think we owe ourselves to consider if we have a bug that needs
fixing. Not sure we are there yet given a single example (the one in
Colin's email). However, I be interested to know how frequent this
pattern is and whether we therefore should look at fixing this at a
different layer.
Best regards,
Niels
Re: Another take on package relationship substvars
Simon McVittie:
On Thu, 22 Feb 2024 at 20:43:21 +0100, Niels Thykier wrote:
Simon McVittie:
On Thu, 22 Feb 2024 at 19:32:21 +0100, Niels Thykier wrote:
We could also make unused substvars a hard failure (FTBFS).
I'd prefer not this
Reminder: My proposal only covers ${foo:Depends} and similar substvars. The
first example you present uses substvars that do not match that pattern.
Sorry, did you mean that your alternate proposal is: we could make unused
substvars **that match *:Depends** (and the other related patterns) a hard
failure, without affecting substvars not matching that pattern? If so,
that wasn't obvious to me!
Indeed, that is what I meant. I see that others had the same confusion,
which could probably avoided if I had said "relationship substvars"
instead. Sorry for the confusion.
If the scope of "unused substvars => hard failure" is limited to
foo:Depends and so on, then yes, I agree that the concern I described
doesn't apply.
Thanks for confirming.
Another reason to be cautious about making missing foo:Depends a hard
failure is that it would mean tools usually can't add new :Depends without
either gating it behind a debhelper compat level bump (or equivalent),
or making potentially large numbers of dependent packages regress
(usually discouraged). So I still prefer your initial proposal.
smcv
Thank you. I agree with this counter argument. That seems like a very
annoying problem to deal with that will likely cause a lot of extra
"infrastructure" that I think nobody wants.
Thanks for the feedback. It was very useful :)
Best regards,
Niels
Re: Another take on package relationship substvars
Hi! On Thu, 2024-02-22 at 23:14:13 +0100, gregor herrmann wrote: > On Thu, 22 Feb 2024 19:32:21 +0100, Niels Thykier wrote: > > If you forget to add a susbtvars that you should added, it is a latent RC > > bug with only a warning from dpkg-gencontrol that you might miss if you grab > > a coffee while waiting for the build to complete. If you add one that is > > not provided, you get a warning from dpkg-gencontrol that will nag you when > > you *don't* go for coffee while waiting for the build. > > I can relate a lot to this paragraph and would append: With or > without the coffee, each time I see a warning from dpkg-gencontrol I > have to think for some time about which of the two possible problems > is meant (substvar defined in d/control but not generated, or > substvar generated but missing in d/control), and that is really > annoying. If the warnings are not clear, I'm happy to further reword them to try to make them so. I think they were previously even more confusing! If you or anyone else has wording suggestions (which are ideally still succinct) that would be great. Thanks, Guillem
Re: Another take on package relationship substvars
Hi!
On Thu, 2024-02-22 at 19:32:21 +0100, Niels Thykier wrote:
> Our current way of dealing with package relationship substvars such as
> ${misc:Depends} has been annoying me for a while. As it is, we are stuck in
> this way setup where the "Depends" field in debian/control is de facto
> mandatory. It is an RC bug (waiting to happen) if you omit ${misc:Depends}
> or, for arch:any packages, ${shlib:Depends} from the dependency field.
>
> Personally, I feel we have this weird setup where we have tooling that will
> scream at you if you forget. But I lack a compelling reason for why we need
> tooling to remind us when we could just as well have tooling solve this
> problem for us.
> Additionally, our tooling for reminding us rarely knows about anything but
> the 2-3 standard substvars everyone uses even though we should get reminders
> for other substvars like ${python3:Depends}, ${gir:Depends}, etc.
> Particularly the rules for when a substvars is applicable turns out to be
> quite non-trivial.
>
> I have given this some thought and I think I have a conceptually idea for
> how to solve this, which I will go over in this email.
Right, this is annoying. This was actually brought up some time
ago (2010) in debian-devel as part of #597340. There was not much
reaction at the time (one good, a couple bad).
> # How do we get here?
>
> High level, any helper framework above dpkg can provide this feature if it
> wants to. It is basically a question of parsing d/*.substvars to figure out
> which substvars are relevant, compute a new field from substvars file + the
> static provided data from d/control, and then passing -DField=<...> to
> dpkg-gencontrol.
>
> There are probably a bunch of nitty gritty implementation details in
> practice, but the overall approach stands.
>
> I think each helper stack (debhelper, cdbs, debputy, etc.) would define
> their own rule set for how to get there. As an example, I could see this be
> the new default in the next debhelper compat level, possibly with a way to
> be early adopter on existing compat levels.
If we agree we want this implicit mechanism of adding some substvars
to fields, then doing it in helpers seems like a suboptimal path, as
each one would need to replicate the logic, and implement substvar
substitution logic. I think if this is to be done, it should be done
at the dpkg-dev level.
Some time ago Dpkg::Substvars got support for optional substvars via
the ?= operator (prompted by you! :). I've started a WIP dpkg branch
and will try to finish implementing something for this tomorrow:
https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=next/substvars%2b%2b
For now I've added a new (tentative) operator $=, that would declare
the variable is to be implicitly applied. Then the generator can
select whether to use old or new semantics.
> ## Alternative solutions
>
> We could also make unused substvars a hard failure (FTBFS). Personally, I
> feel auto-managing them will be less painful for users. But if the consensus
> goes down this direction, then I would be behind it as it is still better
> than the status quo in my book.
I also implemented in that branch another operator !=, which marks a
substvar as required, so failure to use it will generate an error. I
might have to move the error emission into some other function that
is not called warn_about_unused() though… :)
> That is my proposal for how Debian contributors can spend less mental effort
> tracking relationship substvars and use their new spare mental capacity
> providing value for our users. Thanks for reviewing the proposal and any
> feedback you might have for making relationship substvars be less annoying
> for users and Debian as a whole.
Personally I think I agree with this being a problem and with a
potential solution for this based on implicit substvars. But I'm not
sure I agree at the layer this needs to be implemented. :) I think
going with explicit operators might be what was missing from the
earlier proposal and calm people's qualms about the too much magic
perception. And if there's no major push back on the overall idea
I'm happy to pursue this directly in dpkg-dev.
Thanks,
Guillem
Re: Another take on package relationship substvars
On 2024-02-22 12:32, Niels Thykier wrote: I am omitting Breaks, Conflicts, and Replaces because I am not aware of any users of these at the moment. I am open to adding them, if there is a strong use-case. I think you should include them (and Enhances as Sam Hartman mentioned) for consistency. It seems potentially confusing if some of the relationships fields are included and some are not. This proposal sounds generally good. I'll have to defer to others who know more about potential corner cases. -- Richard OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Another take on package relationship substvars
On Thu, 22 Feb 2024 19:32:21 +0100, Niels Thykier wrote:
> I think our package helper tooling should just automatically aggregate all
> provided substvars of the format ${*:Depends} and append it the Depends
> field. Rinse and repeat for other relationship fields.
I very much like this proposal.
And:
> If you forget to add a susbtvars that you should added, it is a latent RC
> bug with only a warning from dpkg-gencontrol that you might miss if you grab
> a coffee while waiting for the build to complete. If you add one that is
> not provided, you get a warning from dpkg-gencontrol that will nag you when
> you *don't* go for coffee while waiting for the build.
I can relate a lot to this paragraph and would append: With or
without the coffee, each time I see a warning from dpkg-gencontrol I
have to think for some time about which of the two possible problems
is meant (substvar defined in d/control but not generated, or
substvar generated but missing in d/control), and that is really
annoying.
Cheers,
gregor
--
.''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org
: :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06
`. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
`-
signature.asc
Description: Digital Signature
Re: Another take on package relationship substvars
> "Niels" == Niels Thykier writes:
Niels> # The proposal
Niels> I think our package helper tooling should just automatically
Niels> aggregate all provided substvars of the format ${*:Depends}
Niels> and append it the Depends field. Rinse and repeat for other
Niels> relationship fields.
Niels> The list of fields where this is applied would be curated, so
Niels> it only applies to known relationship fields where we feel it
Niels> makes sense. My starting list would be:
Niels> * Any dependency field, that is: Pre-Depends, Depends,
Niels> Recommends, and Suggests
Niels> * The Provides field.
I think you should explicitly mention Enhances, and suggest that it be
included even though I am unaware of uses. I'd be fine though with
saying Enhances is not currently included.
I think this proposal makes a lot of sense.
Re: Another take on package relationship substvars
On 22.02.24 20:43, Niels Thykier wrote:
Simon McVittie:
We could also make unused substvars a hard failure (FTBFS).
I'd prefer not this, because this would be really painful if you're
using substvars for something other than a simple list of dependencies,
like gobject-introspection does:
[...]
Reminder: My proposal only covers ${foo:Depends} and similar substvars.
The first example you present uses substvars that do not match that
pattern. Therefore initial reaction is that this effectively ends up
being a straw man argument because it does not correctly account for the
scope of this proposal.
I'd prefer not to put lots of glue into debian/rules to generate these
substvars for precisely the packages that use them, because that's
another thing to keep in sync between d/control and d/rules (so at the
moment I'm ignoring dpkg-gencontrol's complaints about these substvars
being mentioned in some but not all binary packages).
Conversely, I could have implemented most of this by generating
per-package
${local:Provides} and ${local:Depends}, but if I did that, then I'd
have to
encode half of each binary package's dependencies in d/rules.
I'd prefer to be able to look at d/control and see the overall "shape" of
the interdependencies - "this Depends on binutils-something:any,
gcc-something, ..." - rather than having that information in d/rules.
smcv
Could you please clarify if the rest of this argument is still relevant
given the example seemed to be off? I am happy to entertain it, but I
would prefer working from a state where we both agree that it is
relevant, before I invest effort to understand it.
Best regards,
Niels
Package: libgcc-13-dev
Recommends: ${dep:libcdev}
Depends: gcc-13-base (= ${gcc:Version}), ${dep:libgcc}, ${dep:libssp},
${dep:libgomp}, ${dep:libitm},
${dep:libatomic}, ${dep:libbtrace}, ${dep:libasan}, ${dep:liblsan},
${dep:libtsan}, ${dep:libubsan}, ${dep:libhwasan}, ${dep:libvtv},
${dep:libqmath}, ${dep:libunwinddev}, ${shlibs:Depends}, ${misc:Depends}
Some of those are undefined depending on the architecture. And most of
these are unused in other packages, however passing every macro into a
package, independent if it's used or not.
Not seeing any benefit in this feature (hard failure).
Matthias
Re: Another take on package relationship substvars
On Thu, Feb 22, 2024 at 08:52:36PM +0100, IOhannes m zmölnig wrote:
> Am 22. Februar 2024 20:25:32 MEZ schrieb Boyuan Yang :
> >在 2024-02-22星期四的 19:32 +0100，Niels Thykier写道：
> >> I am omitting Breaks, Conflicts, and Replaces because I am not aware of
> >> any users of these at the moment. I am open to adding them, if there is
> >> a strong use-case.
Once upon a time, dh_suidmanager recommended that people add "Conflicts:
suidmanager (<< 0.50)" when migrating away from it to normal
statoverrides. With your proposal, I can imagine it having made sense
to do that (or at least something similar) using substvars. And while
dbgsym control files aren't actually generated using substvars,
dh_gencontrol does use -DReplaces and -DBreaks when generating them, and
it's not too hard to imagine something similar being needed for some
kind of automatic transition.
I'd include these three fields, if you're going to do this. I think I'm
in favour, as long as testing indicates that it doesn't cause much
breakage.
> While I like the idea in general, I wonder how I could override these
> automatic additions.
> I think there are some packages that even demote `${shlibs:Depends}` to
> Recommends.
That rings a bell, but I can't think where it was.
One thing I've done occasionally when I had no better option is to
postprocess the output of dpkg-shlibdeps. For example:
https://salsa.debian.org/ssh-team/openssh/-/blob/b9671cc7/debian/adjust-openssl-dependencies
https://salsa.debian.org/ssh-team/openssh/-/blob/b9671cc7/debian/rules#L217-219
(Possibly obsolete now - I should check. But anyway.)
If you really had to, then postprocessing debian/*.substvars to remove a
particular field would be simpler than that - probably just two lines in
debian/rules. I think that would be tolerable if the need is as rare as
I think it is, though of course it'd be worth documenting.
--
Colin Watson (he/him) [cjwat...@debian.org]
Re: Another take on package relationship substvars
Am 22. Februar 2024 20:25:32 MEZ schrieb Boyuan Yang :
>在 2024-02-22星期四的 19:32 +0100，Niels Thykier写道：
>> I think our package helper tooling should just automatically aggregate
>> all provided substvars of the format ${*:Depends} and append it the
>> Depends field. Rinse and repeat for other relationship fields.
>>
>> The list of fields where this is applied would be curated, so it only
>> applies to known relationship fields where we feel it makes sense. My
>> starting list would be:
>>
>> * Any dependency field, that is: Pre-Depends, Depends, Recommends, and
>> Suggests
>>
>> * The Provides field.
>>
>> I am omitting Breaks, Conflicts, and Replaces because I am not aware of
>> any users of these at the moment. I am open to adding them, if there is
>> a strong use-case.
>
>Can we also consider ${*:Built-Using} as typically seen in
>${sphinxdoc:Built-Using}?
While I like the idea in general, I wonder how I could override these automatic
additions.
I think there are some packages that even demote `${shlibs:Depends}` to
Recommends.
mfh.her.fsr
IOhannes
Re: Another take on package relationship substvars
On Thu, 22 Feb 2024 at 20:43:21 +0100, Niels Thykier wrote:
> Simon McVittie:
> > On Thu, 22 Feb 2024 at 19:32:21 +0100, Niels Thykier wrote:
> > > We could also make unused substvars a hard failure (FTBFS).
> >
> > I'd prefer not this
>
> Reminder: My proposal only covers ${foo:Depends} and similar substvars. The
> first example you present uses substvars that do not match that pattern.
Sorry, did you mean that your alternate proposal is: we could make unused
substvars **that match *:Depends** (and the other related patterns) a hard
failure, without affecting substvars not matching that pattern? If so,
that wasn't obvious to me!
If the scope of "unused substvars => hard failure" is limited to
foo:Depends and so on, then yes, I agree that the concern I described
doesn't apply.
Another reason to be cautious about making missing foo:Depends a hard
failure is that it would mean tools usually can't add new :Depends without
either gating it behind a debhelper compat level bump (or equivalent),
or making potentially large numbers of dependent packages regress
(usually discouraged). So I still prefer your initial proposal.
smcv
Re: Another take on package relationship substvars
Quoting Boyuan Yang (2024-02-22 20:25:32)
> 在 2024-02-22星期四的 19:32 +0100，Niels Thykier写道：
> > I think our package helper tooling should just automatically aggregate
> > all provided substvars of the format ${*:Depends} and append it the
> > Depends field. Rinse and repeat for other relationship fields.
> >
> > The list of fields where this is applied would be curated, so it only
> > applies to known relationship fields where we feel it makes sense. My
> > starting list would be:
> >
> > * Any dependency field, that is: Pre-Depends, Depends, Recommends, and
> > Suggests
> >
> > * The Provides field.
> >
> > I am omitting Breaks, Conflicts, and Replaces because I am not aware of
> > any users of these at the moment. I am open to adding them, if there is
> > a strong use-case.
>
> Can we also consider ${*:Built-Using} as typically seen in
> ${sphinxdoc:Built-Using}?
> This is another field that people keep forget adding. While missing
> this field is not severely harmful, having it automatically handled
> would be beneficial.
...and related to that, also ${*:Static-Built-Using} which is generated
by Rust (and, I seem to recall having read somewhere, Go) tooling.
- Jonas
--
* Jonas Smedegaard - idealist & Internet-arkitekt
* Tlf.: +45 40843136 Website: http://dr.jones.dk/
* Sponsorship: https://ko-fi.com/drjones
[x] quote me freely [ ] ask before reusing [ ] keep private
signature.asc
Description: signature
Re: Another take on package relationship substvars
Boyuan Yang:
[...]
Can we also consider ${*:Built-Using} as typically seen in
${sphinxdoc:Built-Using}?
This is another field that people keep forget adding. While missing
this field is not severely harmful, having it automatically handled
would be beneficial.
Thanks,
Boyuan Yang
Personally, I see no problem with adding this provided the current use
of it correctly matches that of the Built-Using field (which is for
license requirements and not just static-linking as far as I know).
By the same argument, we can add Static-Built-Using as well (or whatever
field people are using to record static linking relationships), if any
tool provides such a variable.
Best regards,
Niels
Re: Another take on package relationship substvars
Simon McVittie:
On Thu, 22 Feb 2024 at 19:32:21 +0100, Niels Thykier wrote:
I think our package helper tooling should just automatically aggregate all
provided substvars of the format ${*:Depends} and append it the Depends
field. Rinse and repeat for other relationship fields.
I recently added ${gir:Provides} to dh_girepository, and the proposed
feature would have meant instant wide adoption without source changes.
So I like this plan!
Thanks for the feedback.
I was not aware that ${gir:Provides} was new. :) But indeed, I was
hoping the proposal would simplify this kind of roll out in the future!
The only thing I'm wary of is that if the helper tools might sometimes
generate a substvar that is well-meaning but inappropriate (even in corner
cases), we might need an off-switch for this behaviour. At the moment you
can just not add the substvar (and silence or ignore the lint warnings),
but the proposed change would make that impossible.
In my original proposal, I did cover how to make all sorts of exceptions
for this case. But I am not aware of a real case where this matters.
So before I invest in a corner case that might not even exist, is anyone
aware of this being a regular case that might warrant general tool
support? Or we are trying to plug a hole that never was?
On the dependency side, dpkg-shlibdeps already has -dRecommends, -dSuggests
and so on, but not all other helpers have this (e.g. #934893), and
maybe it would be useful to have a conventional option to turn off the
dependency-generation altogether in the hopefully rare cases where it
does more harm than good?
Again, before we invest a lot in creating an API for features for this,
do we see cases that need fixing that cannot "just" be solving by
improving the tool providing the substvars itself?
On the Provides side, I don't think there's any similar convention.
We could also make unused substvars a hard failure (FTBFS).
I'd prefer not this, because this would be really painful if you're
using substvars for something other than a simple list of dependencies,
like gobject-introspection does:
[...]
Reminder: My proposal only covers ${foo:Depends} and similar substvars.
The first example you present uses substvars that do not match that
pattern. Therefore initial reaction is that this effectively ends up
being a straw man argument because it does not correctly account for the
scope of this proposal.
I'd prefer not to put lots of glue into debian/rules to generate these
substvars for precisely the packages that use them, because that's
another thing to keep in sync between d/control and d/rules (so at the
moment I'm ignoring dpkg-gencontrol's complaints about these substvars
being mentioned in some but not all binary packages).
Conversely, I could have implemented most of this by generating per-package
${local:Provides} and ${local:Depends}, but if I did that, then I'd have to
encode half of each binary package's dependencies in d/rules.
I'd prefer to be able to look at d/control and see the overall "shape" of
the interdependencies - "this Depends on binutils-something:any,
gcc-something, ..." - rather than having that information in d/rules.
smcv
Could you please clarify if the rest of this argument is still relevant
given the example seemed to be off? I am happy to entertain it, but I
would prefer working from a state where we both agree that it is
relevant, before I invest effort to understand it.
Best regards,
Niels
Re: Another take on package relationship substvars
在 2024-02-22星期四的 19:32 +0100，Niels Thykier写道：
> I think our package helper tooling should just automatically aggregate
> all provided substvars of the format ${*:Depends} and append it the
> Depends field. Rinse and repeat for other relationship fields.
>
> The list of fields where this is applied would be curated, so it only
> applies to known relationship fields where we feel it makes sense. My
> starting list would be:
>
> * Any dependency field, that is: Pre-Depends, Depends, Recommends, and
> Suggests
>
> * The Provides field.
>
> I am omitting Breaks, Conflicts, and Replaces because I am not aware of
> any users of these at the moment. I am open to adding them, if there is
> a strong use-case.
Can we also consider ${*:Built-Using} as typically seen in
${sphinxdoc:Built-Using}?
This is another field that people keep forget adding. While missing
this field is not severely harmful, having it automatically handled
would be beneficial.
Thanks,
Boyuan Yang
signature.asc
Description: This is a digitally signed message part
Re: Another take on package relationship substvars
On Thu, 22 Feb 2024 at 19:32:21 +0100, Niels Thykier wrote:
> I think our package helper tooling should just automatically aggregate all
> provided substvars of the format ${*:Depends} and append it the Depends
> field. Rinse and repeat for other relationship fields.
I recently added ${gir:Provides} to dh_girepository, and the proposed
feature would have meant instant wide adoption without source changes.
So I like this plan!
The only thing I'm wary of is that if the helper tools might sometimes
generate a substvar that is well-meaning but inappropriate (even in corner
cases), we might need an off-switch for this behaviour. At the moment you
can just not add the substvar (and silence or ignore the lint warnings),
but the proposed change would make that impossible.
On the dependency side, dpkg-shlibdeps already has -dRecommends, -dSuggests
and so on, but not all other helpers have this (e.g. #934893), and
maybe it would be useful to have a conventional option to turn off the
dependency-generation altogether in the hopefully rare cases where it
does more harm than good?
On the Provides side, I don't think there's any similar convention.
> We could also make unused substvars a hard failure (FTBFS).
I'd prefer not this, because this would be really painful if you're
using substvars for something other than a simple list of dependencies,
like gobject-introspection does:
Package: gobject-introspection
Depends: binutils-${local:DEB-HOST-GNU-TYPE-DASHES}:any,
gcc-${local:DEB-HOST-GNU-TYPE-DASHES},
gobject-introspection-${local:DEB-HOST-ARCH-OS}-${local:DEB-HOST-ARCH-ENDIAN}-endian
(= ${binary:Version}),
...
Description: ...
... The versions of these tools with
architecture name prefixes, such as ${local:DEB-HOST-GNU-TYPE}-g-ir-scanner,
...
I'd prefer not to put lots of glue into debian/rules to generate these
substvars for precisely the packages that use them, because that's
another thing to keep in sync between d/control and d/rules (so at the
moment I'm ignoring dpkg-gencontrol's complaints about these substvars
being mentioned in some but not all binary packages).
Conversely, I could have implemented most of this by generating per-package
${local:Provides} and ${local:Depends}, but if I did that, then I'd have to
encode half of each binary package's dependencies in d/rules.
I'd prefer to be able to look at d/control and see the overall "shape" of
the interdependencies - "this Depends on binutils-something:any,
gcc-something, ..." - rather than having that information in d/rules.
smcv
