Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Felipe Sateler]

On Tue, 25 Jan 2022 21:38:01 +0100, Vincent Bernat wrote:

> 
> I think we should forego the NEW queue. If people want to check
> packages, they can do it once they are in unstable with regular bugs.
> Current checks are partly done by Lintian and I suppose people could
> watch new Lintian warnings and detect bad packages quickly. This could
> be done when src is not NEW as a test. People could loose their upload
> rights if they are caught abusing the system (and get DM rights for some
> selected packages instead).

+1. If we have a good remediation mechanism, this bottleneck is not 
necessary.

Re: Do we need to hide packages in NEW queue (Was: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not))

[Francesco Poli]

On Wed, 26 Jan 2022 07:38:10 +0100 Andreas Tille wrote:

> Am Tue, Jan 25, 2022 at 01:45:11PM -0800 schrieb Russ Allbery:
[...]
> > The question, which keeps being raised in part
> > because I don't think it's gotten a good answer, is what the basis is for
> > treating copyright and licensing bugs differently than any other bug in
> > Debian?

I thought the basis was the fact that copyright and licensing bugs may
have bad legal consequences (lawsuits against the Project for
distributing legally undistributable packages, things like that), while
technical bugs do not cause issues with lawyers and are, in this sense,
"easier" to fix.
The consequences of introducing a "legally botched" package into the
archive are thus harder to undo, with respect to introducing a
technically flawed package...

> > 
> > The need for pre-screening was obvious when we had export control issues,
> > but my understanding is that those have gone away.  Are we working from
> > legal advice telling us that this pre-screening is required for some legal
> > purpose?  If so, is it effective for the legal purpose at which it is
> > aimed?  Is this system left over from old advice?  Have we checked our
> > assumptions recently?

I am under the impression that the pre-screening in the NEW queue is an
attempt to catch legal issues *before* the package is introduced into
the archive.
As far as I remember, the FTP masters are the people responsible for
what the Debian Project distributes through its archive...

Is this wrong (or no longer valid)?

[...]
> > NEW processing is a lot of friction for the project as a whole and a lot
> > of work for the ftp team.  If we were able to do less work at the cost of
> > a minimal increase in bugs, or at the cost of handling bugs a bit
> > differently, maybe that would be a good thing?
> > 
> > In other words, it's unclear what requirements we're attempting to meet
> > and what the basis of those requirements is, which makes it hard to have a
> > conversation about whether the current design is the best design for the
> > problem we're trying to solve.
> 
> I'm CCing debian-legal for this branch of the discussion (but I do not
> read this list and think keeping debian-devel in the row is a good idea).

Personally, I think the legal pre-screening by the FTP masters in the
NEW queue is useful and should be kept.

In fact, I wish the pre-screening were stricter.

I've seen cases, where a bug is reported against a legally
undistributable package and the issue is left unaddressed for ages with
nobody apparently caring enough.
Maybe it's better, if such issues are addressed *before* the package is
accepted into the archive...


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
. Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE


pgpWiRMgvfmm3.pgp
Description: PGP signature

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Alec Leamas]

Hi,

Not a DD, still raising my voice. I'm *not* advocating that Fedora's 
processes are "better", just trying to add ideas.


On 26/01/2022 11:43, Adam Borowski wrote:

On Tue, Jan 25, 2022 at 09:38:01PM +0100, Vincent Bernat wrote:



I think we should forego the NEW queue. If people want to check
packages, they can do it once they are in unstable with regular bugs.


Without the NEW queue, there would be no point at which packaging receives
any sort of review.  I'd prefer Debian to deliver at least some level of
quality.


Perhaps wrong to focus on the queue as such. The focus should be the 
need for a manual review -- this is IMHO the important point.


The current ftpmaster review model is somehow modeled after a 
"supervisor" idea. Fedora uses peer reviews. The advantage is the 
incentive to make reviews:  I can review your package if you review 
mine. One could of course imagine that this would lead to sloppy 
reviews. However, this is not my experience.


It also means a more transparent process.



Otherwise, we'd fall to the level of NPM.  And there's ample examples what
that would mean.


Indeed.


Current checks are partly done by Lintian and I suppose people could
watch new Lintian warnings and detect bad packages quickly.


Lintian is just a dumb machine that can ease human reviews but not replace
them.



Yes. It's interesting to compare to Fedora's tooling fedora-review which 
has another focus: It outputs list of things to check when reviewing a 
package. Some of those are automatically checked, others are just a 
checkbox which should be manually checked. Lintian is a good tool, but 
not IMHO a review support.




This could be done when src is not NEW as a test.


I've managed to trample upon someone else's package just yesterday -- and it
escaped automated checks because a binary of that name already existed in
the archive, just not on any arch which I test.



Yes, one of those manual checks...

On 26/01/2022 11:29, Adam Borowski wrote:

> For practical reasons we have to obey the laws, no matter how oppressive
> they are.  But I don't see why we should do more than eg. Fedora which
> has corporate backing with an actual legal team.

Also note that this legal team *not* is used to review all packages. 
Instead, they are a resource which are contacted by packagers when we 
need advice. The typical situation is in a (peer) review where things 
cannot be settled. The legal team also files bugs as required, and 
maintain the packaging manual's copyright part.


Of course, this creates a very different social relation between the 
legal team and the rest.


Just my 5 öre
--alec

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Gard Spreemann]

Adam Borowski  writes:

> On Tue, Jan 25, 2022 at 09:38:01PM +0100, Vincent Bernat wrote:
>> For me, the copyright check is just a bad excuse. People upload
>> non-distributable stuff everywhere and it seems the world continue to go
>> round. What amount of non-distributable packages is stopped by the NEW
>> queue?
>> 
>> I think we should forego the NEW queue. If people want to check
>> packages, they can do it once they are in unstable with regular bugs.
>
> Without the NEW queue, there would be no point at which packaging receives
> any sort of review.  I'd prefer Debian to deliver at least some level of
> quality.
>
> Otherwise, we'd fall to the level of NPM.  And there's ample examples what
> that would mean.

Is this not the job of the Policy and of community self-policing by
means of RC bugs?


 Best,
 Gard


signature.asc
Description: PGP signature

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Stephan Lachnit]

On Wed, Jan 26, 2022 at 11:43 AM Adam Borowski  wrote:
>
> On Tue, Jan 25, 2022 at 09:38:01PM +0100, Vincent Bernat wrote:
> >
> > I think we should forego the NEW queue. If people want to check
> > packages, they can do it once they are in unstable with regular bugs.
>
> Without the NEW queue, there would be no point at which packaging receives
> any sort of review.  I'd prefer Debian to deliver at least some level of
> quality.
>
> Otherwise, we'd fall to the level of NPM.  And there's ample examples what
> that would mean.

I disagree with the comparison to NPM. Simply because not everyone can
upload - you have to be DD or DM to do that, which means you have to
go through a non-trivial process where it is checked that you know
what you do. As of right now, a malicious acting DD can already upload
harmful packages without NEW stopping this at all.

Regards,
Stephan

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Adam Borowski]

On Tue, Jan 25, 2022 at 09:38:01PM +0100, Vincent Bernat wrote:
> For me, the copyright check is just a bad excuse. People upload
> non-distributable stuff everywhere and it seems the world continue to go
> round. What amount of non-distributable packages is stopped by the NEW
> queue?
> 
> I think we should forego the NEW queue. If people want to check
> packages, they can do it once they are in unstable with regular bugs.

Without the NEW queue, there would be no point at which packaging receives
any sort of review.  I'd prefer Debian to deliver at least some level of
quality.

Otherwise, we'd fall to the level of NPM.  And there's ample examples what
that would mean.

> Current checks are partly done by Lintian and I suppose people could
> watch new Lintian warnings and detect bad packages quickly.

Lintian is just a dumb machine that can ease human reviews but not replace
them.

> This could be done when src is not NEW as a test.

I've managed to trample upon someone else's package just yesterday -- and it
escaped automated checks because a binary of that name already existed in
the archive, just not on any arch which I test.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ Aryans: split from other Indo-Europeans ~2900-2000BC → Ural →
⣾⠁⢠⠒⠀⣿⡁ Bactria → settled 2000-1000BC in northwest India.
⢿⡄⠘⠷⠚⠋⠀ Gypsies: came ~1000AD from northern India; aryan.
⠈⠳⣄ Germans: IE people who came ~2800BC to Scandinavia; not aryan.

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Adam Borowski]

On Wed, Jan 26, 2022 at 10:44:37AM +0100, Gard Spreemann wrote:
> Jonas Smedegaard  writes:
> > Quoting Vincent Bernat (2022-01-25 21:38:01)
> >> I didn't comment at first because I thought someone else would raise 
> >> the idea. But it seems people still like the idea of a NEW queue. Not 
> >> me. The NEW queue is a hindrance.

> > I don't like current copyright laws, and I suspect a fair amount of 
> > people involved in Free Software doesn't.

I for one consider copyright theft, a crime against humanity[1], and a waste
of time.  Any second spent dealing with copyright is a second not spent adding
features or fixing technical bugs.

For practical reasons we have to obey the laws, no matter how oppressive
they are.  But I don't see why we should do more than eg. Fedora which
has corporate backing with an actual legal team.

For those of you without a Fedora box at hand, I made a tarball:
  https://angband.pl/tmp/fedora-licenses.tar.xz
This is so less than we do!

> > I just don't think the solution is to ignore copyright or licensing 
> > statements.

On the other hand, "grep -r Copyright|uniq" plus a copy of non-standard
licenses would be enough.

> To me, the elephant in the room is this question: Does the way the NEW
> queue currently works provide good (good enough?) assurances to
> ourselves that we are *not* ignoring copyright or licensing?

I think the NEW review is much needed, and currently grossly inadequate
-- and that's because 95% of the FTPmaster time being spent on copyright
crap rather than technical matters.


Meow!

[1]. The needs of a human go into two groups: 1. those shared with
non-human animals (food, air, freedom, shelter, reproduction, not being
killed, not being hurt, etc), and 2. those dependant on transmission
of ideas.  And transmission of ideas is directly hindered by copyright.
-- 
⢀⣴⠾⠻⢶⣦⠀ Ash nazg durbatulûk,
⣾⠁⢠⠒⠀⣿⡁   ash nazg gimbatul,
⢿⡄⠘⠷⠚⠋⠀ ash nazg thrakatulûk
⠈⠳⣄   agh burzum-ishi krimpatul.

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Gard Spreemann]

Jonas Smedegaard  writes:

> Quoting Vincent Bernat (2022-01-25 21:38:01)
>> I didn't comment at first because I thought someone else would raise 
>> the idea. But it seems people still like the idea of a NEW queue. Not 
>> me. The NEW queue is a hindrance.
>
> For the record, I don't "like" the NEW queue.
>
> I don't like current copyright laws, and I suspect a fair amount of 
> people involved in Free Software doesn't.
>
> I just don't think the solution is to ignore copyright or licensing 
> statements.

To me, the elephant in the room is this question: Does the way the NEW
queue currently works provide good (good enough?) assurances to
ourselves that we are *not* ignoring copyright or licensing?

It feels like we are answering "yes" based on the amount of heroic work
the FTP masters put in, and the amount of waiting developers have to
suffer through. We're gauging our solution to a problem solely by the
amount of effort, sweat and tears we put in, so to speak.


 Best,
 Gard
 


signature.asc
Description: PGP signature

Re: Do we need to hide packages in NEW queue (Was: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not))

[Philip Hands]

Andreas Tille  writes:

...
> May be some intermediate step would be to not hide packages in NEW queue
> but exposing them as an apt source.  If I'm correct this is not the case
> since it had certain legal consequences for the project if code with
> certain non-free licenses would be downloadable from some debian.org
> address.  May be NEW could be considered as some kind of pre-non-free as
> long as it is not checked and the legal consequences are not valid for
> us any more.  But I'm not educated in international law - just asking
> whether somebody might know better.

I have a repository on salsa: 
https://salsa.debian.org/installer-team/branch2repo
that allows one to easily take a collection of branches (with the same
branch name) from several repos, and assemble the (u)debs that one can
build from all those branches into an apt repo.

The motivation for that is for testing patches to Debian-Installer, but
it should work for anything, so if that (or something like it) got
merged into the main salsa-CI pipeline then people could more easily
decouple the testing of new packages from their progress through NEW.

This does of course raise the question of whether I ought to be able to
do that, since it creates apt repos, such as this (trivial) example:

  
https://salsa.debian.org/installer-team/branch2repo/-/jobs/2365384/artifacts/browse/public/

that publish .debs from a debian.org host, that could easily be created
from sources that have never been near NEW.

Of course, the URL is not exactly obvious there, and the artifacts will
get deleted, so maybe that's a difference, but I don't suppose it would
be too hard to make that into a stable 'pages' URL and ensure that it
got built often enough to keep the repo there permanently.  Would that
cross the line?

I think the important distinction is probably that once packages get
through NEW they are mirrored all over the world, by unsuspecting third
parties, who live in every jurisdiction under the sun. They are also
incorporated into down-stream distros, often with little/no manual
oversight, some of whom then do things like selling their resulting
distribution for profit.

That involves other people in a lot of risks that might not apply to
Debian itself, so I'd suggest requires rather more caution than trying
to see what we alone can expect to get away with.

Do we need to shut down salsa's ability to do the above (given that one
can do all of that from a guest account, using any old code you
uploaded), or is that OK because the URLs are unstable and/or obscure?
(obviously, given that I did it, I think it's OK)

If obscure URLs are enough, I'd think it would be OK to have things in
the NEW queue available from repo URLs that were not something that one
could easily mirror, and would not be an "all of current NEW repo" but
rather something like an apt repo per upload, so that one could easily
test stuff stuck in NEW, but wouldn't be tempted to just install
everything that hits NEW by default.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Do we need to hide packages in NEW queue (Was: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not))

[Andreas Tille]

Am Tue, Jan 25, 2022 at 01:45:11PM -0800 schrieb Russ Allbery:
> Jonas Smedegaard  writes:
> 
> > I just don't think the solution is to ignore copyright or licensing
> > statements.
> 
> That's not the goal.  The question, which keeps being raised in part
> because I don't think it's gotten a good answer, is what the basis is for
> treating copyright and licensing bugs differently than any other bug in
> Debian?
> 
> The need for pre-screening was obvious when we had export control issues,
> but my understanding is that those have gone away.  Are we working from
> legal advice telling us that this pre-screening is required for some legal
> purpose?  If so, is it effective for the legal purpose at which it is
> aimed?  Is this system left over from old advice?  Have we checked our
> assumptions recently?

May be some intermediate step would be to not hide packages in NEW queue
but exposing them as an apt source.  If I'm correct this is not the case
since it had certain legal consequences for the project if code with
certain non-free licenses would be downloadable from some debian.org
address.  May be NEW could be considered as some kind of pre-non-free as
long as it is not checked and the legal consequences are not valid for
us any more.  But I'm not educated in international law - just asking
whether somebody might know better.

I'd consider it a big step forward to develop against packages from NEW
queue.  (And yes, *I* know how to help myself with a local mirror but
team wide development is something else.)
 
> NEW processing is a lot of friction for the project as a whole and a lot
> of work for the ftp team.  If we were able to do less work at the cost of
> a minimal increase in bugs, or at the cost of handling bugs a bit
> differently, maybe that would be a good thing?
> 
> In other words, it's unclear what requirements we're attempting to meet
> and what the basis of those requirements is, which makes it hard to have a
> conversation about whether the current design is the best design for the
> problem we're trying to solve.

I'm CCing debian-legal for this branch of the discussion (but I do not
read this list and think keeping debian-devel in the row is a good idea).

Kind regards

  Andreas. 

-- 
http://fam-tille.de

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Erik Huelsmann]

Hi Russ,

> > I just don't think the solution is to ignore copyright or licensing
> > statements.
>
> That's not the goal.  The question, which keeps being raised in part
> because I don't think it's gotten a good answer, is what the basis is for
> treating copyright and licensing bugs differently than any other bug in
> Debian?

Sorry to barge in (not being a Debian dev): Having seen this
discussion go back and forth on the topic, I agree that this is the
question. However, after all these mails - including yours - I'm left
with: Posing the question is one thing, but getting it answered is
another. Is there anybody specifically more likely to have an answer
here than anybody else? And if so, is that person/group involved in
this discussion now? If not, shouldn't they be made aware of it?

> The need for pre-screening was obvious when we had export control issues,
> but my understanding is that those have gone away.  Are we working from
> legal advice telling us that this pre-screening is required for some legal
> purpose?  If so, is it effective for the legal purpose at which it is
> aimed?  Is this system left over from old advice?  Have we checked our
> assumptions recently?
>
> NEW processing is a lot of friction for the project as a whole and a lot
> of work for the ftp team.  If we were able to do less work at the cost of
> a minimal increase in bugs, or at the cost of handling bugs a bit
> differently, maybe that would be a good thing?
>
> In other words, it's unclear what requirements we're attempting to meet
> and what the basis of those requirements is, which makes it hard to have a
> conversation about whether the current design is the best design for the
> problem we're trying to solve.


-- 
Bye,

Erik.

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Russ Allbery]

Jonas Smedegaard  writes:

> I just don't think the solution is to ignore copyright or licensing
> statements.

That's not the goal.  The question, which keeps being raised in part
because I don't think it's gotten a good answer, is what the basis is for
treating copyright and licensing bugs differently than any other bug in
Debian?

The need for pre-screening was obvious when we had export control issues,
but my understanding is that those have gone away.  Are we working from
legal advice telling us that this pre-screening is required for some legal
purpose?  If so, is it effective for the legal purpose at which it is
aimed?  Is this system left over from old advice?  Have we checked our
assumptions recently?

NEW processing is a lot of friction for the project as a whole and a lot
of work for the ftp team.  If we were able to do less work at the cost of
a minimal increase in bugs, or at the cost of handling bugs a bit
differently, maybe that would be a good thing?

In other words, it's unclear what requirements we're attempting to meet
and what the basis of those requirements is, which makes it hard to have a
conversation about whether the current design is the best design for the
problem we're trying to solve.

-- 
Russ Allbery (r...@debian.org)  

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Vincent Bernat]

 ❦ 25 January 2022 21:51 +01, Jonas Smedegaard:

>> I didn't comment at first because I thought someone else would raise 
>> the idea. But it seems people still like the idea of a NEW queue. Not 
>> me. The NEW queue is a hindrance.
>
> For the record, I don't "like" the NEW queue.
>
> I don't like current copyright laws, and I suspect a fair amount of 
> people involved in Free Software doesn't.
>
> I just don't think the solution is to ignore copyright or licensing 
> statements.

I mentioned it. No need to ignore, just file regular bugs. I said:

> For me, the copyright check is just a bad excuse. People upload
> non-distributable stuff everywhere and it seems the world continue to
> go round. What amount of non-distributable packages is stopped by the
> NEW queue?
-- 
Use the "telephone test" for readability.
- The Elements of Programming Style (Kernighan & Plauger)

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Jonas Smedegaard]

Quoting Vincent Bernat (2022-01-25 21:38:01)
> I didn't comment at first because I thought someone else would raise 
> the idea. But it seems people still like the idea of a NEW queue. Not 
> me. The NEW queue is a hindrance.

For the record, I don't "like" the NEW queue.

I don't like current copyright laws, and I suspect a fair amount of 
people involved in Free Software doesn't.

I just don't think the solution is to ignore copyright or licensing 
statements.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Vincent Bernat]

 ❦ 21 January 2022 09:51 -05, M. Zhou:

> I'd rather propose choice C. Because I to some extent understand
> both sides who support either A or B. I maintain bulky C++ packages,
> and I also had a little experience reviewing packages on behalf of
> ftp-team.

I didn't comment at first because I thought someone else would raise the
idea. But it seems people still like the idea of a NEW queue. Not me.
The NEW queue is a hindrance. I have stopped contributing to Python
stuff for this reason. Packaging something can take weeks because you
need to upload one package, wait for it to be reviewed, then package the
next one, etc. You could upload many packages at once, but it makes
testing and building more difficult. New contributors may just give up
by the time their first package is accepted. I think we have many
unmaintained packages for this reason (no real stats on my side, but
when a package is several versions late, it's usually a sponsored upload
or one of my packages).

I would propose that there should be a reputation system. You can get
points by uploading stuff without issues and lose them if there are
errors. If you have enough points, you can spend them to skip the check.
But someone would have to implement it and the team being understaffed
for whatever reason (and maybe not convinced by this system), I know
this won't be done (we don't have PPA because FTP team wants to
implement it but no time, we don't have autosigned packages because
nobody has time to implement it, etc).

For me, the copyright check is just a bad excuse. People upload
non-distributable stuff everywhere and it seems the world continue to go
round. What amount of non-distributable packages is stopped by the NEW
queue?

I think we should forego the NEW queue. If people want to check
packages, they can do it once they are in unstable with regular bugs.
Current checks are partly done by Lintian and I suppose people could
watch new Lintian warnings and detect bad packages quickly. This could
be done when src is not NEW as a test. People could loose their upload
rights if they are caught abusing the system (and get DM rights for some
selected packages instead). This could be opt-in if people find this
idea offensive.
-- 
Avoid temporary variables.
- The Elements of Programming Style (Kernighan & Plauger)

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Andreas Tille]

Am Mon, Jan 24, 2022 at 06:50:17PM -0500 schrieb Theodore Y. Ts'o:
> 
> So could the Release Team figure out a way to automatically rebuild
> packages that have source dependencies on static libraries?
> 
> This would solve the problem of new binary packages causing a full
> ftpmasters policy review of packages, at least for those who need to
> create new binary packages each time SONAME gets bumped.

If I were a member of the release team I would wait until this thread
might uncover some statement by ftpmaster whether this full review
is actually intended or the waiting time is rather caused by the lack
of an appropriate automatic tool that needs to be written.

Kind regards

 Andreas.

-- 
http://fam-tille.de

Re: Back to the topic of changed binary named (Was: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not))

[Stephan Lachnit]

On Mon, Jan 24, 2022 at 8:15 AM Andreas Tille  wrote:
>
> However, my point was that I want to know what policy ftpmaster applies
> to new binary names and to focus on this topic.  I really want to know
> that policy of ftpmaster and I really would like to see that documented
> and I'm afraid that thread is drifting away from the original topic
> that I will not get any answer on this.
>
> So again: I see a conflict in my interpretation of the mail[1] (original
> posters again in CC) which suggests "an auto-approver" and what I'm
> currently observing and I would be really happy if we can document the
> policy for changed (and new) binary names of existing source packages.

Since I feel my mail went lost in the discussion, here again my opinion:

Option C. New binary packages where the src is already in Debian can
still go through NEW for sanity checks, but do not require a
d/copyright review. These packages should be checked with priority
since they should be quick to review. Same goes for source packages
renames.

Instead, I suggest starting a "d/copyright review lottery" working
group with the goal to review the d/copyright of every package in
testing if changed since the last stable release, especially before
the next stable release. I would personally join this endeavor and
help to write tools to keep track of which packages are "eligible" for
the lottery.
This offloads a lot of work from the ftp-masters and in making regular
d/copyright reviews for all packages split among project members
should also increase the quality of these.

On Mon, Jan 24, 2022 at 8:15 AM Andreas Tille  wrote:
>
> To give another example which has nothing todo with ABI changes:
> Currently I'm afraid of splitting some Arch: all data from some
> Arch: any package since I'm simply afraid of the changed package
> sticking long in new queue.  I know this is bad - but I consider
> users waiting for package updates worse.

On Mon, Jan 24, 2022 at 7:49 PM Theodore Y. Ts'o  wrote:
>
> As far as I know, ftpmaster requires a long, laborious review every
> single time there is a new binary package released.  As a result, it
> strongly disincentivizes maintainers from packaging up new releases
> because it's a pain in the posterior.  A while back, people asked me I
> could update f2fs-tools, and it was shortly before the release freeze,
> and even though there was apparently security fixes involved that
> would be fixed in the latest version of f2fs-tools, I knew there would
> be no point, since there was no way the package would squirt out the
> review pipeline before the release freeze.
>
> Even if it isn't formal policy, the long delay has happened enough
> times that I just assume it will be there, and it influences my
> behavior accordingly.

These are just two examples where the delay of updates in the NEW
queue affects the quality of a package or a Debian release - while it
should do the opposite. I'm sure there are many more, I'm not a DD for
a long time but I heard the "won't make improvement xyz because of the
NEW queue" argument regularly. IMHO if there is not a sudden increase
of review time from the ftp-masters, something needs to be done before
packages start losing quality due to NEW.

Regards,
Stephan

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Theodore Y. Ts'o]

On Mon, Jan 24, 2022 at 08:48:28PM +0100, Paul Gevers wrote:
> Hi Ted,
> 
> I think this is the second time you write something like this, but for
> dynamically linked libraries, the rebuild happens (by the Release Team,
> (please use transition trackers for that) because we automatically track
> transitions [1]). Unless people don't follow the convention that your binary
> matches the SONAME. But nowadays we find those more and more due to
> autopkgtest (reverse dependencies that fail because they can't find the
> appropriate library). It becomes increasingly more difficult to hide the
> fact that your package is not named appropriately.

So could the Release Team figure out a way to automatically rebuild
packages that have source dependencies on static libraries?

This would solve the problem of new binary packages causing a full
ftpmasters policy review of packages, at least for those who need to
create new binary packages each time SONAME gets bumped.

- Ted

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Paul Gevers]

Hi Ted,

On 24-01-2022 19:44, Theodore Y. Ts'o wrote:

No, dpkg-shlibsdeps doesn't save you.  Again, consider the
hypothetical package libshaky, which over the period of 9 months, has
soname changes which generate (over time) packages libshaky3,
libshaky4, libshaky6, libshaky7, and libshaky8.

The latest version of libshaky sources will create the binary packages
libshaky8, libshaky-bin, and libshaky-dev.  Other various external
packages such as say, shaky-cli uses libshaky4, shaky-gtk uses
libshaky6, shaky-qt might use libshaky7, etc.

Now suppose that there is a critical security bug fixed in the latest
version of libshaky sources.  So the security fix is might be fixed in
libshaky8, but the same security bug that allows remote code execution
as well as privileged escalation might apply to libshaky[3467] as
well, but since the fix was only in the latest version of libshaky,
you might as well have been using static libraries in libshaky.
Except that is apparently not allowed by policy.  Oops.


I think this is the second time you write something like this, but for 
dynamically linked libraries, the rebuild happens (by the Release Team, 
(please use transition trackers for that) because we automatically track 
transitions [1]). Unless people don't follow the convention that your 
binary matches the SONAME. But nowadays we find those more and more due 
to autopkgtest (reverse dependencies that fail because they can't find 
the appropriate library). It becomes increasingly more difficult to hide 
the fact that your package is not named appropriately.


Paul

[1] https://release.debian.org/transitions/


OpenPGP_signature
Description: OpenPGP digital signature

Re: Back to the topic of changed binary named (Was: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not))

[Theodore Y. Ts'o]

On Mon, Jan 24, 2022 at 08:15:30AM +0100, Andreas Tille wrote:
> Hi Paul and others,
> 
> its surely an interesting topic how to avoid binary name changes and its
> also interesting to discuss ABI changes and workarounds.
> 
> However, my point was that I want to know what policy ftpmaster applies
> to new binary names and to focus on this topic.  I really want to know
> that policy of ftpmaster and I really would like to see that documented
> and I'm afraid that thread is drifting away from the original topic
> that I will not get any answer on this.

As far as I know, ftpmaster requires a long, laborious review every
single time there is a new binary package released.  As a result, it
strongly disincentivizes maintainers from packaging up new releases
because it's a pain in the posterior.  A while back, people asked me I
could update f2fs-tools, and it was shortly before the release freeze,
and even though there was apparently security fixes involved that
would be fixed in the latest version of f2fs-tools, I knew there would
be no point, since there was no way the package would squirt out the
review pipeline before the release freeze.

Even if it isn't formal policy, the long delay has happened enough
times that I just assume it will be there, and it influences my
behavior accordingly.

- Ted

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Theodore Y. Ts'o]

On Mon, Jan 24, 2022 at 10:20:48AM +0800, Paul Wise wrote:
> On Sun, 2022-01-23 at 17:43 -0500, Theodore Y. Ts'o wrote:
> 
> > That only works if there are no other packages depending on those
> > shared libraries which are coming from other source packages.
> 
> I don't think that is true, I believe you can put multiple things in
> the depends section of an shlibs file and dpkg-shlibdeps will propagate
> that to reverse dependencies just fine. From the manual pages it looks
> like the same applies to the symbols files. I found on my system that
> there are *already* packages that do something similar (see below).

No, dpkg-shlibsdeps doesn't save you.  Again, consider the
hypothetical package libshaky, which over the period of 9 months, has
soname changes which generate (over time) packages libshaky3,
libshaky4, libshaky6, libshaky7, and libshaky8.

The latest version of libshaky sources will create the binary packages
libshaky8, libshaky-bin, and libshaky-dev.  Other various external
packages such as say, shaky-cli uses libshaky4, shaky-gtk uses
libshaky6, shaky-qt might use libshaky7, etc.

Now suppose that there is a critical security bug fixed in the latest
version of libshaky sources.  So the security fix is might be fixed in
libshaky8, but the same security bug that allows remote code execution
as well as privileged escalation might apply to libshaky[3467] as
well, but since the fix was only in the latest version of libshaky,
you might as well have been using static libraries in libshaky.
Except that is apparently not allowed by policy.  Oops.

  - Ted

P.S.  Let's assume that the reason why SONAME needs to be bumped every
single time is not because upstream is lazy, and just bumps the SONAME
"just in case", but because they did a terrible job designing the
library ABI, and there are complex structures that need to all the
time because they are exposed in the public headers, yet the
structures depend on internal implementation details, so they are
constantly fluid.  This is not a hypothetical situation; the openssl
libraries are very much in the case, and when we were trying to get
them to agree to create a stable ABI for the Linux Standard Base,
usptream basically said, "no can do".

So complex ABI analysis isn't going to help you; you basically have to
rebuild all of the dependent packages whenever the SONAME gets bumped.

Back to the topic of changed binary named (Was: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not))

[Andreas Tille]

Hi Paul and others,

its surely an interesting topic how to avoid binary name changes and its
also interesting to discuss ABI changes and workarounds.

However, my point was that I want to know what policy ftpmaster applies
to new binary names and to focus on this topic.  I really want to know
that policy of ftpmaster and I really would like to see that documented
and I'm afraid that thread is drifting away from the original topic
that I will not get any answer on this.

So again: I see a conflict in my interpretation of the mail[1] (original
posters again in CC) which suggests "an auto-approver" and what I'm
currently observing and I would be really happy if we can document the
policy for changed (and new) binary names of existing source packages.

To give another example which has nothing todo with ABI changes:
Currently I'm afraid of splitting some Arch: all data from some
Arch: any package since I'm simply afraid of the changed package
sticking long in new queue.  I know this is bad - but I consider
users waiting for package updates worse.

Kind regards
Andreas.


[1] https://lists.debian.org/debian-devel/2021/07/msg00231.html

Am Mon, Jan 24, 2022 at 10:20:48AM +0800 schrieb Paul Wise:
> On Sun, 2022-01-23 at 17:43 -0500, Theodore Y. Ts'o wrote:
> 
> > That only works if there are no other packages depending on those
> > shared libraries which are coming from other source packages.
> 
> I don't think that is true, I believe you can put multiple things in
> the depends section of an shlibs file and dpkg-shlibdeps will propagate
> that to reverse dependencies just fine. From the manual pages it looks
> like the same applies to the symbols files. I found on my system that
> there are *already* packages that do something similar (see below).
> ...

-- 
http://fam-tille.de

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Paul Wise]

On Sun, 2022-01-23 at 17:43 -0500, Theodore Y. Ts'o wrote:

> That only works if there are no other packages depending on those
> shared libraries which are coming from other source packages.

I don't think that is true, I believe you can put multiple things in
the depends section of an shlibs file and dpkg-shlibdeps will propagate
that to reverse dependencies just fine. From the manual pages it looks
like the same applies to the symbols files. I found on my system that
there are *already* packages that do something similar (see below).

> But my claim is that if the upstream can't manage to maintain a stable
> ABI, then maybe we shouldn't be trying to ship shared libraries.  But
> officially, that's not allowed, since it's considered bad.

Personally, to detect such upstreams I think Debian needs a service
tracking ABI using abipkgdiff (from abigail-tools) and pkg-abidiff.
Once we have that it isn't too much more work for Debian to maintain
the SONAME instead of upstream.

> If we have that solution for Rust and Golang, the maybe it can also
> make life easier for upstreams that can't maintain an ABI.

I initially mainly wanted it for static linking detection, I expect
there is some of that (at least in -static packages) in Debian and that
we are not thinking to rebuild such packages after security issues.

Packages that have complex dependencies in shlibs/symbols files:

$ grep -h '^lib.*,' /var/lib/dpkg/info/*.shlibs
libbfd 2.37.50-multiarch.20220106 binutils-multiarch (>= 2.37.50.20220106), 
binutils-multiarch (<< 2.37.50.20220107)
libopcodes 2.37.50-multiarch.20220106 binutils-multiarch (>= 2.37.50.20220106), 
binutils-multiarch (<< 2.37.50.20220107)
libbctoolbox 1 libbctoolbox1 (>= 4.4.0), libbctoolbox1 (<< 4.5.0)
libbellesip 1 libbellesip1 (>= 4.4.0), libbellesip1 (<< 4.5.0)
libbfd 2.37.50-system.20220106 libbinutils (>= 2.37.50.20220106), libbinutils 
(<< 2.37.50.20220107)
libopcodes 2.37.50-system.20220106 libbinutils (>= 2.37.50.20220106), 
libbinutils (<< 2.37.50.20220107)
libboost_python39 1.74.0 libboost-python1.74.0 (>= 1.74.0), 
libboost-python1.74.0-py39
libeabutil 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libeabwidgets 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libecontacteditor 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libecontactlisteditor 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libecontactprint 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libemail-engine 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libessmime 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libevolution-addressbook-importers 0 libevolution (>= 3.42.3), libevolution (<< 
3.43)
libevolution-calendar-importers 0 libevolution (>= 3.42.3), libevolution (<< 
3.43)
libevolution-calendar 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libevolution-mail-composer 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libevolution-mail-formatter 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libevolution-mail-importers 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libevolution-mail 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libevolution-shell 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libevolution-smime 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libevolution-util 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libgnomecanvas 0 libevolution (>= 3.42.3), libevolution (<< 3.43)
libgconf-2 4 libgconf-2-4 (>= 2.31.1), gconf-service
libgnustep-base 1.28 libgnustep-base1.28 (>= 1.28.0), gnustep-base-runtime (>= 
1.28.0)
libortp 15 libortp15 (>= 1:4.4.0), libortp15 (<< 1:4.5.0)
libphonon4qt5 4 libphonon4qt5-4 (>= 4:4.11.1), phonon4qt5
libtotem 0 libtotem0 (>= 3.38.2-1), libtotem0 (<< 3.39)

$ grep -h ',.*<<' /var/lib/dpkg/info/*.symbols
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6 (>> 2.33), libc6 (<< 2.34)
| libc6-i386 (>> 2.33), libc6-i386 (<< 2.34)
| libc6-i386 (>> 2.33), libc6-i386 (<< 2.34)
| libc6-i386 (>> 2.33), libc6-i386 (<< 2.34)
| libc6-i386 (>> 2.33), libc6-i386 (<< 2.34)
| libc6-i386 (>> 2.33), libc6-i386 (<< 2.34)
| libc6-i386 (>> 2.33), libc6-i386 (<< 2.34)
| libc6-i386 (>> 2.33), libc6-i386 (<< 2.34)
| libc6-i386 (>> 2.33), libc6-i386 (<< 2.34)
| libc6-i386 (>> 2.33), libc6-i386 (<< 2.34)
| libc6-i386 (>> 2.33), libc6-i386 (<< 2.34)
| libc6-i386 (>> 2.33), libc6-i386 (<< 2.34)
| libc6-i386 (>> 2.33), libc6-i386 (<< 2.34)

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Theodore Y. Ts'o]

On Sat, Jan 22, 2022 at 08:58:37AM +0800, Paul Wise wrote:
> > The other thing that's perhaps considering here is that unfortunately,
> > there are some upstreams that are extremely irresponsible with library
> > ABI backwards compatibility, where they bump the SONAME essentially at
> > every release.  I recall one extreme case a few years ago where there
> > were over ten(!) SONAME bumps for a particular library over 12 months.
> 
> You could avoid NEW for these SONAME bumps by using a single binary
> package and ensuring that the symbols/shlibs depend on the right
> version ranges. Or add Provides libfoo-abi-N or libfoo-abi (= N)
> and have the symbols and or shlibs generate dependencies on that.

That only works if there are no other packages depending on those
shared libraries which are coming from other source packages.  After
all, if the only consumers of those shared libraries is source package
for those libraries, there's a much simpler solution --- just
compiling the d*mned package using static linking, and just moving on.

But if there are other packages which are using those shared
libraries, then the official party line is that just shipping static
libraries in libshaky-dev is bad, Bad, BAD, since it means that when
there are security bugs fixed in the sources for libshaky, they aren't
automatically fixed for all of the users of libshaky until they
relink.

But my claim is that if the upstream can't manage to maintain a stable
ABI, then maybe we shouldn't be trying to ship shared libraries.  But
officially, that's not allowed, since it's considered bad.
Unfortunately, that's effectively punishing maintainers who are
supporting sources which support shared library.  For those languages
like Rust, etc., which don't support shared libraries, life is *much*
simpler.

> In the past I've suggested a solution to static linking and binary
> packages containing source could be to have a service scanning every
> binary package for static/source files (.a, Rust, Golang etc), noting
> the relevant package/version tuples and then searching the buildinfo
> files for binary packages that built with those packages installed and
> automatically rebuilding affected packages, or having a service that
> would let you manually rebuild packages affected by security issues.
> 
> https://wiki.debian.org/StaticLinking

If we have that solution for Rust and Golang, the maybe it can also
make life easier for upstreams that can't maintain an ABI.  (And yes,
I don't have much patience with those folks given that e2fsprogs has
had a stable library since 1997, which is the last time I've had to
bump an SONAME for libext2fs.  But that's because I'm careful, where
as some other developers like for f2fs-tools, bump their SONAME every
!@#@?!  release.  Sigh...)

- Ted

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Stephan Lachnit]

On Fri, Jan 21, 2022 at 7:04 PM Paul Gevers  wrote:
>
> It's not only the copyright that the ftp-master are responsible for. New
> binaries fill a place in the Debian namespace and they *are* the keepers
> of that.

One could say that for new binaries packages whose src is already in
Debian, the ftp-masters skip the d/copyright review and only do the
other tasks. This should allow for way quicker transitions of simple
SOVERSION bumps.

In general I prefer a random d/copyright check more than one based on
the introduction of a new binary, as I just don't see any connection
between a new binary name and a change of copyright.

Regards,
Stephan

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Andreas Tille]

Hi,

Am Fri, Jan 21, 2022 at 07:04:33PM +0100 schrieb Paul Gevers:
> > I have heard this argument and my mail was simply to find out what
> > fellow developers think about this.  IMHO the issue is sufficiently
> > important to have some kind of documented consensus about this.
> 
> It's not only the copyright that the ftp-master are responsible for. New
> binaries fill a place in the Debian namespace and they *are* the keepers of
> that.

I absolutely agree that a four eyes principle.  On the other hand several
maintainers consider uploading to experimental (as in said case of onetbb)
and trust a team to verify the issues mentioned above.
 
> And https://lists.debian.org/debian-devel/2021/07/msg00231.html

Yes, this mail was for sure my motivation to write my mail.  IMHO it
says that there is no need for manual inspection ... and I would love
if this would be clarified and documented.

Kind regards

  Andreas.

-- 
http://fam-tille.de

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Simon McVittie]

On Sat, 22 Jan 2022 at 08:58:37 +0800, Paul Wise wrote:
> On Fri, 2022-01-21 at 13:55 -0500, Theodore Ts'o wrote:
> > The other thing that's perhaps considering here is that unfortunately,
> > there are some upstreams that are extremely irresponsible with library
> > ABI backwards compatibility, where they bump the SONAME essentially at
> > every release.

I don't think characterizing that as irresponsible is entirely fair,
and it risks setting up incentives that harm us: if the upstream for
a package I maintain is breaking backwards-compat, I'd prefer them to
acknowledge it and bump the SONAME, rather than saying "well, it isn't
*that* big a break" and ignoring it.

> You could avoid NEW for these SONAME bumps by using a single binary
> package and ensuring that the symbols/shlibs depend on the right
> version ranges. Or add Provides libfoo-abi-N or libfoo-abi (= N)
> and have the symbols and or shlibs generate dependencies on that.

I used that method for early GTK 4 prereleases in experimental, when
upstream were breaking ABI in each 3.9x prerelease. They specifically
weren't guaranteeing API or ABI stability yet at that stage in
development, but I wanted to start packaging early, before it was too
late to incorporate feedback that could result in further incompatible
changes. (At the time the source package was called src:gtk+4.0, but
it's in the git history of src:gtk4 now.)

I think this is fine for prereleases and experimental, but for packages
with more than a trivial number of reverse-dependencies it is likely
to cause issues for testing/unstable transitions, because it defeats
the release team's "smooth upgrades" mechanism (in which they keep the
old libfoo1 binaries and the old src:foo in testing until there are no
more rdeps, even after they have been superseded by a new src:foo with
libfoo2, so that migration doesn't have to all happen in one go). I
would say that maintainers who are considering doing this in unstable
should talk to the release team first.

smcv

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Paul Wise]

On Fri, 2022-01-21 at 13:55 -0500, Theodore Ts'o wrote:

> Can we have better automated tooling, either in Lintian, or in when
> source packages are rebuilt, that can take care of this?
> 
> The other thing that's perhaps considering here is that unfortunately,
> there are some upstreams that are extremely irresponsible with library
> ABI backwards compatibility, where they bump the SONAME essentially at
> every release.  I recall one extreme case a few years ago where there
> were over ten(!) SONAME bumps for a particular library over 12 months.

You could avoid NEW for these SONAME bumps by using a single binary
package and ensuring that the symbols/shlibs depend on the right
version ranges. Or add Provides libfoo-abi-N or libfoo-abi (= N)
and have the symbols and or shlibs generate dependencies on that.

> But if we're going to do that, then we could also just support static
> libraries, and just rebuilt all of the pacakges that link statically
> with libshaky, thus solving the security argument for shared
> libraries.  This also avoids the fairness problem where some packages
> are reguarly going through ftpmaster review, and others aren't...

In the past I've suggested a solution to static linking and binary
packages containing source could be to have a service scanning every
binary package for static/source files (.a, Rust, Golang etc), noting
the relevant package/version tuples and then searching the buildinfo
files for binary packages that built with those packages installed and
automatically rebuilding affected packages, or having a service that
would let you manually rebuild packages affected by security issues.

https://wiki.debian.org/StaticLinking

In addition there are toolchain changes coming in at least Golang and
possibly GCC (pushed by Fedora IIRC) for adding source file/package
information into generated binaries.

The scanning/buildinfo/rebuilding idea would produce more builds than
strictly necessary, but perhaps the coming toolchain changes could be
helpful in filtering down the lists.

Ultimately the best option would be for all build toolchains to have
instrumentation that enables us to completely graph the flow of source
data to -dev binary packages etc to final binary packages. Perhaps each
build system, compiler etc would communicate with a daemon that would
collect the file/etc paths/hashes/etc, map those back to package,
version tuples and upload those to dak in buildgraph files.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Theodore Ts'o]

On Fri, Jan 21, 2022 at 01:28:54PM -0500, Scott Kitterman wrote:
> 
> 1.  When the SO name changes and the binary package name is adjusted 
> accordingly, it is not super rare for the maintainer to mess something up in 
> the renaming and end up with an empty binary package, which does no one any 
> good.  I note that for debhelper compat 15 there appears to be some related 
> work in progress.  Perhaps this is, or can be extended to be, sufficient to 
> eventually make this kind of error a thing of the past.

Can we have better automated tooling, either in Lintian, or in when
source packages are rebuilt, that can take care of this?

The other thing that's perhaps considering here is that unfortunately,
there are some upstreams that are extremely irresponsible with library
ABI backwards compatibility, where they bump the SONAME essentially at
every release.  I recall one extreme case a few years ago where there
were over ten(!) SONAME bumps for a particular library over 12 months.

The problem with this is that it makes for a massive headache when it
comes to security updates.  The claim for why we want to use shared
libraries, despite the library dependency hell problem, is that when a
security problem gets fixed, all we need to do is to upload a new
shared library package, and all of the packages which depend on it
automatically get updated.  Well, if during the course of a testing
release, we have binary packages depending on libshaky3, libshaky5,
libshaky6, libshaky7 and libshaky8, now if there's a long-standing
security bug that gets fixed, it's not necessarily the case that when
the Debian maintainer which uploads an updated libshaky source
package, which might result in binary packages libshaky-dev,
libshaky-bin, and libshaky8, that there will be updates for
libshaky{3,5,6,7}.

Now that we are requiring source uploads for everything entering
testing, there's easy answer to this --- which is to simply have an
automated system which rebuilds all of the packages that have a
build-depends on libshaky-dev, so all the packages will now have a
dependency on libshaky8.  Huzzah!

But if we're going to do that, then we could also just support static
libraries, and just rebuilt all of the pacakges that link statically
with libshaky, thus solving the security argument for shared
libraries.  This also avoids the fairness problem where some packages
are reguarly going through ftpmaster review, and others aren't...

Just a thought

- Ted

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Scott Kitterman]

On Friday, January 21, 2022 1:33:07 PM EST Adam Borowski wrote:
> On Fri, Jan 21, 2022 at 01:28:54PM -0500, Scott Kitterman wrote:
> > 2.  New binary package "steals" binary from another source.  This is
> > sometimes OK.  Sometimes it's accidental.  It could also be malicious (I
> > don't remember if I've every actually seen this done for an intentional
> > "steal" or not, I might have).
> 
> Stealing a binary does not go through NEW.

It does.  Binary is new to that source so there is no existing override.

Scott K

signature.asc
Description: This is a digitally signed message part.

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Adam Borowski]

On Fri, Jan 21, 2022 at 01:28:54PM -0500, Scott Kitterman wrote:
> 2.  New binary package "steals" binary from another source.  This is 
> sometimes 
> OK.  Sometimes it's accidental.  It could also be malicious (I don't remember 
> if I've every actually seen this done for an intentional "steal" or not, I 
> might have).

Stealing a binary does not go through NEW.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ Aryans: split from other Indo-Europeans ~2900-2000BC → Ural →
⣾⠁⢠⠒⠀⣿⡁ Bactria → settled 2000-1000BC in northwest India.
⢿⡄⠘⠷⠚⠋⠀ Gypsies: came ~1000AD from northern India; aryan.
⠈⠳⣄ Germans: IE people who came ~2800BC to Scandinavia; not aryan.

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Scott Kitterman]

On Friday, January 21, 2022 12:19:12 PM EST Andreas Tille wrote:
> Hi Mo,
> 
> Am Fri, Jan 21, 2022 at 09:51:12AM -0500 schrieb M. Zhou:
> > I'd rather propose choice C. Because I to some extent understand
> > both sides who support either A or B. I maintain bulky C++ packages,
> > and I also had a little experience reviewing packages on behalf of
> > ftp-team.
> > 
> > A -- Some (e.g. C++) packages may frequently enter the NEW queue,
> > with OLD src and NEW bins (e.g. SOVERSION bump). A portion of devs
> > feel it is not necessary for frequently because it drastically
> > slows down the maintainer's work. In the worst case, when the
> > package finally passed the NEW queue, the maintainer may have
> > to go through NEW queue again upon the next upload. (This is very
> > likely to happen for tensorflow, etc).
> 
> I have heard this argument and my mail was simply to find out what
> fellow developers think about this.  IMHO the issue is sufficiently
> important to have some kind of documented consensus about this.

Speaking only for myself, not the FTP Team.

There are other considerations.  Here's a few I thought of without investing a 
lot of time in it (not necessarily comprehensive):

1.  When the SO name changes and the binary package name is adjusted 
accordingly, it is not super rare for the maintainer to mess something up in 
the renaming and end up with an empty binary package, which does no one any 
good.  I note that for debhelper compat 15 there appears to be some related 
work in progress.  Perhaps this is, or can be extended to be, sufficient to 
eventually make this kind of error a thing of the past.

2.  New binary package "steals" binary from another source.  This is sometimes 
OK.  Sometimes it's accidental.  It could also be malicious (I don't remember 
if I've every actually seen this done for an intentional "steal" or not, I 
might have).

3.  New package added for reasons that make sense to the maintainer, but not 
for the archive as a whole.  I've seen this come up multiple times, usually in 
the context of the binary being too trivial (which is a judgment call).

It's not just let's do extra copyright/license checks.

Scott K

signature.asc
Description: This is a digitally signed message part.

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Paul Gevers]

Hi,

I'm not involved in ftp-master, but...

On 21-01-2022 18:19, Andreas Tille wrote:

Am Fri, Jan 21, 2022 at 09:51:12AM -0500 schrieb M. Zhou:

I'd rather propose choice C. Because I to some extent understand
both sides who support either A or B. I maintain bulky C++ packages,
and I also had a little experience reviewing packages on behalf of
ftp-team.

A -- Some (e.g. C++) packages may frequently enter the NEW queue,
with OLD src and NEW bins (e.g. SOVERSION bump). A portion of devs
feel it is not necessary for frequently because it drastically
slows down the maintainer's work. In the worst case, when the
package finally passed the NEW queue, the maintainer may have
to go through NEW queue again upon the next upload. (This is very
likely to happen for tensorflow, etc).


I have heard this argument and my mail was simply to find out what
fellow developers think about this.  IMHO the issue is sufficiently
important to have some kind of documented consensus about this.


It's not only the copyright that the ftp-master are responsible for. New 
binaries fill a place in the Debian namespace and they *are* the keepers 
of that.


And https://lists.debian.org/debian-devel/2021/07/msg00231.html

Paul


OpenPGP_signature
Description: OpenPGP digital signature

Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[Andreas Tille]

Hi Mo,

Am Fri, Jan 21, 2022 at 09:51:12AM -0500 schrieb M. Zhou:
> I'd rather propose choice C. Because I to some extent understand
> both sides who support either A or B. I maintain bulky C++ packages,
> and I also had a little experience reviewing packages on behalf of
> ftp-team.
> 
> A -- Some (e.g. C++) packages may frequently enter the NEW queue,
> with OLD src and NEW bins (e.g. SOVERSION bump). A portion of devs
> feel it is not necessary for frequently because it drastically
> slows down the maintainer's work. In the worst case, when the
> package finally passed the NEW queue, the maintainer may have
> to go through NEW queue again upon the next upload. (This is very
> likely to happen for tensorflow, etc).

I have heard this argument and my mail was simply to find out what
fellow developers think about this.  IMHO the issue is sufficiently
important to have some kind of documented consensus about this.
 
> B -- Uploads with OLD src and OLD bin are not required to go through
> NEW queue, even if a decade as passed as long as the src names and
> bin names are kept unchanged. One of the supports for B is that
> the d/copyright file may silently rot (get outdated), as uploads 
> without updated d/copyright won't be rejected. Checking packages
> when they bump SOVERSION is to some extent a "periodical" check.
> This worked very well for packages with stable ABI. But for pacakges
> without stable ABI, especially bulky (C++) packages, this is
> painful for both uploaders and ftp checkers.

I also heard that argument that passing the new queue is a good chance
to review d/copyright.  But as you said it will never catch code that
never needs passing new queue again.  So also here I would seek for
opinions since this argument is in conflict with my interpretation
of the said mail I was refering to[1].

> Given the understanding of both options, I propose choice C:
> 
> C. Lottery NEW queue:
> 
> if (src is new):
> # completely new package
> require manual-review
> elif (src is old) but (bin is new):
> if not-checked-since-last-stable-release:
> # approximate advantage of choice B.
> require manual-review
> elif src.version already exists in archive
> # choice A wants to avoid this case.
> auto-accept
> else:
> if (lottery := random()) < threshold
> require manual-review
> else:
> # I expect a faster pace of debian development.
> auto-accept
> 
> In this way concerns for both people supporting A and B can be partly
> addressed. The old-src-new-bin case have a large chance to pass NEW
> as long as they have been reviewed once after the last stable release.
> The burden for ftp-team can be reduced. And pace of maitnainers can
> be faster with less chance of being blocked and unstable to do anything
> but wait.

I agree that this solution possibly covers both sides.  However my
question was a bit simpler since I wanted to find out whether we really
have supporters of both sides and how strong their arguments might be
for the respective side.  Than we can document the issue and possibly
your suggestion might be an acceptable solution (if and only if we
realise that there are those conflicting opinions).

> > I would love to have this clearly documented since in case B. I would
> > stop wasting my and ftpmaster time with nagging which is not
> > rectified
> > than.
> > 
> > I personally clearly prefer A. and I wish we could clarify this
> > situation.
> 
> Me too. I perfer A personally as well. Debian might be the only major
> distribution which checks the license so thoroughly. Unconditionally
> allow an old-src-new-bin upload to pass is basically impossible, I
> speculate. Choice C might be more practical and feasible.

"Speculate" is the term we need to get rid of first. ;-)
This was my motivation to write the initial mail.
 
> It must be many outdated d/copyright in our archive. Letting eligible
> uploads automatically pass with a high probability is not likely
> causing problem even if yet another outdated d/copyright sneaked in.

I perfectly agree that we have a high probablility for outdated
d/copyright files in our archive.  However, I would like to consider
this problem as orthogonal to the binary name change issue.  It puts
an absolutely not rectified bias on those packages that are typically
changing binary packages frequently - at least I do not see any good
reason to prefer a package that by chance has a new binary name over
any other package inside the archive.

That's why I would rather consider to do a call for volunteers to
verify d/copyright files of *completely* random packages.  I do not
see any point in putting this workload onto ftpmaster but may be as
a first interesting step to join the ftpmaster team.

Kind regards

 Andreas.
 
> > [1] https://lists.debian.org/debian-devel/2021/07/msg00231.html
> > [2] https://ftp-master.debian.org/new/onetbb_2021.4.0-1~exp1.html

-- 

Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not

[M. Zhou]

Hi Andreas,

Thank you for mentioning this. Your post inspired me to came up a
new choice.

On Fri, 2022-01-21 at 11:33 +0100, Andreas Tille wrote:
> 
> This recently happed for me in the case of onetbb (which was not
> uploaded by myself - so I'm not even asking for myself while other
> packages of mine (new ones and ones with just renames) are waiting in
> the queue.)  There are lots of other packages (namely numba and lots
> of
> other packages depending from numba that FTBFS) depending from onetbb
> -
> thus I pinged on #debian-ftp more than once (which I really hate).

When I was once an ftp-trainee (now I'm not in ftp-team), there was no
popcon query when doing dak review. Whether a package is of high
priority depends on one's experience to some extent. Even if I knew
some packages must have a high popcon, their bulky size make the
review process (checking files one by one) not quite easy.

> Due to this I'd like to have a clear statement here (which would
> prove myself in pinging either right or wrong and thus I'm asking):
> 
>   A. Packages with new binary package names should not undergo
>  the copyright inspection by ftpmaster and can be
>  auto-approved (either technically or manually)
> 
>   B. Any package in the new queue needs to be inspected by
>  ftpmaster for copyright issues and it takes as long as
>  it takes no matter whether it is a new package or a
>  package with changed binary name.
> 

I'd rather propose choice C. Because I to some extent understand
both sides who support either A or B. I maintain bulky C++ packages,
and I also had a little experience reviewing packages on behalf of
ftp-team.

A -- Some (e.g. C++) packages may frequently enter the NEW queue,
with OLD src and NEW bins (e.g. SOVERSION bump). A portion of devs
feel it is not necessary for frequently because it drastically
slows down the maintainer's work. In the worst case, when the
package finally passed the NEW queue, the maintainer may have
to go through NEW queue again upon the next upload. (This is very
likely to happen for tensorflow, etc).

B -- Uploads with OLD src and OLD bin are not required to go through
NEW queue, even if a decade as passed as long as the src names and
bin names are kept unchanged. One of the supports for B is that
the d/copyright file may silently rot (get outdated), as uploads 
without updated d/copyright won't be rejected. Checking packages
when they bump SOVERSION is to some extent a "periodical" check.
This worked very well for packages with stable ABI. But for pacakges
without stable ABI, especially bulky (C++) packages, this is
painful for both uploaders and ftp checkers.

Given the understanding of both options, I propose choice C:

C. Lottery NEW queue:

if (src is new):
# completely new package
require manual-review
elif (src is old) but (bin is new):
if not-checked-since-last-stable-release:
# approximate advantage of choice B.
require manual-review
elif src.version already exists in archive
# choice A wants to avoid this case.
auto-accept
else:
if (lottery := random()) < threshold
require manual-review
else:
# I expect a faster pace of debian development.
auto-accept

In this way concerns for both people supporting A and B can be partly
addressed. The old-src-new-bin case have a large chance to pass NEW
as long as they have been reviewed once after the last stable release.
The burden for ftp-team can be reduced. And pace of maitnainers can
be faster with less chance of being blocked and unstable to do anything
but wait.

> I would love to have this clearly documented since in case B. I would
> stop wasting my and ftpmaster time with nagging which is not
> rectified
> than.
> 
> I personally clearly prefer A. and I wish we could clarify this
> situation.

Me too. I perfer A personally as well. Debian might be the only major
distribution which checks the license so thoroughly. Unconditionally
allow an old-src-new-bin upload to pass is basically impossible, I
speculate. Choice C might be more practical and feasible.

It must be many outdated d/copyright in our archive. Letting eligible
uploads automatically pass with a high probability is not likely
causing problem even if yet another outdated d/copyright sneaked in.

> Kind regards
> 
>  Andreas.
> 
> [1] https://lists.debian.org/debian-devel/2021/07/msg00231.html
> [2] https://ftp-master.debian.org/new/onetbb_2021.4.0-1~exp1.html
>