Re: removal instead of orphaning?

[Adam D. Barratt]

On 2016-09-19 12:28, Jakub Wilk wrote:

* Adam D. Barratt , 2016-09-18, 11:28:
"Fixed in NMU" has not been a distinct state for several years, since 
the introduction of BTS version tracking.


To clarify, the state still exists:
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=fixed
... but I guess most packages were tagged like this by mistake.


Yes, I was referring to the workflow implied by your mail, where bugs 
marked as closed in an NMU were not in fact closed but merely tagged 
+fixed.


I'd forgotten that the BTS labels bugs with the "fixed" tag set as 
"fixed in NMU", which seems wrong given that the documentation for that 
tag explicitly lists NMUs as an example of use, not the only one - 
#382717 being a good example of the "not completely fixed yet" usage 
(although tbh in that case it should probably have the tag removed).


Regards,

Adam

Re: removal instead of orphaning?

[Jakub Wilk]

* Adam D. Barratt , 2016-09-18, 11:28:
"Fixed in NMU" has not been a distinct state for several years, since the 
introduction of BTS version tracking.


To clarify, the state still exists:
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=fixed
... but I guess most packages were tagged like this by mistake.

BTS version tracking was introduced in July 2005:
https://lists.debian.org/20050718110629.ga...@riva.ucam.org

dak started closing bugs for NMUs and experimental uploads only in February 
2007:

https://anonscm.debian.org/cgit/mirror/dak.git/commit/?id=141673ae65e8160c78464c1eac1ecb615aedfc54

It's been less than a decade, so it's normal people aren't used to it yet. :-P

--
Jakub Wilk

Re: removal instead of orphaning?

[Stephen Kitt]

On Sun, 18 Sep 2016 15:39:38 +0100, "Adam D. Barratt"
 wrote:
[...]
> In order to determine whether a particular upload is "descended" from
> any other previous upload, the information used by the BTS is generated
> by parsing changelogs and building a tree of uploads, with each stanza
> being considered as the descendent of the stanza prior to it. This is
> why an upload that does not include the changelog stanza from an NMU
> will cause the BTS to consider that any bugs closed in the NMU have been
> subsequently reintroduced in the later upload.

Thanks, I didn't know about that — that's rather nifty!

Regards,

Stephen


pgp3xKpsQtxIf.pgp
Description: OpenPGP digital signature

Re: removal instead of orphaning?

[Adam D. Barratt]

On Sun, 2016-09-18 at 14:30 +0200, Stephen Kitt wrote:
> On Sun, 18 Sep 2016 11:28:55 +0100, "Adam D. Barratt"
>  wrote:
> > If your next maintainer upload includes the changelog stanza for the NMU
> > in its changelog then the BTS will automatically know that your upload
> > includes those fixes, without your mentioning them in your own stanza
> > for your upload.
> 
> Doesn't it just consider that any version after the "fixed" version contains
> the fixes? Otherwise I thought you'd need to specify the appropriate -v
> option.

You're talking about two different things. -v is needed to include
information in the .changes file for the purposes of e.g. dak closing
bugs in a previous version which was never uploaded to Debian; the BTS
does not see that information and does not base its decisions on bug
status on whether the version has increased.

> There are still cases sometimes of NMU changes being ommitted from the
> following maintainer upload, so "fixed in NMU" could still be a meaningful
> state. Admittedly patches get dropped accidentally between maintainer uploads
> sometimes too, so the distinction isn't all that important...

Indeed. The important question is "which versions of the package does
the changelog claim that bug X is closed in?", regardless of who
performed those uploads.

Regards,

Adam

Re: removal instead of orphaning?

[Adam D. Barratt]

On Sun, 2016-09-18 at 16:01 +0200, Santiago Vila wrote:
> On Sun, Sep 18, 2016 at 02:25:13PM +0100, Ben Hutchings wrote:
> 
> > That depends on what you mean by 'after'.  If you mean 'with a greater
> > version', then the answer is no.  The BTS parses changelogs to
> > determine whether a version currently in the archive is derived from a
> > version where the bug was fixed.
> 
> Are you really sure the BTS parses changelogs in full? I think it's
> more that the BTS parses the *.changes files being uploaded.

The BTS never sees .changes files, ttbomk.

> In fact, the closing feature is honored by the BTS not because of the
> "Closes:" word in the changelog itself, but because dpkg-genchanges
> parses the changelog and adds a "Closes:" header to the *.changes file,
> but this header could, in principle, be added by hand and be missing
> from the changelog itself.

We've deviated somewhat from the original discussion, and I think the
point under discussion may have become confused.

The list of bugs closed in a particular upload is derived from the
Closes: header in the .changes file, indeed - the action of closing them
occurs in dak, during upload processing.

In order to determine whether a particular upload is "descended" from
any other previous upload, the information used by the BTS is generated
by parsing changelogs and building a tree of uploads, with each stanza
being considered as the descendent of the stanza prior to it. This is
why an upload that does not include the changelog stanza from an NMU
will cause the BTS to consider that any bugs closed in the NMU have been
subsequently reintroduced in the later upload.

[
adam@jacala:~/code/debian/debbugs$ git grep -C1 changelog
Debbugs/Versions.pm-Gathering the version data is beyond the scope of this 
module: in the case
Debbugs/Versions.pm:of Debian it is carried out by mechanical analysis of 
package changelogs.
Debbugs/Versions.pm-Debbugs::Versions takes version data for a package 
generated by this or any
]

Regards,

Adam

Re: removal instead of orphaning?

[Santiago Vila]

On Sun, Sep 18, 2016 at 02:25:13PM +0100, Ben Hutchings wrote:

> That depends on what you mean by 'after'.  If you mean 'with a greater
> version', then the answer is no.  The BTS parses changelogs to
> determine whether a version currently in the archive is derived from a
> version where the bug was fixed.

Are you really sure the BTS parses changelogs in full? I think it's
more that the BTS parses the *.changes files being uploaded.

In fact, the closing feature is honored by the BTS not because of the
"Closes:" word in the changelog itself, but because dpkg-genchanges
parses the changelog and adds a "Closes:" header to the *.changes file,
but this header could, in principle, be added by hand and be missing
from the changelog itself.

Thanks.

Re: removal instead of orphaning?

[Ben Hutchings]

On Sun, 2016-09-18 at 14:30 +0200, Stephen Kitt wrote:
> On Sun, 18 Sep 2016 11:28:55 +0100, "Adam D. Barratt"
> >  wrote:
> > 
> > On Sun, 2016-09-18 at 11:45 +0200, Marc Haber wrote:
> > > 
> > > So when I do the first upload after an NMU all bugs that the BTS has
> > > as "fixed in NMU" get changed to "closed"?  
> > 
> > "Fixed in NMU" has not been a distinct state for several years, since
> > the introduction of BTS version tracking. The NMU upload simply marks
> > the bugs as closed in the version of that upload, the same as it would
> > for any other upload.
> > 
> > If your next maintainer upload includes the changelog stanza for the NMU
> > in its changelog then the BTS will automatically know that your upload
> > includes those fixes, without your mentioning them in your own stanza
> > for your upload.
> 
> Doesn't it just consider that any version after the "fixed" version contains
> the fixes? Otherwise I thought you'd need to specify the appropriate -v
> option.

That depends on what you mean by 'after'.  If you mean 'with a greater
version', then the answer is no.  The BTS parses changelogs to
determine whether a version currently in the archive is derived from a
version where the bug was fixed.

> There are still cases sometimes of NMU changes being ommitted from the
> following maintainer upload, so "fixed in NMU" could still be a meaningful
> state. Admittedly patches get dropped accidentally between maintainer uploads
> sometimes too, so the distinction isn't all that important...

When that happens, the changelog entry for the fixed version is usually
missing and the BTS logic determines that the fix is not present.

Ben.

-- 
Ben Hutchings
Klipstein's 4th Law of Prototyping and Production:
A fail-safe circuit will destroy
others.


signature.asc
Description: This is a digitally signed message part

Re: removal instead of orphaning?

[Stephen Kitt]

On Sun, 18 Sep 2016 11:28:55 +0100, "Adam D. Barratt"
 wrote:
> On Sun, 2016-09-18 at 11:45 +0200, Marc Haber wrote:
> > So when I do the first upload after an NMU all bugs that the BTS has
> > as "fixed in NMU" get changed to "closed"?  
> 
> "Fixed in NMU" has not been a distinct state for several years, since
> the introduction of BTS version tracking. The NMU upload simply marks
> the bugs as closed in the version of that upload, the same as it would
> for any other upload.
> 
> If your next maintainer upload includes the changelog stanza for the NMU
> in its changelog then the BTS will automatically know that your upload
> includes those fixes, without your mentioning them in your own stanza
> for your upload.

Doesn't it just consider that any version after the "fixed" version contains
the fixes? Otherwise I thought you'd need to specify the appropriate -v
option.

There are still cases sometimes of NMU changes being ommitted from the
following maintainer upload, so "fixed in NMU" could still be a meaningful
state. Admittedly patches get dropped accidentally between maintainer uploads
sometimes too, so the distinction isn't all that important...

Regards,

Stephen


pgpAZAA3tesnU.pgp
Description: OpenPGP digital signature

Re: removal instead of orphaning?

[Adam D. Barratt]

On Sun, 2016-09-18 at 11:45 +0200, Marc Haber wrote:
> On Sun, 18 Sep 2016 10:10:14 +0800, Paul Wise  wrote:
> >With the BTS version tracking feature, acking NMUs is no longer needed
> >as the BTS tracks changelog heritage IIRC, so I'd not mention that in
> >the description.
> 
> So when I do the first upload after an NMU all bugs that the BTS has
> as "fixed in NMU" get changed to "closed"?

"Fixed in NMU" has not been a distinct state for several years, since
the introduction of BTS version tracking. The NMU upload simply marks
the bugs as closed in the version of that upload, the same as it would
for any other upload.

If your next maintainer upload includes the changelog stanza for the NMU
in its changelog then the BTS will automatically know that your upload
includes those fixes, without your mentioning them in your own stanza
for your upload.

Regards,

Adam

Re: removal instead of orphaning?

[Marc Haber]

On Sun, 18 Sep 2016 10:10:14 +0800, Paul Wise  wrote:
>With the BTS version tracking feature, acking NMUs is no longer needed
>as the BTS tracks changelog heritage IIRC, so I'd not mention that in
>the description.

So when I do the first upload after an NMU all bugs that the BTS has
as "fixed in NMU" get changed to "closed"?

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: removal instead of orphaning?

[Paul Gevers]

Hi Manuel,

On 09/17/16 20:53, Manuel A. Fernandez Montecelo wrote:
>> On Sun, Aug 28, 2016 at 10:13:44AM +0100, Manuel A. Fernandez
>> Montecelo wrote:
>>> I think that one measure to improve the current situation is that, for
>>> the people doing NMUs, to orphan the package when the number of NMUs
>>> exceeds for example 3 or 5 in a row, or 1 year since the oldest NMU not
>>> acked by the maintainers and there are bugs or reasons requiring new
>>> intervention.  (Exceptions can apply if there are specific reasons to
>>> avoid to do that, e.g. keep packages as part of a team).  With this,
>>> future NMUs will be QA uploads.

> Also, I hoped to gather some feedback to the message at least, to see if
> the above looks reasonable to most people.

In general, the above seems reasonable to me and an improvement. Except
maybe not only have 5 NMUs in a row, but also no response in e.g. half a
year. That way quick NMUs in succession don't qualify.

Paul



signature.asc
Description: OpenPGP digital signature

Re: removal instead of orphaning?

[Paul Wise]

On Sun, Sep 18, 2016 at 2:53 AM, Manuel A. Fernandez Montecelo wrote:
> 2016-08-28 21:50 Sean Whitton:
>> On Sun, Aug 28, 2016 at 10:13:44AM +0100, Manuel A. Fernandez Montecelo
>> wrote:
>>>
>>> I think that one measure to improve the current situation is that, for
>>> the people doing NMUs, to orphan the package when the number of NMUs
>>> exceeds for example 3 or 5 in a row, or 1 year since the oldest NMU not
>>> acked by the maintainers and there are bugs or reasons requiring new
>>> intervention.  (Exceptions can apply if there are specific reasons to
>>> avoid to do that, e.g. keep packages as part of a team).  With this,
>>> future NMUs will be QA uploads.
>>>
>>> If there's a general agreement that this is a good thing, and somehow
>>> becomes part of the NMU process or similar guidelines [2], it would help
>>> to avoid conflicts.
>>
>>
>> If you have time to make this idea into a patch, a bug against
>> developers-reference would be a good place to discuss it.
>
>
> I failed to do this in the last few weeks and it looks like the next
> will be worse in terms of spare time, so I'll see what I can do, but I
> cannot promise anything :-/
>
> Also, I hoped to gather some feedback to the message at least, to see if
> the above looks reasonable to most people.

With the BTS version tracking feature, acking NMUs is no longer needed
as the BTS tracks changelog heritage IIRC, so I'd not mention that in
the description.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: removal instead of orphaning?

[Manuel A. Fernandez Montecelo]

2016-08-28 21:50 Sean Whitton:

Hello,

On Sun, Aug 28, 2016 at 10:13:44AM +0100, Manuel A. Fernandez Montecelo wrote:

I think that one measure to improve the current situation is that, for
the people doing NMUs, to orphan the package when the number of NMUs
exceeds for example 3 or 5 in a row, or 1 year since the oldest NMU not
acked by the maintainers and there are bugs or reasons requiring new
intervention.  (Exceptions can apply if there are specific reasons to
avoid to do that, e.g. keep packages as part of a team).  With this,
future NMUs will be QA uploads.

If there's a general agreement that this is a good thing, and somehow
becomes part of the NMU process or similar guidelines [2], it would help
to avoid conflicts.


If you have time to make this idea into a patch, a bug against
developers-reference would be a good place to discuss it.


I failed to do this in the last few weeks and it looks like the next
will be worse in terms of spare time, so I'll see what I can do, but I
cannot promise anything :-/

Also, I hoped to gather some feedback to the message at least, to see if
the above looks reasonable to most people.


Cheers.
--
Manuel A. Fernandez Montecelo 

Re: removal instead of orphaning?

[Paul Wise]

On Wed, 2016-08-31 at 00:19 +, Sean Whitton wrote:

> The nice thing about the homepage being there is that the user can get
> it by running `apt-cache show foo`.  Unless you plan to pull in that
> information when building the binary package?

It woudn't need to be present when building the binary package for it
to be present in the output of `apt show`, dak could pull it from
whatever service it is maintained in and add it to the apt metadata.

> Is your reason for not including the watch file primarily because it
> doesn't make sense that updating it requires a new source package revision?

The contents of Maintainers, Homepage, debian/watch, debian/upstream/*
and possibly other information can change independently of the source
package so it makes sense to not maintain them in the source package.

Personally I think Debian is too attached to the current model of
storing everything in the source package (and increasingly in git) to
be able to change to external maintenance of these things soon.

That said, we do have separate watch file maintenance already:

https://alioth.debian.org/projects/sepwatch/

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: removal instead of orphaning?

[Sean Whitton]

Hello,

On Tue, Aug 30, 2016 at 10:20:21AM +0800, Paul Wise wrote:
> On Tue, Aug 30, 2016 at 1:56 AM, Niels Thykier wrote:
> 
> > Frankly, I do not think that the source package is the correct place for
> > the Maintainer / Uploaders data.  There are plenty of cases where it
> > would make sense to update it "retroactively" after the package has been
> > uploaded (E.g. orphaned or adopting without an upload).
> 
> Indeed. Same for Homepage and watch files.

The nice thing about the homepage being there is that the user can get
it by running `apt-cache show foo`.  Unless you plan to pull in that
information when building the binary package?

Is your reason for not including the watch file primarily because it
doesn't make sense that updating it requires a new source package revision?

-- 
Sean Whitton

Re: removal instead of orphaning?

[Paul Wise]

On Tue, Aug 30, 2016 at 1:56 AM, Niels Thykier wrote:

> Frankly, I do not think that the source package is the correct place for
> the Maintainer / Uploaders data.  There are plenty of cases where it
> would make sense to update it "retroactively" after the package has been
> uploaded (E.g. orphaned or adopting without an upload).

Indeed. Same for Homepage and watch files.

> If we are planning on adding this kind of extra way of being "related"
> to a package, which would not be stored in the package itself, could we
> then look at pulling the Maintainer + Uploaders field out as well?

Here is an old proposal for that:

http://dep.debian.net/deps/dep2/

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: removal instead of orphaning?

[Niels Thykier]

Ian Jackson:
> Holger Levsen writes ("Re: removal instead of orphaning?"):
>> Maybe a solution would be a third kind of maintainer/uploader, so
>> people could indicate that they are happy to do house-cleaning work on
>> this package, even though they are not apt to maintain it properly.
>>
>> Maintainers: debian...@lists.debian.org
>> House-cleaners: f...@example.org
> 
> I think it might be worth a facility for documenting this kind of
> thing, indeed.
> 
> I'm not sure that a source package control file field is the right
> place for this information, though.  The set of housekeepers might
> change more quickly than the package is updated, and one wouldn't want
> to need to do an upload just because of a housekeepers change.
> 
> [...]
> 
> Ian.
> 

Frankly, I do not think that the source package is the correct place for
the Maintainer / Uploaders data.  There are plenty of cases where it
would make sense to update it "retroactively" after the package has been
uploaded (E.g. orphaned or adopting without an upload).

If we are planning on adding this kind of extra way of being "related"
to a package, which would not be stored in the package itself, could we
then look at pulling the Maintainer + Uploaders field out as well?

Thanks,
~Niels

Re: removal instead of orphaning?

[Ian Jackson]

Holger Levsen writes ("Re: removal instead of orphaning?"):
> Maybe a solution would be a third kind of maintainer/uploader, so
> people could indicate that they are happy to do house-cleaning work on
> this package, even though they are not apt to maintain it properly.
> 
> Maintainers: debian...@lists.debian.org
> House-cleaners: f...@example.org

I think it might be worth a facility for documenting this kind of
thing, indeed.

I'm not sure that a source package control file field is the right
place for this information, though.  The set of housekeepers might
change more quickly than the package is updated, and one wouldn't want
to need to do an upload just because of a housekeepers change.

Personally if I care about a package enough to want fix it if
something really bad were to happen to it, but not enough to actually
become its maintainer when it's orphaned, I subscribe to the package
in the PTS.

Perhaps one way of documenting this information would be to have an
optional `public subscription' flag in the PTS.  Public subscribers to
the package would be listed on a web page and maybe in some API or
deb822 file somewhere or something ?

Ian.

Re: removal instead of orphaning?

[Sean Whitton]

Hello,

On Sun, Aug 28, 2016 at 10:13:44AM +0100, Manuel A. Fernandez Montecelo wrote:
> I think that one measure to improve the current situation is that, for
> the people doing NMUs, to orphan the package when the number of NMUs
> exceeds for example 3 or 5 in a row, or 1 year since the oldest NMU not
> acked by the maintainers and there are bugs or reasons requiring new
> intervention.  (Exceptions can apply if there are specific reasons to
> avoid to do that, e.g. keep packages as part of a team).  With this,
> future NMUs will be QA uploads.
> 
> If there's a general agreement that this is a good thing, and somehow
> becomes part of the NMU process or similar guidelines [2], it would help
> to avoid conflicts.

If you have time to make this idea into a patch, a bug against
developers-reference would be a good place to discuss it.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: removal instead of orphaning?

[Manuel A. Fernandez Montecelo]

2016-08-28 02:22 gregor herrmann:

On Sat, 27 Aug 2016 11:40:03 +0200, Paul Gevers wrote:


On 26-08-16 23:40, Julien Cristau wrote:
> off the top of my head:
> - it's wasting time of anyone doing QA work
> - it's wasting time of any user who looks for a piece of software to
>   do
>   $stuff and gets to eliminate all the noise from unmaintained
>   probably-broken cruft
These were indeed the two items I was mostly thinking of. I felt the
pain of the first item last year with the dh-python migration at
Debconf.


Speaking about [perl, in my case] transitions, in my experience
orphaned packages are less annyoing (I can just do a QA upload, where
I can fix whatever I want) than
officially-maintained-but-de-facto-neglected packages where I do
minimal NMUs, and then again next year, and then again the year
afterwards, etc.

(I know, the answer to that is salvaging^Winvolving MIA and getting
the packages orphaned; I just wanted to point out that orphaned
packages are not that much of a burden in all cases.)


I completely agree with this.

Sometimes there's the feeling from many people (including MIA team) that
reporting possibly inactive people and expecting MIA to follow the
procedures is a way of "harassing them out of the project".  I
understand that, specially people working on the MIA team, they can feel
sometimes like an "evicting squad", and the people being pinged by MIA
might feel "pushed" or "on risk of eviction".

However, I think that it's quite important from the point of view of
Quality Assurance processes to detect such cases and strip packages of
inactive maintainers (not necessarily remove people from DM/DD
positions) precisely for the reason that Gregor explains here.

In that way, people working on a wide range of packages that they don't
particularly care about (e.g. porters, people helping with transitions,
RC bugfixing in BSPs or during the freezes, contributors or DMs which
need sponsoring, or people just wanting to fix a particular problem)
will have a much better time doing QA uploads [2] than following the NMU
processes, thus are more likely to fix things, thus increasing overall
quality.


So, as Gregor, I also believe that orphaned packages are much better [1]
than the many other packages maintained by NMUs for years, which are in
that state because they're not orphaned by the maintainers and nobody
cares to change the situation.


I think that one measure to improve the current situation is that, for
the people doing NMUs, to orphan the package when the number of NMUs
exceeds for example 3 or 5 in a row, or 1 year since the oldest NMU not
acked by the maintainers and there are bugs or reasons requiring new
intervention.  (Exceptions can apply if there are specific reasons to
avoid to do that, e.g. keep packages as part of a team).  With this,
future NMUs will be QA uploads.

If there's a general agreement that this is a good thing, and somehow
becomes part of the NMU process or similar guidelines [2], it would help
to avoid conflicts.


There is also the additional advantage that, when they're maintained by
QA and neglected, it's easier to get them removed from the archive as
well (the original purpose of the first post of the thread).


Cheers.


[1] A different question is that orphaned packages also tend to not get
   many upstream updates (specially if they require transitions) unless
   they block somebody's way (RC bugs, blocking a transition of another
   package, etc); but officially-maintained-but-de-facto-neglected
   packages are not better in that front either.

[2] I think that I see recommended somewhere that I cannot find right
   now, but it is not "officially" endorsed or not general practice, as
   far as I know.

--
Manuel A. Fernandez Montecelo 

Re: removal instead of orphaning?

[gregor herrmann]

On Sat, 27 Aug 2016 11:40:03 +0200, Paul Gevers wrote:

> On 26-08-16 23:40, Julien Cristau wrote:
> > off the top of my head:
> > - it's wasting time of anyone doing QA work
> > - it's wasting time of any user who looks for a piece of software to
> >   do
> >   $stuff and gets to eliminate all the noise from unmaintained
> >   probably-broken cruft
> These were indeed the two items I was mostly thinking of. I felt the
> pain of the first item last year with the dh-python migration at
> Debconf. 

Speaking about [perl, in my case] transitions, in my experience
orphaned packages are less annyoing (I can just do a QA upload, where
I can fix whatever I want) than
officially-maintained-but-de-facto-neglected packages where I do
minimal NMUs, and then again next year, and then again the year
afterwards, etc.

(I know, the answer to that is salvaging^Winvolving MIA and getting
the packages orphaned; I just wanted to point out that orphaned
packages are not that much of a burden in all cases.)


Cheers,
gregor

-- 
 .''`.  Homepage https://info.comodo.priv.at/ - OpenPGP key 0xBB3A68018649AA06
 : :' : Debian GNU/Linux user, admin, and developer -  https://www.debian.org/
 `. `'  Member of VIBE!AT & SPI, fellow of the Free Software Foundation Europe
   `-   NP: Beatles


signature.asc
Description: Digital Signature

Re: removal instead of orphaning?

[Holger Levsen]

On Sat, Aug 27, 2016 at 02:02:39PM +0200, Paul Gevers wrote:
> And how do we balance the work it takes for those doing QA on those
> packages (for whatever reason) versus the value mentioned by Paul? As
> mentioned so often, popcon has it's value, but is definitely not the answer.

We have several kinds of orphaned packages, eg. at least the distinction
between "commonly used" (I'd say syslinux, docbook and sgml-base are examples
of those), "not so commonly used but having a user base which cares"
(eg sysvinit) and all the rest.

I don't think it makes sense to treat this different kinds of orphaned
packages the same and I don't think anybody does, though I don't see
this reflected in this threat yet…

(and FWIW+btw I'm glad we have popcon data, as incomplete as it might
be, it's one of the view data points we have about package usage.)

Maybe a solution would be a third kind of maintainer/uploader, so
people could indicate that they are happy to do house-cleaning work on
this package, even though they are not apt to maintain it properly.

Maintainers: debian...@lists.debian.org
House-cleaners: f...@example.org

Once we have this we could be more agressive about removing packages
with low popcon scores (below a to be defined threshold) from sid
automatically - and not do that if House-cleaners is set.

Looking at the output of wnpp-alert on my system there are a few
packages where I would step up and do some work to prevent removal from
Debian but which I cannot sensibly commit to maintain properly.


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: removal instead of orphaning?

[Paul Gevers]

Hi Paul,

On 27-08-16 13:15, Paul Wise wrote:
> The long tail of less used and orphaned packages has value.
> Deletionism is bad on Wikipedia and in Debian too.

Can you also please explain WHY you have this opinion? Because I don't
understand it. I think I understand it for Wikipedia, but why for
Debian? In my opinion we keep to much, but I'd like to be convinced
otherwise.

Maybe I can phrase an alternative question for one of the two cases why
I started this thread: do we have good ways to warn the user when (s)he
is installing (via depends even?) a knowingly unmaintained package? If
not, shouldn't we have those? E.g. maybe an option in apt? If we believe
that we should keep unmaintained packages in the archive for the users
sake rather than removal, I truly believe the user should know without
doing the research.

And how do we balance the work it takes for those doing QA on those
packages (for whatever reason) versus the value mentioned by Paul? As
mentioned so often, popcon has it's value, but is definitely not the answer.

Paul



signature.asc
Description: OpenPGP digital signature

Re: removal instead of orphaning?

[Paul Gevers]

Hi

On 27-08-16 13:33, Vincent Bernat wrote:
> Sometimes, a package is still worthwhile even without a maintainer.

And all I ask is the original maintainer to take this into account (of
course only making an estimate for the users) when (s)he is deciding
what to do with the package.

Paul



signature.asc
Description: OpenPGP digital signature

Re: removal instead of orphaning?

[Vincent Bernat]

 ❦ 27 août 2016 12:23 CEST, Andrew Shadura  :

>> Today I was, once again, surprised to see how many (low popcon) orphaned
>> packages we have. I believe that orphanage is a burden to our community
>> in the sense that not all packages are picked up by a new maintainer and
>> these packages need some QA once in a while and often don't get enough
>> of that (at least most packages that I touched).
>
> Sorry, no.
>
> Packages have not only maintainers, but users. And users mainly have
> no skills to maintain packages. And when packages are removed, users
> become unhappy.
>
> A bunch of packages I use have been removed instead of orphaning them.
> I'm a DD myself, but I don't have enough time (and sometimes skills)
> to maintain them.
>
> I am indeed very unhappy.

I share your concerns. Sometimes, a package is still worthwhile even
without a maintainer. If there is an RC bug, a concerned DD can do an
NMU fixing the bug while still not wanting to become a maintainer
because they don't have time.
-- 
I fell asleep reading a dull book, and I dreamt that I was reading on,
so I woke up from sheer boredom.


signature.asc
Description: PGP signature

Re: removal instead of orphaning?

[Paul Wise]

On Fri, Aug 26, 2016 at 10:12 PM, Paul Gevers wrote:

> I suggest that everybody that orphans a package makes a statement in the
> wnpp bug report on why (s)he believes orphaning is better than removal.
> If people agree with my idea here, I'll suggest a change to the
> reportbug template for orphaning bugs.

I for one will be removing that change from any O bugs I submit.
The long tail of less used and orphaned packages has value.
Deletionism is bad on Wikipedia and in Debian too.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: removal instead of orphaning?

[Paul Gevers]

On 26-08-16 23:40, Julien Cristau wrote:
>> Who is this a burden for? As long as there are no RC bugs filed for
>> the orphaned packages, I don't see any a direct reason to remove
>> them. If no-one used the package, then sure, the package is really
>> useless. But if at least some people are using it, it has value.
>>
> off the top of my head:
> - it's wasting time of anyone doing QA work
> - it's wasting time of any user who looks for a piece of software to
>   do
>   $stuff and gets to eliminate all the noise from unmaintained
>   probably-broken cruft

These were indeed the two items I was mostly thinking of. I felt the
pain of the first item last year with the dh-python migration at
Debconf. And I have felt the pain of the second item multiple times in
the past. Nowadays, I check all the tracker.d.o pages of packages before
I install a package which function is provided by multiple packages. I
think we could do better on that front. This is a first simple step.

And I don't agree that if a package is non-RC, it is worth keeping in
our archive. I like to see our archive in good quality. A package
without an active maintainer is always in danger of slipping in quality,
with only the users noticing, who may file bugs, but if the bugs are
non-RC, there is nobody to notice.

On 26-08-16 21:52, Guus Sliepen wrote:
> On Fri, Aug 26, 2016 at 07:43:20AM -1000, David Prévot wrote:
>> What about, e.g., security issues: if nobody cares about maintaining
>> code, whether dormant or dead upstream, or simply no maintainer to
>> include security fixes or upload new upstream versions, then I believe
>> it may cause direct harm to the project.
> 
> Perhaps. But consider this: people who don't need a package don't
> install it. Those who do need it do. If Debian, for whatever reason,
> does not provide the package they need, they will have to download it
> themselves and install it on their machine. Which for them will take
> more time and effort than apt-get install would.

Yes, but consider the same case but where upstream has newer releases,
at least this user can get proper upstream support, while he isn't
getting any support in Debian. And as Julien noted, there may be more
packages that can do the same. This user should find the maintained
package instead of this one. (So yes, if your package provides
functionality available elsewhere in the archive, you have a higher
chance of saying, yes, let's remove it instead).

> So while Debian the project can
> wash its hands of the package in question, the harm done to the end-user
> is still the same or maybe even larger.

I don't agree. I think users expect good packages from Debian, and if we
don't maintain a package, it is a service to the users to say so clearly
(e.g. by not providing it). By the way, remember that we have
derivatives that don't pull from stable, but from unstable to develop
on. They get our unmaintained packages, even if we would keep it out of
testing/the next release.

> I'm quite sure there are many packages with active maintainers for which
> nobody cares enough to file RC-bugs either. Are you actively checking
> for security problems in all of your packages?

Yes I do and I think that is a task of you as a maintainer. Things like
DMD¹ make that easy (at least that is what I use currently).

> If you haven't automated
> it in some way, do you manually check for new versions and upstream bug
> reports every day or week?

New versions, yes, uscan and watch files do that, so also that shows up
in QA pages like DMD and DDPO². DMD even pushes that to you if you
subscribe to the RSS feed.

> I personally find the criterion "package is orphaned" too arbitrary to
> say it should be removed.

And I never said that. I wanted to say: instead of orphaning, consider
if your package should be removed from Debian instead of being orphaned.

Paul

¹ https://udd.debian.org/dmd.cgi?email1=elbrus%40debian.org
² https://qa.debian.org/developer.php?email=elbrus%40debian.org



signature.asc
Description: OpenPGP digital signature

Re: removal instead of orphaning?

[Jakub Wilk]

* Sebastian Reichel , 2016-08-27, 02:24:

rc-alert -dU `wnpp-alert | grep "^O " | cut -d' ' -f 3`


I don't think this handles the case when the RC bug was filed against a 
binary package that has a different name than the source package.


Anyway, it's not like orphaned are the only packages that are neglected 
(or that all orphaned packages are neglected). Please consider running 
rc-alert against all packages you have installed (which incidentally 
make the command-line much simpler):


$ rc-alert -dU

--
Jakub Wilk

Re: removal instead of orphaning?

[Sebastian Reichel]

Hi,

On Fri, Aug 26, 2016 at 09:47:39PM +0200, Adam Borowski wrote:
> On Fri, Aug 26, 2016 at 09:38:02PM +0200, Guus Sliepen wrote:
> > > Should unrelated people spend time on packages they don't care about?
> > 
> > No, that's why they are orphaned in the first place.
> 
> You can run the attached script to see if there are any RC bugs on an
> orphaned package you might give a damn about.

> rc-alert -dU "$@" \
>   `wget -qO- http://www.debian.org/devel/wnpp/orphaned|
>   sed -ne 's/.* href="https\?:\/\/bugs.debian.org\/\([0-9]*\)">\([^:<]*\)[: 
> ]*\([^<]*\)<\/a>.*/O \1 \2 -- \3/; T d; p; : d'|
>   cut -d' ' -f 3`

I was a bit scared by the length of the list on my system, but
it seems to list all RC bugs for all orphaned packages, not
just those being installed. This is because of rc-alert's
behaviour:

$ dpkg -l bootchart
dpkg-query: no packages found matching bootchart
$ rc-alert -dU bootchart
Package: bootchart
Bug: 731669
Title:   bootchart: outdated version should not be shipped with jessie
Flags:   [] (none)
Dists:   [U] (unstable)

Package: src:bootchart
Bug: 817382
Title:   bootchart: Removal of debhelper compat 4
Flags:   [] (none)
Dists:   [U] (unstable)

This modification should output only rc-bugs of orphaned and
installed packages, which ony my system is just a single bug
instead of houndreds:

rc-alert -dU `wnpp-alert | grep "^O " | cut -d' ' -f 3`

-- Sebastian




signature.asc
Description: PGP signature

Re: removal instead of orphaning?

[Guus Sliepen]

On Fri, Aug 26, 2016 at 08:49:00PM +, Niels Thykier wrote:

> > Possible things this tool could do:
> > 
> > - Notify about orphaned packages the DD is using
> > - Notify about installed packages with RC bugs
> > - Notify about installed packages with RFH/RFA bugs
> > [...]
> 
> Aren't these 3 covered by "rc-alert" and "wnpp-alert" already?

Andrey Rahmatullin wrote:

> > Aren't these 3 covered by "rc-alert" and "wnpp-alert" already?
> Also how-can-i-help(1).

Ah, I actualy knew about wnpp-alert, but not rc-alert or how-can-i-help.

-- 
Met vriendelijke groet / with kind regards,
  Guus Sliepen 


signature.asc
Description: Digital signature

Re: removal instead of orphaning?

[Julien Cristau]

On Fri, Aug 26, 2016 at 19:01:52 +0200, Guus Sliepen wrote:

> On Fri, Aug 26, 2016 at 04:12:46PM +0200, Paul Gevers wrote:
> 
> > Today I was, once again, surprised to see how many (low popcon) orphaned
> > packages we have. I believe that orphanage is a burden to our community
> > in the sense that not all packages are picked up by a new maintainer and
> > these packages need some QA once in a while and often don't get enough
> > of that (at least most packages that I touched).
> 
> Who is this a burden for? As long as there are no RC bugs filed for the
> orphaned packages, I don't see any a direct reason to remove them.
> If no-one used the package, then sure, the package is really useless.
> But if at least some people are using it, it has value.
> 
off the top of my head:
- it's wasting time of anyone doing QA work
- it's wasting time of any user who looks for a piece of software to do
  $stuff and gets to eliminate all the noise from unmaintained
  probably-broken cruft

Cheers,
Julien

Re: removal instead of orphaning?

[Andrey Rahmatullin]

On Fri, Aug 26, 2016 at 08:49:00PM +, Niels Thykier wrote:
> > Possible things this tool could do:
> > 
> > - Notify about orphaned packages the DD is using
> > - Notify about installed packages with RC bugs
> > - Notify about installed packages with RFH/RFA bugs
> > [...]
> > 
> 
> Aren't these 3 covered by "rc-alert" and "wnpp-alert" already?
Also how-can-i-help(1).

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: removal instead of orphaning?

[Niels Thykier]

Guus Sliepen:
> On Fri, Aug 26, 2016 at 09:47:39PM +0200, Adam Borowski wrote:
> [...]
> Possible things this tool could do:
> 
> - Notify about orphaned packages the DD is using
> - Notify about installed packages with RC bugs
> - Notify about installed packages with RFH/RFA bugs
> [...]
> 

Aren't these 3 covered by "rc-alert" and "wnpp-alert" already?

Thanks,
~Niels

Re: removal instead of orphaning?

[Guus Sliepen]

On Fri, Aug 26, 2016 at 09:47:39PM +0200, Adam Borowski wrote:

> You can run the attached script to see if there are any RC bugs on an
> orphaned package you might give a damn about.

Interesting. Might it be an idea to create a package that contains this
kind of wisdom, and makes it easy for DDs to query things they should
have a look at? And maybe have a cron job to remind them of it? Getting
DD's a bit more involved in the things that might affect them would be a
good idea. I myself would certainly benefit from such a tool. 

Possible things this tool could do:

- Notify about orphaned packages the DD is using
- Notify about installed packages with RC bugs
- Notify about installed packages with RFH/RFA bugs
- Notify about open bugs the DD filed that have seen no activity for a while
- Notify about bugs against packages the DD maintains that have seen no
  activity for a while
- Notify about any issues found in the DD's packages that you can also
  find on qa.debian.org (lintian, piuparts, reproducible build warnings
  and so on)
- Notify about bugs in packages that are (reverse-)depends of the
  packages the DD maintains.

The above is things specific for a DD. But other ideas could be:

- Find a list of bugs that might be relevant to the user, based on
  debtags for example.
- Pick a random package so you see something different for a change, and
  can maybe test it, audit it, create translations for it, etc.
- Find packages on mentors.debian.net that need reviewing/uploading.
- Your idea here.

-- 
Met vriendelijke groet / with kind regards,
  Guus Sliepen 


signature.asc
Description: Digital signature

Re: removal instead of orphaning?

[Guus Sliepen]

On Fri, Aug 26, 2016 at 07:43:20AM -1000, David Prévot wrote:

> > As long as there are no RC bugs filed for the
> > orphaned packages, I don't see any a direct reason to remove them.
> 
> What about, e.g., security issues: if nobody cares about maintaining
> code, whether dormant or dead upstream, or simply no maintainer to
> include security fixes or upload new upstream versions, then I believe
> it may cause direct harm to the project.

Perhaps. But consider this: people who don't need a package don't
install it. Those who do need it do. If Debian, for whatever reason,
does not provide the package they need, they will have to download it
themselves and install it on their machine. Which for them will take
more time and effort than apt-get install would. And then for sure they
will not get automatic updates, and I don't think that most end-users
will start tracking security fixes and new upstream versions themselves,
unless they really need new features. So while Debian the project can
wash its hands of the package in question, the harm done to the end-user
is still the same or maybe even larger.

> The fact that nobody cared enough to track issues and eventually file
> RC-bugs may not be the best way to claim that a package is good
> enough.

I'm quite sure there are many packages with active maintainers for which
nobody cares enough to file RC-bugs either. Are you actively checking
for security problems in all of your packages? If you haven't automated
it in some way, do you manually check for new versions and upstream bug
reports every day or week?

I personally find the criterion "package is orphaned" too arbitrary to
say it should be removed.

> > If no-one used the package, then sure, the package is really useless.
> > But if at least some people are using it, it has value.
> 
> Maybe it is worth considering alternative instead of using unmaintained
> code, or stepping up in proper maintenance, rather than leaving
> unaudited code in some of our user machines.

Which fraction of code in Debian with active maintainers is actually audited?

-- 
Met vriendelijke groet / with kind regards,
  Guus Sliepen 


signature.asc
Description: Digital signature

Re: removal instead of orphaning?

[Adam Borowski]

On Fri, Aug 26, 2016 at 09:38:02PM +0200, Guus Sliepen wrote:
> > Should unrelated people spend time on packages they don't care about?
> 
> No, that's why they are orphaned in the first place.

You can run the attached script to see if there are any RC bugs on an
orphaned package you might give a damn about.

-- 
An imaginary friend squared is a real enemy.
#!/bin/sh

rc-alert -dU "$@" \
  `wget -qO- http://www.debian.org/devel/wnpp/orphaned|
  sed -ne 's/.*\([^:<]*\)[: 
]*\([^<]*\)<\/a>.*/O \1 \2 -- \3/; T d; p; : d'|
  cut -d' ' -f 3`

Re: removal instead of orphaning?

[Andrey Rahmatullin]

On Fri, Aug 26, 2016 at 09:38:02PM +0200, Guus Sliepen wrote:
> > > > I believe that orphanage is a burden to our community [...]
> > > 
> > > Who is this a burden for? 
> > Transitions. Mandatory packaging changes, like python helper one.
> > True, a package can get an RC bug and be removed during these processes
> > but that requires more time and more effort
> 
> Both pro-active and reactive removals cost time and effort. I don't
> think the difference is significant. But I don't have hard numbers. A
> study on what method is best would cost time and effort in itself.
If that RoM is done instead of O it's hardly any effort.
But it takes a lot of effort to study a package affected by your
transition and find out what to do with it.

> > and do we even remove orphaned packages if they have an RC bug?
> 
> Yes, packages with RC bugs are automatically removed from testing
> nowadays.
Which, of course, is not the same as removing them from unstable.

> > > If no-one used the package, then sure, the package is really useless.
> > > But if at least some people are using it, it has value.
> > Should unrelated people spend time on packages they don't care about?
> No, that's why they are orphaned in the first place.
But Debian is a distribution, not a pile of random packages, and we spend
effort to make those packages work together. This requires spending time
on packages which nobody else cares about but which are still a part of
the distribution.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: removal instead of orphaning?

[Guus Sliepen]

On Fri, Aug 26, 2016 at 10:34:29PM +0500, Andrey Rahmatullin wrote:

> > > I believe that orphanage is a burden to our community [...]
> > 
> > Who is this a burden for? 
> Transitions. Mandatory packaging changes, like python helper one.
> True, a package can get an RC bug and be removed during these processes
> but that requires more time and more effort

Both pro-active and reactive removals cost time and effort. I don't
think the difference is significant. But I don't have hard numbers. A
study on what method is best would cost time and effort in itself.

> and do we even remove orphaned packages if they have an RC bug?

Yes, packages with RC bugs are automatically removed from testing
nowadays.

> > If no-one used the package, then sure, the package is really useless.
> > But if at least some people are using it, it has value.
> Should unrelated people spend time on packages they don't care about?

No, that's why they are orphaned in the first place.

-- 
Met vriendelijke groet / with kind regards,
  Guus Sliepen 


signature.asc
Description: Digital signature

Re: removal instead of orphaning?

[David Prévot]

Hi,

Le 26/08/2016 à 07:01, Guus Sliepen a écrit :
> On Fri, Aug 26, 2016 at 04:12:46PM +0200, Paul Gevers wrote:
> 
>> Today I was, once again, surprised to see how many (low popcon) orphaned
>> packages we have. I believe that orphanage is a burden to our community
>> in the sense that not all packages are picked up by a new maintainer and
>> these packages need some QA once in a while and often don't get enough
>> of that (at least most packages that I touched).
> 
> Who is this a burden for? As long as there are no RC bugs filed for the
> orphaned packages, I don't see any a direct reason to remove them.

What about, e.g., security issues: if nobody cares about maintaining
code, whether dormant or dead upstream, or simply no maintainer to
include security fixes or upload new upstream versions, then I believe
it may cause direct harm to the project. The fact that nobody cared
enough to track issues and eventually file RC-bugs may not be the best
way to claim that a package is good enough.

> If no-one used the package, then sure, the package is really useless.
> But if at least some people are using it, it has value.

Maybe it is worth considering alternative instead of using unmaintained
code, or stepping up in proper maintenance, rather than leaving
unaudited code in some of our user machines.

Regards

David



signature.asc
Description: OpenPGP digital signature

Re: removal instead of orphaning?

[Andrey Rahmatullin]

On Fri, Aug 26, 2016 at 07:01:52PM +0200, Guus Sliepen wrote:
> > Today I was, once again, surprised to see how many (low popcon) orphaned
> > packages we have. I believe that orphanage is a burden to our community
> > in the sense that not all packages are picked up by a new maintainer and
> > these packages need some QA once in a while and often don't get enough
> > of that (at least most packages that I touched).
> Who is this a burden for? 
Transitions. Mandatory packaging changes, like python helper one.
True, a package can get an RC bug and be removed during these processes
but that requires more time and more effort and do we even remove orphaned
packages if they have an RC bug?

> If no-one used the package, then sure, the package is really useless.
> But if at least some people are using it, it has value.
Should unrelated people spend time on packages they don't care about?

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: removal instead of orphaning?

[Guus Sliepen]

On Fri, Aug 26, 2016 at 04:12:46PM +0200, Paul Gevers wrote:

> Today I was, once again, surprised to see how many (low popcon) orphaned
> packages we have. I believe that orphanage is a burden to our community
> in the sense that not all packages are picked up by a new maintainer and
> these packages need some QA once in a while and often don't get enough
> of that (at least most packages that I touched).

Who is this a burden for? As long as there are no RC bugs filed for the
orphaned packages, I don't see any a direct reason to remove them.
If no-one used the package, then sure, the package is really useless.
But if at least some people are using it, it has value.

> I suggest that everybody that orphans a package makes a statement in the
> wnpp bug report on why (s)he believes orphaning is better than removal.
> If people agree with my idea here, I'll suggest a change to the
> reportbug template for orphaning bugs.

I certainly won't hurt to help the old maintainer to provide some more
information about the reasons for orphaning, and about the state of the
package. But I wouldn't formulate it in a way as to suggest that removal
is better.

-- 
Met vriendelijke groet / with kind regards,
  Guus Sliepen 


signature.asc
Description: Digital signature