Re: A proposal for improving transparency of the FTP NEW process

[Andreas Tille]

Hi Chris,

On Sat, Mar 03, 2018 at 07:33:22PM +, Chris Lamb wrote:
> Steffen Möller wrote:
> 
> > But you are a celebrity. Just not the kind of celebrity that gets
> > free coffee at Starbucks, though. Except for when you fix their Wifi,
> > I mean. But if I was an ftpadmin and saw a package of yours uploaded,
> > you'd find me priorising this up ...
> 
> (Just for clarity, as the ftp-team member who — I think? — accepted
> the packages in question, it was nothing to do with "celebrity.")

(I guess you forgot   brackets around your
statement :-P )

I'm happy that you confirm my assumption that picking from the new queue
is orthogonal to the person who uploaded the package.

To Steffen: How demotivating would it be for a newcomer if a package
needs to wait for a long time for the only reason that this maintainer
ID is showing up the first time in the new queue?

Kind regards

Andreas. 

-- 
http://fam-tille.de

Re: A proposal for improving transparency of the FTP NEW process

[Andreas Tille]

On Fri, Mar 02, 2018 at 03:11:48PM +0200, Lars Wirzenius wrote:
> On Fri, 2018-03-02 at 13:51 +0100, Gert Wollny wrote:
> > How do you want to achieve this with a source package that has 13k+
> > source files and where upstream does not provide a standard license
> > header for each file? I.e. there is some license text and it needs to
> > be quoted, but licensecheck doesn't detect the license or doesn't
> > detect the copyright entry, so one has to manually inspect many files
> > to get it right. 
> 
> If upstream and the package uploader togehter don't make the copyright
> status so clear it's easy for ftp master (or anyone else) to review,
> then yes, I think we'd be better off with not adding that software to
> Debian. Ftp master time is a scarce resource, I think we should try to
> be careful of how we spend it.

I Gert's initial mail he wrote that the second pass of the package took
six monthes.  So ftpmaster has managed to do the large amount of work.
The question was how we can make dealing with the remaining issues that
should probably cost less work then one day more efficient.
 
> > Do you really want to reject these packages outright from Debian, even
> > though they follow the DFSG?
> 
> Do we really want to pile on more work on an already busy team just so
> you can do less?

IMHO this question does not fit the problem Gert wants to address.  My
answer to it would be:  I'm fine if ftpmaster is taking low hanging
fruits first.  The question what is actually a low hanging fruit and how
we can make fruits hanging lower remains unanswered.

Another very personal opinion:  For my definition of "universal
operating system" also complex packages belong to our targets.

Kind regards

Andreas.

-- 
http://fam-tille.de

Re: A proposal for improving transparency of the FTP NEW process

[Dominique Dumont]

On Saturday, 3 March 2018 07:54:00 CET Lars Wirzenius wrote:
> We have licencecheck, and if that isn't good enough, we can improve
> it.

That's my cue to advertise "cme update dpkg-copyright" that uses licencecheck 
output to provide a debian/copyright file

See [1] for details (and limitations)

HTH

[1] 
https://github.com/dod38fr/config-model/wiki/Updating-debian-copyright-file-with-cme

-- 
 https://github.com/dod38fr/   -o- http://search.cpan.org/~ddumont/
http://ddumont.wordpress.com/  -o-   irc: dod at irc.debian.org

Re: A proposal for improving transparency of the FTP NEW process

[Andrey Rahmatullin]

On Sun, Mar 04, 2018 at 11:16:29PM +0100, Thomas Goirand wrote:
> P.S: Why on earth do we need to have the ftpmaster@d.o as Cc? Don't you
> guys believe they read debian-devel without cc-ing them?
Well, at least some active DDs *don't* read d-d@ and I can understand why.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: A proposal for improving transparency of the FTP NEW process

[Paul Wise]

On Sun, Mar 4, 2018 at 5:53 PM, Philip Hands wrote:

> Perhaps it's more work than licensecheck, or doesn't suit your
> requirements, but there is also license-reconcile.

As well as a bunch of other tools, some of which need packaging:

https://wiki.debian.org/CopyrightReviewTools

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Salsa issue tracker disabled for Debian group (was: A proposal for improving transparency of the FTP NEW process)

[Ben Finney]

Thomas Goirand  writes:

> Also, I would really have preferred if Salsa's issue tracker feature
> was simply removed/desactivated, because every other day, there's
> someone proposing to replace debbug with it. Thanks but no thanks.

As best I can tell, any project created in the Debian group
 already has the Issues permission
off, without anything else needing to be done.

That seems entirely appropriate, for a Debian packaging project, for the
reason you state (any Debian package should have bugs reported to the
Debian BTS).

Does that meet the preference you expressed above?

-- 
 \   “I distrust those people who know so well what God wants them |
  `\to do to their fellows, because it always coincides with their |
_o__)  own desires.” —Susan Brownell Anthony, 1896 |
Ben Finney

Re: A proposal for improving transparency of the FTP NEW process

[Alexander Wirt]

On Sun, 04 Mar 2018, Thomas Goirand wrote:

> On 03/02/2018 01:00 PM, Gert Wollny wrote:
> > Since ftp-master also sometimes sends messages like "I let it pass for
> > now, but please fix it with the next upload", using the package issue
> > tracker would also be a way to keep track of these minor issues.
> 
> For this, we have the BTS. If the issue is RC, it will prevent shit from
> migrating. Salsa's issue tracker doesn't have this feature.
> 
> Also, I would really have preferred if Salsa's issue tracker feature was
> simply removed/desactivated, because every other day, there's someone
> proposing to replace debbug with it. Thanks but no thanks. One place is
> enough to look into. If you wish to write somewhere, the ITP bug is the
> correct place to go.
Every project/group can disable it at will. But it makes sense for some
things (like the salsa support tracker). So the answer is no for global
disabling it. 

Alex

Re: A proposal for improving transparency of the FTP NEW process

[Thomas Goirand]

On 03/02/2018 01:00 PM, Gert Wollny wrote:
> Since ftp-master also sometimes sends messages like "I let it pass for
> now, but please fix it with the next upload", using the package issue
> tracker would also be a way to keep track of these minor issues.

For this, we have the BTS. If the issue is RC, it will prevent shit from
migrating. Salsa's issue tracker doesn't have this feature.

Also, I would really have preferred if Salsa's issue tracker feature was
simply removed/desactivated, because every other day, there's someone
proposing to replace debbug with it. Thanks but no thanks. One place is
enough to look into. If you wish to write somewhere, the ITP bug is the
correct place to go.

On 03/02/2018 01:15 PM, Lars Wirzenius wrote:
> Counter proposal: let's work on ways in which uploaders can make it
> easy and quick for ftp masters to review packages in NEW.

I've sent so many packages through NEW that I sometimes feel guilty
about it. Though I don't know how to make it easy for them.

On 03/02/2018 01:15 PM, Lars Wirzenius wrote:
> The idea
> should be, in my opinion, that any package that requires more than a
> day of work to review should be rejected by default.

Let's reject the Linux kernel, Qemu, etc.: they are too big... :)

More seriously: big software are more complex, but probably also more
useful for our users. So your proposal doesn't feel right.

Cheers,

Thomas Goirand (zigo)

P.S: Why on earth do we need to have the ftpmaster@d.o as Cc? Don't you
guys believe they read debian-devel without cc-ing them?

Re: A proposal for improving transparency of the FTP NEW process

[Philip Hands]

Gert Wollny  writes:

> Am Freitag, den 02.03.2018, 17:49 +0100 schrieb Philip Hands:
>> Gert Wollny  writes:
>> 
>> > Am Freitag, den 02.03.2018, 14:01 +0100 schrieb Iustin Pop:
>> > > 
>> > > How do you (we) know the package indeed is DFSG-compliant, if
>> > > there
>> > > is  no license information? If upstream cannot bother to provide
>> > > headers, how do we know the code is indeed licenced under the
>> > > claimed
>> > > licence? 
>> > > Etc.
>> > > Note: I haven't looked at the package. Maybe I misunderstand the
>> > > situation…
>> > 
>> > The information is all there big parts of it just can't be
>> > automatically extracted (mostly the copyright information),
>> 
>> Would you be so kind as to cite some examples of copyright
>> information that is there but not automatically extractable, just so
>> that we can get an idea of what you have in mind?
>
> Sspecifically in vtk7 there are two main issues, one is that in nearly
> all files the main copyright header is 
>
>   Copyright (c) Ken Martin, Will Schroeder, Bill Lorensen
>   All rights reserved.
>   See Copyright.txt or http://www.kitware.com/Copyright.htm for 
>   details.
>
>  This software is distributed WITHOUT ANY WARRANTY; without even
>  the implied warranty of MERCHANTABILITY or FITNESS FOR A ARTICULAR
>  PURPOSE.  See the above copyright notice for more information.
>
> Which means licensecheck will report an unknown license,

While licensecheck might not be able to do that right now, it is clear
that it would be trivial to automatically detect that text, which is why
I asked.

Perhaps it's more work than licensecheck, or doesn't suit your
requirements, but there is also license-reconcile.

license-reconcile lets you add rules to deal with things that it doesn't
understand out of the box:

  
http://git.hands.com/?p=freeswitch.git;a=blob;f=debian/license-reconcile.yml;h=0e40cba01eeb67f82d18ca8f11210271848d0549;hb=refs/heads/copyright2
  

(as you can see, freeswitch is quite a jumble when it comes to
copyright)

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: A proposal for improving transparency of the FTP NEW process

[Ben Finney]

Paul Gevers  writes:

> Hi,
>
> On 03-03-18 11:57, Ben Finney wrote:
> > A large code base with complex interacting works of copyright can be
> > broken into smaller packages, that are each individually easier to
> > review and pass through NEW; either built as separate source
> > packages, or combined at build time.
>
> Except if there is one upstream tar ball. Do you really suggest we
> should repack one upstream tar ball into multiple smaller tar balls
> just for the review process?

To say “should” is rather too strong. I'm presenting it as one option to
try, for those who are frustrated at the slow review process for a large,
complex source package with complex copyright interactions: Break it
into more easily-reviewed pieces.

The review process is an acknowledged bottleneck for NEW queue
processing, and we've seen stated that large, complex source packages
are especially difficult to get through.

I'm proposing that the same code base as a set of smaller source
packages will get through that bottleneck easier; I don't say that as a
recommendation nor a “should”, but as an option to try, for those of us
who not FTPmaster and are looking for ways to work better with the
process.

-- 
 \   “Whenever you read a good book, it's like the author is right |
  `\   there, in the room talking to you, which is why I don't like to |
_o__)   read good books.” —Jack Handey |
Ben Finney

Re: A proposal for improving transparency of the FTP NEW process

[Pierre-Elliott Bécue]

Le samedi 03 mars 2018 à 21:57:42+1100, Ben Finney a écrit :
> Lars Wirzenius  writes:
> 
> > Admittedly, it is quite a small package, but that's kind-of my point.
> > Making it easy for others to do the thing you need them to do is what
> > you can do to make things easier for you. Asking them to do more work,
> > or to change how they do their thing, is less likely to go well.
> >
> > The Linux kernel maintainers flat-out reject large patches that dump
> > big changes without prior discussion. This is because it's easier for
> > them to digest changes in smaller chunks, and they put the
> > responsibility for presenting the changes that way on the people
> > producing the patches.
> >
> > For Debian, I think we should have the same approach. […]
> 
> Your recounted experience suggests another way (compatible with the ways
> you discussed) to reduce the work of reviewing a code base with complex
> copyright interactions:
> 
> A large code base with complex interacting works of copyright can be
> broken into smaller packages, that are each individually easier to
> review and pass through NEW; either built as separate source packages,
> or combined at build time.
> 
> Will that work for every large package? Maybe that's too much to hope
> for. But in those cases where the maintainers are frustrated that the
> NEW queue processing takes a long time to pass their new package: isn't
> it worth the effort to try making it easier to review by breaking the
> package into smaller more easily-reviewed chunks?
> 
> A maintainer (or a team) who tries that might find it's not so hard; and
> having done that, it becomes that much easier to understand not only the
> copyright status of each smaller package, but for a newcomer to
> understand how to maintain it. This is a benefit not only in getting the
> package reviewed through the NEW queue, but also for attracting
> additional co-maintainers.
> 
> Breaking large complex tangles into smaller manageable pieces is thus,
> I'd suggest, an investment in reducing the overall work of Debian
> package maintenance.

God.

I can't imagine what kind of hell it'd actually be to split up some thing as
vtk into smaller packages without massive redesign upstream.

I'm actually pretty sure I'd not like being part of it.

Cheers,

-- 
Pierre-Elliott Bécue
GPG: 9AE0 4D98 6400 E3B6 7528  F493 0D44 2664 1949 74E2
It's far easier to fight for one's principles than to live up to them.

Re: A proposal for improving transparency of the FTP NEW process

[Chris Lamb]

Steffen Möller wrote:

> But you are a celebrity. Just not the kind of celebrity that gets
> free coffee at Starbucks, though. Except for when you fix their Wifi,
> I mean. But if I was an ftpadmin and saw a package of yours uploaded,
> you'd find me priorising this up ...

(Just for clarity, as the ftp-team member who — I think? — accepted
the packages in question, it was nothing to do with "celebrity.")


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Re: A proposal for improving transparency of the FTP NEW process

[Paul Gevers]

Hi,

On 03-03-18 11:57, Ben Finney wrote:
> A large code base with complex interacting works of copyright can be
> broken into smaller packages, that are each individually easier to
> review and pass through NEW; either built as separate source packages,
> or combined at build time.

Except if there is one upstream tar ball. Do you really suggest we
should repack one upstream tar ball into multiple smaller tar balls just
for the review process? In the examples I have in mind (fpc and lazarus)
I wouldn't even know what binary packages to build from the smaller
pieces until they are combined again.

I don't believe this is a great recommendation.

Paul



signature.asc
Description: OpenPGP digital signature

Re: A proposal for improving transparency of the FTP NEW process

[Steffen Möller]

On 3/3/18 7:54 AM, Lars Wirzenius wrote:

On Fri, 2018-03-02 at 22:05 -0600, Steve Robbins wrote:

I assume that the reason my packages have been processed is due to one

of two reasons: a) I get quoted on LWN several times a year, so I'm a
celebrity and get special treatment

I expect that's it.

For the avoidance of doubt, especially for onlookers: I was, of
course, being sarcastic with that LWN command, and I think Steve is
continuing by being deadpan sarcastic in response.

I wish text/plain carried font information so I could use a font to
indicate when I'm being sarcastic (Times, Helvetica, or Courier).


But you are a celebrity. Just not the kind of celebrity that gets
free coffee at Starbucks, though. Except for when you fix their Wifi,
I mean. But if I was an ftpadmin and saw a package of yours uploaded,
you'd find me priorising this up ... and if there is not something
more interesting like a new version of bsdgames that one needs to
quality-control ... being a tech-celebrity must be good for something.

And nobody diss sarcasm, please. It is a tremendous helper,
not only to stay mentally afloat but also for brain storming to help
stimulating lateral thinking. It is mainly problematic in broadcast
communication like mailing lists when you do not know with whom
you are working with.


Or possibly you have a more generous notion of "fast".  Currently there are
five or six dozen packages waiting more than 2 months.  That's not fast in my
books.

By "fast" I mean "less than 24 hours". I uploaded vmdb2 (new source
package) about a month ago. The timestamp of the mail saying it's
going into the NEW queue is 16:27. The ACCEPTED mail has a timestamp
of 18:00. That was on February 10.

I had this, too. Just yesterday. Thrice. My fourth package had a problem,
which I kind of knew when it was not going through as quickly.

Admittedly, it is quite a small package, but that's kind-of my point.
Making it easy for others to do the thing you need them to do is what
you can do to make things easier for you. Asking them to do more work,
or to change how they do their thing, is less likely to go well.

The Linux kernel maintainers flat-out reject large patches that dump
big changes without prior discussion. This is because it's easier for
them to digest changes in smaller chunks, and they put the
responsibility for presenting the changes that way on the people
producing the patches.

For Debian, I think we should have the same approach. Not literally
the same approach, of course, since that would mean having the Debian
package maintainer refactor the upstream code into smaller programs,
and that would just be silly.


I disagree here. We think in units. What is released together should not
be split into multiple source packages with Debian. I am still living
through that with my mgltools packages that made everyone unhappy,
also making my communication with upstream more difficult. And the
overall workto review it all is the same.


But the same approach of making it the
uploader's responsibility to present a new package in a way that makes
it easy to review it. This includes making copyright information easy,
and working with upstream to make it clear, possibly by using SPDX
markup for copyrights and licensing. For all of Debian it meants
finding or developing tools for automating extraction of copyright
information into debian/copyright in whatever manner is needed.
We have licencecheck, and if that isn't good enough, we can improve
it.


Ha! And here I agree again. We should somehow improve our infrastructure
to allow for

 * an increase automation, e.g. by something like debian/licencecheck
   to  help customizing the otherwise unknown licenses* replies to the 
ITP with


 * issues to address prior to a reload? issues on salsa?Just a nything that
   renders the ftpadmins' reviewing reentrant.

 * maybe ftpmasters can delegate some work to an individual they trust
   for some particular checks when the source tree is on salsa? We would
   effectively then have temporary volunteer ftpadmin assistants.


There may be other reasons why some packages stay in NEW for a long
time. Getting information from ftp masters about the reasons why, and
a discussion of how we can make easier for them to make NEW review
easier would, I feel, take us forward better than another than us
complaining that things are taking too long.


Yes. I am very happy not to be an ftpadmin and cannot thank our
ftpadminning peers enough for their efforts. Our current infrastructure
is not really set out to be communicative in that process.

Steffen

Re: A proposal for improving transparency of the FTP NEW process

[Ben Finney]

Lars Wirzenius  writes:

> Admittedly, it is quite a small package, but that's kind-of my point.
> Making it easy for others to do the thing you need them to do is what
> you can do to make things easier for you. Asking them to do more work,
> or to change how they do their thing, is less likely to go well.
>
> The Linux kernel maintainers flat-out reject large patches that dump
> big changes without prior discussion. This is because it's easier for
> them to digest changes in smaller chunks, and they put the
> responsibility for presenting the changes that way on the people
> producing the patches.
>
> For Debian, I think we should have the same approach. […]

Your recounted experience suggests another way (compatible with the ways
you discussed) to reduce the work of reviewing a code base with complex
copyright interactions:

A large code base with complex interacting works of copyright can be
broken into smaller packages, that are each individually easier to
review and pass through NEW; either built as separate source packages,
or combined at build time.

Will that work for every large package? Maybe that's too much to hope
for. But in those cases where the maintainers are frustrated that the
NEW queue processing takes a long time to pass their new package: isn't
it worth the effort to try making it easier to review by breaking the
package into smaller more easily-reviewed chunks?

A maintainer (or a team) who tries that might find it's not so hard; and
having done that, it becomes that much easier to understand not only the
copyright status of each smaller package, but for a newcomer to
understand how to maintain it. This is a benefit not only in getting the
package reviewed through the NEW queue, but also for attracting
additional co-maintainers.

Breaking large complex tangles into smaller manageable pieces is thus,
I'd suggest, an investment in reducing the overall work of Debian
package maintenance.

> There may be other reasons why some packages stay in NEW for a long
> time. Getting information from ftp masters about the reasons why, and
> a discussion of how we can make easier for them to make NEW review
> easier would, I feel, take us forward better than another than us
> complaining that things are taking too long.

Thanks for keeping the options open.

-- 
 \ Contentsofsignaturemaysettleduringshipping. |
  `\   |
_o__)  |
Ben Finney

Re: A proposal for improving transparency of the FTP NEW process

[W. Martin Borgert]

On 2018-03-03 10:22, Chris Lamb wrote:
> This is, of course, not very obvious or initiutive and improved
> transparency on this would obviously be a beneficial to all parties,
> let alone Debian at large.

Indeed. Sometimes I see interesting packages in NEW, but don't
know why they don't pass. A public place with all relevant
information, maybe comments by FTP team members, dialogue with
maintainer etc. would be helpful for the community in general.

> I solved this by making sarcasm my default mode.

I hope, that this was meant sarcastic. Wait...

Re: A proposal for improving transparency of the FTP NEW process

[Chris Lamb]

Hi Lars & Steve,

> > Or possibly you have a more generous notion of "fast".  Currently there
> > are five or six dozen packages waiting more than 2 months
[…]
> There may be other reasons why some packages stay in NEW for a long
> time. Getting information from ftp masters about the reasons why

A good proportion of these have outstanding issues that are blocking
on the maintainer, not the FTP team.

This is, of course, not very obvious or initiutive and improved
transparency on this would obviously be a beneficial to all parties,
let alone Debian at large.

> I wish text/plain carried font information so I could use a font to
> indicate when I'm being sarcastic (Times, Helvetica, or Courier).

I solved this by making sarcasm my default mode.


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Re: A proposal for improving transparency of the FTP NEW process

[Lars Wirzenius]

On Fri, 2018-03-02 at 22:05 -0600, Steve Robbins wrote:
> I assume that the reason my packages have been processed is due to one
> > of two reasons: a) I get quoted on LWN several times a year, so I'm a
> > celebrity and get special treatment
> 
> I expect that's it.  

For the avoidance of doubt, especially for onlookers: I was, of
course, being sarcastic with that LWN command, and I think Steve is
continuing by being deadpan sarcastic in response.

I wish text/plain carried font information so I could use a font to
indicate when I'm being sarcastic (Times, Helvetica, or Courier).

> Or possibly you have a more generous notion of "fast".  Currently there are 
> five or six dozen packages waiting more than 2 months.  That's not fast in my 
> books.

By "fast" I mean "less than 24 hours". I uploaded vmdb2 (new source
package) about a month ago. The timestamp of the mail saying it's
going into the NEW queue is 16:27. The ACCEPTED mail has a timestamp
of 18:00. That was on February 10.

Admittedly, it is quite a small package, but that's kind-of my point.
Making it easy for others to do the thing you need them to do is what
you can do to make things easier for you. Asking them to do more work,
or to change how they do their thing, is less likely to go well.

The Linux kernel maintainers flat-out reject large patches that dump
big changes without prior discussion. This is because it's easier for
them to digest changes in smaller chunks, and they put the
responsibility for presenting the changes that way on the people
producing the patches.

For Debian, I think we should have the same approach. Not literally
the same approach, of course, since that would mean having the Debian
package maintainer refactor the upstream code into smaller programs,
and that would just be silly. But the same approach of making it the
uploader's responsibility to present a new package in a way that makes
it easy to review it. This includes making copyright information easy,
and working with upstream to make it clear, possibly by using SPDX
markup for copyrights and licensing. For all of Debian it meants
finding or developing tools for automating extraction of copyright
information into debian/copyright in whatever manner is needed.
We have licencecheck, and if that isn't good enough, we can improve
it.

There may be other reasons why some packages stay in NEW for a long
time. Getting information from ftp masters about the reasons why, and
a discussion of how we can make easier for them to make NEW review
easier would, I feel, take us forward better than another than us
complaining that things are taking too long.

signature.asc
Description: This is a digitally signed message part

Re: A proposal for improving transparency of the FTP NEW process

[Steve Robbins]

On Friday, March 2, 2018 6:15:54 AM CST Lars Wirzenius wrote:

> I'm not involved with the ftp master team in any way, except I
> occasionally make them do work by uploading things that go to thew NEW
> queue. In the past decade ago, the NEW processing has almost always
> been fast, and when it hasn't, it's been due to issues like a Debian
> release happening.
> 
> I assume that the reason my packages have been processed is due to one
> of two reasons: a) I get quoted on LWN several times a year, so I'm a
> celebrity and get special treatment

I expect that's it.  

Or possibly you have a more generous notion of "fast".  Currently there are 
five or six dozen packages waiting more than 2 months.  That's not fast in my 
books.

-Steve


signature.asc
Description: This is a digitally signed message part.

Re: A proposal for improving transparency of the FTP NEW process

[gregor herrmann]

On Fri, 02 Mar 2018 18:53:07 -0500, Scott Kitterman wrote:

> Because we don't know if a package is even distributable until after it's 
> reviewed, packages in New are not available outside the FTP Team to review.  
> I 
> don't expect that to change.

That's the theory.
The practice since many years is that most packages are available
on Alioth or Salsa.
 

Cheers,
gregor

-- 
 .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
   `-   NP: Paco de Lucia: Manteca Colora [Rumba]


signature.asc
Description: Digital Signature

Re: A proposal for improving transparency of the FTP NEW process

[Scott Kitterman]

On Friday, March 02, 2018 02:23:00 PM Gert Wollny wrote:
> Am Freitag, den 02.03.2018, 07:39 -0500 schrieb Scott Kitterman:
> > On Friday, March 02, 2018 01:00:57 PM Gert Wollny wrote:
> > > Dear all,
> > > 
> > > as the one who is the uploader of the package that is currently
> > > longest
> > > in the NEW pipeline (vtk7), I'd like to make a proposal how
> > > transparency and also the interaction from non ftp-master members
> > > to
> > > review packages could be improved.
> > > 
> > > Short version: Use the salsa per-package issue tracker for problems
> > > that come up with the review in NEW.
> > 
> > No.  I think the short version of your proposal is:
> > 
> > If the FTP team spent more time on tracking status more stuff would
> > get through New.
> 
> There is a reason why I wrote "reviewes" in my proposal, and not "ftp-
> master": this kind of tracking could be an incentive for more people to
> get involved reviewing other peoples packages, because it gives a clear
> path for contribution, and adding to it what I wrote as answer to
> Samuel, there is also the possibility for recognition of this review
> work by adding an "Reviewed-By:" entry to the changelog.
> 
> Apart from that, I would guess is that the ftp team already does some
> tracking.

Because we don't know if a package is even distributable until after it's 
reviewed, packages in New are not available outside the FTP Team to review.  I 
don't expect that to change.

When an issue is noted or a question raised in a package review, we have a 
standard mechanism for prodding the uploader (person who prepared the change) 
via email.  For the case of a package that's been a long time in New with now 
feedback, the best way to find out the status is to ask in #debian-ftp.  There 
isn't always someone available to answer immediately, but questions to 
generally get answered.

Scott K

Re: A proposal for improving transparency of the FTP NEW process

[Don Armstrong]

On Fri, 02 Mar 2018, Gert Wollny wrote:
> In salsa you get the links to the commits automatically, in the BTS
> one would have to set these manually I guess. That was my main
> incentive to propose this.

There's nothing stoping us from linking to commits "automatically" in
the BTS; I'd just need someone to propose how this would work, and a
link format that seems sensible.


-- 
Don Armstrong  https://www.donarmstrong.com

But if, after all, we are on the wrong track, what then? Only
disappointed human hopes, nothing more. And even if we perish, what
will it matter in the endless cycles of eternity?
 -- Fridtjof Nansen _Farthest North_ p152

Re: A proposal for improving transparency of the FTP NEW process

[Jonas Smedegaard]

Quoting Gert Wollny (2018-03-02 19:02:44)
> Am Freitag, den 02.03.2018, 17:49 +0100 schrieb Philip Hands:
> > Gert Wollny  writes:
> > 
> > > Am Freitag, den 02.03.2018, 14:01 +0100 schrieb Iustin Pop:
> > > > 
> > > > How do you (we) know the package indeed is DFSG-compliant, if 
> > > > there is no license information? If upstream cannot bother to 
> > > > provide headers, how do we know the code is indeed licenced 
> > > > under the claimed licence?
> > > > Etc.
> > > > Note: I haven't looked at the package. Maybe I misunderstand the 
> > > > situation…
> > > 
> > > The information is all there big parts of it just can't be 
> > > automatically extracted (mostly the copyright information),
> > 
> > Would you be so kind as to cite some examples of copyright 
> > information that is there but not automatically extractable, just so 
> > that we can get an idea of what you have in mind?
> 
> Sspecifically in vtk7 there are two main issues, one is that in nearly 
> all files the main copyright header is
> 
>   Copyright (c) Ken Martin, Will Schroeder, Bill Lorensen
>   All rights reserved.
>   See Copyright.txt or http://www.kitware.com/Copyright.htm for 
>   details.
> 
>  This software is distributed WITHOUT ANY WARRANTY; without even
>  the implied warranty of MERCHANTABILITY or FITNESS FOR A ARTICULAR
>  PURPOSE.  See the above copyright notice for more information.
> 
> Which means licensecheck will report an unknown license, and one has 
> to check what is actually written in these files. Copyright.txt is 
> then simply a BSD-3 clause license, but obviously one has to check.
> 
> The second issue is that there are often two or more distinct 
> copyright notices in different blocks with different statements, think 
> variations of:
>   
>  Copyright 2008 Sandia Corporation.
>  Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,
>  the U.S. Government retains certain rights in this software.
> 
>  Copyright (c) Ken Martin, Will Schroeder, Bill Lorensen
>  All rights reserved.
>  See Copyright.txt or http://www.kitware.com/Copyright.htm for details.
> 
> Another licenses that is not recognized is 
> 
>  Copyright 2008 Sandia Corporation.
>  Under the terms of Contract DE-AC04-94AL85000, there is a non-
>  exclusive license for use of this work by or on behalf of the
>  U.S. Government. Redistribution and use in source and binary forms, 
>  with or without modification, are permitted provided that this Notice
>  and any statement of authorship are reproduced on all copies.

Thanks for elaborating.

Please consider filing bugreports against licensecheck.  I can be slow 
to respond to them, but dearly appreciate examples of challenging cases.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: signature

Re: A proposal for improving transparency of the FTP NEW process

[Gert Wollny]

Am Freitag, den 02.03.2018, 17:49 +0100 schrieb Philip Hands:
> Gert Wollny  writes:
> 
> > Am Freitag, den 02.03.2018, 14:01 +0100 schrieb Iustin Pop:
> > > 
> > > How do you (we) know the package indeed is DFSG-compliant, if
> > > there
> > > is  no license information? If upstream cannot bother to provide
> > > headers, how do we know the code is indeed licenced under the
> > > claimed
> > > licence? 
> > > Etc.
> > > Note: I haven't looked at the package. Maybe I misunderstand the
> > > situation…
> > 
> > The information is all there big parts of it just can't be
> > automatically extracted (mostly the copyright information),
> 
> Would you be so kind as to cite some examples of copyright
> information that is there but not automatically extractable, just so
> that we can get an idea of what you have in mind?

Sspecifically in vtk7 there are two main issues, one is that in nearly
all files the main copyright header is 

  Copyright (c) Ken Martin, Will Schroeder, Bill Lorensen
  All rights reserved.
  See Copyright.txt or http://www.kitware.com/Copyright.htm for 
  details.

 This software is distributed WITHOUT ANY WARRANTY; without even
 the implied warranty of MERCHANTABILITY or FITNESS FOR A ARTICULAR
 PURPOSE.  See the above copyright notice for more information.

Which means licensecheck will report an unknown license, and one has to
check what is actually written in these files. Copyright.txt is then
simply a BSD-3 clause license, but obviously one has to check.

The second issue is that there are often two or more distinct copyright
notices in different blocks with different statements, think variations
of: 
  
 Copyright 2008 Sandia Corporation.
 Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,
 the U.S. Government retains certain rights in this software.

 Copyright (c) Ken Martin, Will Schroeder, Bill Lorensen
 All rights reserved.
 See Copyright.txt or http://www.kitware.com/Copyright.htm for details.

Another licenses that is not recognized is 

 Copyright 2008 Sandia Corporation.
 Under the terms of Contract DE-AC04-94AL85000, there is a non-
 exclusive license for use of this work by or on behalf of the
 U.S. Government. Redistribution and use in source and binary forms, 
 with or without modification, are permitted provided that this Notice
 and any statement of authorship are reproduced on all copies.

Cheers, 
Gert

Re: A proposal for improving transparency of the FTP NEW process

[Philip Hands]

Gert Wollny  writes:

> Am Freitag, den 02.03.2018, 14:01 +0100 schrieb Iustin Pop:
>> 
>> How do you (we) know the package indeed is DFSG-compliant, if there
>> is  no license information? If upstream cannot bother to provide
>> headers, how do we know the code is indeed licenced under the claimed
>> licence? 
>> Etc.
>
>> Note: I haven't looked at the package. Maybe I misunderstand the
>> situation…
>
> The information is all there big parts of it just can't be
> automatically extracted (mostly the copyright information),

Would you be so kind as to cite some examples of copyright information
that is there but not automatically extractable, just so that we can get
an idea of what you have in mind?

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: A proposal for improving transparency of the FTP NEW process

[Ole Streicher]

Philip Hands  writes:
> Gert Wollny  writes:
> ...
>> Short version: Use the salsa per-package issue tracker for problems
>> that come up with the review in NEW.
>
> Is there any significant benefit that this brings over having the same
> interaction in the BTS?
>
> I realise that Gitlab is the new shiny thing, but there is a cost to
> using two issue tracking mechanisms when one would do, and for packages
> where the maintainer is not actually using salsa, what then?

I would rather like to see the salsa issue page to the BTS (so that "new
issue" would create a new BTS entry).

Cheers

Ole

Re: A proposal for improving transparency of the FTP NEW process

[Gert Wollny]

Am Freitag, den 02.03.2018, 07:39 -0500 schrieb Scott Kitterman:
> On Friday, March 02, 2018 01:00:57 PM Gert Wollny wrote:
> > Dear all,
> > 
> > as the one who is the uploader of the package that is currently
> > longest
> > in the NEW pipeline (vtk7), I'd like to make a proposal how
> > transparency and also the interaction from non ftp-master members
> > to
> > review packages could be improved.
> > 
> > Short version: Use the salsa per-package issue tracker for problems
> > that come up with the review in NEW.
> > 
> 
> No.  I think the short version of your proposal is:
> 
> If the FTP team spent more time on tracking status more stuff would
> get through New.

There is a reason why I wrote "reviewes" in my proposal, and not "ftp-
master": this kind of tracking could be an incentive for more people to
get involved reviewing other peoples packages, because it gives a clear
path for contribution, and adding to it what I wrote as answer to
Samuel, there is also the possibility for recognition of this review
work by adding an "Reviewed-By:" entry to the changelog.

Apart from that, I would guess is that the ftp team already does some
tracking.
  
best, 
Gert

Re: A proposal for improving transparency of the FTP NEW process

[Gert Wollny]

Am Freitag, den 02.03.2018, 14:01 +0100 schrieb Iustin Pop:
> 
> How do you (we) know the package indeed is DFSG-compliant, if there
> is  no license information? If upstream cannot bother to provide
> headers, how do we know the code is indeed licenced under the claimed
> licence? 
> Etc.

> Note: I haven't looked at the package. Maybe I misunderstand the
> situation…

The information is all there big parts of it just can't be
automatically extracted (mostly the copyright information), which would
bring us back to the discussion about 

  "Has Copyright summarizing outlived its usefulness?" 

we've seen here some time ago. 

best, 
Gert

Re: A proposal for improving transparency of the FTP NEW process

[Lars Wirzenius]

On Fri, 2018-03-02 at 13:51 +0100, Gert Wollny wrote:
> How do you want to achieve this with a source package that has 13k+
> source files and where upstream does not provide a standard license
> header for each file? I.e. there is some license text and it needs to
> be quoted, but licensecheck doesn't detect the license or doesn't
> detect the copyright entry, so one has to manually inspect many files
> to get it right. 

If upstream and the package uploader togehter don't make the copyright
status so clear it's easy for ftp master (or anyone else) to review,
then yes, I think we'd be better off with not adding that software to
Debian. Ftp master time is a scarce resource, I think we should try to
be careful of how we spend it.

> Do you really want to reject these packages outright from Debian, even
> though they follow the DFSG?

Do we really want to pile on more work on an already busy team just so
you can do less?


signature.asc
Description: This is a digitally signed message part

Re: A proposal for improving transparency of the FTP NEW process

[Gert Wollny]

Am Freitag, den 02.03.2018, 13:38 +0100 schrieb Philip Hands:
> Gert Wollny  writes:
> ...
> > Short version: Use the salsa per-package issue tracker for problems
> > that come up with the review in NEW.
> 
> Is there any significant benefit that this brings over having the
> same interaction in the BTS?

In salsa you get the links to the commits automatically, in the BTS one
would have to set these manually I guess. That was my main incentive to
propose this.  

> I realise that Gitlab is the new shiny thing, but there is a cost to
> using two issue tracking mechanisms when one would do, 
> and for packages where the maintainer is not actually using salsa,
> what then?

Then add to my proposal: For packages that are maintained on salsa, but
I guess one could also manage everything in the BTS. 

Re: A proposal for improving transparency of the FTP NEW process

[Iustin Pop]

On 2018-03-02 13:51:24, Gert Wollny wrote:
> Am Freitag, den 02.03.2018, 14:15 +0200 schrieb Lars Wirzenius:
> > 
> > 
> > Counter proposal: let's work on ways in which uploaders can make it
> > easy and quick for ftp masters to review packages in NEW. The idea
> > should be, in my opinion, that any package that requires more than a
> > day of work to review should be rejected by default.
> 
> How do you want to achieve this with a source package that has 13k+
> source files and where upstream does not provide a standard license
> header for each file? I.e. there is some license text and it needs to
> be quoted, but licensecheck doesn't detect the license or doesn't
> detect the copyright entry, so one has to manually inspect many files
> to get it right. 
> 
> Do you really want to reject these packages outright from Debian, even
> though they follow the DFSG?

How do you (we) know the package indeed is DFSG-compliant, if there is
no license information? If upstream cannot bother to provide headers,
how do we know the code is indeed licenced under the claimed licence?
Etc.

Note: I haven't looked at the package. Maybe I misunderstand the
situation…

Re: A proposal for improving transparency of the FTP NEW process

[Gert Wollny]

Am Freitag, den 02.03.2018, 14:15 +0200 schrieb Lars Wirzenius:
> 
> 
> Counter proposal: let's work on ways in which uploaders can make it
> easy and quick for ftp masters to review packages in NEW. The idea
> should be, in my opinion, that any package that requires more than a
> day of work to review should be rejected by default.

How do you want to achieve this with a source package that has 13k+
source files and where upstream does not provide a standard license
header for each file? I.e. there is some license text and it needs to
be quoted, but licensecheck doesn't detect the license or doesn't
detect the copyright entry, so one has to manually inspect many files
to get it right. 

Do you really want to reject these packages outright from Debian, even
though they follow the DFSG?


Best,
Gert

Re: A proposal for improving transparency of the FTP NEW process

[Philip Hands]

Gert Wollny  writes:
...
> Short version: Use the salsa per-package issue tracker for problems
> that come up with the review in NEW.

Is there any significant benefit that this brings over having the same
interaction in the BTS?

I realise that Gitlab is the new shiny thing, but there is a cost to
using two issue tracking mechanisms when one would do, and for packages
where the maintainer is not actually using salsa, what then?

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: A proposal for improving transparency of the FTP NEW process

[Scott Kitterman]

On Friday, March 02, 2018 01:00:57 PM Gert Wollny wrote:
> Dear all,
> 
> as the one who is the uploader of the package that is currently longest
> in the NEW pipeline (vtk7), I'd like to make a proposal how
> transparency and also the interaction from non ftp-master members to
> review packages could be improved.
> 
> Short version: Use the salsa per-package issue tracker for problems
> that come up with the review in NEW.
> 
No.  I think the short version of your proposal is:

If the FTP team spent more time on tracking status more stuff would get 
through New.

Scott K

Re: A proposal for improving transparency of the FTP NEW process

[Gert Wollny]

Am Freitag, den 02.03.2018, 13:10 +0100 schrieb Samuel Thibault:
> Hello,
> 
> This reminds me a discussion at debconf: it could be useful that
> anybody be able to submit issues with the NEW package, so that for
> obvious things ftpmaster doesn't even have to spend time, and ideally
> ftpmaster would only look at packages which have already been
> reviewed not only by the uploader.

Well, in a way sponsored packages already have this. 

Anyway, while I don't think that this doesn't make that much sense for
packages that need to go through NEW only because of a so-name change
or because some additional binary package was added, for new source
packages this would be very useful and it was also my intent. 

To formalized this one could, for instance, add a pseudo-package salsa
"ftp-new-source" where for every new source package the uploader would
have to file a "request for review " bug, that must then be closed by
someone else who would then be added to the changelog with "Reviewed-
by: name ", and any  upload that introduces a new source package
could be required to have at least one such entry.

For large package this would also make it easier to have a shared
review, because in the bug tracker reviewers could claim responsibility
for parts of the source package. 

Best, 
Gert 

Re: A proposal for improving transparency of the FTP NEW process

[Lars Wirzenius]

On Fri, 2018-03-02 at 13:00 +0100, Gert Wollny wrote:
> as the one who is the uploader of the package that is currently longest
> in the NEW pipeline (vtk7), I'd like to make a proposal how
> transparency and also the interaction from non ftp-master members to
> review packages could be improved.

I'm not involved with the ftp master team in any way, except I
occasionally make them do work by uploading things that go to thew NEW
queue. In the past decade ago, the NEW processing has almost always
been fast, and when it hasn't, it's been due to issues like a Debian
release happening.

I assume that the reason my packages have been processed is due to one
of two reasons: a) I get quoted on LWN several times a year, so I'm a
celebrity and get special treatment or b) I take care to make my
packages easy to review. I've had a couple of rejections, and those
were clear bugs, and after I fixed them, the re-review went fast.

> Short version: Use the salsa per-package issue tracker for problems
> that come up with the review in NEW.

Counter proposal: let's work on ways in which uploaders can make it
easy and quick for ftp masters to review packages in NEW. The idea
should be, in my opinion, that any package that requires more than a
day of work to review should be rejected by default.


signature.asc
Description: This is a digitally signed message part

Re: A proposal for improving transparency of the FTP NEW process

[Samuel Thibault]

Hello,

This reminds me a discussion at debconf: it could be useful that anybody
be able to submit issues with the NEW package, so that for obvious
things ftpmaster doesn't even have to spend time, and ideally ftpmaster
would only look at packages which have already been reviewed not only by
the uploader.

Samuel

A proposal for improving transparency of the FTP NEW process

[Gert Wollny]

Dear all, 

as the one who is the uploader of the package that is currently longest
in the NEW pipeline (vtk7), I'd like to make a proposal how
transparency and also the interaction from non ftp-master members to
review packages could be improved.

Short version: Use the salsa per-package issue tracker for problems
that come up with the review in NEW.

Long version: 

First a short piece of history about this package: The version
currently in the pipeline is a second upload, after ftp master
(rightfully) complained about a bunch of files with questionable
copyright (not even upstream could clarify the situation), and some
other issues that we all fixed, I re-uploaded the package and now it
sits there for six month. Granted, the package is quite big, and being
a reviewer as ftp-master is a tedious job, but I would have hoped that
a second upload dedicated at fixing the indicated problems would result
in a shortened review time, because the review could focus on these
issues only. 

For this reason, to make the review process a bit more transparent, and
also to make it easier for others who want to help reviewing packages
in a visible way, I'd like to propose the following: With the new
gitlab environment on salsa every package hosted there has an issue
tracker that can be enabled. Given that Debian user-visible bugs are
tracked in the usual Debian bug tracker, I'd suggest to use these issue
trackers (also) for problems that relate to packages in the NEW
pipeline. Some teams already use the issue tracker for tracking work on
packages that is of not much interest to Debian users. Adding the
issues related to packages are still under review by ftp-master before
they can enter the archive seems like a logical step to me.

Now, if the reviewers of a package finds a problem, they could open an
issue in the issue tracker for this package as they go. This has the
nice effect for the maintainers that they could see early in the
process if something needs fixing, and they could already work on it
before ftp-master send the summary reject message. In this reject
message ftp-master cold simply list the issues like given in the
tracker. Since teams and maintainers might use the issue tracker for
other work as well, some unique label, like "new-pipeline" could be
helpful here.   

Since ftp-master also sometimes sends messages like "I let it pass for
now, but please fix it with the next upload", using the package issue
tracker would also be a way to keep track of these minor issues.
Specifically, because some package have to go through NEW only once in
there lifetime, issues like these might get overlooked if they are not
tracked properly.

Because closing an issue with a commit message nicely adds a link to
the corresponding commit, the reviewer of a re-upload has an easy way
to check how the issues with the package were closed without navigating
through the file hierarchy of the package, hopefully making a second
review faster.  

In order to not interfere with the Debian bugtracker, the commit
messages used to close these issues on salsa could only use "fixes" or
"resolved" instead of "closing" to close an issue.

The nice thing about this approach is that all the infrastructure is
already in place and maintainers only need to enable the issue tracker
for their packages before they do an upload to new. 

What do you think about this? 

Best,
Gert