Re: feedback for NEW packages: switch to using the BTS?

[Sean Whitton]

Hello,

On Sun 01 May 2022 at 07:52am +08, Paul Wise wrote:

> On Sat, 2022-04-30 at 14:20 -0700, Sean Whitton wrote:
>
>> This would really slow things down;
>> I don't think we could work that way.
>
> Using separate bug reports or not would of course be up to the person
> filing them, but I expect the process could be automated well enough
> that it wouldn't be much different to the current process (which I am
> not familiar with). An automated multi-bug process might be something
> like this; press the button for filing a bug, get a template with the
> right version etc, type out the problem details, press send, repeat the
> process. Is the current process any different other than the repeating?

It's quicker to to edit a single text file, review the whole thing and
then send it.

-- 
Sean Whitton

Re: feedback for NEW packages: switch to using the BTS?

[Don Armstrong]

On Sat, 30 Apr 2022, Paul Wise wrote:
> I suppose debbugs could allow the package state to go out of sync with
> NEW and instead retain the addresses for a certain period of time
> after packages are removed from NEW and while there are open bugs on
> the packages that were removed from NEW. 

This might be a reasonable process to figure out. We currently don't
disambiguate between "this package never existed" and "this package used
to exist".

Doing that would allow the current FTP master process to work while
tracking things in the BTS.

REJECT process would create a bug against the source package which can
be optionally cloned by the maintainer if they want to fix issues
separately.

Packages uploaded to NEW would create entries in the to-be-created
"historical Maintainers" file. We wouldn't bother to track detailed
versioning information; that will get fixed up when the package actually
transits is ACCEPTED.

Still unresolved is how this intersects with the search that people use
to reassign bugs against outdated packages to the current package (or
close bugs in removed packages).

-- 
Don Armstrong  https://www.donarmstrong.com

Grimble left his mother in the food store and went to the launderette
and watched the clothes go round. It was a bit like color television
only with less plot.
 -- Clement Freud _Grimble_

Re: feedback for NEW packages: switch to using the BTS?

[Timo Röhling]

What does it help to have it sitting there instead of rejected?

I gave it more thought, and I don't think it has to be sitting in
the NEW queue at all. As far as I understand it, the main advantage
to gain is additional context; a bug report provides documentation
and a place to discuss solutions for packaging issues, especially if
the NEW package is maintained by a team. None of this requires the
packages to keep waiting in NEW, it just requires the BTS to learn
about them, so the bug reports can be assigned properly.

There are probably edge cases to consider (such as invalid package
names or hijacking uploads) where it is not possible to assign the
bugs. I still like the idea to expand the BTS scope to include
pre-archive packaging bugs, provided that the assumption that most
rejected package uploads will be fixed and accepted eventually,
holds.


Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: feedback for NEW packages: switch to using the BTS?

[Scott Kitterman]

On May 1, 2022 4:44:21 PM UTC, "Timo Röhling"  wrote:
>* Scott Kitterman  [2022-04-29 23:32]:
>> I don't understand why this is any better than just rejecting the
>> package?  Once it's been determined that the upload won't be
>> accepted, I don't see a benefit to having it remain in New.
>I think this is a matter of perspective: for the FTP team, the
>upload is basically a stateless process; a package is either fit for
>inclusion in the archive or it is not. For the uploading DD, it is
>merely a particular (undesirable) state in the packaging process.
>The NEW queue is a mandatory review of their packaging effort, and
>the REJECT is feedback which they will (hopefully) address, and then
>try again. Maybe, in some circumstances, they will abandon the
>package, but I'd say that this not typical.
>
>Converting the NEW review process into a bug-centric approach
>instead of a mail-centric approach reinforces the DD's point of
>view, focusing more on the package upload as a part of package
>maintainership.
>
>The question is: how will this affect the FTP team's point of view?
>If it will clutter the NEW queue with half-processed packages, I'd
>say it is a bad tradeoff, because it adds mental load to the FTP
>team. At a minimum, there needs to be appropriate tooling which
>allows to distinguish easily between new packages requiring FTP team
>review and packages waiting for fixes to be applied.

I don't think that answers my question at all.

Yes, someone could invest a bunch of time in updating DAK so that known 
unacceptable packages wouldn't clutter the review que, but how would that be 
better than what we have now?

What does it help to have it sitting there instead of rejected?

Scott K

Re: feedback for NEW packages: switch to using the BTS?

[Timo Röhling]

* Scott Kitterman  [2022-04-29 23:32]:

I don't understand why this is any better than just rejecting the
package?  Once it's been determined that the upload won't be
accepted, I don't see a benefit to having it remain in New.

I think this is a matter of perspective: for the FTP team, the
upload is basically a stateless process; a package is either fit for
inclusion in the archive or it is not. For the uploading DD, it is
merely a particular (undesirable) state in the packaging process.
The NEW queue is a mandatory review of their packaging effort, and
the REJECT is feedback which they will (hopefully) address, and then
try again. Maybe, in some circumstances, they will abandon the
package, but I'd say that this not typical.

Converting the NEW review process into a bug-centric approach
instead of a mail-centric approach reinforces the DD's point of
view, focusing more on the package upload as a part of package
maintainership.

The question is: how will this affect the FTP team's point of view?
If it will clutter the NEW queue with half-processed packages, I'd
say it is a bad tradeoff, because it adds mental load to the FTP
team. At a minimum, there needs to be appropriate tooling which
allows to distinguish easily between new packages requiring FTP team
review and packages waiting for fixes to be applied.


Cheers
Timo


--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: feedback for NEW packages: switch to using the BTS?

[nick black]

Paul Wise left as an exercise for the reader:
> I think the same problem and suggestions also apply with the current
> system of prods and rejects, so please do add or propose adding it in
> the appropriate documentation and templates. I will of course seek to
> preserve these statements if I end up working on the BTS proposal.

absolutely, there's just enough opacity in a current REJECT (in
my fairly minimal experience) that i think it less susceptible
to such invalid inferences. but yes, this is something that
could go in the Mentors notes now.

(to be sure, at no time did i intend to impugn your suggestions
 in this thread, which seem a definite improvement from the
 user's perspective.)

-- 
nick black -=- https://www.nick-black.com
to make an apple pie from scratch,
you need first invent a universe.


signature.asc
Description: PGP signature

Re: feedback for NEW packages: switch to using the BTS?

[Paul Wise]

On Sat, 2022-04-30 at 22:47 -0400, nick black wrote:

> i understand. i suppose that what i'm saying is it will probably
> be worthwhile to convey in Mentors etc. documentation that
> "resolving the bugs filed thus far [regarding the package in
> NEW] does not imply that no further bugs will be filed."
> 
> i'm just worried that people will get a bug filed that is not a
> complete, exhaustive analysis, and think that it's the only
> thing standing between them and archive glory. perhaps whatever
> files the bug ought indicate this in the text of the bug itself.

I think the same problem and suggestions also apply with the current
system of prods and rejects, so please do add or propose adding it in
the appropriate documentation and templates. I will of course seek to
preserve these statements if I end up working on the BTS proposal.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: feedback for NEW packages: switch to using the BTS?

[nick black]

Paul Wise left as an exercise for the reader:
> > if resolving the resulting bug report is the bar needing clearing to
> > enter the archive, that does seem to require FTPmasters to create a
> > complete statement of REJECT-worthy problems in each uploaded
> > package, which seems like more work than they must currently do.
> 
> The bar for acceptance would be the same as it is now, the proposal
> just changes how the issues blocking acceptance are communicated.
> Now you get a REJECT mail, under the proposal you get a serious bug.

i understand. i suppose that what i'm saying is it will probably
be worthwhile to convey in Mentors etc. documentation that
"resolving the bugs filed thus far [regarding the package in
NEW] does not imply that no further bugs will be filed."

i'm just worried that people will get a bug filed that is not a
complete, exhaustive analysis, and think that it's the only
thing standing between them and archive glory. perhaps whatever
files the bug ought indicate this in the text of the bug itself.

-- 
nick black -=- https://www.nick-black.com
to make an apple pie from scratch,
you need first invent a universe.


signature.asc
Description: PGP signature

Re: feedback for NEW packages: switch to using the BTS?

[Paul Wise]

On Sat, 2022-04-30 at 14:20 -0700, Sean Whitton wrote:

> This would really slow things down;
> I don't think we could work that way.

Using separate bug reports or not would of course be up to the person
filing them, but I expect the process could be automated well enough
that it wouldn't be much different to the current process (which I am
not familiar with). An automated multi-bug process might be something
like this; press the button for filing a bug, get a template with the
right version etc, type out the problem details, press send, repeat the
process. Is the current process any different other than the repeating?

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: feedback for NEW packages: switch to using the BTS?

[Sean Whitton]

Hello,

On Sat 30 Apr 2022 at 07:44AM +08, Paul Wise wrote:

> It also means the ftpmasters can file separate bugs for each problem,
> rather than combining them all into one mail.

This would really slow things down; I don't think we could work that
way.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: feedback for NEW packages: switch to using the BTS?

[Paul Wise]

On Sat, 2022-04-30 at 03:09 -0400, nick black wrote:

> currently, as far as i understand things, a REJECT can be issued
> for the first REJECT-worthy problem.

The same would occur under the proposal, except that there would
presumably be separate bug reports for separate issues.

> if resolving the resulting bug report is the bar needing clearing to
> enter the archive, that does seem to require FTPmasters to create a
> complete statement of REJECT-worthy problems in each uploaded
> package, which seems like more work than they must currently do.

The bar for acceptance would be the same as it is now, the proposal
just changes how the issues blocking acceptance are communicated.
Now you get a REJECT mail, under the proposal you get a serious bug.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: feedback for NEW packages: switch to using the BTS?

[nick black]

Paul Wise left as an exercise for the reader:
> Packages with incomplete or incorrect debian/copyright information
> currently would recieve a REJECT rather than acceptance. The proposal
> would not change that, just turn that REJECT into a bug report, which
> you could fix in a second upload to NEW and then the package would be
> reprocessed and get an ACCEPT or another bug.

currently, as far as i understand things, a REJECT can be issued
for the first REJECT-worthy problem. if resolving the resulting
bug report is the bar needing clearing to enter the archive,
that does seem to require FTPmasters to create a complete
statement of REJECT-worthy problems in each uploaded package,
which seems like more work than they must currently do.

if the resulting bug is *not* a complete statement of problems,
and that is understood, this isn't an issue. just a thought.

-- 
nick black -=- https://www.nick-black.com
to make an apple pie from scratch,
you need first invent a universe.


signature.asc
Description: PGP signature

Re: feedback for NEW packages: switch to using the BTS?

[Paul Wise]

On Sat, 2022-04-30 at 00:13 +, Scott Kitterman wrote:

> I'm still not understanding how any of that needs to have a package
> we've decided not to accept sitting in New?

My thinking is that debbugs would require a package (imported from
new.822) to exist for maintainer addresses. If you file a bug, then
remove the package from NEW, then it disappears from the BTS too and
subsequent mails to the bug wouldn't have a place to be forwarded to
and would go to the address for unknown packages.

I suppose debbugs could allow the package state to go out of sync with
NEW and instead retain the addresses for a certain period of time after
packages are removed from NEW and while there are open bugs on the
packages that were removed from NEW. Then some process could regularly
close and archive the bugs filed by ftpmasters that were not solved.

I think it would be simpler to handle this on the dak side though,
by keeping the packages around, perhaps in OLD instead of NEW :)
and then rejecting them after a certain time and closing the bugs.

> In my mind if a package is in New it's in one of two states:
> 
> 1.  Not reviewed yet.
> 
> 2.  Partly/completely reviewed, pending resolution of a question
> (e.g. prod sent to maintainer, waiting for reply).
> 
> What's the advantage of added a third:
> 
> 3.  Not going to be accepted, waiting for a new upload or for enough
> time to pass before rejecting.

States 2 and 3 are very similar; there is something to be resolved with
the package and it is waiting on maintainer/uploader action for that.
The only difference is 2 is waiting on a reply, 3 waits on an upload.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: feedback for NEW packages: switch to using the BTS?

[Scott Kitterman]

On April 29, 2022 11:44:54 PM UTC, Paul Wise  wrote:
>On Fri, 2022-04-29 at 23:32 +, Scott Kitterman wrote:
>
>> I don't understand why this is any better than just rejecting the
>> package?  Once it's been determined that the upload won't be
>> accepted, I don't see a benefit to having it remain in New.
>
>The initial upload might not get an ACCEPT, but later revisions of the
>package might be suitable for inclusion in Debian.
>
>The main aim of the proposal is transparency for all problems getting
>the package into NEW. Often it isn't clear what happened to a package
>that was rejected, the proposal makes it easy to find out; just check
>the bug reports for the source package.
>
>It also means the ftpmasters can file separate bugs for each problem,
>rather than combining them all into one mail.
>
>It also means the ftpmasters can easily point out other issues they
>noticed that don't block the package from being accepted and ensure
>that those issues won't be forgotten.
>

Most of that seems mostly reasonable, but I'm still not understanding how any 
of that needs to have a package we've decided not to accept sitting in New?

In my mind if a package is in New it's in one of two states:

1.  Not reviewed yet.

2.  Partly/completely reviewed, pending resolution of a question (e.g. prod 
sent to maintainer, waiting for reply).

What's the advantage of added a third:

3.  Not going to be accepted, waiting for a new upload or for enough time to 
pass before rejecting.

Scott K

Re: feedback for NEW packages: switch to using the BTS?

[Paul Wise]

On Fri, 2022-04-29 at 23:32 +, Scott Kitterman wrote:

> I don't understand why this is any better than just rejecting the
> package?  Once it's been determined that the upload won't be
> accepted, I don't see a benefit to having it remain in New.

The initial upload might not get an ACCEPT, but later revisions of the
package might be suitable for inclusion in Debian.

The main aim of the proposal is transparency for all problems getting
the package into NEW. Often it isn't clear what happened to a package
that was rejected, the proposal makes it easy to find out; just check
the bug reports for the source package.

It also means the ftpmasters can file separate bugs for each problem,
rather than combining them all into one mail.

It also means the ftpmasters can easily point out other issues they
noticed that don't block the package from being accepted and ensure
that those issues won't be forgotten.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: feedback for NEW packages: switch to using the BTS?

[Paul Wise]

On Fri, 2022-04-29 at 18:08 +0200, Andreas Tille wrote:

> To give some actual examples that IMHO are candidates for accept + bug
> report:

Packages with incomplete or incorrect debian/copyright information
currently would recieve a REJECT rather than acceptance. The proposal
would not change that, just turn that REJECT into a bug report, which
you could fix in a second upload to NEW and then the package would be
reprocessed and get an ACCEPT or another bug.

> I think Paul was not talking about non-distributable software but rather
> code that is considered DFSG free but missing proper paragraphs inside
> d/copyright which can be easily fixed in BTS.

I think completely non-distributable software should get a bug against
the package in NEW tagged as wontfix and then the NEW package should
get removed with a REJECT mail mentioning the reason.

There may be cases where it might be possible to fix some issue with a
unredistributable file in an otherwise redistributable package, in
those cases a serious bug against the package in NEW should be filed
and the maintainer be given a chance to fix the issue.

The proposal doesn't change anything with regard to packages with
incomplete or incorrect debian/copyright information, that is a
separate issue to what this proposal aims to solve.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: feedback for NEW packages: switch to using the BTS?

[Scott Kitterman]

On April 29, 2022 11:04:57 PM UTC, Paul Wise  wrote:
>On Fri, 2022-04-29 at 13:36 +0100, Steve McIntyre wrote:
>
>> Just to clarify: is this suggesting that packages from NEW would end
>> up in the archive even with serious bugs? If not, what's the point of
>> the "eventual removal" above? I'm not following you here...
>
>No, the bugs I propose to be filed would be filed *while* the package
>is in NEW and then *after* the serious ones are all fixed, that is a
>positive indication that the package is probably suitable for an ACCEPT
>action by the ftpmasters, although of course a final review is needed.

I don't understand why this is any better than just rejecting the package?  
Once it's been determined that the upload won't be accepted, I don't see a 
benefit to having it remain in New.

Scott K

Re: feedback for NEW packages: switch to using the BTS?

[Paul Wise]

On Fri, 2022-04-29 at 11:26 -0400, Scott Kitterman wrote:

> 1.  Making reject/prod emails more public:
> 
> There are actually two different types of messages we can send while 
> reviewing 
> packages in DAK:
> 
> a.  Reject: As indicated by the name, this is an email that accompanies the 
> package being rejected from the New queue.
> 
> b. Prod: These leave the package in New and are used to ask the maintainer a 
> question.

The proposal essentially turns most Reject mails into Prod mails and
turns all Prod mails to bugs. The only Reject mails that will be sent
are if the bugs aren't being solved after a certain amount of time,
then the packages will be removed from NEW and get a REJECT.

> My recollection is that these go to the maintainer and changed by.  In many 
> cases the maintainer is a team, so these are already not considered private.

That makes it seem more reasonable to make everything public,
also I note the REJECT mails are available to all Debian members. 

ssh://mirror.ftp-master.debian.org/srv/ftp-master.debian.org/queue/reject/

I have been reading these files and note that many of them are
automated, probably the most common one is people mistakenly doing
source-only uploads to NEW. I have another idea to add a pre-reject API
to dakweb for use by dupload/dput/dput-ng but haven't a proposal yet.

> Whatever is done in this area should definitely be integrated into DAK as 
> part 
> of process-new.  I don't think it would be helpful to try to add steps to the 
> work the FTP Team has to do.  New is slow enough that we should definitely 
> not 
> make is slower.

That seems reasonable.

> Also, if any of this will depend on the existence of a WNPP bug, then policy 
> should be changed to require them.  They are currently not required and not 
> everyone files them.  If something is going to be required to get a package 
> accepted, it should be in policy.

The proposal was created because I think filing WNPP bugs for every NEW
upload is very much suboptimal, so I came up with the proposal as an
alternative to that, so I'd avoid filing additional WNPP bugs for NEW
above the existing ITP/etc bugs.

> On a related note: It is not unusual for me to notice packaging issues which 
> are not serious enough to reject the package, but should be documented.

One advantage of this proposal is that it allows filing bugs against
any package already in NEW, so you could do that even if you notice a
minor issue while processing a package, then have to move on to
something else, then come back to the package the next day.

> This is particularly true when an existing package goes through New due to a 
> new 
> binary package.  It would be useful to have the option to accept with bug 
> report so that we don't need to go through the steps of filing a bug by hand. 
>  
> When I do this I also send a prod to the uploader/maintainer and it's
> effectively doing work twice.  Making New processing more efficient will help 
> make it faster.

That seems like a useful feature, but I think I would separate this
into two steps, first file the bug, maybe from dak and then accept.

> 2.  Not rejecting packages with serious defects:
> 
> I'm not sure I understand what it proposed:
> 
> > The ftpmasters could simply file severity serious bug reports against
> > NEW packages that have issues blocking their entry into Debian. When
> > there are minor issues noticed at the same time, then file bugs of a
> > lower severity. Only when a NEW package has not had its serious bugs
> > fixed in a long time would an eventual removal and REJECT mail happen,
> > perhaps after a few months of zero action on the bug reports.
> 
> I think this proposes to accept all packages regardless of how defective they 
> are and then remove them later if the bugs aren't fixed.  If that's what is 
> proposed, my thought is absolutely not.

That is definitely *not* what was proposed, and I think it is a bad
idea to accept packages that are not suitable for the archive. The
proposal is simply to change the NEW packages feedback mechanism and
change the procedure for packages that don't yet meet Debian standards.

> If a package is not suitable for the archive then it should be rejected with 
> appropriate feedback and re-uploaded.

The proposal is to instead of rejecting the package, keep it in NEW,
file a serious bug against the package in NEW, wait for it to be fixed
and then reprocess the package.

> Note: Although I am a member of the FTP Team, I am only speaking for myself 
> here, not the team as a whole.

Thanks for your response.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: feedback for NEW packages: switch to using the BTS?

[Paul Wise]

On Fri, 2022-04-29 at 13:36 +0100, Steve McIntyre wrote:

> Just to clarify: is this suggesting that packages from NEW would end
> up in the archive even with serious bugs? If not, what's the point of
> the "eventual removal" above? I'm not following you here...

No, the bugs I propose to be filed would be filed *while* the package
is in NEW and then *after* the serious ones are all fixed, that is a
positive indication that the package is probably suitable for an ACCEPT
action by the ftpmasters, although of course a final review is needed.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: feedback for NEW packages: switch to using the BTS?

[Scott Kitterman]

On Friday, April 29, 2022 12:08:21 PM EDT Andreas Tille wrote:
> Hi Scott,
> 
> thanks a lot for becoming involved into this discussion.
> 
> Am Fri, Apr 29, 2022 at 11:26:33AM -0400 schrieb Scott Kitterman:
> > 2.  Not rejecting packages with serious defects:
> > 
> > I'm not sure I understand what it proposed:
> > > The ftpmasters could simply file severity serious bug reports against
> > > NEW packages that have issues blocking their entry into Debian. When
> > > there are minor issues noticed at the same time, then file bugs of a
> > > lower severity. Only when a NEW package has not had its serious bugs
> > > fixed in a long time would an eventual removal and REJECT mail happen,
> > > perhaps after a few months of zero action on the bug reports.
> > 
> > I think this proposes to accept all packages regardless of how defective
> > they are and then remove them later if the bugs aren't fixed.  If that's
> > what is proposed, my thought is absolutely not.
> > 
> > If a package is not suitable for the archive then it should be rejected
> > with appropriate feedback and re-uploaded.
> 
> To give some actual examples that IMHO are candidates for accept + bug
> report:
> 
>1. In case versioneer.py (Creative Commons "Public Domain Dedication"
>   license (CC0-1.0) is missing in d/copyright like in propka[1]
> 
>2. Packages which are DFSG free but might miss some single copyright
>   statement.
>   My favourite example would be r-bioc-basilisk[2].  In this specific
>   example I even disagree with ftpmaster[3] but I see no real chance
>   as a maintainer to discuss my point and can only re-re-re-upload
>   what I consider correct.  So I gave up and this package is not yet
>   inside Debian.  This could be discussed more sensibly in a bug
>   report IMHO.
> 
> I think Paul was not talking about non-distributable software but rather
> code that is considered DFSG free but missing proper paragraphs inside
> d/copyright which can be easily fixed in BTS.

The in that case, it's just a variant of accept plus file a bug.

I don't think it's productive to get into a discussion on the distinction 
between "buggy" and "too buggy to go in the archive".  It's very dependent on 
the specifics of the situation.

Also, accept with severe bugs and remove later if not fixed requires someone to 
track these issues and follow-up on getting them removed.  That seems like 
more work that isn't easily automated (we don't do automatic removals from 
Unstable now - they are at most semi-automatic where an FTP Team member still 
needs to review and do the actual removal).

To summarize:

The reject/prod messages are already not considered private.  Any additional 
addressing of them to whatever location in the BTS is a matter of have the BTS 
be ready for it and adding it to process-new to send it.

If the answer is that ITP bugs are now required, then policy should be updated 
first.

It would be useful to have a mechanism within DAK process-new to file a bug 
with the prod note of the appropriate severity.  I think if this existed you 
would be inclined to see more accept with bug resolutions.

I think accepting something to buggy to remain in unstable and removing it 
later if not fixed is not something that should be pursued.

Still not speaking for the FTP Team.

Scott K

signature.asc
Description: This is a digitally signed message part.

Re: feedback for NEW packages: switch to using the BTS?

[Andreas Tille]

Hi Scott,

thanks a lot for becoming involved into this discussion.

Am Fri, Apr 29, 2022 at 11:26:33AM -0400 schrieb Scott Kitterman:
> 2.  Not rejecting packages with serious defects:
> 
> I'm not sure I understand what it proposed:
> 
> > The ftpmasters could simply file severity serious bug reports against
> > NEW packages that have issues blocking their entry into Debian. When
> > there are minor issues noticed at the same time, then file bugs of a
> > lower severity. Only when a NEW package has not had its serious bugs
> > fixed in a long time would an eventual removal and REJECT mail happen,
> > perhaps after a few months of zero action on the bug reports.
> 
> I think this proposes to accept all packages regardless of how defective they 
> are and then remove them later if the bugs aren't fixed.  If that's what is 
> proposed, my thought is absolutely not.
> 
> If a package is not suitable for the archive then it should be rejected with 
> appropriate feedback and re-uploaded.

To give some actual examples that IMHO are candidates for accept + bug
report:

   1. In case versioneer.py (Creative Commons "Public Domain Dedication"
  license (CC0-1.0) is missing in d/copyright like in propka[1]

   2. Packages which are DFSG free but might miss some single copyright
  statement.
  My favourite example would be r-bioc-basilisk[2].  In this specific
  example I even disagree with ftpmaster[3] but I see no real chance
  as a maintainer to discuss my point and can only re-re-re-upload
  what I consider correct.  So I gave up and this package is not yet
  inside Debian.  This could be discussed more sensibly in a bug
  report IMHO.

I think Paul was not talking about non-distributable software but rather
code that is considered DFSG free but missing proper paragraphs inside
d/copyright which can be easily fixed in BTS.

> Note: Although I am a member of the FTP Team, I am only speaking for myself 
> here, not the team as a whole.

Thanks a lot for speaking at a "competent yourself" ;-)

Kind regards

   Andreas.
 

[1] 
https://alioth-lists.debian.net/pipermail/debian-med-packaging/2022-February/098605.html
[2] 
https://alioth-lists.debian.net/pipermail/r-pkg-team/2022-February/024165.html
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=991859#42

-- 
http://fam-tille.de

Re: feedback for NEW packages: switch to using the BTS?

[Scott Kitterman]

On Wednesday, April 27, 2022 8:54:05 PM EDT Paul Wise wrote:
> Hi all,
> 
> During the discussions about NEW on debian-devel in recent times, I had
> the idea that instead of the current mechanism of sending REJECT mails,
> Debian could switch to using the BTS for most feedback on NEW packages.
> 
> This means that most discussion about NEW packages would become public,
> but of course the ftpmasters could opt to send private mail instead if
> in some cases if there were sensitive issues to be discussed.
> 
> The ftpmasters could simply file severity serious bug reports against
> NEW packages that have issues blocking their entry into Debian. When
> there are minor issues noticed at the same time, then file bugs of a
> lower severity. Only when a NEW package has not had its serious bugs
> fixed in a long time would an eventual removal and REJECT mail happen,
> perhaps after a few months of zero action on the bug reports.
> 
> The changes that are needed to make this happen include:
> 
> The community needs to decide that this change is a good idea and be
> willing to recieve most feedback on their work in public.
> 
> The BTS and ftpmaster teams need to agree with this change.
> One BTS admin said this is feasible in principle and one of the
> ftpmasters seemed to like the idea when I mentioned it on IRC.
> 
> Where the bug mail should go to needs to be decided on. Personally I'm
> thinking the person who did the upload plus the Maintainer field.
> 
> The people doing processing of NEW packages need to be willing to file
> bug reports against them where necessary.
> 
> dak needs to export debian/changelog version lists for NEW packages.
> 
> debbugs needs to import packages/versions from the new.822 file:
> 
> https://ftp-master.debian.org/new.822
> 
> dak needs to link to the BTS from new.822 and the NEW packages info.
> 
> https://ftp-master.debian.org/new.html
> 
> The technical work needed here is minimal, so if there are no other
> volunteers to do that work, then I would be willing to do it. I have
> experience with Perl/Python but only a small amount of experience with
> working on the debbugs and dak codebases.
> 
> Thoughts welcome, especially from those who don't like this idea.

This proposal has a number of parts and I'm not entirely confident I understand 
what it being proposed.  A few thoughts/clarifications.

1.  Making reject/prod emails more public:

There are actually two different types of messages we can send while reviewing 
packages in DAK:

a.  Reject: As indicated by the name, this is an email that accompanies the 
package being rejected from the New queue.

b. Prod: These leave the package in New and are used to ask the maintainer a 
question.

My recollection is that these go to the maintainer and changed by.  In many 
cases the maintainer is a team, so these are already not considered private.

Whatever is done in this area should definitely be integrated into DAK as part 
of process-new.  I don't think it would be helpful to try to add steps to the 
work the FTP Team has to do.  New is slow enough that we should definitely not 
make is slower.

Also, if any of this will depend on the existence of a WNPP bug, then policy 
should be changed to require them.  They are currently not required and not 
everyone files them.  If something is going to be required to get a package 
accepted, it should be in policy.

On a related note: It is not unusual for me to notice packaging issues which 
are not serious enough to reject the package, but should be documented.  This 
is particularly true when an existing package goes through New due to a new 
binary package.  It would be useful to have the option to accept with bug 
report so that we don't need to go through the steps of filing a bug by hand.  
When I do this I also send a prod to the uploader/maintainer and it's 
effectively doing work twice.  Making New processing more efficient will help 
make it faster.

2.  Not rejecting packages with serious defects:

I'm not sure I understand what it proposed:

> The ftpmasters could simply file severity serious bug reports against
> NEW packages that have issues blocking their entry into Debian. When
> there are minor issues noticed at the same time, then file bugs of a
> lower severity. Only when a NEW package has not had its serious bugs
> fixed in a long time would an eventual removal and REJECT mail happen,
> perhaps after a few months of zero action on the bug reports.

I think this proposes to accept all packages regardless of how defective they 
are and then remove them later if the bugs aren't fixed.  If that's what is 
proposed, my thought is absolutely not.

If a package is not suitable for the archive then it should be rejected with 
appropriate feedback and re-uploaded.

Note: Although I am a member of the FTP Team, I am only speaking for myself 
here, not the team as a whole.

Scott K

signature.asc
Description: This is a digitally signed message part.

Re: feedback for NEW packages: switch to using the BTS?

[The Wanderer]

On 2022-04-29 at 08:36, Steve McIntyre wrote:

> Paul Wise wrote:
> 
>> During the discussions about NEW on debian-devel in recent times, I
>> had the idea that instead of the current mechanism of sending
>> REJECT mails, Debian could switch to using the BTS for most
>> feedback on NEW packages.
>> 
>> This means that most discussion about NEW packages would become
>> public, but of course the ftpmasters could opt to send private mail
>> instead if in some cases if there were sensitive issues to be
>> discussed.
>> 
>> The ftpmasters could simply file severity serious bug reports
>> against NEW packages that have issues blocking their entry into
>> Debian. When there are minor issues noticed at the same time, then
>> file bugs of a lower severity. Only when a NEW package has not had
>> its serious bugs fixed in a long time would an eventual removal and
>> REJECT mail happen, perhaps after a few months of zero action on
>> the bug reports.
> 
> Just to clarify: is this suggesting that packages from NEW would end 
> up in the archive even with serious bugs? If not, what's the point
> of the "eventual removal" above? I'm not following you here...

I parsed that not as "removal from the archive" but as "removal from the
NEW queue", much as now happens (in some order and via some mechanism,
perhaps a manual one) when a REJECT mail is sent.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw



signature.asc
Description: OpenPGP digital signature

Re: feedback for NEW packages: switch to using the BTS?

[Steve McIntyre]

Paul Wise wrote:
>
>During the discussions about NEW on debian-devel in recent times, I had
>the idea that instead of the current mechanism of sending REJECT mails,
>Debian could switch to using the BTS for most feedback on NEW packages.
>
>This means that most discussion about NEW packages would become public,
>but of course the ftpmasters could opt to send private mail instead if
>in some cases if there were sensitive issues to be discussed.
>
>The ftpmasters could simply file severity serious bug reports against
>NEW packages that have issues blocking their entry into Debian. When
>there are minor issues noticed at the same time, then file bugs of a
>lower severity. Only when a NEW package has not had its serious bugs
>fixed in a long time would an eventual removal and REJECT mail happen,
>perhaps after a few months of zero action on the bug reports.

Just to clarify: is this suggesting that packages from NEW would end
up in the archive even with serious bugs? If not, what's the point of
the "eventual removal" above? I'm not following you here...

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"We're the technical experts.  We were hired so that management could
 ignore our recommendations and tell us how to do our jobs."  -- Mike Andrews

Re: feedback for NEW packages: switch to using the BTS?

[Andreas Tille]

Hi Paul,

Am Fri, Apr 29, 2022 at 07:13:42AM +0800 schrieb Paul Wise:
> 
> The packaging work is supposed to start *after* the ITP is filed,

Sure.

> so
> this is a suboptimal order to do things in and doesn't provide much
> value,

The "packaging work" to create the debian/ dir is a 1min process since
for R this is automated.  Do you expect a racing condition in this
minute? 

> except as a way to prevent people packaging the same R library
> during the wait in NEW, but the presence of the R library in the R-pkg
> team VCS already provides that, so the ITP bug isn't really needed. 

The automated process also checks existing WNPP bugs so it makes sense
to file such a bug.  Moreover it is used to document rejects.

I don't want to say that WNPP bugs are a better solution than your
proposal.  My point was simply that there is some existing relation
between BTS and NEW which might be usable or not.
 
> Even when filing ITPs before packaging, for language ecosystem teams
> where all library packages for the language go through the same team,
> team co-ordination mechanisms are probably enough of a way to prevent
> duplication of work, and that work is mostly automated anyway for
> several such teams, so the ITP bug mostly isn't really needed.

I confirm that it is not strictly needed (that's another reason why we
feel pretty safe to file the ITP after the packaging) but as I explained
it has some use anyway and since it is so brain dead simple to create we
do it.

> Some teams make exceptions for packages that aren't strictly just
> libraries; for eg Rust folks to ITPs for things that ship executables,
> so that people can hear about things they might want to use on Debian.
> 
> > That's true.  I just wanted to mention that some of your ideas
> > are in a way used even now.
> 
> Yeah, the proposal was derived from the suggestion to file ITPs for all
> NEW packages, mostly aimed at avoiding the deficiencies with that idea.

Nice.

Kind regards

   Andreas.

-- 
http://fam-tille.de

Re: feedback for NEW packages: switch to using the BTS?

[Paul Wise]

On Thu, 2022-04-28 at 20:24 +0200, Andreas Tille wrote:

> But filing an ITP bug is cheap.  The R-pkg team has a script
> itp_from_debian_dir[1] which creates this bug automatically once the
> packaging work is done.

The packaging work is supposed to start *after* the ITP is filed, so
this is a suboptimal order to do things in and doesn't provide much
value, except as a way to prevent people packaging the same R library
during the wait in NEW, but the presence of the R library in the R-pkg
team VCS already provides that, so the ITP bug isn't really needed. 

Even when filing ITPs before packaging, for language ecosystem teams
where all library packages for the language go through the same team,
team co-ordination mechanisms are probably enough of a way to prevent
duplication of work, and that work is mostly automated anyway for
several such teams, so the ITP bug mostly isn't really needed.

Some teams make exceptions for packages that aren't strictly just
libraries; for eg Rust folks to ITPs for things that ship executables,
so that people can hear about things they might want to use on Debian.

> That's true.  I just wanted to mention that some of your ideas
> are in a way used even now.

Yeah, the proposal was derived from the suggestion to file ITPs for all
NEW packages, mostly aimed at avoiding the deficiencies with that idea.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: feedback for NEW packages: switch to using the BTS?

[Paul Wise]

On Thu, 2022-04-28 at 13:48 +0200, Marco d'Itri wrote:

> This looks like a good idea to me, but I think that the big problem 
> which needs to be solved is not discussing REJECTs but the packages 
> which stay in NEW for many months with no feedback.

Thanks. The idea is narrowly aimed at improving the transparency and
communication issues with NEW. It would probably mean packages packages
stay in NEW for longer, rather than getting a REJECT, then reuploaded,
more issues found, another REJECT etc (not sure if that happens tho).

As I understand it, there is currently no specified criteria for
prioritisation of processing of NEW packages, but requesting a review
on #debian-ftp for particular reasons (RC bug fixes, new binaries,
waiting time etc) often means that someone will review it much sooner.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: feedback for NEW packages: switch to using the BTS?

[Andreas Tille]

Hi Paul,

Am Thu, Apr 28, 2022 at 06:46:43PM +0800 schrieb Paul Wise:
> There are two main categories of NEW packages; source and binary.
> Packages adding an new source should have an ITP bug, but don't
> always, for eg the Rust/Golang teams don't file them for every
> library. Packages adding a new binary package often have a transition
> bug, but those are usually filed after the upload is accepted.
> 
> Filing a new WNPP bug for every NEW package upload seems a bit tedious
> and I think lots of people aren't going to bother/remember to do it.

But filing an ITP bug is cheap.  The R-pkg team has a script
itp_from_debian_dir[1] which creates this bug automatically once the
packaging work is done.
 
> The proposal I make would mean that the manual bug filing for every
> upload wouldn't be needed, instead the BTS would know about packages
> in NEW and ftpmasters and others could file bugs against them, which
> would mainly only be needed when there are REJECTable issues present.
> 
> In addition there could be separate bugs for each individual issue,
> rather than one long discussion about multiple different issues.
> This is probably especially important for low severity issues.

I'm fine with this suggestion as well.
 
> > (BTW, I usually bounce any reject to the according WNPP bug to have
> > the issue documented in a publicly visible place where any interested
> > person should look.)
> 
> That seems reasonable if the ftpmasters are fine with that.

Ftpmaster did not agreed with my suggestion so far.

> If the
> BTS for NEW proposal gets implemented it would no longer be needed.

That's true.  I just wanted to mention that some of your ideas
are in a way used even now.
 
> Am Thu, Apr 28, 2022 at 08:54:05AM +0800 schrieb Paul Wise:
> 
> > > The changes that are needed to make this happen include:
> 
> I guess tracker.d.o would also need to show NEW packages.

This would be extremely helpful.

Kind regards

Andreas.


[1] 
https://salsa.debian.org/r-pkg-team/dh-r/-/blob/master/scripts/itp_from_debian_dir

-- 
http://fam-tille.de

Re: feedback for NEW packages: switch to using the BTS?

[Marco d'Itri]

On Apr 28, Paul Wise  wrote:

> During the discussions about NEW on debian-devel in recent times, I had
> the idea that instead of the current mechanism of sending REJECT mails,
> Debian could switch to using the BTS for most feedback on NEW packages.
This looks like a good idea to me, but I think that the big problem 
which needs to be solved is not discussing REJECTs but the packages 
which stay in NEW for many months with no feedback.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: feedback for NEW packages: switch to using the BTS?

[Paul Wise]

On Thu, 2022-04-28 at 09:36 +0200, Andreas Tille wrote:

> What about WNPP bug?  When I asked ftpmaster to kindly CC their
> rejects to the WNPP bug I was told that not all packages in new have WNPP
> bugs. If we want to formalise this it could probably be enforced that new
> packages really need to have such a bug and we only need to decide
> for existing packages that need to pass NEW.

There are two main categories of NEW packages; source and binary.
Packages adding an new source should have an ITP bug, but don't
always, for eg the Rust/Golang teams don't file them for every
library. Packages adding a new binary package often have a transition
bug, but those are usually filed after the upload is accepted.

Filing a new WNPP bug for every NEW package upload seems a bit tedious
and I think lots of people aren't going to bother/remember to do it.

The proposal I make would mean that the manual bug filing for every
upload wouldn't be needed, instead the BTS would know about packages
in NEW and ftpmasters and others could file bugs against them, which
would mainly only be needed when there are REJECTable issues present.

In addition there could be separate bugs for each individual issue,
rather than one long discussion about multiple different issues.
This is probably especially important for low severity issues.

> (BTW, I usually bounce any reject to the according WNPP bug to have
> the issue documented in a publicly visible place where any interested
> person should look.)

That seems reasonable if the ftpmasters are fine with that. If the
BTS for NEW proposal gets implemented it would no longer be needed.

Am Thu, Apr 28, 2022 at 08:54:05AM +0800 schrieb Paul Wise:

> > The changes that are needed to make this happen include:

I guess tracker.d.o would also need to show NEW packages.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: feedback for NEW packages: switch to using the BTS?

[Andreas Tille]

Hi Paul,

Am Thu, Apr 28, 2022 at 08:54:05AM +0800 schrieb Paul Wise:
> During the discussions about NEW on debian-devel in recent times, I had
> the idea that instead of the current mechanism of sending REJECT mails,
> Debian could switch to using the BTS for most feedback on NEW packages.
> 
> This means that most discussion about NEW packages would become public,
> but of course the ftpmasters could opt to send private mail instead if
> in some cases if there were sensitive issues to be discussed.
> 
> The ftpmasters could simply file severity serious bug reports against
> NEW packages that have issues blocking their entry into Debian. When
> there are minor issues noticed at the same time, then file bugs of a
> lower severity. Only when a NEW package has not had its serious bugs
> fixed in a long time would an eventual removal and REJECT mail happen,
> perhaps after a few months of zero action on the bug reports.

I consider this a really nice solution for the majority of rejects and I
think a lot of good arguments were given inside the discussion you was
refering to.
 
> The changes that are needed to make this happen include:
> 
> The community needs to decide that this change is a good idea and be
> willing to recieve most feedback on their work in public.
> 
> The BTS and ftpmaster teams need to agree with this change.
> One BTS admin said this is feasible in principle and one of the
> ftpmasters seemed to like the idea when I mentioned it on IRC.

I admit I would be happy if this "seemed to like the idea" would
be confirmed here in a public place. ;-)
 
> Where the bug mail should go to needs to be decided on. Personally I'm
> thinking the person who did the upload plus the Maintainer field.

What about WNPP bug?  When I asked ftpmaster to kindly CC their rejects
to the WNPP bug I was told that not all packages in new have WNPP bugs.
If we want to formalise this it could probably be enforced that new
packages really need to have such a bug and we only need to decide for
existing packages that need to pass NEW.

(BTW, I usually bounce any reject to the according WNPP bug to have
the issue documented in a publicly visible place where any interested
person should look.)
 
> The people doing processing of NEW packages need to be willing to file
> bug reports against them where necessary.
> 
> dak needs to export debian/changelog version lists for NEW packages.
> 
> debbugs needs to import packages/versions from the new.822 file:
> 
> https://ftp-master.debian.org/new.822
> 
> dak needs to link to the BTS from new.822 and the NEW packages info.
> 
> https://ftp-master.debian.org/new.html
> 
> The technical work needed here is minimal, so if there are no other
> volunteers to do that work, then I would be willing to do it. I have
> experience with Perl/Python but only a small amount of experience with
> working on the debbugs and dak codebases.
> 
> Thoughts welcome, especially from those who don't like this idea.

Sorry, I'm amongst those who absolutely like this idea. ;-)
Thanks a lot for the follow up on the past discussion.

Kind regards

  Andreas.

-- 
http://fam-tille.de

feedback for NEW packages: switch to using the BTS?

[Paul Wise]

Hi all,

During the discussions about NEW on debian-devel in recent times, I had
the idea that instead of the current mechanism of sending REJECT mails,
Debian could switch to using the BTS for most feedback on NEW packages.

This means that most discussion about NEW packages would become public,
but of course the ftpmasters could opt to send private mail instead if
in some cases if there were sensitive issues to be discussed.

The ftpmasters could simply file severity serious bug reports against
NEW packages that have issues blocking their entry into Debian. When
there are minor issues noticed at the same time, then file bugs of a
lower severity. Only when a NEW package has not had its serious bugs
fixed in a long time would an eventual removal and REJECT mail happen,
perhaps after a few months of zero action on the bug reports.

The changes that are needed to make this happen include:

The community needs to decide that this change is a good idea and be
willing to recieve most feedback on their work in public.

The BTS and ftpmaster teams need to agree with this change.
One BTS admin said this is feasible in principle and one of the
ftpmasters seemed to like the idea when I mentioned it on IRC.

Where the bug mail should go to needs to be decided on. Personally I'm
thinking the person who did the upload plus the Maintainer field.

The people doing processing of NEW packages need to be willing to file
bug reports against them where necessary.

dak needs to export debian/changelog version lists for NEW packages.

debbugs needs to import packages/versions from the new.822 file:

https://ftp-master.debian.org/new.822

dak needs to link to the BTS from new.822 and the NEW packages info.

https://ftp-master.debian.org/new.html

The technical work needed here is minimal, so if there are no other
volunteers to do that work, then I would be willing to do it. I have
experience with Perl/Python but only a small amount of experience with
working on the debbugs and dak codebases.

Thoughts welcome, especially from those who don't like this idea.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: Seeking feedback on a meta package builder

[Sean Whitton]

Hello,

On Fri 04 Jun 2021 at 06:39PM +02, Helmut Grohne wrote:

> Hi Sean,
>
> On Thu, Jun 03, 2021 at 04:47:44PM -0700, Sean Whitton wrote:
>> dgit wraps some of the existing tools.  While dgit is mainly for humans,
>> one role it can have in automated toolchains is producing an ephemeral
>> source package for the purpose of performing a build where the real
>> input is a git tree.  dgit knows how to convert various forms of git
>> tree into source packages and there are TODOs for some others.
>
> While dgit does wrap build tools and thus could be a possible mdbp user,
> dgit does so in a way that forwards the flexibility of those build tools
> through dgit. As such, it has a subcommand for each implementation and
> lets the user fiddle with the underlying implementation in the way they
> want without providing any kind of abstraction. I'm vaguely convinced
> that for dgit, this is best. An abstraction could be added as another
> subcommand if desired, but does buy little in the end. After all, dgit
> users tend to want tight control over the build and know their favourite
> build tool well.
>
> The intended audience for the abstraction on the other hand would be
> performing many builds in a mechanical way.

Right.  I was thinking that if you wanted to perform many builds from
git trees, dgit could help obtain the .dscs.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Seeking feedback on a meta package builder

[Raphael Hertzog]

Hi,

On Thu, 10 Jun 2021, Helmut Grohne wrote:
> On Thu, Jun 10, 2021 at 03:00:31PM +0200, Raphael Hertzog wrote:
> > I want to know it precisely in the context of selecting a worker. I don't
> > want to schedule a task on a worker and later get back an answer "sorry I
> > can't handle your task", and then have to schedule it on some other
> > worker.
> 
> I'm a little unsure what you mean precisely here. [...]
> Or do you want to locally decide whether your worker can handle
> it (e.g. using data previously acquired from the worker and cached on
> the scheduler)?

That's my plan, yes.

> I guess it is not the last variant, because you cannot validate the
> distribution in any way. I guess it also is not the first variant,
> because you don't want to call into every single worker for every single
> build object. Do you confirm that you want the second variant?

Yes.

> you can disable such a worker of course. Still, I suggest that you
> cannot rule out all possible worker-specific failure modes ahead of
> time.

Right.

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS

Re: Seeking feedback on a meta package builder

[Helmut Grohne]

Hi Raphaël,

On Thu, Jun 10, 2021 at 03:00:31PM +0200, Raphael Hertzog wrote:
> I want to know it precisely in the context of selecting a worker. I don't
> want to schedule a task on a worker and later get back an answer "sorry I
> can't handle your task", and then have to schedule it on some other
> worker.

I'm a little unsure what you mean precisely here. Do you want your
scheduler call out to your worker and ask it whether it can handle the
build? Or do you want to locally decide whether your worker can handle
it (e.g. using data previously acquired from the worker and cached on
the scheduler)? Or do you want to know whether a backend is capable of
satisfying a request in principle without considering the concrete
system performing the build?

I guess it is not the last variant, because you cannot validate the
distribution in any way. I guess it also is not the first variant,
because you don't want to call into every single worker for every single
build object. Do you confirm that you want the second variant?

That might be a little more difficult due to the data collection part.
I'm not sure I can reasonably cover that.

> When I have selected a worker, I want to be sure that the worker
> is properly configured to be able to run that specific task.

That much makes sense in principle, but what do you want debusine to do
when one of your workers is misconfigured and cannot reach its primary
mirror? All builds there will fail and likely you can identify that the
failure happened too early for the package to be at fault. You can
either pass down the failure such that users have to deal with temporary
issues or automatically reschedule the build elsewhere. Additionally,
you can disable such a worker of course. Still, I suggest that you
cannot rule out all possible worker-specific failure modes ahead of
time.

> Yes, but using the ssh backend will require specific user configuration
> while the sbuild/pbuilder one could potientially be auto-selected based
> on whether the user did run the appropriate sbuild-create-chroot or
> pbuilder --create earlier.

That definitely is implementable. It could be a simple "use sbuild if it
works, else try pbuilder, else try mmdebstrap with a default mirror". I
don't think I'd use it, but it might be a good default value for tools
that need to perform builds. I've taken a note and will look into
implementing this.

> "justadeb" or "gimmeadeb" or "buildadeb" because I just want a .deb (and I
> don't care how it gets built!).

Indeed, one of my use cases is to get a log and no .debs. You can
explicitly request no artifacts and that'll prevent them from being
transferred over mdbp-ssh.

> "deblord" or "debring" - the lord of the ring to tie all package builders
> together

Hmm. The more I think about this, the more I like the untypable mdbp:
You only type it once to configure your application that does perform
builds. If you actually type mdbp regularly, you're doing it wrong. It
really is not meant as a frontend to be used interactively.

Helmut

Re: Seeking feedback on a meta package builder

[Matt Zagrabelny]

On Thu, Jun 10, 2021 at 8:01 AM Raphael Hertzog  wrote:

>
> > > PS: I already hate the "mdbp" name after having it typed so many times.
> >
> > I'm not attached to either. Any suggestions?
>
> sbp for "standardized build package" is easier to type but not necessarily
> nicer.
>
> "justadeb" or "gimmeadeb" or "buildadeb" because I just want a .deb (and I
> don't care how it gets built!).
>

vendeb - vending machine for .deb


> "deblord" or "debring" - the lord of the ring to tie all package builders
> together
>

omegabuild - the last build system (also sort of a joke about being far
from alpha software.)

-m

Re: Seeking feedback on a meta package builder

[Raphael Hertzog]

Hi,

On Tue, 08 Jun 2021, Helmut Grohne wrote:
> With "look behind the abstraction", I think that you mean that debusine
> would have to implement the mdbp api to perform worker selection. While
> that would be possible indeed, I don't see this as required though.
[…]
> I do wonder though, in what kind of situation would you want to merely
> know whether a backend can perform a build as opposed to just attempting
> it and being able to tell backend issues from actual build failures
> apart.

I want to know it precisely in the context of selecting a worker. I don't
want to schedule a task on a worker and later get back an answer "sorry I
can't handle your task", and then have to schedule it on some other
worker.

When I have selected a worker, I want to be sure that the worker
is properly configured to be able to run that specific task.

> > It would not be enough for debusine yet (for debusine I need to be able to
> > export data from the worker and then make that decision on the server and
> > not on the build machine itself) but it would be nice step forward for the
> > local use case where you want "mdbp" to figure out alone which backend to
> > use based on what the user did setup earlier.
> 
> Yes, that makes sense. I note that the decision is meant to be made on
> the build-issuing side for mdbp as well. If you use the ssh backend, the
> relevant command might look like this:
> 
> mdbp-ssh someserver.somesite mdbp-mmdebstrap --mirror 
> http://mirror.somesite/debian

Yes, but using the ssh backend will require specific user configuration
while the sbuild/pbuilder one could potientially be auto-selected based
on whether the user did run the appropriate sbuild-create-chroot or
pbuilder --create earlier.

> > This abstraction definitely makes sense to me. Before looking closely
> > at your build_schema.json, but after having looked at your mail here,
> > I wrote this as a try to go in your direction:
> > https://freexian-team.pages.debian.net/debusine/devel/ontology.html#task-packagebuild
> 
> Great. Maybe that's the level where we can make best progress?

Likely, yes.

> I've taken the liberty to rather open a discussion issue at
> https://salsa.debian.org/freexian-team/debusine/-/issues/10. Hope this
> works for you.

Yes, thanks!

> > PS: I already hate the "mdbp" name after having it typed so many times.
> 
> I'm not attached to either. Any suggestions?

sbp for "standardized build package" is easier to type but not necessarily
nicer.

"justadeb" or "gimmeadeb" or "buildadeb" because I just want a .deb (and I
don't care how it gets built!).

"deblord" or "debring" - the lord of the ring to tie all package builders
together

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS

Re: Seeking feedback on a meta package builder

[Holger Levsen]

On Tue, Jun 08, 2021 at 10:07:50PM +0200, Helmut Grohne wrote:
> > PS: I already hate the "mdbp" name after having it typed so many times.
> I'm not attached to either. Any suggestions?
 
even medebupa seems better to me.

(assuming mdbp is meta debian build package?)

or meta-debuild or something^wanything more pronouncable.

meta-debuild would imply that 'meta' is the main feature here...


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄


signature.asc
Description: PGP signature

Re: Seeking feedback on a meta package builder

[Helmut Grohne]

Hi Raphaël,

On Tue, Jun 08, 2021 at 12:05:44AM +0200, Raphael Hertzog wrote:
> The point is that if you have to look behind the abstraction to do some
> other related tasks like selecting the worker, then using the abstraction
> to actually execute the build doesn't bring you much, just supplementary
> issues if the assumptions that you make no longer match the assumptions
> implemented in the abstraction.

You do have a point here. My focus definitely was on making it easy for
the one issuing a build. debusine seems to assume both roles here.

I felt that worker selection (let alone worker addressing) would be so
inhomogeneous that covering it in a general abstraction does not seem
feasible to me. That's why I left it out.

With "look behind the abstraction", I think that you mean that debusine
would have to implement the mdbp api to perform worker selection. While
that would be possible indeed, I don't see this as required though.

> And it seems to me that aiming for "it works out of the box without having
> to configure mdbp (if you already have sbuild schroot or pbuilder
> tarball)" is a nice thing to aim for. So somehow it would be nice if your
> various mdbp-* implementations could report whether they are able to
> perform a given build.

This is mostly true. One of the ideas is that when you encounter an
application or service that uses mdbp, it should be easy to direct it at
your existing builder. Some backends will need configuration though, so
a magic autoconfiguration seems infeasible to me. For instance, the ssh
one will need a remote host and for mmdebstrap you most likely want to
specify a local mirror.

I do wonder though, in what kind of situation would you want to merely
know whether a backend can perform a build as opposed to just attempting
it and being able to tell backend issues from actual build failures
apart.

> It would not be enough for debusine yet (for debusine I need to be able to
> export data from the worker and then make that decision on the server and
> not on the build machine itself) but it would be nice step forward for the
> local use case where you want "mdbp" to figure out alone which backend to
> use based on what the user did setup earlier.

Yes, that makes sense. I note that the decision is meant to be made on
the build-issuing side for mdbp as well. If you use the ssh backend, the
relevant command might look like this:

mdbp-ssh someserver.somesite mdbp-mmdebstrap --mirror 
http://mirror.somesite/debian

A user would append their build object to the command line and run it.

This is not meant as a recommendation for debusine to use mdbp-ssh. We
already agreed that it is not a good match.  It is meant to illustrate
that coming up with an automatic backend choice is not obvious though.

> That said if you are willing to complexify your mdbp abstraction to
> cover this use case, then there might be room to actually use mdbp in

There is a trade-off involved here. The reason we're talking now is that
I do want to consider and enable other uses even if that adds
complexity.

> debusine. I would still be annoyed by having to interface with CLI calls
> to make basic requests like those and I would hope that we could define a
> Python API instead.

Let us compare this to gettext. gettext also is an interface, one that
exists in multiple languages though. There is a CLI interface, but for
many languages there are individual bindings. The same can be done here
without loosing the gist.

In fact I already had to do this. The CLI uses arguments, stdout and
files. The latter tend to work badly over ssh, so I already added an
alternative API called mdbp-streamapi that only uses file descriptors to
interface with. The mdbp-streamapi command just converts this
alternative representation back to the "normal" CLI.

Sure, we can add more APIs if needed. The one thing that I'd like to
keep though is for them being compatible or convertible. Being able to
plug them into one another with ease.

> This abstraction definitely makes sense to me. Before looking closely
> at your build_schema.json, but after having looked at your mail here,
> I wrote this as a try to go in your direction:
> https://freexian-team.pages.debian.net/debusine/devel/ontology.html#task-packagebuild

Great. Maybe that's the level where we can make best progress?

> Feel free to submit merge requests to update my document to more
> closely match yours (many entries are missing).

I've taken the liberty to rather open a discussion issue at
https://salsa.debian.org/freexian-team/debusine/-/issues/10. Hope this
works for you. Bringing debusine's PackageTask and mdbp's build_schema is
the thing I'm really after. If these match, converting between the
relevant APIs should become a simple matter.

> Yes, the "command line interface" does not really help for debusine,
> unless you extend it to cover the requirements I explained above. And even
> then I don't really like to have to run an external executable to be 

Re: Seeking feedback on a meta package builder

[Raphael Hertzog]

Hi,

On Mon, 07 Jun 2021, Helmut Grohne wrote:
> My abstraction says nothing about workers or how to select them. The
> idea behind that was that a user should not have to care. Yes, you
> (debusine) do need a mapping between workers and available distributions
> somewhere. You can implement scheduler as a middleware (or router) in
> the abstraction I'm proposing to do this selection. What I don't see is
> where this hurts. The worker selection is something that you have to do
> anyway regardless of whether you use mdbp for the build abstraction.

The point is that if you have to look behind the abstraction to do some
other related tasks like selecting the worker, then using the abstraction
to actually execute the build doesn't bring you much, just supplementary
issues if the assumptions that you make no longer match the assumptions
implemented in the abstraction.

And it seems to me that aiming for "it works out of the box without having
to configure mdbp (if you already have sbuild schroot or pbuilder
tarball)" is a nice thing to aim for. So somehow it would be nice if your
various mdbp-* implementations could report whether they are able to
perform a given build.

It would not be enough for debusine yet (for debusine I need to be able to
export data from the worker and then make that decision on the server and
not on the build machine itself) but it would be nice step forward for the
local use case where you want "mdbp" to figure out alone which backend to
use based on what the user did setup earlier.

That said if you are willing to complexify your mdbp abstraction to
cover this use case, then there might be room to actually use mdbp in
debusine. I would still be annoyed by having to interface with CLI calls
to make basic requests like those and I would hope that we could define a
Python API instead.

> I'm actually interested in figuring this out as the fewer competing
> abstractions we end up with, the better they will be supported.

Sure.

> I note that my abstraction kinda has two levels. For one thing there is
> a build object (represented as json, but that really doesn't matter). It
> describes the action that is requested by the user (what to build,
> architectures, options, etc.).

This abstraction definitely makes sense to me. Before looking closely
at your build_schema.json, but after having looked at your mail here,
I wrote this as a try to go in your direction:
https://freexian-team.pages.debian.net/debusine/devel/ontology.html#task-packagebuild

There are some differences that we should strive to eliminate but I leave
that up to you to comment, it's too late for me right now. :-)

Feel free to submit merge requests to update my document to more
closely match yours (many entries are missing).

> The other is a command line interface that receives a build object and
> transmits (via files) the relevant artifacts. Maybe the build object
> level better addresses the needs of debusine while leaving some aspects
> unanswered (e.g. artifact transmission).

Yes, the "command line interface" does not really help for debusine,
unless you extend it to cover the requirements I explained above. And even
then I don't really like to have to run an external executable to be able
to answer a question like "can this worker handle this build request".

On the opposite, the fact that debusine will expose a "PackageBuild" task
matching your abstraction means that it will be trivial to write
mdbp-debusine.

Cheers,

PS: I already hate the "mdbp" name after having it typed so many times.
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS

Re: Seeking feedback on a meta package builder

[Helmut Grohne]

Hi Raphaël,

Reading your reply makes me think there must be a misunderstanding. Much
of what you write only makes sense to me that way. I hope my reply is
not too repetitive and clears up some of that.

On Sun, Jun 06, 2021 at 06:00:28PM +0200, Raphael Hertzog wrote:
> I don't want to sound too negative but you can't put at the same level
> "mdbp-ssh" and "mdbp-sbuild". Your abstraction is going too far by trying
> to include remote building IMO.

The abstraction says precisely nothing about where a build is being
performed. The use of ssh for a remote middleware is a detail
(hopefully) invisible to the user.

> And while abstractions are nice, for debusine I need to know how each
> implementation work to be able to answer questions like "can I build for
> that distribution on this worker" (i.e. do I have the required chroots) ?
> And here your current abstraction is hurting more than it helps.

My abstraction says nothing about workers or how to select them. The
idea behind that was that a user should not have to care. Yes, you
(debusine) do need a mapping between workers and available distributions
somewhere. You can implement scheduler as a middleware (or router) in
the abstraction I'm proposing to do this selection. What I don't see is
where this hurts. The worker selection is something that you have to do
anyway regardless of whether you use mdbp for the build abstraction.

I'm actually interested in figuring this out as the fewer competing
abstractions we end up with, the better they will be supported.

> So I'm afraid that mdbp is just not suited for the needs of debusine at
> this point.

I don't fully follow your reasoning yet and hope that we can get this
sorted out.

> Also given that you want to include support of building across multiple
> machines, I don't really understand why you select a command line API
> as the interface to drive this all. It's just not a good fit. Having one
> process drive all the machines via ssh is just fragile and not really
> scalable. And it requires all the machines to be directly adressable. In
> debusine, only the server needs a public IP, the workers connect to it.

Good questions. A command line API gives freedom to the implementer of
the API. It allows easy wrapping ("middleware") of implementations and
choice of implementation language. While my implementations have been
done in Python, I don't think Python is a requirement here. It just
happens to be convenient to me. Actually having competing
implementations can be a nice thing as long as their interface is
compatible.

I'm not sure where you got this "one process drive all machines". The
proposed abstraction calls for one process per build. One invocation
will certainly only deal with one machine at a time. The thing that
might not scale is the number of processes.

You mention the need for machines to be directly addressable. That is
true for mdbp-ssh, but it is only one implementation of the abstraction.
You are free to implement other remote middlewares that lack this
requirement.

Your paragraph gives good reasons for why the mdbp-ssh middleware is a
bad fit for debusine. What I don't see is good reasons for the
abstraction not being a good fit.

I note that my abstraction kinda has two levels. For one thing there is
a build object (represented as json, but that really doesn't matter). It
describes the action that is requested by the user (what to build,
architectures, options, etc.). The other is a command line interface
that receives a build object and transmits (via files) the relevant
artifacts. Maybe the build object level better addresses the needs of
debusine while leaving some aspects unanswered (e.g. artifact
transmission).

Helmut

Re: Seeking feedback on a meta package builder

[Raphael Hertzog]

Hi,

On Fri, 04 Jun 2021, Helmut Grohne wrote:
> This is very sad. The whole point of me reaching out where was
> specifically trying to avoid this kind of fragmentation and now you're
> adding to it. While mdbp also is an implementation, it first and
> foremost is an API and I'd hope that it would be generic enough to be
> reused here.

I don't want to sound too negative but you can't put at the same level
"mdbp-ssh" and "mdbp-sbuild". Your abstraction is going too far by trying
to include remote building IMO.

And while abstractions are nice, for debusine I need to know how each
implementation work to be able to answer questions like "can I build for
that distribution on this worker" (i.e. do I have the required chroots) ?
And here your current abstraction is hurting more than it helps.

So I'm afraid that mdbp is just not suited for the needs of debusine at
this point.

Also given that you want to include support of building across multiple
machines, I don't really understand why you select a command line API
as the interface to drive this all. It's just not a good fit. Having one
process drive all the machines via ssh is just fragile and not really
scalable. And it requires all the machines to be directly adressable. In
debusine, only the server needs a public IP, the workers connect to it.

> Of course, you can turn this around and say that I'm the one fragmenting
> and I should be using the debusine abstraction. Unfortunately, I'd need
> a time machine to get it from the future. Otherwise, I'd really do that.

:-)

We managed to live without this for years, we can wait a few more months.
But it's not up to me to tell you what to work on.

> > As I said, I don't want to tightly couple it, it's just the easiest and
> > most useful first step for me.
> 
> I doubt this is true. You already learned the hard way that coming up
> with an abstraction for just sbuild is difficult. Using an existing
> abstraction saves you from a pile of thinking and quite certainly is
> easier.

Thanks for your feedback indeed, it improved the initial design and will
avoid some rework in the future when we add the abstraction.

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS

Re: Seeking feedback on a meta package builder

[Helmut Grohne]

Hi Sean,

On Thu, Jun 03, 2021 at 04:47:44PM -0700, Sean Whitton wrote:
> dgit wraps some of the existing tools.  While dgit is mainly for humans,
> one role it can have in automated toolchains is producing an ephemeral
> source package for the purpose of performing a build where the real
> input is a git tree.  dgit knows how to convert various forms of git
> tree into source packages and there are TODOs for some others.

While dgit does wrap build tools and thus could be a possible mdbp user,
dgit does so in a way that forwards the flexibility of those build tools
through dgit. As such, it has a subcommand for each implementation and
lets the user fiddle with the underlying implementation in the way they
want without providing any kind of abstraction. I'm vaguely convinced
that for dgit, this is best. An abstraction could be added as another
subcommand if desired, but does buy little in the end. After all, dgit
users tend to want tight control over the build and know their favourite
build tool well.

The intended audience for the abstraction on the other hand would be
performing many builds in a mechanical way.

Helmut

Re: Seeking feedback on a meta package builder

[Helmut Grohne]

Hi Raphaël,

On Fri, Jun 04, 2021 at 08:42:23AM +0200, Raphael Hertzog wrote:
> I don't intend to restrict too much the number of "tasks" that I will
> accept in debusine, we can have both if it makes sense. Though I believe
> that this is an abstraction worth having at the debusine level without
> delegating that abstraction to "mdbp".

This is very sad. The whole point of me reaching out where was
specifically trying to avoid this kind of fragmentation and now you're
adding to it. While mdbp also is an implementation, it first and
foremost is an API and I'd hope that it would be generic enough to be
reused here. Long term, I'd hope that each build tool would provide
their own implementation of the mdbp API, but adding multiple competing
build abstraction APIs is not going to get us there. Build tool authors
will rightly refuse saying that people should agree on an API first.

Choice is nice, but fragmentation has a cost.

Of course, you can turn this around and say that I'm the one fragmenting
and I should be using the debusine abstraction. Unfortunately, I'd need
a time machine to get it from the future. Otherwise, I'd really do that.

> I tried to fix it based on your feedback in the ticket but it looks like
> it doesn't match your expectation. When I request a build for the "arm64"
> architecture, I want "arm64.deb" as output, that's defined by the "host
> architecture" right?

Correct.

> (In this context, I would not care if it's build by cross-compilation in
> an amd64 build chroot or if it's build natively in an arm64 chroot)

A different way to put this is to say that the build architecture does
not matter and is left unspecified.

> As I said, I don't want to tightly couple it, it's just the easiest and
> most useful first step for me.

I doubt this is true. You already learned the hard way that coming up
with an abstraction for just sbuild is difficult. Using an existing
abstraction saves you from a pile of thinking and quite certainly is
easier.

Helmut

Re: Seeking feedback on a meta package builder

[Raphael Hertzog]

Hi,

On Wed, 02 Jun 2021, Helmut Grohne wrote:
>  * debusine's build milestone implements a lock-in on sbuild. The
>abstraction that I'm seeking doesn't happen here.

Just to be clear, I think I want that kind of abstraction at some point.
Because one should be able to schedule a package build and it should be
able to build without preference on a worker where we have sbuild + the
appropriate chroot or on a machine where pbuilder has been setup or
on a machine with mdbp ;-)

But Freexian is paying someone to get this code so I'm doing it with small
steps that lead to a minimal viable product on which we can cooperate
further. Release early, release often. :-)

That abstraction will likely come later when someone implements a second
way to build packages.

> Possibly debusine could replace its sbuild task with a mdbp task?

I don't intend to restrict too much the number of "tasks" that I will
accept in debusine, we can have both if it makes sense. Though I believe
that this is an abstraction worth having at the debusine level without
delegating that abstraction to "mdbp".

> would automatically work on pbuilder and mmdebstrap for free. It also
> doesn't have to implement another method for issuing builds remotely and
> can concentrate on the scheduling part.

That's not really relevant, part of the goal of debusine is to let
you leverage a network of workers for any kind of task, not only for
package builds.

> > In the current design
> > (https://salsa.debian.org/freexian-team/debusine/-/issues/7) for the
> > current iteration, I don't have that level of details but it should
> > not be too hard to add this on top. Only the differentiated handling of
> > build/host architecture might be non-trivial as it will impact the 
> > scheduling
> > (identification of an appopriate worker).
> 
> Please get the architecture part right from the start. It is not that
> hard actually. Your issue already mentions an unspecific "architecture"
> field, so you already get to choose suitable workers. It is only the
> build architecture that affects worker choice. The host architecture is
> only a setting to pass down and debusine does not have to care about it
> at all.

I tried to fix it based on your feedback in the ticket but it looks like
it doesn't match your expectation. When I request a build for the "arm64"
architecture, I want "arm64.deb" as output, that's defined by the "host
architecture" right?

(In this context, I would not care if it's build by cross-compilation in
an amd64 build chroot or if it's build natively in an arm64 chroot)

> Maybe we can use some AWS credits to provide PBaaS to developers at some
> point. Or even archive rebuilds as a service.

Definitely.

> While Debian uses sbuild in official buildds, reproducible.d.n is fully
> built on pbuilder instead. It isn't that homogeneous in the end. In
> order to make debusine easier to set up, I think it should not tightly
> couple with sbuild.

As I said, I don't want to tightly couple it, it's just the easiest and
most useful first step for me.

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS

Re: Seeking feedback on a meta package builder

[Sean Whitton]

Hello Helmut,

On Mon 31 May 2021 at 08:25AM +02, Helmut Grohne wrote:

>
> So with this mail I seek multiple sorts of replies:
>  * What other tools perform builds as part of their job, but don't
>exactly care how they're being performed?
>  * What build tools am I missing?
>  * Does the problem scope I've chosen make sense?
>  * Feedback on the proposed API
>  * What feature is missing for your tool to use mdbp?
>  * Which build tools should mdbp support to make it useful to you?

dgit wraps some of the existing tools.  While dgit is mainly for humans,
one role it can have in automated toolchains is producing an ephemeral
source package for the purpose of performing a build where the real
input is a git tree.  dgit knows how to convert various forms of git
tree into source packages and there are TODOs for some others.

-- 
Sean Whitton

Re: Seeking feedback on a meta package builder

[Jonas Smedegaard]

Hi Helmut,

Quoting Helmut Grohne (2021-06-03 09:34:32)
> On Thu, Jun 03, 2021 at 09:22:05AM +0200, Jonas Smedegaard wrote:
> > And sure, a frontend wrapper tool may not need to care about some 
> > options, if possible to configure the backend independently.
> 
> Thank you for putting this so clearly.
> 
> For now, I've been focussing on the interface between build tool users 
> and build tools. Limiting the amount of options there is important to 
> provide some sort of tool-independence. This is why I'm pushing back 
> here. I've not invested into flexible backend configuration yet, but 
> that certainly is something worth pursuing. So very likely, ccache and 
> eatmydata fall into the category of backend configuration and should 
> be possible somehow. They're not the sort of thing that you typically 
> want to vary with each individual build.

Makes good sense.

Thanks for working on this!


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: Seeking feedback on a meta package builder

[Helmut Grohne]

Hi Jonas,

On Thu, Jun 03, 2021 at 09:22:05AM +0200, Jonas Smedegaard wrote:
> And sure, a frontend wrapper tool may not need to care about some 
> options, if possible to configure the backend independently.

Thank you for putting this so clearly.

For now, I've been focussing on the interface between build tool users
and build tools. Limiting the amount of options there is important to
provide some sort of tool-independence. This is why I'm pushing back
here. I've not invested into flexible backend configuration yet, but
that certainly is something worth pursuing. So very likely, ccache and
eatmydata fall into the category of backend configuration and should be
possible somehow. They're not the sort of thing that you typically want
to vary with each individual build.

Helmut

Re: Seeking feedback on a meta package builder

[Jonas Smedegaard]

Quoting Helmut Grohne (2021-06-03 07:41:07)
> On Wed, Jun 02, 2021 at 05:07:16PM +0200, Alex Muntada wrote:
> > This reminded me of my using of ccache and eatmydata in sbuild. They 
> > may not be relevant for the context of mdbp depending on the 
> > resources available for building, indeed.
> > 
> > I'm pretty fond of them because they speed up a lot some heavy 
> > builds on my laptop, therefore I'd appreciate to have some speed 
> > toggles in any service where I want to send those builds to.
> 
> TL;DR: Don't add speedup flags, make it fast by default.
> 
> I don't think speedups are relevant for any of the prospective users. 
> ccache either needs a small set of packages or a huge cache directory 
> to perform well. For many use cases (rebootstrap in particular), the 
> toolchain changes frequently, so the cache would have to be 
> invalidated very often. Most mdbp users would work with at least 
> hundreds of packages.
> 
> As for eatmydata, I don't think that adding this as toggle is useful. 
> After all, we don't want eatmydata to have any influence on the build. 
> So a mdbp backend can just unconditionally enable it if it makes sense 
> without concerning a prospective user of the API.
> 
> A very reliably alternative to eatmydata is building on tmpfs with a 
> huge swap partition. It also performs significantly better than 
> eatmydata in my experience as it fully eliminates the need to write to 
> disk a lot of the time.
> 
> When you use pbuilder's "extract tarball" approach or a type=file 
> schroot with sbuild on a tmpfs (optionally speeding things up by using 
> zstd as compressor), the chroot setup can feel quite instantaneous.
> 
> Ultimately, speedups should just become the default for the benefit of 
> everyone rather than opt-in. If I remember correctly, pbuilder now 
> defaults to disable man database generation via debconf, which also 
> provides a useful speedup.

I think those are more accurately described as "trade-off flags" and 
that your argument is to limit the scope of tools instead of supporting 
a wider variety of needs.

Most notably, ccache is a tradeoff between speed+cost of network 
bandwidth versus local disk space.

I agree that any speedup without tradeoff should be just enabled by 
default.

And sure, a frontend wrapper tool may not need to care about some 
options, if possible to configure the backend independently.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: Seeking feedback on a meta package builder

[Helmut Grohne]

Hi Alex,

On Wed, Jun 02, 2021 at 05:07:16PM +0200, Alex Muntada wrote:
> This reminded me of my using of ccache and eatmydata in sbuild.
> They may not be relevant for the context of mdbp depending on
> the resources available for building, indeed.
> 
> I'm pretty fond of them because they speed up a lot some heavy
> builds on my laptop, therefore I'd appreciate to have some speed
> toggles in any service where I want to send those builds to.

TL;DR: Don't add speedup flags, make it fast by default.

I don't think speedups are relevant for any of the prospective users.
ccache either needs a small set of packages or a huge cache directory to
perform well. For many use cases (rebootstrap in particular), the
toolchain changes frequently, so the cache would have to be invalidated
very often. Most mdbp users would work with at least hundreds of
packages.

As for eatmydata, I don't think that adding this as toggle is useful.
After all, we don't want eatmydata to have any influence on the build.
So a mdbp backend can just unconditionally enable it if it makes sense
without concerning a prospective user of the API.

A very reliably alternative to eatmydata is building on tmpfs with a
huge swap partition. It also performs significantly better than
eatmydata in my experience as it fully eliminates the need to write to
disk a lot of the time.

When you use pbuilder's "extract tarball" approach or a type=file
schroot with sbuild on a tmpfs (optionally speeding things up by using
zstd as compressor), the chroot setup can feel quite instantaneous.

Ultimately, speedups should just become the default for the benefit of
everyone rather than opt-in. If I remember correctly, pbuilder now
defaults to disable man database generation via debconf, which also
provides a useful speedup.

Helmut

Re: Seeking feedback on a meta package builder

[Alex Muntada]

Hi Helmut,

> On Mon, May 31, 2021 at 6:26 AM Helmut Grohne wrote:
> 
> > Some aspects that you may want to set for a build are:

Paul said:

> What to run in the build environment at each stage.

This reminded me of my using of ccache and eatmydata in sbuild.
They may not be relevant for the context of mdbp depending on
the resources available for building, indeed.

I'm pretty fond of them because they speed up a lot some heavy
builds on my laptop, therefore I'd appreciate to have some speed
toggles in any service where I want to send those builds to.

Cheers,
Alex

--
  ⢀⣴⠾⠻⢶⣦⠀
  ⣾⠁⢠⠒⠀⣿⡁   Alex Muntada 
  ⢿⡄⠘⠷⠚⠋   Debian Developer  log.alexm.org
  ⠈⠳⣄



signature.asc
Description: PGP signature

Re: Seeking feedback on a meta package builder

[Helmut Grohne]

Hi Paul,

On Tue, Jun 01, 2021 at 01:00:55AM +, Paul Wise wrote:
> > Some aspects that you may want to set for a build are:
> 
> Looking at my pbuilder configs, some more:

Note that you are not a computer program. The thing I'm working on is
not supposed to be run by humans, but by other tools. As a human, please
continue to use the underlying builder directly to exert the
customizations that you need.

> Where to place the build results.

Covered by mdbp.

> Which dependency resolver to use when installing deps.

Unsure. This is an aspect where the various build implementations
differ. Achieving uniformity on this is hard.

> Whether suite/backports to build under.

Base suite is required by mdbp.

> Which packages not yet in Debian to make available to the build process.

I know this isn't the same, but an extra repository gives close to the
same flexibility at less convenience. However, convenience is less
relevant if the other side is a machine instead of a human.

> Extra environment variables to set.

Covered by mdbp.

> Set the parallelism level.

Covered by mdbp.

> What to run in the build environment at each stage.

Unsure. This also is an aspect where various tools differ widely. Much
of the time, you can achieve this by other means. For instance, you can
install a special package with a strange postinst. Or you can build a
modified source package. Do you have a case where this flexibility
really is needed and cannot be achieved by other means?

Helmut

Re: Seeking feedback on a meta package builder

[Helmut Grohne]

Hi Raphaël,

On Wed, Jun 02, 2021 at 09:12:24AM +0200, Raphael Hertzog wrote:
> On Mon, 31 May 2021, Helmut Grohne wrote:
> > I see this tight coupling as a problem for mainly two reasons.
> [...]
> >  * As tasks grow, there is a desire to distribute builds to multiple
> >machines. As such, the various tools typically grow their own
> >strategy for moving build tasks between machines.
> 
> We already exchanged a while ago about "debusine", it ought to solve this
> part a least.
> 
> https://freexian-team.pages.debian.net/debusine/

Yes, we did.

> We're just starting to work on the project with the goal of handling
> package builds as a first step:
> https://salsa.debian.org/freexian-team/debusine/-/milestones/1
> 
> I invite you to "watch" the project on
> https://salsa.debian.org/freexian-team/debusine so that you get
> notification of comments on issues as that's where we are discussing
> the design.
> 
> And feel free to file feature requests.

I think debusine does more and less than what I want at the same time:
 * debusine thinks of a build more generically than building Debian
   packages. Building a Debian package just happens to be a special case
   that is very relevant to debusine, but it shall be able to build
   other kinds of things as well.
 * Beyond building, debusine also handles scheduling of builds and
   storing of artifacts.
 * debusine's build milestone implements a lock-in on sbuild. The
   abstraction that I'm seeking doesn't happen here.

Possibly debusine could replace its sbuild task with a mdbp task? It
would automatically work on pbuilder and mmdebstrap for free. It also
doesn't have to implement another method for issuing builds remotely and
can concentrate on the scheduling part.

> In the current design
> (https://salsa.debian.org/freexian-team/debusine/-/issues/7) for the
> current iteration, I don't have that level of details but it should
> not be too hard to add this on top. Only the differentiated handling of
> build/host architecture might be non-trivial as it will impact the scheduling
> (identification of an appopriate worker).

Please get the architecture part right from the start. It is not that
hard actually. Your issue already mentions an unspecific "architecture"
field, so you already get to choose suitable workers. It is only the
build architecture that affects worker choice. The host architecture is
only a setting to pass down and debusine does not have to care about it
at all.

> >  * Does the problem scope I've chosen make sense?
> 
> "Package Build as a Service" :-)

Kinda. That would be awesome, yes. Indeed, the abstraction I am thinking
of should allow you to swap out a local builder for PBaaS as a simple
configuration change.

Maybe we can use some AWS credits to provide PBaaS to developers at some
point. Or even archive rebuilds as a service.

> When thinking of debusine, I thought something of something similar, I
> have set out to use sbuild in the first iteration because that what I
> (and Debian) uses for official package builds but given the number of
> different ways to build packages, I was wondering whether I should
> consider to have a tool-agnostic "PackageBuild" task that can be scheduled
> and that could then be performed by tool-specific implementations.

While Debian uses sbuild in official buildds, reproducible.d.n is fully
built on pbuilder instead. It isn't that homogeneous in the end. In
order to make debusine easier to set up, I think it should not tightly
couple with sbuild.

Consider using mdbp to implement your tool-agnostic Debian package
building task. What is missing on the mdbp side to make this work for
you?

Helmut

Re: Seeking feedback on a meta package builder

[Raphael Hertzog]

Hi,

On Mon, 31 May 2021, Helmut Grohne wrote:
> I see this tight coupling as a problem for mainly two reasons.
[...]
>  * As tasks grow, there is a desire to distribute builds to multiple
>machines. As such, the various tools typically grow their own
>strategy for moving build tasks between machines.

We already exchanged a while ago about "debusine", it ought to solve this
part a least.

https://freexian-team.pages.debian.net/debusine/

We're just starting to work on the project with the goal of handling
package builds as a first step:
https://salsa.debian.org/freexian-team/debusine/-/milestones/1

I invite you to "watch" the project on
https://salsa.debian.org/freexian-team/debusine so that you get
notification of comments on issues as that's where we are discussing
the design.

And feel free to file feature requests.

> So I set out to solve this. More precisely, I looked for an API that
> could abstract both sbuild and pbuilder (because they're the most common
> ones) and allow executing builds on remote machines. A tool performing
> builds should not have to care which one is in use and whether the build
> is performed locally or not. Since describing the settings of a build
> can be complex, I chose json for representation. Some aspects that you
> may want to set for a build are:
>  * build architecture / host architecture
>  * arch-only / indep-only / full
>  * DEB_BUILD_PROFILES
>  * DEB_BUILD_OPTIONS
>  * whether network access is permissible
>  * forcing a build path

In the current design
(https://salsa.debian.org/freexian-team/debusine/-/issues/7) for the
current iteration, I don't have that level of details but it should
not be too hard to add this on top. Only the differentiated handling of
build/host architecture might be non-trivial as it will impact the scheduling
(identification of an appopriate worker).

>  * Does the problem scope I've chosen make sense?

"Package Build as a Service" :-)

When thinking of debusine, I thought something of something similar, I
have set out to use sbuild in the first iteration because that what I
(and Debian) uses for official package builds but given the number of
different ways to build packages, I was wondering whether I should
consider to have a tool-agnostic "PackageBuild" task that can be scheduled
and that could then be performed by tool-specific implementations.

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS

Re: Seeking feedback on a meta package builder

[Paul Wise]

On Mon, May 31, 2021 at 6:26 AM Helmut Grohne wrote:

> Here are a number of lock-ins I happen to know:

This was mentioned on IRC:

cowpoke (from devscripts) -> remote cowbuilder

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Seeking feedback on a meta package builder

[Paul Wise]

On Mon, May 31, 2021 at 6:26 AM Helmut Grohne wrote:

> When someone writes a tool, that happens to build packages as part of
> its job (the earlier list), typically one of the build tools (the second
> list) is chosen and fixed. It's not like you can mix and match them.
> Here are a number of lock-ins I happen to know:

A couple more:

debuild -> dpkg-buildpackage
(pdebuild works around that coupling)

apt source -b -> dpkg-buildpackage

> Some aspects that you may want to set for a build are:

Looking at my pbuilder configs, some more:

Where to place the build results.

Which dependency resolver to use when installing deps.

Whether suite/backports to build under.

Which packages not yet in Debian to make available to the build process.

Extra environment variables to set.

Set the parallelism level.

What to run in the build environment at each stage.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Seeking feedback on a meta package builder

[Helmut Grohne]

Hi fellow developers,

Building Debian packages is a job that is not only part of our daily
development practice, but also a task regularly performed by tools.
These include:
 * QA tools frequently rebuild packages to find bugs
   * Archive rebuilds (e.g. Lucas Nussbaum, Matthias Klose, Santiago
 Vila, et al)
   * reproducible.debian.net
   * crossqa.debian.net
   * ratt
 * Tools whose task is larger than building packages, but happens to
   include doing so
   * https://wiki.debian.org/HelmutGrohne/rebootstrap
   * libdvd-pkg
   * apt-src

I suspect that this list is incomplete. My primary motivation for
looking into this came from rebootstrap though.

The task of building Debian packages has been solved numerous times.
Here are some known solutions:
 * sbuild
 * pbuilder
 * cowbuilder
 * qemubuilder
 * docker-based tools:
   + debocker
   + debdocker
   + docker-buildpackage
   + whalebuilder
 * systemd-nspawn-based tools
   + conbuilder
   + debspawn
 * manually without chroot
 * Create a chroot for each build using mmdebstrap

Again, the list is incomplete and if you want to improve it, please edit
https://wiki.debian.org/SystemBuildTools instead.

When someone writes a tool, that happens to build packages as part of
its job (the earlier list), typically one of the build tools (the second
list) is chosen and fixed. It's not like you can mix and match them.
Here are a number of lock-ins I happen to know:

 * reproducible.debian.net -> pbuilder
 * crossqa.debian.net -> sbuild
 * ratt -> sbuild
 * libdvd-pkg -> manually
 * rebootstrap -> manually

I see this tight coupling as a problem for mainly two reasons.
 * When someone else works on your tool, they have to set up your
   preferred builder. As a developer, you cannot just choose between
   pbuilder and sbuild. You effectively get to use both, because some
   tools use one and others use the other.
 * As tasks grow, there is a desire to distribute builds to multiple
   machines. As such, the various tools typically grow their own
   strategy for moving build tasks between machines.

Lengthy problem description, I know. If you've read some of my bug
reports, you may have noticed that only few come without solutions.

So I set out to solve this. More precisely, I looked for an API that
could abstract both sbuild and pbuilder (because they're the most common
ones) and allow executing builds on remote machines. A tool performing
builds should not have to care which one is in use and whether the build
is performed locally or not. Since describing the settings of a build
can be complex, I chose json for representation. Some aspects that you
may want to set for a build are:
 * build architecture / host architecture
 * arch-only / indep-only / full
 * DEB_BUILD_PROFILES
 * DEB_BUILD_OPTIONS
 * whether network access is permissible
 * forcing a build path

I called it "Meta Dpkg-BuildPackage" or simply mdbp for now and you can
find it at:
https://git.subdivi.de/?p=~helmut/mdbp.git;a=blob;f=README.md
git://git.subdivi.de/~helmut/mdbp.git

So with this mail I seek multiple sorts of replies:
 * What other tools perform builds as part of their job, but don't
   exactly care how they're being performed?
 * What build tools am I missing?
 * Does the problem scope I've chosen make sense?
 * Feedback on the proposed API
 * What feature is missing for your tool to use mdbp?
 * Which build tools should mdbp support to make it useful to you?

Thank you for considering. If you want to discuss on irc, I think both
#debian-bootstrap and #debian-qa would be appropriate.

Helmut

Bug#956682: ITP: feedbackd -- DBus service for haptic/visual/audio feedback

[Arnaud Ferraris]

Package: wnpp
Severity: wishlist
Owner: Arnaud Ferraris 

* Package name: feedbackd
  Version : 0.0.0+git20200305
  Upstream Author : Purism SPC
* URL : https://source.puri.sm/Librem5/feedbackd
* License : GPL-3+ and LGPL-2.1+
  Programming Lang: C
  Description : DBus service for haptic/visual/audio feedback

Feedbackd is a DBus activated daemon that provides haptic/visual/audio
feedback based on events.

It is a dependency to Phosh, a Wayland shell for mobile devices,
which I also plan to package, and as such will be useful for bringing
Debian to mobile phones.

I take the responsibility for maintaining this package.

Bug#954035: ITP: qtfeedback-opensource-src -- Qt Feedback module

[Mike Gabriel]

Package: wnpp
Severity: wishlist
Owner: Mike Gabriel 

* Package name: qtfeedback-opensource-src
  Version : 5.0~git20180329.a14bd0bb-1
  Upstream Author : Chris Adams 
* URL : https://code.qt.io/qt/qtsystems.git
* License : LGPL-2.1 (with exception) or GPL-3
  Programming Lang: C++
  Description : Qt Feedback module

 Qt is a cross-platform C++ application framework. Qt's primary feature
 is its rich set of widgets that provide standard GUI functionality.
 .
 This package contains Qt Feedback module.
 .
 This Qt module will be packaged under the umbrella of the Debian Qt/KDE
 Maintainers and will be maintained by the Debian UBports Team.

Re: Feedback request about the Alba Upstream to Debian packaging effort

[Paul Wise]

On Sat, Jun 2, 2018 at 3:41 PM, PICCA Frederic-Emmanuel wrote:

> The next meeting of this community will be held in Prague[4] next
> week. During this meeting, Alba will present their plan about
> packaging "Collaborative and automated Packaging"[5].

You might be interested in the autodeb GSoC 2018 project:

https://debconf18.debconf.org/talks/49-autodeb-automatic-packages-for-everything/
https://wiki.debian.org/autodeb
https://salsa.debian.org/autodeb-team/autodeb

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Feedback request about the Alba Upstream to Debian packaging effort

[Carlos Pascual]

(Hi. Sorry if you already got this message. There was some problem with my 
previous attempts to reply, so I am trying again after our email admin changed 
some settings)


Hi,

I work at the Alba Synchrotron and I authored the presentation that Frederic 
sent (thanks to him for initiating this discussion). I'll try to reply to some 
comments:

On Sunday, June 3, 2018 1:07:26 PM CEST PICCA Frederic-Emmanuel wrote:
> Hello Ian
> > I reviewed the version number proposal and it seems sound to me.
> 
> It just comes to my mind that Maybe it does not fit well with  my convention
> for exeprimental numbering whcih is
> blablab_x.y.z-t~exp1
> so maybe the best way would be to use
> blalbla_x.y.z-t~~alba+1

So, you would not use the "bpo9" part for the packages built for stretch?


> > I have some observations:
> > You probably want to keep the delta between the upstream and the
> > debianised branch as small as possible.
>
> Yes you are right this should be kept as small as possible.

Not sure if I understand well, but I think that this is exactly what we 
achieve for the packages for which we are upstream and we use the whole 
pipeline (including upstream-packaging-and deploy) 

> 
> > On build systems and debian/ruless: if (i) you can get as much of your
> > upstream build system looking as standard as possible (ii) you can get
> > as much of the rest supported by dh as possible, then your
> > debian/rules files could possibly be very small indeed.
> 

This is generally what we have.

> 
> > debian/changelog is a bit of a pain and you will have to write code to
> > generate it.  [gbp-]dch can be helpful.
> 
> gbp allows to generate this automatically, but this is true that the quality
> of the changelog is not necessary good. Most of the time because the commit
> are not that great either...

That is exactly our approach. We let gbp dch do its magic and it is mostly a 
matter of educating ourselves to do adequate commits when committing to the 
debian branch(es).

Our goal is to have "not-terrible" automated changelogs for our local packages 
but rely on manual editions (via a dedicated commit)) before public debian 
releases.
 
> > Ideally you would treat debian/control as an output file: generate it
> > entirely from upstream stuff, using some tool, and commit the result
> > as a committed-artefact to the debianised branch.
> 
> I agreed with you that it would be great to have a way to generate a debian
> package from the upstream git repository and  some minimalist metadata
> purely descriptive.
> > Your debianisation tool becomes part of the source code for your
> > packages.  You need to publish it in your repository, track the
> > version used, etc.  So it probably needs to be debianised.  I think
> > you need to mention it in Built-Using or something similar.
> good point, but for now this is just the gitlab file.
> Do you think that the guix way can be modified in order to generate Debian
> packages instead of guix binaries ? I like a lot the idea to maintain only
> the metadata in a repository. Indeed in our case scientific software are
> most of the time not that standard and need lot's of tweak in order to be
> packages.

Auto-generating the package would be a good addition, but as Fred says, it is 
out of the current scope.
That said, we do  use some scripts that automate the debianization of our 
python upstream sources (and also creation of the repos, etc). For now these 
are too tied to our local infrastructure. I see a lot of room for improvement 
and generalization in this area.

But still, in our experience, full automation of the first debianization of 
anything-but-trivial packages is too hard. So our approach is to assume that 
the first debianisation is done *before* we apply this pipeline

> > Finally, and VERY CONTROVERSIALLY: consider whether you want to bother
> >> > with source packages.  You could just use git branches instead.
> > Source packages are a pain.
> 
> Just use dgit ;)

The source packages are created in the "upstream" part of the pipeline and are 
of interest only to upstream (and totally optional). They are not used at all 
in the debian packaging part of the pipeline. For the packaging, we use git 
branches and merges only (see https://people.debian.org/~picca/gitlab-ci.yml )


> > Good luck and have fun!


Cheers!

And thanks again for your comments!

Carlos

-- 
++
 Carlos Pascual Izarra
 Scientific Software Coordinator
 Computing Division
 ALBA Synchrotron  [http://www.albasynchrotron.es]
 Carrer de la Llum 2-26
 E-08290 Cerdanyola del Valles (Barcelona), Spain
 E-mail: cpasc...@cells.es
 Phone: +34 93 592 4428
++

Re: Feedback request about the Alba Upstream to Debian packaging effort

[Ian Jackson]

PICCA Frederic-Emmanuel writes ("Feedback request about the Alba Upstream to 
Debian packaging effort"):
> Since I am not at all a specialist of gitlab-ci, I would like your
> advice on the pipeline, and also on the numbering scheme propose by
> Alba. In fact all feedback which should smooth the upstream to debian
> flow.
...
> [5] https://people.debian.org/~picca/CollabPkg-v3.pdf

I didn't have a massive amount of time to review this in detail, but
it sounds cool.  I looked at the slides in the pdf [5] above.
(Shame there isn't a technical report...)

I reviewed the version number proposal and it seems sound to me.

I have some observations:

You probably want to keep the delta between the upstream and the
debianised branch as small as possible.

On build systems and debian/ruless: if (i) you can get as much of your
upstream build system looking as standard as possible (ii) you can get
as much of the rest supported by dh as possible, then your
debian/rules files could possibly be very small indeed.

debian/changelog is a bit of a pain and you will have to write code to
generate it.  [gbp-]dch can be helpful.

Ideally you would treat debian/control as an output file: generate it
entirely from upstream stuff, using some tool, and commit the result
as a committed-artefact to the debianised branch.

Your debianisation tool becomes part of the source code for your
packages.  You need to publish it in your repository, track the
version used, etc.  So it probably needs to be debianised.  I think
you need to mention it in Built-Using or something similar.

Finally, and VERY CONTROVERSIALLY: consider whether you want to bother
with source packages.  You could just use git branches instead.
Source packages are a pain.

Good luck and have fun!

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Feedback request about the Alba Upstream to Debian packaging effort

[PICCA Frederic-Emmanuel]

Hello,

the Alba[1] synchrotron radiation facilities, recently switch to
Debian for their OS. They are part of the Tango[2] control system
community which contain most of the European Synchrotron Radiation
Facilities and others[3]. At least three instituts have already
choosen Debian (partially Soleil, ESRF, petraIII, and Alba).

The next meeting of this community will be held in Prague[4] next
week. During this meeting, Alba will present their plan about
packaging "Collaborative and automated Packaging"[5].

The idea is to propose a pipeline via a .gitlab-ci.yml[6] file which
should be added to the upstream repository and/or in a repository
dedicated to packaging, in order to generate debian packages ready to
install software on their facility or ready to upload into Debian.

Since I am not at all a specialist of gitlab-ci, I would like your
advice on the pipeline, and also on the numbering scheme propose by
Alba. In fact all feedback which should smooth the upstream to debian
flow.

Thanks for considering.

Frederic

Ps: Please keep the CC, they are not yet subscribers of debian-devel

[1] https://www.cells.es/en
[2] http://www.tango-controls.org/
[3] http://www.tango-controls.org/partners/institutions/
[4] 
https://indico.eli-beams.eu/event/310/other-view?view=standard#20180605.detailed
[5] https://people.debian.org/~picca/CollabPkg-v3.pdf
[6] https://people.debian.org/~picca/gitlab-ci.yml

Bug#895829: ITP: snore -- sleep with feedback

[Paride Legovini]

Package: wnpp
Severity: wishlist
Owner: Paride Legovini <p...@ninthfloor.org>

* Package name: snore
  Version : 0.1
  Upstream Author : Claudio Alessi 
* URL : https://github.com/clamiax/snore/
* License : Expat
  Programming Lang: C
  Description : sleep with feedback

This tool is similar to sleep(1), but visual feedback is given by
printing the flowing of time in both ascending and descending order.

This is useful as a generic countdown timer, or when sleep(1) is used in
an interactive script.

Re: Feedback on 3.0 source format problems

[Thomas Goirand]

On 01/04/2017 06:30 AM, Nikolaus Rath wrote:
> But I am not sure if a package structure like
> 
> mypkg/upstream/*
> mypkg/debian/*
> mypkg/patches/* (?)
> 
> would have any *practical* benefits over the current situation

It would have the huge benefit that upstream .git* files would stay in
the upstream folder, and wouldn't conflict. I had for example issues
with conflicting .gitreview files when using Gerrit for the Debian
packaging.

I don't see however any advantage of moving the patches folder outside
of the debian folder.

> because
> this transformation could be trivially automated in either direction.

Would you mind explaining who and how it would be automated?

Cheers,

Thomas Goirand (zigo)

Bug#853014: ITP: visual-regexp-el -- in-buffer visual feedback while using Emacs regexps

[Sean Whitton]

Package: wnpp
Severity: wishlist
Owner: Sean Whitton <spwhit...@spwhitton.name>

* Package name: visual-regexp-el
  Version : 1.0.0
  Upstream Author : Marko Bencun <mben...@gmail.com>
* URL : https://github.com/benma/visual-regexp.el
* License : GPL-3+
  Programming Lang: Emacs Lisp
  Description : in-buffer visual feedback while using Emacs regexps

I intend to maintain this under pkg-emacsen.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Feedback on 3.0 source format problems

[Raphael Hertzog]

Hi,

On Sun, 01 Jan 2017, Guillem Jover wrote:
> On Sun, 2017-01-01 at 10:47:59 -0800, Nikolaus Rath wrote:
> > On Jan 01 2017, Guillem Jover  wrote:
> > > (I'm not using  because
> > > TBH it read more like a sales brochure than a more neutral page…)
> > 
> > TBH this feels like you're sniping at Raphael here, which I think is
> > pretty sad and inappropriate.
> 
> Hmm, I brought it up because it does read like that to me (please have
> a look if you haven't), and it does not feel right for me to change the
> tone of that page on my own.

I agree with your choice of using a separate page and I even agree with
your reasons.

I don't agree that you had to write the above sentence at the start of the
discussion. It did not bring anything to the discussion and I confirm what
Nikolaus said, I felt attacked for having promoted the new source format.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/

Re: Feedback on 3.0 source format problems

[Guido Günther]

On Tue, Jan 10, 2017 at 10:14:09AM -0800, Nikolaus Rath wrote:
> On Jan 10 2017, Guido Günther  wrote:
> > On Wed, Jan 04, 2017 at 04:38:11PM -0800, Nikolaus Rath wrote:
> >> On Jan 05 2017, Brian May  wrote:
> >> > Vincent Bernat  writes:
> >> >
> >> >> There have been a lot of complaints about it. For me, it is a pain to
> >> >> use. Its integration with gbp is poor, it produces a messy history when
> >> >> you are working on your patches and I often run into problems with
> >> >> .debian/.git-dpm file it maintains (import a new upstream, make some
> >> >> changes, notice that somebody else also pushed a change, pull --rebase,
> >> >> everything is broken). Since we started using it, we opened a lot of bug
> >> >> reports and not a single one of them has been fixed. I think that nobody
> >> >> wants to work on it because it is an extremely fragile tool and the
> >> >> first one to try to fix it will inherit of all the problems to solve.
> >> >
> >> > It also has a number of bugs that are not getting fixed.
> >> 
> >> Yeah, I think we heard before that git-dpm is not being maintained. I
> >> said it, Vincent said it in his reply, and now you are saying it
> >> again. No one is disputing the point.
> >> 
> >> > Plus if conflicts occur because multiple people unexpectedly make
> >> > changes at the same time it (i.e. you can't push because somebody else
> >> > already pushed changes) can be a world of confusion trying to sort out
> >> > the mess.
> >> 
> >> Yes, it is a mess. But I don't think it's any worse than having to
> >> resolve conflicts in debian/patches/, which is the equivalent problem
> >> when multiple people use gbp at the same time.
> >
> > When this happens you do a "gbp pq import" and have the full power of
> > git rebas at your hands.
> 
> Are you sure? The problem we're talking about is when two conflicting
> changes to debian/patches have been committed. I think in that case you
> first have to solve the git conflict before you can call gbp pq - or can
> gbp pq import really deal with conflict markers *inside the patch*?

You abort the merge instead of resolving the conflict then do a gbp pq
on your packaging branch and on the (know known to be) conflicting one
(say master and origin/master). Then you can use rebase and
whatnot. Not ideal but much batter than wading through merge conflicts
in diffs.

Cheers,
 -- Guido

Re: Feedback on 3.0 source format problems

[Nikolaus Rath]

On Jan 10 2017, Guido Günther  wrote:
> On Wed, Jan 04, 2017 at 04:38:11PM -0800, Nikolaus Rath wrote:
>> On Jan 05 2017, Brian May  wrote:
>> > Vincent Bernat  writes:
>> >
>> >> There have been a lot of complaints about it. For me, it is a pain to
>> >> use. Its integration with gbp is poor, it produces a messy history when
>> >> you are working on your patches and I often run into problems with
>> >> .debian/.git-dpm file it maintains (import a new upstream, make some
>> >> changes, notice that somebody else also pushed a change, pull --rebase,
>> >> everything is broken). Since we started using it, we opened a lot of bug
>> >> reports and not a single one of them has been fixed. I think that nobody
>> >> wants to work on it because it is an extremely fragile tool and the
>> >> first one to try to fix it will inherit of all the problems to solve.
>> >
>> > It also has a number of bugs that are not getting fixed.
>> 
>> Yeah, I think we heard before that git-dpm is not being maintained. I
>> said it, Vincent said it in his reply, and now you are saying it
>> again. No one is disputing the point.
>> 
>> > Plus if conflicts occur because multiple people unexpectedly make
>> > changes at the same time it (i.e. you can't push because somebody else
>> > already pushed changes) can be a world of confusion trying to sort out
>> > the mess.
>> 
>> Yes, it is a mess. But I don't think it's any worse than having to
>> resolve conflicts in debian/patches/, which is the equivalent problem
>> when multiple people use gbp at the same time.
>
> When this happens you do a "gbp pq import" and have the full power of
> git rebas at your hands.

Are you sure? The problem we're talking about is when two conflicting
changes to debian/patches have been committed. I think in that case you
first have to solve the git conflict before you can call gbp pq - or can
gbp pq import really deal with conflict markers *inside the patch*?

Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«

Re: Feedback on 3.0 source format problems

[Guido Günther]

On Wed, Jan 04, 2017 at 04:38:11PM -0800, Nikolaus Rath wrote:
> On Jan 05 2017, Brian May  wrote:
> > Vincent Bernat  writes:
> >
> >> There have been a lot of complaints about it. For me, it is a pain to
> >> use. Its integration with gbp is poor, it produces a messy history when
> >> you are working on your patches and I often run into problems with
> >> .debian/.git-dpm file it maintains (import a new upstream, make some
> >> changes, notice that somebody else also pushed a change, pull --rebase,
> >> everything is broken). Since we started using it, we opened a lot of bug
> >> reports and not a single one of them has been fixed. I think that nobody
> >> wants to work on it because it is an extremely fragile tool and the
> >> first one to try to fix it will inherit of all the problems to solve.
> >
> > It also has a number of bugs that are not getting fixed.
> 
> Yeah, I think we heard before that git-dpm is not being maintained. I
> said it, Vincent said it in his reply, and now you are saying it
> again. No one is disputing the point.
> 
> > Plus if conflicts occur because multiple people unexpectedly make
> > changes at the same time it (i.e. you can't push because somebody else
> > already pushed changes) can be a world of confusion trying to sort out
> > the mess.
> 
> Yes, it is a mess. But I don't think it's any worse than having to
> resolve conflicts in debian/patches/, which is the equivalent problem
> when multiple people use gbp at the same time.

When this happens you do a "gbp pq import" and have the full power of
git rebas at your hands.
Cheers,
 -- Guido

Re: Feedback on 3.0 source format problems

[Guido Günther]

On Tue, Jan 03, 2017 at 07:24:18PM -0800, Russ Allbery wrote:
> Nikolaus Rath  writes:
> 
> > Are there really upstreams that do that? I'd expect that the primary
> > consumer of Debian patches are other distributions, downstreams, and
> > users.
> 
> > I'd think that anything that's relevant for upstream development is
> > forwarded to upstream by the maintainer in whatever format upstream
> > prefers. This requires extra time, but I would be surprised to hear if
> > there are maintainers that have sufficient time to create patches that
> > are suitable for upstream, but don't have the little extra time to send
> > them upstream.
> 
> There are definitely upstreams who like to look at what patches Debian is
> shipping at their convenience, rather than having to ask the maintainer.
> The maintainer may have already sent along the patches, but it's easy to
> lose track of what patches are still being applied.

> The other case where being able to point upstream at a directory of
> patches is very nice is when upstream has been dormant for a long time and
> then comes back to life.  It's an easy way for them to pull a bunch of
> changes at once for a new development cycle.
> 
> I've also found it significantly smooth over relations with upstreams who
> are otherwise quite prickly about modifications to their distributed
> releases.  Having all the patches be published and clearly documented
> somewhere without anyone having to ask wins points for transparency and
> makes them feel like the whole process is less out of control and more
> congenial and collaborative, even if you were also in the habit of sending
> changes upstream.  It eliminates the fear that you're also applying other
> ugly hacks you're not telling them about that might be maintenance burdens
> for them.

Thanks for the nice summary. I now wonder:

If we want a way to present this as a git view for a 3.0 (quilt)
packages one could make gbp pq create a branch that contains the patched
tree (like "gbp pq import" but on top of upstream branch and not on top
of the debian branch). This branch would be directly mergeable by
upstream.  If we tag after the import these histories would be stable
and if one wants these to be fast-forwardable then the topmost commits
can be joined by pseudo merges and one could track the history of each
commit. I did not yet see the point to add this since:

a) these branches usually have debian specific upstream modifications,
cherry-picks _from_ upstream and new fixes _for_ upstream intermixed so
upstream would cherry-pick at best but never merge

b) upstream usually wants specific patches on top of its current version
not of the version we're currently working in Debian. This can be done
with gbp pq import and git cherry-pick or rebase already.

so I usually:

* use tarballs (since uscan is so convenient)
* add upstreams history as a detached history
* cherry pick between the debian branch and upstreams master (or any
  other of upstream's branches)
* point upstream to debian/patches on alioth (or much more often
  git-send-email directly)

so would adding such a view make any sense nevertheless?
Cheers
 -- Guido

Re: Feedback on 3.0 source format problems

[Andrey Rahmatullin]

On Tue, Jan 10, 2017 at 08:36:13AM +1000, Russell Stuart wrote:
> To state the bleeding obvious, it arises because on day 1 Debian
> decided to do the builds in the original source tree, then tries to
> recover the original source at the end by running "debian/rules clean".
>  When I moved from rpm's to deb over a decade ago, I was surprised by
> this.  Rpm's create temp build directory, so the "debian/rules clean"
> step can be handled reliably 100% of the time by the rpm build tool.
Indeed.

> Later when I started to work on other peoples packages it became
> apparent that many of the Romans didn't bother with it.  So the
> debian/rules binary; dpkg --install; test; debian/rules clean; fix;
> rinse and repeat cycle doesn't work at all for maybe 1/3 of packages,
> and another 1/3 occasionally fail when something goes wrong during the
> patch / build process.
This is a policy violation and people are filing bug reports about such
problems. Still, I've always seen that as "more work for nothing" because
of the same reasons as you. The feature of exporting the build tree in 
{svn,git}-buildpackage is a blessing and I always use it for packages that
use those tools, including all of mine.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Feedback on 3.0 source format problems

[Johannes Schauer]

Hi Mattia,

Quoting Mattia Rizzolo (2017-01-09 11:27:30)
> On Sun, Jan 08, 2017 at 01:05:37PM +0100, Johannes Schauer wrote:
> > Mattia, please see below for a pbuilder-specific question.
> 
> Thanks for CCing me; I'm not following this thread anymore (as it
> surpassed the threshold above which I stop to dedicate my time to it),
> so please CC me if you need my input on anything.

then you should probably instruct your mailer to not exclude you from the
Mail-Followup-To header. :)

> > I'm not very familiar with pbuilder. Looking at the man page it seems that
> > pbuilder itself exclusively accepts a source package .dsc and for building
> > a source directory one needs the pdebuild wrapper?
> 
> Exactly.
> 
> just calling plain `pdebuild` without any option will (roughly) call
>   1. dpkg-buildpackage -S -d -us -uc -rfakeroot $other_user_options
>   2. sudo pbuilder build $some_options $user_options ../$pkg_$ver.dsc
> 
> > If that is the case, then it seems sensible for sbuild to do the same as
> > pdebuild if given a source directory only. Both tools should behave the same
> > way when executed in the same context, I think.
> 
> I don't understand what you're planning to do to sbuild.
> 
> > Are there any reasons against this plan? (targeting Buster)
> 
> And I haven't understood what "this plan" is, either.
> FYI, James asked me whether I was fine to move from that
> `dpkg-buildpackage` invocation to call `dpkg-source` instead (with the
> --before-build and --after-build things too), and my only concerns is
> people who are relying on finding a _source.changes in '..'.
> My current workflow is basically calling `pdebuild` in a source tree
> and if the build works `debsign ../$pkg_$ver_source.changes` and then
> dput it (not that my BUILDRESULT is a directory elsewhere that I clean
> often).  I don't think what I'm doing is exotic enough to be fine disrupting
> it.

hrm... this is even more confusing now. Okay, let me give some context.

Thibaut pointed out that it would be cool if sbuild would revert the patches
after building with sbuild to which James replied that this is something that
pdebuild by default. So I wanted to get more details about what pdebuild is
doing with respect to how it leaves the unpacked source directory after
building the source package so that we can have sbuild do the same thing.

Now reading the pdebuild source code it indeed seems that pdebuild is using
'dpkg-buildpackage -S -d' instead of using 'dpkg-source -b' as sbuild does.
From reading the dpkg-buildpackage code, it seems that that one is doing:

$ dpkg-source --before-build .
$ dpkg-source -b .
$ dpkg-source --after-build .

which is what takes care of the patches independent in which state they were.
On the other hand, using dpkg-buildpackage also creates a changes file, a
buildinfo file and tries to sign the whole kaboodle. So maybe both tools are
wrong and pbuilder should replace 'dpkg-buildpackage -S -d' with above three
dpkg-source invocations while sbuild should add the --before-build and
--after-build invocations?

What do the dpkg developers think?

Thanks!

cheers, josch


signature.asc
Description: signature

Re: Feedback on 3.0 source format problems

[Adam D. Barratt]

On Mon, 2017-01-09 at 21:01 +0100, Johannes Schauer wrote:
> Hi,
> 
> Quoting Ian Jackson (2017-01-09 18:33:51)
> > Johannes Schauer writes ("Re: Feedback on 3.0 source format problems"):
> > > Sbuild could do this cleanup itself if there was a way to
> > > automatically determine whether the user would like their tree to be
> > > patches applied or unapplied.
> > 
> > This would have to be some kind of (perhaps package-specific) personal
> > configuration, I think.
> 
> is that what debian/source/local-options is about?
> 
> The only docs I find about it are:
> 
> https://www.debian.org/doc/manuals/maint-guide/dother.en.html#sourcel

dkpg-source(1), as referenced from the URL above.

Regards,

Adam

Re: Feedback on 3.0 source format problems

[Russell Stuart]

On Mon, 2017-01-09 at 17:33 +, Ian Jackson wrote:
> All of this applying and unapplying of patches around build
> operations is complete madness if you ask me - but I don't see a
> better approach given the constraints.  dgit sometimes ends up doing
> this (and moans about it), which is even madder given that dgit has
> git to help it out.

To state the bleeding obvious, it arises because on day 1 Debian
decided to do the builds in the original source tree, then tries to
recover the original source at the end by running "debian/rules clean".
 When I moved from rpm's to deb over a decade ago, I was surprised by
this.  Rpm's create temp build directory, so the "debian/rules clean"
step can be handled reliably 100% of the time by the rpm build tool.
Debian insisting you write it creates extra work.  But when in Rome do
as the Romans do and all that.

Later when I started to work on other peoples packages it became
apparent that many of the Romans didn't bother with it.  So the
debian/rules binary; dpkg --install; test; debian/rules clean; fix;
rinse and repeat cycle doesn't work at all for maybe 1/3 of packages,
and another 1/3 occasionally fail when something goes wrong during the
patch / build process.

When it goes wrong your only option is "rm -r; dpkg-source -x; manually
reapply your changes" which is tedious, error prone, and for me least
conducive to losing a days work.  It is indeed utter madness that over
a decade later we still allow this work flow. 

signature.asc
Description: This is a digitally signed message part

Re: dput: Call for feedback: What should change? What should stay the same?

[Ben Finney]

Barry Warsaw  writes:

> Unfortunately, the documentation you find on extending upload
> permissions to DMs doesn't tell you that only dput-ng supports the dm
> subcommand.

Likewise, I'm having no luch finding comprehensive reference
documentation for commands accepted by ‘dak’.

Is there a bug report I've missed which addresses this?

-- 
 \ “I still have my Christmas Tree. I looked at it today. Sure |
  `\   enough, I couldn't see any forests.” —Steven Wright |
_o__)  |
Ben Finney

Re: dput: Call for feedback: What should change? What should stay the same?

[Barry Warsaw]

On Dec 28, 2016, at 10:25 AM, Steve Langasek wrote:

>Last I looked, the dcut command in dput doesn't support the 'dm' subcommand;
>this led me to switching to dput-ng when I needed it.

Same here, as I recently needed to `dcut dm` allow for a maintainer of a
package I had been sponsoring while he went through the process.
Unfortunately, the documentation you find on extending upload permissions to
DMs doesn't tell you that only dput-ng supports the dm subcommand.

That's about the only difference I've noticed so far, so I'd be happy enough
to switch back if dput also had a dm subcommand (although truthfully, I rarely
use that anyway).

I think it's fairly confusing that there's dput and dput-ng and would love to
see functional and cli convergence so that eventually there's only one package
that supports current use cases.

Cheers,
-Barry


pgpJv0ozO18GG.pgp
Description: OpenPGP digital signature

Re: Feedback on 3.0 source format problems

[Johannes Schauer]

Hi,

Quoting Ian Jackson (2017-01-09 18:33:51)
> Johannes Schauer writes ("Re: Feedback on 3.0 source format problems"):
> > Sbuild could do this cleanup itself if there was a way to
> > automatically determine whether the user would like their tree to be
> > patches applied or unapplied.
> 
> This would have to be some kind of (perhaps package-specific) personal
> configuration, I think.

is that what debian/source/local-options is about?

The only docs I find about it are:

https://www.debian.org/doc/manuals/maint-guide/dother.en.html#sourcel

> > I do not even know of a way to determine upfront whether a source
> > tree is patches applied or unapplied (that check has to be
> > independent of the source format).
> 
> This is, in the general case, clearly impossible.  As a simple
> example, consider the result of the following:
> 
>   # .oO{ somepackage is broken }
>   dgit clone somepackage && cd somepackage
>   # .oO{ hrm I wonder why it is broken - oh there is only one patch }
>   # .oO{ oh the breakage is in the busted patch "add zorkmids" }
>   git revert -n :/'add zorkmids'
>   git commit
> 
> Now the tree is exactly identical to a patches-unapplied tree.  But
> the user wanted it to drop the patch.  Tools should not reapply it.

Then maybe I don't understand or there is at least some confusion about what
pdebuild is doing. At least from James' email I understand that it is trying to
somehow restore the original state (whether it was patches applied or patches
unapplied) by calling:

 $ dpkg-source --before-build .
 $ dpkg-source -b .
 $ dpkg-source --after-build .

It would be great if somebody could clarify all this and maybe help get us to a
state where we can have the involved tools all do the same sensible thing
independent of the source package format and packaging workflow.

Thanks!

cheers, josch


signature.asc
Description: signature

Re: Feedback on 3.0 source format problems

[Sean Whitton]

On Mon, Jan 09, 2017 at 01:25:44PM +, Ian Jackson wrote:
> Sean Whitton writes ("Re: Feedback on 3.0 source format problems"):
> >  My first worry is that pseudomerges are weird.  In fact, I've never
> > seen them outside of weird Debian git workflows :) Someone might
> > look at the interchange view, see all these pseudomerges, and have
> > no idea how to interpret what the Debian maintainer is doing.  Do
> > you have any thoughts on mitigating the potential confusion?
> 
> I think it might be useful to have an option to git-rev-list to
> disregard non-contributing edges of pseudomerges.  That would mean you
> could see the results in `gitk'.

That would be really cool.

> > The advantage of thinking of the Debian packaging as just another
> > branch of development is that the branching structure itself is easy
> > to interpret for anyone who uses git.  "Ah, I see they merged my
> > release tag into their branch, they must have been bringing Debian
> > up-to-date with the latest release" -- this is very natural for git
> > users.  We call it "packaging a new upstream release" but it's
> > easier for an outsider to think of it as bringing a feature branch
> > up-to-date with the latest mainline developments.
> 
> I can see why you might think this.  But this really does depend on
> how big the delta is from upstream.

Agreed.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Feedback on 3.0 source format problems

[Ian Jackson]

Johannes Schauer writes ("Re: Feedback on 3.0 source format problems"):
> Sbuild could do this cleanup itself if there was a way to
> automatically determine whether the user would like their tree to be
> patches applied or unapplied.

This would have to be some kind of (perhaps package-specific) personal
configuration, I think.

> I do not even know of a way to determine upfront whether a source
> tree is patches applied or unapplied (that check has to be
> independent of the source format).

This is, in the general case, clearly impossible.  As a simple
example, consider the result of the following:

  # .oO{ somepackage is broken }
  dgit clone somepackage && cd somepackage
  # .oO{ hrm I wonder why it is broken - oh there is only one patch }
  # .oO{ oh the breakage is in the busted patch "add zorkmids" }
  git revert -n :/'add zorkmids'
  git commit

Now the tree is exactly identical to a patches-unapplied tree.  But
the user wanted it to drop the patch.  Tools should not reapply it.

> This also brings me to a question about the --unapply-patches option. The man
> page says:

All of this applying and unapplying of patches around build operations
is complete madness if you ask me - but I don't see a better approach
given the constraints.  dgit sometimes ends up doing this (and moans
about it), which is even madder given that dgit has git to help it
out.

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Feedback on 3.0 source format problems

[Ian Jackson]

Sean Whitton writes ("Re: Feedback on 3.0 source format problems"):
> I take it that only the maintainer is meant to look at the
> merging-baseline, and everyone else looks at the interchange view.

Anyone wanting to look at the patch stack can look at the interchange
view.  git blame, and git log, for example, will show:
  * For unchanged upstream files, the upstream history
  * For debian/ files (except patches), the Debian merging history
  * For changed upstream files, the upstream history followed
by a commit representing the most recent version of each patch

>  My first worry is that pseudomerges are weird.  In fact, I've never
> seen them outside of weird Debian git workflows :) Someone might
> look at the interchange view, see all these pseudomerges, and have
> no idea how to interpret what the Debian maintainer is doing.  Do
> you have any thoughts on mitigating the potential confusion?

I think it might be useful to have an option to git-rev-list to
disregard non-contributing edges of pseudomerges.  That would mean you
could see the results in `gitk'.

> The advantage of thinking of the Debian packaging as just another branch
> of development is that the branching structure itself is easy to
> interpret for anyone who uses git.  "Ah, I see they merged my release
> tag into their branch, they must have been bringing Debian up-to-date
> with the latest release" -- this is very natural for git users.  We call
> it "packaging a new upstream release" but it's easier for an outsider to
> think of it as bringing a feature branch up-to-date with the latest
> mainline developments.

I can see why you might think this.  But this really does depend on
how big the delta is from upstream.  People who are used to doing
significant upstream development in projects managed in a
`traditional' way, rather than `just press the big "merge" button'
style, will want to see a patch stack.

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Feedback on 3.0 source format problems

[Mattia Rizzolo]

On Sun, Jan 08, 2017 at 01:05:37PM +0100, Johannes Schauer wrote:
> Mattia, please see below for a pbuilder-specific question.

Thanks for CCing me; I'm not following this thread anymore (as it
surpassed the threshold above which I stop to dedicate my time to it),
so please CC me if you need my input on anything.

> I'm not very familiar with pbuilder. Looking at the man page it seems that
> pbuilder itself exclusively accepts a source package .dsc and for building a
> source directory one needs the pdebuild wrapper?

Exactly.

just calling plain `pdebuild` without any option will (roughly) call
  1. dpkg-buildpackage -S -d -us -uc -rfakeroot $other_user_options
  2. sudo pbuilder build $some_options $user_options ../$pkg_$ver.dsc

> If that is the case, then it seems sensible for sbuild to do the same as
> pdebuild if given a source directory only. Both tools should behave the same
> way when executed in the same context, I think.

I don't understand what you're planning to do to sbuild.

> Are there any reasons against this plan? (targeting Buster)

And I haven't understood what "this plan" is, either.
FYI, James asked me whether I was fine to move from that
`dpkg-buildpackage` invocation to call `dpkg-source` instead (with the
--before-build and --after-build things too), and my only concerns is
people who are relying on finding a _source.changes in '..'.
My current workflow is basically calling `pdebuild` in a source tree
and if the build works `debsign ../$pkg_$ver_source.changes` and then
dput it (not that my BUILDRESULT is a directory elsewhere that I clean
often).  I don't think what I'm doing is exotic enough to be fine
disrupting it.

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: Feedback on 3.0 source format problems

[Sean Whitton]

Hello Ian,

On Fri, Jan 06, 2017 at 03:29:38PM +, Ian Jackson wrote:
> Sean Whitton writes ("Re: Feedback on 3.0 source format problems"):
> > Could you explain in general terms the difference between the
> > interchange and packaging-only branches
> 
> See modified diagram below.  Are the annotations I have added (and the
> name change) any help ?

Yes, thanks, I think I see most of what's going on now.  Thank you for
taking the time to draw the diagrams.  It's certainly an ingenious use
of git.

I take it that only the maintainer is meant to look at the
merging-baseline, and everyone else looks at the interchange view.  My
first worry is that pseudomerges are weird.  In fact, I've never seen
them outside of weird Debian git workflows :)  Someone might look at the
interchange view, see all these pseudomerges, and have no idea how to
interpret what the Debian maintainer is doing.  Do you have any thoughts
on mitigating the potential confusion?

The advantage of thinking of the Debian packaging as just another branch
of development is that the branching structure itself is easy to
interpret for anyone who uses git.  "Ah, I see they merged my release
tag into their branch, they must have been bringing Debian up-to-date
with the latest release" -- this is very natural for git users.  We call
it "packaging a new upstream release" but it's easier for an outsider to
think of it as bringing a feature branch up-to-date with the latest
mainline developments.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Feedback on 3.0 source format problems

[Paul Wise]

On Sun, Jan 8, 2017 at 8:05 PM, Johannes Schauer wrote:

> I'm not very familiar with pbuilder. Looking at the man page it seems that
> pbuilder itself exclusively accepts a source package .dsc and for building a
> source directory one needs the pdebuild wrapper?

Right.

> If that is the case, then it seems sensible for sbuild to do the same as
> pdebuild if given a source directory only. Both tools should behave the same
> way when executed in the same context, I think.

An sdebuild script sounds good to me.

> Are there any reasons against this plan? (targeting Buster)

Longer-term, I think we want debuild and possibly pdebuild to become
just `exec dpkg-buildpackage` so we should probably think about where
the code for this should live.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Feedback on 3.0 source format problems

[Johannes Schauer]

Hi all,

Mattia, please see below for a pbuilder-specific question.

Quoting James Clarke (2017-01-08 12:14:07)
> This turns out to be true. Working in a patches-applied tree:
> 
> $ dpkg-source --before-build .
> $ dpkg-source -b .
> $ dpkg-source --after-build .
> 
> leaves the patches applied. Working in a patches-unapplied tree, the
> patches are unapplied during the --after-build. This seems to do exactly
> what you want, with the caveat that, if patches are only *part*-applied,
> they will be completely unapplied by the --after-build, but this seems
> like a very strange edge-case.
> 
> This behaviour is exactly what pdebuild has been doing forever (and
> therefore git-buildpackage --git-pbuilder and others).

I'm not very familiar with pbuilder. Looking at the man page it seems that
pbuilder itself exclusively accepts a source package .dsc and for building a
source directory one needs the pdebuild wrapper?

If that is the case, then it seems sensible for sbuild to do the same as
pdebuild if given a source directory only. Both tools should behave the same
way when executed in the same context, I think.

Are there any reasons against this plan? (targeting Buster)

Thanks!

cheers, josch


signature.asc
Description: signature

Re: Feedback on 3.0 source format problems

[James Clarke]

On Sun, Jan 08, 2017 at 01:53:51AM +0100, Johannes Schauer wrote:
> Quoting Thibaut Paumard (2017-01-07 07:12:59)
> > I manage my patches using quilt. I would really prefer if sbuild et al.
> > would revert the patches after building by default, but that's life. I
> > respect that other people have other views.
>
> you could always file a wishlist bug against sbuild with your feature request.
> ;)
>
> I guess you are using sbuild from within an unpacked source package?
>
> The input for sbuild is always a source package. If all you give to sbuild is 
> a
> directory, then it first needs to create that source package. It uses
> "dpkg-source -b ." for this purpose. It is dpkg-source which applies the
> patches.
>
> What you could already do today with sbuild is to say:
>
> sbuild --pre-build-commands="dpkg-source --after-build $(pwd)" ...
>
> You can also put this into your ~/.sbuildrc but that would then prevent you
> from using sbuild outside of unpacked source package directories.
>
> Sbuild could do this cleanup itself if there was a way to automatically
> determine whether the user would like their tree to be patches applied or
> unapplied. I do not even know of a way to determine upfront whether a source
> tree is patches applied or unapplied (that check has to be independent of the
> source format). Heck, with quilt, the source tree could even have patches half
> applied. I'm not so sure how fragile it will be to let sbuild try to put your
> source directory back into the state that it found it.

dpkg-source can almost do this for you; see later :)

> This also brings me to a question about the --unapply-patches option. The man
> page says:
>
>--no-unapply-patches, --unapply-patches
>   By  default,  dpkg-source will automatically unapply the patches
>   in  the  --after-build  hook  if  it  did  apply   them   during
>   --before-build(--unapply-patchessincedpkg1.15.8,
>   --no-unapply-patches since dpkg 1.16.5).   Those  options  allow
>   you  to  forcefully  disable  or  enable the patch unapplication
>   process.Thoseoptions are only allowed in
>   debian/source/local-options   so   that   all  generated  source
>   packages have the same behavior by default.
>
> Is --before-build not also run when using --build? That patches are applied
> when using --build indicates to me that it is.

No, it's not, but per dpkg-source(1) you can see that for 3.0 (quilt),
both --before-build and --build apply patches.

> But the --unapply-patches option
> seems to have no effect when used with --build. So is there a way to apply and
> unapply patches with a single dpkg-source call?

No, there isn't. The standard invocation by dpkg-buildpackage -S is:

dpkg-source --before-build .
fakeroot debian/rules clean
dpkg-source -b .
dpkg-source --after-build .
(also dpkg-gencontrol and dpkg-genbuildinfo)

> And if --unapply-patches does
> not have any effect together with --build, why do I not get a warning on the
> command line about this?

Wishlist bug against dpkg? :)

> And then there is the mysterious hint about
> debian/source/local-options which doesn't increase my understanding of the
> matter

Indeed, but they seem to work when given on the command-line for
--after-build.

Now, if you look at the --(no-)unapply-patches description, you will see

> By default, dpkg-source will automatically unapply the patches in the
> --after-build hook if it did apply them during --before-build

This turns out to be true. Working in a patches-applied tree:

$ dpkg-source --before-build .
$ dpkg-source -b .
$ dpkg-source --after-build .

leaves the patches applied. Working in a patches-unapplied tree, the
patches are unapplied during the --after-build. This seems to do exactly
what you want, with the caveat that, if patches are only *part*-applied,
they will be completely unapplied by the --after-build, but this seems
like a very strange edge-case.

This behaviour is exactly what pdebuild has been doing forever (and
therefore git-buildpackage --git-pbuilder and others).

Regards,
James

Re: Feedback on 3.0 source format problems

[Ritesh Raj Sarraf]

On Mon, 2017-01-02 at 12:36 +, Sean Whitton wrote:
> Dear Josh,
> 
> On Mon, Jan 02, 2017 at 03:25:29AM -0800, Josh Triplett wrote:
> > Currently working on some improvements in that direction, to separate
> > repository format from workflow.
> 
> I'd like to encourage you to read my dgit-maint-merge(7) workflow
> tutorial.  Perhaps the work to which you refer could result in some
> improvements to what I wrote there.  Thanks in advance.

Thank you very much for creating and mentioning about such tutorials. I wasn't
aware of them. I'm going to try some of the exampled workflows.


-- 
Given the large number of mailing lists I follow, I request you to CC
me in replies for quicker response

signature.asc
Description: This is a digitally signed message part

Re: Feedback on 3.0 source format problems

[Johannes Schauer]

Hi,

Quoting Thibaut Paumard (2017-01-07 07:12:59)
> I manage my patches using quilt. I would really prefer if sbuild et al.
> would revert the patches after building by default, but that's life. I
> respect that other people have other views.

you could always file a wishlist bug against sbuild with your feature request.
;)

I guess you are using sbuild from within an unpacked source package?

The input for sbuild is always a source package. If all you give to sbuild is a
directory, then it first needs to create that source package. It uses
"dpkg-source -b ." for this purpose. It is dpkg-source which applies the
patches.

What you could already do today with sbuild is to say:

sbuild --pre-build-commands="dpkg-source --after-build $(pwd)" ...

You can also put this into your ~/.sbuildrc but that would then prevent you
from using sbuild outside of unpacked source package directories.

Sbuild could do this cleanup itself if there was a way to automatically
determine whether the user would like their tree to be patches applied or
unapplied. I do not even know of a way to determine upfront whether a source
tree is patches applied or unapplied (that check has to be independent of the
source format). Heck, with quilt, the source tree could even have patches half
applied. I'm not so sure how fragile it will be to let sbuild try to put your
source directory back into the state that it found it.

This also brings me to a question about the --unapply-patches option. The man
page says:

   --no-unapply-patches, --unapply-patches
  By  default,  dpkg-source will automatically unapply the patches
  in  the  --after-build  hook  if  it  did  apply   them   during
  --before-build(--unapply-patchessincedpkg1.15.8,
  --no-unapply-patches since dpkg 1.16.5).   Those  options  allow
  you  to  forcefully  disable  or  enable the patch unapplication
  process.Thoseoptions are only allowed in
  debian/source/local-options   so   that   all  generated  source
  packages have the same behavior by default.

Is --before-build not also run when using --build? That patches are applied
when using --build indicates to me that it is. But the --unapply-patches option
seems to have no effect when used with --build. So is there a way to apply and
unapply patches with a single dpkg-source call? And if --unapply-patches does
not have any effect together with --build, why do I not get a warning on the
command line about this? And then there is the mysterious hint about
debian/source/local-options which doesn't increase my understanding of the
matter

Thoughts?

Thanks!

cheers, josch


signature.asc
Description: signature

Re: Feedback on 3.0 source format problems

[Thibaut Paumard]

Le 07/01/2017 à 22:10, Nikolaus Rath a écrit :
> On Jan 07 2017, Thibaut Paumard <thibaut.paum...@obspm.fr> wrote:
>> Well, just to say, I'm personally quite happy with '3.0 (quilt)'. I try
>> to maintain all my packages in git in unapplied state, because in my
>> opinion this is the sensible thing to do. When I do a
>>   git diff upstream master
>> I want to see only debian/ in there.
> 
> What's the point of that? If the only difference is the addition of the
> debian/ directory, you can simply look at the debian directory. There's
> no need to use git.

The point is to make sure I don't make changes to upstream by mistake.

>> I much prefer to check a diff of diff over a simple diff,
> 
> Wow, ok. That is astonishing to hear, but I'll take your word for it.

The goal is to check that all my changes to the packaging are
intentional, again. I find it easier this way, I appreciate other people
find it easier the other way round.

>> For me the patch is the final product, I like the clear separation
>> between upstream and debian/, so it's for me a very appealing design to
>> have individual patches in debian/patches. I use git to keep the history
>> of the patch, not to manage it.
> 
> And so is this. Well, I definitely learned something new in this thread.

Me too. My point is that this thread encourages criticism of the 3.0
source format. Most participants in this thread are not quite happy (or
quite unhappy?) with this format, but like often presumably people who
*are* happy with the format will not speak up.

I would not like the readers of the thread to come to the conclusion
that the majority is unhappy with the format, because it's not a poll.

My "Feedback on 3.0 source format problems" is that I have nothing to
declare. I take note of the various tools that have been mentioned and
will look at whether they can enhance my packaging experience when time
permits...

Kind regards, Thibaut.

Re: Feedback on 3.0 source format problems

[Nikolaus Rath]

On Jan 07 2017, Thibaut Paumard  wrote:
> Well, just to say, I'm personally quite happy with '3.0 (quilt)'. I try
> to maintain all my packages in git in unapplied state, because in my
> opinion this is the sensible thing to do. When I do a
>   git diff upstream master
> I want to see only debian/ in there.

What's the point of that? If the only difference is the addition of the
debian/ directory, you can simply look at the debian directory. There's
no need to use git.

> I much prefer to check a diff of diff over a simple diff,

Wow, ok. That is astonishing to hear, but I'll take your word for it.

> For me the patch is the final product, I like the clear separation
> between upstream and debian/, so it's for me a very appealing design to
> have individual patches in debian/patches. I use git to keep the history
> of the patch, not to manage it.

And so is this. Well, I definitely learned something new in this thread.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«

Re: Feedback on 3.0 source format problems

[Ian Jackson]

Thibaut Paumard writes ("Re: Feedback on 3.0 source format problems"):
> I'm not interested at the moment in dgit or other wrappers because
>  1- they seem to me to add complexity to the process;
>  2- I prefer to understand what I'm doing.

Those are good reasons.  (And I'm trying to make them a bit less
true.)  But:

dgit users (including downstreams) who use `dgit clone' to get your
packages will appreciate it if you used `dgit push', because they will
get (a slightly modified version of) your history, rather than just
imports of .dsc's.

Of course that's not of very much direct benefit to you.

> Well, just to say, I'm personally quite happy with '3.0 (quilt)'. I try
> to maintain all my packages in git in unapplied state,

This is supported by dgit.

Thanks,
Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Feedback on 3.0 source format problems

[Thibaut Paumard]

Le 06/01/2017 à 16:37, Ian Jackson a écrit :
> Sebastian Andrzej Siewior writes ("Re: Feedback on 3.0 source format 
> problems"):
>> On 2017-01-03 16:58:21 [+], Ian Jackson wrote:
>>> Looked at another way, it is trying to be a version control system,
>>> layered on top of the Debian archive.  But it is only about a quarter
>>> of a VCS.  There are no formal interfaces to do proper VCS operations.
>>> If there is a formal interface, it is quilt(1) (which is itself very
>>> poor.  NB that is not quilt's fault: quilt inevitably has a hard job
>>> because can't make enough assumptions).
>>
>> there quilt push, pop and header which seems enough.
> 
> Well, it seems you don't really think so :-), because:
> 
>> I usually have git-dpm which creates the quilt series and I try to
>> keep patches documented.
> 
> So you are using git operations to manipulate your patch stack.
> git-dpm is one of the tools we have which lets you do that.

Well, just to say, I'm personally quite happy with '3.0 (quilt)'. I try
to maintain all my packages in git in unapplied state, because in my
opinion this is the sensible thing to do. When I do a
  git diff upstream master
I want to see only debian/ in there. I much prefer to check a diff of
diff over a simple diff, because most of the time I want to see what
changed in the packaging, not in the final state. When I want to do the
later, I use debdiff.

For me the patch is the final product, I like the clear separation
between upstream and debian/, so it's for me a very appealing design to
have individual patches in debian/patches. I use git to keep the history
of the patch, not to manage it.

I manage my patches using quilt. I would really prefer if sbuild et al.
would revert the patches after building by default, but that's life. I
respect that other people have other views.

I'm not interested at the moment in dgit or other wrappers because
 1- they seem to me to add complexity to the process;
 2- I prefer to understand what I'm doing.

Kind regards, Thibaut.

-- 
* Dr Thibaut Paumard   | LESIA/CNRS - Table équatoriale (bât. 5)   *
* Tel: +33 1 45 07 78 60   | Observatoire de Paris - Section de Meudon *
* Fax: +33 1 45 07 79 17   | 5, Place Jules Janssen*
* thibaut.paum...@obspm.fr | 92195 MEUDON CEDEX (France)   *



smime.p7s
Description: Signature cryptographique S/MIME

Re: Feedback on 3.0 source format problems

[gregor herrmann]

On Fri, 06 Jan 2017 15:37:48 +, Ian Jackson wrote:

> I think only a minority of people are actually using quilt on
> debian/patches.

I have a different impression (but no proof either of course).


Cheers,
gregor

-- 
 .''`.  https://info.comodo.priv.at/ - Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member of VIBE!AT & SPI, fellow of the Free Software Foundation Europe
   `-   NP: Die Quote: Tonight


signature.asc
Description: Digital Signature

Re: Feedback on 3.0 source format problems

[Ian Jackson]

Ian Jackson writes ("Re: Feedback on 3.0 source format problems"):
>   !   NMUer's HEAD was here when they said `dgit push'.
>   Rebase branch launderer turns each ! into an
>   equivalent *.

I mean it turns each % into an equivalent *, but it work on !s too.

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Feedback on 3.0 source format problems

[Ian Jackson]

Sebastian Andrzej Siewior writes ("Re: Feedback on 3.0 source format problems"):
> On 2017-01-03 16:58:21 [+], Ian Jackson wrote:
> > Looked at another way, it is trying to be a version control system,
> > layered on top of the Debian archive.  But it is only about a quarter
> > of a VCS.  There are no formal interfaces to do proper VCS operations.
> > If there is a formal interface, it is quilt(1) (which is itself very
> > poor.  NB that is not quilt's fault: quilt inevitably has a hard job
> > because can't make enough assumptions).
> 
> there quilt push, pop and header which seems enough.

Well, it seems you don't really think so :-), because:

> I usually have git-dpm which creates the quilt series and I try to
> keep patches documented.

So you are using git operations to manipulate your patch stack.
git-dpm is one of the tools we have which lets you do that.

> > I think if we want to be storing patch queues we should be doing so in
> > a real version control system.  Indeed, most people are doing that.
> > For now many people are using `3.0 (quilt)' as an interchange format,
> > but ideally we would switch to a useable git workflow that did a
> > similar thing, and then we could use git as our history interchange
> > format.
> 
> I read this a few times in this thread that people want the patches in a
> VCS. I never saw this a missing feature or requirement on my side.
...
> I can't think of an example where having a patch history somewhere else
> than within the patch itself is useful. My thinking is probably limited
> by my workflow :) Would you have an example where and how this could be
> usefull?

You have misunderstood, I think.  I don't mean (and I think most
other people don't mean) that they want the history _of_ a patch.
That is, we aren't saying we want to look at the output of
`git blame debian/patches/01-sponge.patch'.

Rather, we think manipulating a stack of patches is much easier if one
converts the patches to a git branch, with each patch becoming a
commit (using a tool morally equivalent to git-am, such as gbp pq
import).

Naturally that doesn't end up recording a history _of_ the patches,
because the patches _become_ `the history'.  Then one manipulates the
patches with git-rebase (or similar tools), git-commit,
git-cherry-pick, etc.

Finally, when doing an upload, one converts the git branch back into
patches (with a tool morally equivalent to git-format-patch).

I think only a minority of people are actually using quilt on
debian/patches.

Regards,
Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Feedback on 3.0 source format problems

[Ian Jackson]

Sean Whitton writes ("Re: Feedback on 3.0 source format problems"):
> Could you explain in general terms the difference between the
> interchange and packaging-only branches

See modified diagram below.  Are the annotations I have added (and the
name change) any help ?

>  Does the packaging-only branch contain debian/ alone?

No, it also contains a complete unmodified copy of the upstream code.
(Except that if upstream had a debian/ directory, it is completely
replaced.)  Perhaps this is the wrong name.  Let's try
`merging-baseline'

For `3.0 (quilt)' the merging-baseline branch contains roughly what
you would get if you untarred the origs and the debian.tar.gz, and
then deleted all the patches without applying them.

Not shown on the diagram is the commit `add patch queue to
debian/patches', which would be needed for `3.0 (quilt)'.  This is
because the diagram is in terms of a sane source format, not `3.0
(quilt)'.  For use with quilty sources, there would be such a commit
(probably dgit-generated) on top of the actual upstream change
commits:

   --/--A!/--B3!--%--/--> interchange view
//  /  with debian/ directory
   %%  %   all upstream changes applied
  //  /3.0 (quilt) has debian/patches
 /2* 2*
//  /
   2*   2  2
  //  /
 11  1`merging-baseline' branch
//  / unmodified upstream code
---p-p--Ap--B--C-->   plus debian/ (but no debian/patches)
  / /   /
   --#-#---#-> upstream

Key:

  1,2,3   commits touching upstream files only
  A,B,C   commits touching debian/ only
  B3  mixed commit (eg made by an NMUer)
  #   upstream releases

 -p-  special merge, takes contents of debian/ from the
 / previous `merging-baseline' commit and rest from upstream

 -/-  pseudomerge; contents are identical to
 / parent lower on diagram.

  %   dgit-generated commit of debian/patches.
  `3.0 (quilt)' only; dropped by rebase tool.

  *   Maintainer's HEAD was here while they were editing,
  before they said they were done, at which point their
  tools generated [% and] -/- commit[s] to convert to
  the fast-forwarding interchange branch.  (Maybe the
  tooling is simply `dgit push'.)

  !   NMUer's HEAD was here when they said `dgit push'.
  Rebase branch launderer turns each ! into an
  equivalent *.

Ian.

--
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Feedback on 3.0 source format problems

[Matthieu Caneill]

Hi Sean,

On Thu, Jan 05, 2017 at 01:05:54PM -0700, Sean Whitton wrote:
> On Wed, Jan 04, 2017 at 03:10:16AM +0100, gregor herrmann wrote:
> > https://sources.debian.net/patches/ goes in that direction. AFAIK it
> > might not be complete and TTBOMK it hasn't been announced widely but
> > it exists and (I think) works for "3.0 (quilt)" packages.
> > 
> > For an example of a package using git-debcherry cf.
> > https://sources.debian.net/patches/libmodule-build-perl/0.422000-1/
> 
> Just to confirm, are you saying that sources.debian.net/patches would
> execute git-debcherry on a 3.0 (quilt) package using
> single-debian-patch, presumably by cloning Vcs-Git?  Does this have to
> be explicitly requested for that package?

No, sources.debian.net/patches is not aware of any VCS. It parses on
the fly the series and the patches of a package (and only for
3.0 (quilt) presently). Still, this allows having URLs to point
upstream to, for released packages.

Cheers,
--
Matthieu


signature.asc
Description: PGP signature

Re: Feedback on 3.0 source format problems

[Sean Whitton]

Hello Ian,

On Wed, Jan 04, 2017 at 12:08:57PM +, Ian Jackson wrote:
> git-dpm sort of does this.  I have been experimenting with and
> blundering towards another approach, which is closer to raw git.
> 
> Something like this:
> 
>--/--A-/---B3---/--> interchange view
> ///
>//3
>   ///
>  222
> ///
>111
>   ///
> ---p-p--Ap---B---> `packaging-only' branch
>   / / /
>--#-#---#-> upstream

Could you explain in general terms the difference between the
interchange and packaging-only branches?  Does the packaging-only branch
contain debian/ alone?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Feedback on 3.0 source format problems

[Sean Whitton]

Hello,

On Tue, Jan 03, 2017 at 06:48:10PM -0800, Nikolaus Rath wrote:
> I'd think that anything that's relevant for upstream development is
> forwarded to upstream by the maintainer in whatever format upstream
> prefers. This requires extra time, but I would be surprised to hear if
> there are maintainers that have sufficient time to create patches that
> are suitable for upstream, but don't have the little extra time to send
> them upstream.

On Wed, Jan 04, 2017 at 02:47:36PM +1000, Russell Stuart wrote:
> This is not a novel requirement.  Most projects I've worked with insist
> you rebase your patches.  This is not new.  Before git they insisted
> your patches applied cleanly - which amounts to the same thing. 
> Breaking up large patches into a series of smaller independent patches
> each with a simple and documented purpose isn't an unusual requirement
> either.

Indeed.  When forwarding a patch upstream, you'll need to ensure it
applies cleanly.  The easiest way is to start a new branch based on the
current upstream HEAD, and cherry-pick your commit onto it.  git
minimises the amount of manual resolution required.

On Wed, Jan 04, 2017 at 10:28:10PM +0100, Sebastian Andrzej Siewior wrote:
> I can't think of an example where having a patch history somewhere else
> than within the patch itself is useful. My thinking is probably limited
> by my workflow :) Would you have an example where and how this could be
> usefull?

In most cases, when you merge a new upstream release, git transparently
handles dropping and refreshing patches, which saves a lot of time.  For
example, after forwarding a patch upstream as I just described, merging
a new upstream release will not usually result in any conflicts, even if
the cherry-pick required manual resolution.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Feedback on 3.0 source format problems

[Sean Whitton]

Hello gregor,

On Wed, Jan 04, 2017 at 03:10:16AM +0100, gregor herrmann wrote:
> On Tue, 03 Jan 2017 20:15:10 +, Sean Whitton wrote:
> 
> > On Tue, Jan 03, 2017 at 10:54:07AM -0800, Russ Allbery wrote:
> > > Well, if we had one more thing: a patches.debian.org service that would
> > > show the git-debcherry-extracted patches against upstream.  I really like
> > > being able to just point upstream at all the patches relevant to them that
> > > Debian has applied.
> > That would be great.  Then the git-debcherry series would be available
> > for those that want it, without requiring package maintainers to do any
> > curation at all.
> 
> https://sources.debian.net/patches/ goes in that direction. AFAIK it
> might not be complete and TTBOMK it hasn't been announced widely but
> it exists and (I think) works for "3.0 (quilt)" packages.
> 
> For an example of a package using git-debcherry cf.
> https://sources.debian.net/patches/libmodule-build-perl/0.422000-1/

Just to confirm, are you saying that sources.debian.net/patches would
execute git-debcherry on a 3.0 (quilt) package using
single-debian-patch, presumably by cloning Vcs-Git?  Does this have to
be explicitly requested for that package?

-- 
Sean Whitton


signature.asc
Description: PGP signature