Re: Haskell binnmus is there a problem?
On 9/2/2019 9:06 PM, Clint Adams wrote:
> On Mon, Sep 02, 2019 at 06:40:24PM +0200, Philipp Kern wrote:
>> I would prefer JSON as an input format, looking roughly like this:
>>
>> [{"pkg": "haskell-active", "ver": "0.2.0.13-6", "archs": ["mipsel",
>> "mips64el"], "suite": "sid", "reason": "semigroupoids changed from 5.2.2 to
>> 5.3.2"}]
>
> Something like https://people.debian.org/~clint/binNMUs-haskell.json ?
Yup, that passes my parser. But it doesn't answer the question I tried
to ask: How would I signal success/failure properly? And, even more
crucially, how would I signal partial success?
I could settle for two different blobs to return: One text blob in case
something in authz/validation went wrong and one JSON blob in case the
script successfully validated the input and then return "result":
"success"/"failure" (say, maybe with some more data from wannna-build)
on every single entry. I suppose in the first case it'd be an inline
response and in the second a file attachment. But then again you'd need
to actually parse the JSON you get back to see if everything went
according to plan...
Kind regards
Philipp Kern
Re: Haskell binnmus is there a problem?
On Mon, Sep 02, 2019 at 06:40:24PM +0200, Philipp Kern wrote:
> I would prefer JSON as an input format, looking roughly like this:
>
> [{"pkg": "haskell-active", "ver": "0.2.0.13-6", "archs": ["mipsel",
> "mips64el"], "suite": "sid", "reason": "semigroupoids changed from 5.2.2 to
> 5.3.2"}]
Something like https://people.debian.org/~clint/binNMUs-haskell.json ?
Re: Haskell binnmus is there a problem?
On 2019-09-02 11:03, Joachim Breitner wrote:
Am Montag, den 02.09.2019, 10:57 +0200 schrieb Philipp Kern:
FWIW, I started working on this. Unfortunately we do not really have
transactions available to schedule all or nothing. If someone has a
creative idea on how to best signal back partial success, let me know.
:)
(My current best guess is a JSON file served as an attachment to the
API
call with success/failure states...)
The JSON is for the response status, and the request will be the
existing format as seen in
https://people.debian.org/~nomeata/binNMUs-haskell.txt
right?
Right now my guide rail is https://buildd.debian.org/auth/giveback.cgi
(documented in [1]) which spews out a bunch of validation notices and
then performs the action.
I would prefer JSON as an input format, looking roughly like this:
[{"pkg": "haskell-active", "ver": "0.2.0.13-6", "archs": ["mipsel",
"mips64el"], "suite": "sid", "reason": "semigroupoids changed from 5.2.2
to 5.3.2"}]
The rationale for this is that it is way easier for me to validate the
input if presented this way. As we know the output of your script above
is piped into wb, which is a glorified wrapper around wanna-build doing
some of the lifting required. Rather than trying to make sense of a line
that is an input to another script (which also supports different
actions), it seems better to constrain us down in a way where every
field is explicitly presented. Especially as ultimately everything needs
to be passed to another binary through command-line options.
The output is then yet another question to solve...
Kind regards
Philipp Kern
[1]
https://debblog.philkern.de/2019/08/alpha-self-service-buildd-givebacks.html
Re: Haskell binnmus is there a problem?
Fix the Go team mailing list address. (pkg-go-maintain...@lists.alioth.debian.org -> debian...@lists.debian.org) On Mon, Sep 02, 2019 at 11:03:56AM +0200, Joachim Breitner wrote: > Hi, > > Am Montag, den 02.09.2019, 10:57 +0200 schrieb Philipp Kern: > > FWIW, I started working on this. Unfortunately we do not really have > > transactions available to schedule all or nothing. If someone has a > > creative idea on how to best signal back partial success, let me know. > > :) > > > > (My current best guess is a JSON file served as an attachment to the API > > call with success/failure states...) > > > The JSON is for the response status, and the request will be the > existing format as seen in > https://people.debian.org/~nomeata/binNMUs-haskell.txt > right? > I found previous mails failed to reach pkg-go alioth list. So to the Go team, the disscution starts here, https://lists.debian.org/debian-release/2019/08/msg00592.html I want to volunteer to implement the necessary things on the Go team side. -- Shengjing Zhu signature.asc Description: PGP signature
Re: Haskell binnmus is there a problem?
Hi, Am Montag, den 02.09.2019, 10:57 +0200 schrieb Philipp Kern: > FWIW, I started working on this. Unfortunately we do not really have > transactions available to schedule all or nothing. If someone has a > creative idea on how to best signal back partial success, let me know. > :) > > (My current best guess is a JSON file served as an attachment to the API > call with success/failure states...) The JSON is for the response status, and the request will be the existing format as seen in https://people.debian.org/~nomeata/binNMUs-haskell.txt right? Cheers, Joachim -- Joachim “nomeata” Breitner • nome...@debian.org • https://j.oach.im/ signature.asc Description: This is a digitally signed message part
Re: Haskell binnmus is there a problem?
On 2019-09-02 10:33, Emilio Pozuelo Monfort wrote: On 29/08/2019 12:08, Philipp Kern wrote: On 2019-08-29 11:46, Emilio Pozuelo Monfort wrote: On 29/08/2019 11:17, Philipp Kern wrote: On 2019-08-29 11:14, Emilio Pozuelo Monfort wrote: Why don't you let the interested teams run the scripts and generate the required binNMUs (like they do now), and then you pull that from a cronjob in wuiet and schedule the binNMUs? You would just need to define the format and do some sanity checks. What would those sanity checks be? You fetch the list from a known debian host (like several other services already do, including ftp-master or release). If you define the format to be e.g. yaml or json, with an array of binNMUs to schedule, each one with - a source package - a version - a list of architectures - a reason You can just then call 'wb nmu' with the right arguments and the right quoting. The sanity checks would be to make sure the arguments have the right format. So that together with the other mail of not making it automatic: Would we just want to have an endpoint you can call as a member of an approved team(?) that ingests a list of work items, processes it and archives it? Assuming that Release Team has no block in place? Can we require a manual action by a team member rather than it being run continuously? So that we could then track down who initiated it? The reason why I asked the Release Team specifically was if we need to constrain the set of packages to act upon (e.g. golang-%). If you do not see a reason, assuming that we can always tell the team in question to fix their automation, that's fine with me too. On the one hand, adding such a constraint would ensure that the teams don't schedule binNMUs outside their realms. OTOH it would reduce the effectiveness of this solution when they need to schedule them on packages that don't fit whichever criteria we chose (e.g. for apps rather than modules). So I'd start with no constrain and we can add it later if we find out it's preferred. FWIW, I started working on this. Unfortunately we do not really have transactions available to schedule all or nothing. If someone has a creative idea on how to best signal back partial success, let me know. :) (My current best guess is a JSON file served as an attachment to the API call with success/failure states...) Kind regards Philipp Kern
Re: Haskell binnmus is there a problem?
On 29/08/2019 12:08, Philipp Kern wrote: > On 2019-08-29 11:46, Emilio Pozuelo Monfort wrote: >> On 29/08/2019 11:17, Philipp Kern wrote: >>> On 2019-08-29 11:14, Emilio Pozuelo Monfort wrote: Why don't you let the interested teams run the scripts and generate the required binNMUs (like they do now), and then you pull that from a cronjob in wuiet and schedule the binNMUs? You would just need to define the format and do some sanity checks. >>> >>> What would those sanity checks be? >> >> You fetch the list from a known debian host (like several other services >> already >> do, including ftp-master or release). If you define the format to be e.g. >> yaml >> or json, with an array of binNMUs to schedule, each one with >> - a source package >> - a version >> - a list of architectures >> - a reason >> >> You can just then call 'wb nmu' with the right arguments and the right >> quoting. >> The sanity checks would be to make sure the arguments have the right format. > > So that together with the other mail of not making it automatic: Would we just > want to have an endpoint you can call as a member of an approved team(?) that > ingests a list of work items, processes it and archives it? Assuming that > Release Team has no block in place? Can we require a manual action by a team > member rather than it being run continuously? So that we could then track down > who initiated it? > > The reason why I asked the Release Team specifically was if we need to > constrain > the set of packages to act upon (e.g. golang-%). If you do not see a reason, > assuming that we can always tell the team in question to fix their automation, > that's fine with me too. On the one hand, adding such a constraint would ensure that the teams don't schedule binNMUs outside their realms. OTOH it would reduce the effectiveness of this solution when they need to schedule them on packages that don't fit whichever criteria we chose (e.g. for apps rather than modules). So I'd start with no constrain and we can add it later if we find out it's preferred. Emilio
Re: Haskell binnmus is there a problem?
On Thu, Aug 29, 2019 at 12:08PM, Philipp Kern wrote: > So that together with the other mail of not making it automatic: Would we > just want to have an endpoint you can call as a member of an approved > team(?) that ingests a list of work items, processes it and archives it? > Assuming that Release Team has no block in place? Can we require a manual > action by a team member rather than it being run continuously? So that we > could then track down who initiated it? I agree on requiring manual action by a team member, for the reasons stated on my other email. I think the above proposal covers our needs. Please let me know how I can help implementing/testing this. Cheers, -- Ilias
Re: Haskell binnmus is there a problem?
On 2019-08-29 11:46, Emilio Pozuelo Monfort wrote: On 29/08/2019 11:17, Philipp Kern wrote: On 2019-08-29 11:14, Emilio Pozuelo Monfort wrote: Why don't you let the interested teams run the scripts and generate the required binNMUs (like they do now), and then you pull that from a cronjob in wuiet and schedule the binNMUs? You would just need to define the format and do some sanity checks. What would those sanity checks be? You fetch the list from a known debian host (like several other services already do, including ftp-master or release). If you define the format to be e.g. yaml or json, with an array of binNMUs to schedule, each one with - a source package - a version - a list of architectures - a reason You can just then call 'wb nmu' with the right arguments and the right quoting. The sanity checks would be to make sure the arguments have the right format. So that together with the other mail of not making it automatic: Would we just want to have an endpoint you can call as a member of an approved team(?) that ingests a list of work items, processes it and archives it? Assuming that Release Team has no block in place? Can we require a manual action by a team member rather than it being run continuously? So that we could then track down who initiated it? The reason why I asked the Release Team specifically was if we need to constrain the set of packages to act upon (e.g. golang-%). If you do not see a reason, assuming that we can always tell the team in question to fix their automation, that's fine with me too. Kind regards Philipp Kern
Re: Haskell binnmus is there a problem?
On 29/08/2019 11:32, Ilias Tsitsimpis wrote: > On Thu, Aug 29, 2019 at 11:14AM, Emilio Pozuelo Monfort wrote: >> On 27/08/2019 21:50, Philipp Kern wrote: >>> Do you have an opinion if this should actually be automated? I.e. >>> automatically be fed into wb on a regular basis? I think the Release >>> Team would also be the first team who would want to have a lever to stop >>> that kind of automation from happening. Unfortunately I don't know how >>> often those binNMUs would interfere with your day to day work. But I'd >>> rather we run this centrally. >> >> Yes, we are ok with these binNMUs for haskell, ocaml, golang... to happen >> automatically, as long as there's a mechanism to temporarily block them (e.g. >> due to some transitions, or to the freeze). > > For me, it would be best if this would not run automatically, to guard > us against cases were a wrong version of a package is being uploaded to > unstable (instead of experimental) and triggers binNMUs before we have a > chance to revert our mistake. > > In addition, there may be cases where we only want to schedule part of > the script's output. We are experiencing this now where ghc is > (currently) unbuildable on mipsel/mips64el, and scheduling binNMUs for > these architectures would be a waste of resources. > >> Why don't you let the interested teams run the scripts and generate the >> required >> binNMUs (like they do now), and then you pull that from a cronjob in wuiet >> and >> schedule the binNMUs? You would just need to define the format and do some >> sanity checks. > > I believe this would work for us, because we would be able to > enable/disable binNMUs on demand (start/stop the script) and alter the > output before submitting it. But then, whoever runs the script basically > has access to the wanna-build infrastructure, and it seems to me that > the wanna-build team want to avoid that. Well, Philipp is a wb admin and is open to the idea. Besides, 'wb gb' is already open to every DD. Opening 'wb nmu' a little bit (in cases where it makes sense such as these, with proper auditing and logging) would be sensible if you ask me. Emilio
Re: Haskell binnmus is there a problem?
On 29/08/2019 11:17, Philipp Kern wrote: > On 2019-08-29 11:14, Emilio Pozuelo Monfort wrote: >> Why don't you let the interested teams run the scripts and generate the >> required >> binNMUs (like they do now), and then you pull that from a cronjob in wuiet >> and >> schedule the binNMUs? You would just need to define the format and do some >> sanity checks. > > What would those sanity checks be? You fetch the list from a known debian host (like several other services already do, including ftp-master or release). If you define the format to be e.g. yaml or json, with an array of binNMUs to schedule, each one with - a source package - a version - a list of architectures - a reason You can just then call 'wb nmu' with the right arguments and the right quoting. The sanity checks would be to make sure the arguments have the right format. Cheers, Emilio
Re: Haskell binnmus is there a problem?
On Thu, Aug 29, 2019 at 11:14AM, Emilio Pozuelo Monfort wrote: > On 27/08/2019 21:50, Philipp Kern wrote: > > Do you have an opinion if this should actually be automated? I.e. > > automatically be fed into wb on a regular basis? I think the Release > > Team would also be the first team who would want to have a lever to stop > > that kind of automation from happening. Unfortunately I don't know how > > often those binNMUs would interfere with your day to day work. But I'd > > rather we run this centrally. > > Yes, we are ok with these binNMUs for haskell, ocaml, golang... to happen > automatically, as long as there's a mechanism to temporarily block them (e.g. > due to some transitions, or to the freeze). For me, it would be best if this would not run automatically, to guard us against cases were a wrong version of a package is being uploaded to unstable (instead of experimental) and triggers binNMUs before we have a chance to revert our mistake. In addition, there may be cases where we only want to schedule part of the script's output. We are experiencing this now where ghc is (currently) unbuildable on mipsel/mips64el, and scheduling binNMUs for these architectures would be a waste of resources. > Why don't you let the interested teams run the scripts and generate the > required > binNMUs (like they do now), and then you pull that from a cronjob in wuiet and > schedule the binNMUs? You would just need to define the format and do some > sanity checks. I believe this would work for us, because we would be able to enable/disable binNMUs on demand (start/stop the script) and alter the output before submitting it. But then, whoever runs the script basically has access to the wanna-build infrastructure, and it seems to me that the wanna-build team want to avoid that. Best, -- Ilias
Re: Haskell binnmus is there a problem?
On 2019-08-29 11:14, Emilio Pozuelo Monfort wrote: Why don't you let the interested teams run the scripts and generate the required binNMUs (like they do now), and then you pull that from a cronjob in wuiet and schedule the binNMUs? You would just need to define the format and do some sanity checks. What would those sanity checks be? Kind regards Philipp Kern
Re: Haskell binnmus is there a problem?
On 27/08/2019 21:50, Philipp Kern wrote: > On 8/27/2019 7:39 PM, Adam D. Barratt wrote: >> On Tue, 2019-08-27 at 19:40 +0300, Ilias Tsitsimpis wrote: >>> So, how can we move forward? Is the release team willing to give more >>> members of our team access to the wanna-build infrastructure, or >>> should we try to automate this and how? >> That's not the Release Team's decision, but the wanna-build >> maintainers. Nor do we have the ability to grant access, beyond any >> points where team membership might overlap. > > Do you have an opinion if this should actually be automated? I.e. > automatically be fed into wb on a regular basis? I think the Release > Team would also be the first team who would want to have a lever to stop > that kind of automation from happening. Unfortunately I don't know how > often those binNMUs would interfere with your day to day work. But I'd > rather we run this centrally. Yes, we are ok with these binNMUs for haskell, ocaml, golang... to happen automatically, as long as there's a mechanism to temporarily block them (e.g. due to some transitions, or to the freeze). > And yes, I realize that this is a little tricky because of policy > questions on what code to run. But dak already solved this AFAIK and as > long as we have DD signatures on the code it should also be fine to > import from, say, Salsa. That said, there are still awkward questions on > how to build the binaries used for this as the Haskell binNMU thing is > obviously written in Haskell rather than being a script. Why don't you let the interested teams run the scripts and generate the required binNMUs (like they do now), and then you pull that from a cronjob in wuiet and schedule the binNMUs? You would just need to define the format and do some sanity checks. Cheers, Emilio
Re: Haskell binnmus is there a problem?
Hi, about the last point: you can treat it like a script and run it with `runhaskell` Cheers, Joachim Am 27. August 2019 21:50:11 MESZ schrieb Philipp Kern : >On 8/27/2019 7:39 PM, Adam D. Barratt wrote: >> On Tue, 2019-08-27 at 19:40 +0300, Ilias Tsitsimpis wrote: >>> So, how can we move forward? Is the release team willing to give >more >>> members of our team access to the wanna-build infrastructure, or >>> should we try to automate this and how? >> That's not the Release Team's decision, but the wanna-build >> maintainers. Nor do we have the ability to grant access, beyond any >> points where team membership might overlap. > >Do you have an opinion if this should actually be automated? I.e. >automatically be fed into wb on a regular basis? I think the Release >Team would also be the first team who would want to have a lever to >stop >that kind of automation from happening. Unfortunately I don't know how >often those binNMUs would interfere with your day to day work. But I'd >rather we run this centrally. > >And yes, I realize that this is a little tricky because of policy >questions on what code to run. But dak already solved this AFAIK and as >long as we have DD signatures on the code it should also be fine to >import from, say, Salsa. That said, there are still awkward questions >on >how to build the binaries used for this as the Haskell binNMU thing is >obviously written in Haskell rather than being a script. > >Kind regards >Philipp Kern
Re: Haskell binnmus is there a problem?
On 8/27/2019 7:39 PM, Adam D. Barratt wrote: > On Tue, 2019-08-27 at 19:40 +0300, Ilias Tsitsimpis wrote: >> So, how can we move forward? Is the release team willing to give more >> members of our team access to the wanna-build infrastructure, or >> should we try to automate this and how? > That's not the Release Team's decision, but the wanna-build > maintainers. Nor do we have the ability to grant access, beyond any > points where team membership might overlap. Do you have an opinion if this should actually be automated? I.e. automatically be fed into wb on a regular basis? I think the Release Team would also be the first team who would want to have a lever to stop that kind of automation from happening. Unfortunately I don't know how often those binNMUs would interfere with your day to day work. But I'd rather we run this centrally. And yes, I realize that this is a little tricky because of policy questions on what code to run. But dak already solved this AFAIK and as long as we have DD signatures on the code it should also be fine to import from, say, Salsa. That said, there are still awkward questions on how to build the binaries used for this as the Haskell binNMU thing is obviously written in Haskell rather than being a script. Kind regards Philipp Kern
Re: Haskell binnmus is there a problem?
On Tue, 2019-08-27 at 19:40 +0300, Ilias Tsitsimpis wrote: > So, how can we move forward? Is the release team willing to give more > members of our team access to the wanna-build infrastructure, or > should we try to automate this and how? That's not the Release Team's decision, but the wanna-build maintainers. Nor do we have the ability to grant access, beyond any points where team membership might overlap. Regards, Adam
Re: Haskell binnmus is there a problem?
On Wed, Aug 21, 2019 at 10:14AM, Mattia Rizzolo wrote: > On Wed, Aug 21, 2019 at 10:06:00AM +0200, Joachim Breitner wrote: > > I used to schedule these, but for many months it seems it wasn't > > necessary, not sure if someone else scheduled them > > At the very least, quite some times pochu did them the last year, fwiw. Indeed, pochu was very helpful during the last Haskell transition, scheduling binNMUs from [1]. [1] https://people.debian.org/~nomeata/binNMUs-haskell.txt > > Going forward, it is not sustainable for this to depend on manual > > interaction by me. We at least need to allow a more active member of > > the Haskell team do these, or finally automate this tedious task. > > It's the problem all statically liked languages have, be that haskell, > ocaml, or the "new" golang (that came to the spotlight due to a somewhat > worse implementation than the other two). Please do try to coordinate > and decide on a sane method to process auto-binNMUs or something like > that. So, how can we move forward? Is the release team willing to give more members of our team access to the wanna-build infrastructure, or should we try to automate this and how? -- Ilias
Re: Haskell binnmus is there a problem?
Le 21/08/2019 à 10:14, Mattia Rizzolo a écrit : >> I used to schedule these, but for many months it seems it wasn't >> necessary, not sure if someone else scheduled them > > At the very least, quite some times pochu did them the last year, fwiw. > >> Going forward, it is not sustainable for this to depend on manual >> interaction by me. We at least need to allow a more active member of >> the Haskell team do these, or finally automate this tedious task. > > It's the problem all statically liked languages have, be that haskell, > ocaml, or the "new" golang (that came to the spotlight due to a somewhat > worse implementation than the other two). Please do try to coordinate > and decide on a sane method to process auto-binNMUs or something like > that. As far as ocaml is concerned, it is not due to the language being "statically linked", but to the strict and automatic ABI checking (and no notion of backward compatibility at the ABI level). We would have the same problem (maybe even worse) if the language was dynamically linked. Cheers, -- Stéphane
Re: Haskell binnmus is there a problem?
On Wed, Aug 21, 2019 at 10:06:00AM +0200, Joachim Breitner wrote: > I used to schedule these, but for many months it seems it wasn't > necessary, not sure if someone else scheduled them At the very least, quite some times pochu did them the last year, fwiw. > Going forward, it is not sustainable for this to depend on manual > interaction by me. We at least need to allow a more active member of > the Haskell team do these, or finally automate this tedious task. It's the problem all statically liked languages have, be that haskell, ocaml, or the "new" golang (that came to the spotlight due to a somewhat worse implementation than the other two). Please do try to coordinate and decide on a sane method to process auto-binNMUs or something like that. -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. more about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `- signature.asc Description: PGP signature
Re: Haskell binnmus is there a problem?
Hi, I used to schedule these, but for many months it seems it wasn't necessary, not sure if someone else scheduled them or if the Haskell team did sourceful uploads in a way that renders them unnecessary. Going forward, it is not sustainable for this to depend on manual interaction by me. We at least need to allow a more active member of the Haskell team do these, or finally automate this tedious task. Cheers, Joachim Am 20. August 2019 20:06:11 MESZ schrieb peter green : >Recently I noticed that haskell binnmus no longer seemed to be >happening. For example mediawiki2latex has no been BD-Uninstallable >since it was uploaded 24 days ago and >https://buildd.debian.org/status/architecture.php?a=arm64=sid >doesn't show any haskell binnmus currently being built (there are a >handful in BD-Uninstallable but they seem to have been scheduled as >part of non-haskell related transitions). > >As I understand it haskell binnmus are normally handled by Joachim >Breitner, who has a script that outputs what needs binnmuing and feeds >output from that script into wanna-build. Is there some issue that is >blocking this at the moment.
