Re: RFR: Britney - autopkgtest integration
Hi all, On 15-04-18 20:18, Paul Gevers wrote: > Just for reference, I started collecting relevant bugs in the > bts with the ci-t...@tracker.debian.org user¹ in the bts (bunk and > ginggs added a couple using the right usertags as well). It has been brought to my attention that the ci-t...@tracker.debian.org isn't a valid e-mail address and shouldn't be used. I have migrated all the user tags to the debian...@lists.debian.org user². Paul ² https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debian...@lists.debian.org signature.asc Description: OpenPGP digital signature
Re: RFR: Britney - autopkgtest integration
[Re-sending as the autopkgtest list on Alioth has ceased to exist. Sorry
for the dup.]
Hi Niels,
On 15-04-18 12:50, Niels Thykier wrote:
> Ok, a few remarks:
>
>
> AutopkgtestPolicy.__init__:
>> self.pending_tests_file = os.path.join(self.options.state_dir,
>> 'pending.json')
>> self.results_cache_file = os.path.join(self.options.state_dir,
>> 'results.cache')
>
> The basenames ("pending.json" and "results.cache") are rather generic
> and state_dir is reused between all policies. I think we should
> namespace these files somehow (E.g. either put them in a subdir or name
> them "autopkgtests-").
I am not at all attached to the names. They come straight from the
Ubuntu code, so I see no problem changing them.
> AutopkgtestPolicy.apply_policy_impl:
>> # Keep track if this source package has tests of its own for
>> the
>> # bounty system, but only if at least one arch has something
>> else than
>> # running or alwaysfail
>> if testsrc == source_name and r - {'RUNNING',
>> 'RUNNING-ALWAYSFAIL', 'ALWAYSFAIL'}:
>> src_has_own_test = True
>
> How difficult would it be to make the bouncy only apply if the tests are
> successful on all architectures (i.e. no "ALWAYSFAIL")? If it is a
> trivial matter, then I would prefer to only give bonus to packages that
> are a good example on all architectures.
I'll think about it. I don't know by heart. I expect this to be no
problem. Please be aware though that currently ci.debian.net only runs
amd64. I don't know the details of what would be needed to add other
architectures, but I expect that to be: hardware. ci.debian.net
currently runs on Amazon EC2 instances, I think there is no other
hardware available there. We had some arm flavor in the past, but I
believe that couldn't keep up or the setup was buggy, or something. In
principle this is no issue, and there have been hardware offers. They
just haven't materialized.
> AutopkgtestPolicy.tests_for_source:
>> # Hardcode linux-meta → linux, lxc, glibc, systemd triggers until
>> we get a more flexible
>> # implementation: https://bugs.debian.org/779559
>> if src.startswith('linux-meta'):
>> for pkg in ['lxc', 'lxd', 'glibc', src.replace('linux-meta',
>> 'linux'), 'systemd', 'snapd']:
>> if pkg not in reported_pkgs:
>> # does this have any image on this arch?
>> for pkg_id in srcinfo.binaries:
>> if pkg_id.architecture == arch and '-image' in
>> pkg_id.package_name:
>> try:
>> tests.append((pkg,
>> self.britney.sources['unstable'][pkg].version))
>> except KeyError:
>> try:
>> tests.append((pkg,
>> sources_info[pkg].version))
>> except KeyError:
>> # package not in that series? *shrug*,
>> then not
>> pass
>> break
>
> Is this piece still relevant? AFAICT, #779559 is fixed in stable. (I
> admit having a feeling of deja vu about this - apologies if I asked
> about this case earlier and simply forgot the answer).
Coincidentally, I investigated last night and removed the code already
locally. Debian doesn't even have a linux-meta package, that is
something Ubuntu specific. In my opinion, the Ubuntu linux-meta package
should just start do the right thing now.
> Beyond that: I also committed some changes for some nits things I
> discovered while reviewing the code.
Ack. And thanks for those. I am not fluent yet in Python to catch any of
these issues.
>> Secondly, assuming we can come to deployment of the new britney policy,
>> I started a gobby document³ to draft an announcement to d-d-a.
[...]
> My "quick" fix would be removing the emphasis and inject a bit longer
> explanation:
> """
> There is one important note to make here on how to go about regressions
> in test cases from reverse dependencies. We recommend communication
> between the maintainers of the involved packages as one party has
> insight in what changed and the other party insight in what is being
> tested. More information is available in the britney documentation
> [britney].
> """
Ack, will take over.
> On another note: Would you be open for getting hint permissions for
> dealing with autopkgtest policy issues (at least in the initial phase of
> the deployment)?
I recon you already know the answer: yes.
I have an additional note. There is one commit to debci pending review
and deployment that I like to get landed before going live with the
official britney checking autopkgtest: the retrigger link currently only
works with a POST http request with a special debci key; the commit will
add a functional GET request option and optionally works with
Re: RFR: Britney - autopkgtest integration
Hi Niels,
On 15-04-18 12:50, Niels Thykier wrote:
> Ok, a few remarks:
>
>
> AutopkgtestPolicy.__init__:
>> self.pending_tests_file = os.path.join(self.options.state_dir,
>> 'pending.json')
>> self.results_cache_file = os.path.join(self.options.state_dir,
>> 'results.cache')
>
> The basenames ("pending.json" and "results.cache") are rather generic
> and state_dir is reused between all policies. I think we should
> namespace these files somehow (E.g. either put them in a subdir or name
> them "autopkgtests-").
I am not at all attached to the names. They come straight from the
Ubuntu code, so I see no problem changing them.
> AutopkgtestPolicy.apply_policy_impl:
>> # Keep track if this source package has tests of its own for
>> the
>> # bounty system, but only if at least one arch has something
>> else than
>> # running or alwaysfail
>> if testsrc == source_name and r - {'RUNNING',
>> 'RUNNING-ALWAYSFAIL', 'ALWAYSFAIL'}:
>> src_has_own_test = True
>
> How difficult would it be to make the bouncy only apply if the tests are
> successful on all architectures (i.e. no "ALWAYSFAIL")? If it is a
> trivial matter, then I would prefer to only give bonus to packages that
> are a good example on all architectures.
I'll think about it. I don't know by heart. I expect this to be no
problem. Please be aware though that currently ci.debian.net only runs
amd64. I don't know the details of what would be needed to add other
architectures, but I expect that to be: hardware. ci.debian.net
currently runs on Amazon EC2 instances, I think there is no other
hardware available there. We had some arm flavor in the past, but I
believe that couldn't keep up or the setup was buggy, or something. In
principle this is no issue, and there have been hardware offers. They
just haven't materialized.
> AutopkgtestPolicy.tests_for_source:
>> # Hardcode linux-meta → linux, lxc, glibc, systemd triggers until
>> we get a more flexible
>> # implementation: https://bugs.debian.org/779559
>> if src.startswith('linux-meta'):
>> for pkg in ['lxc', 'lxd', 'glibc', src.replace('linux-meta',
>> 'linux'), 'systemd', 'snapd']:
>> if pkg not in reported_pkgs:
>> # does this have any image on this arch?
>> for pkg_id in srcinfo.binaries:
>> if pkg_id.architecture == arch and '-image' in
>> pkg_id.package_name:
>> try:
>> tests.append((pkg,
>> self.britney.sources['unstable'][pkg].version))
>> except KeyError:
>> try:
>> tests.append((pkg,
>> sources_info[pkg].version))
>> except KeyError:
>> # package not in that series? *shrug*,
>> then not
>> pass
>> break
>
> Is this piece still relevant? AFAICT, #779559 is fixed in stable. (I
> admit having a feeling of deja vu about this - apologies if I asked
> about this case earlier and simply forgot the answer).
Coincidentally, I investigated last night and removed the code already
locally. Debian doesn't even have a linux-meta package, that is
something Ubuntu specific. In my opinion, the Ubuntu linux-meta package
should just start do the right thing now.
> Beyond that: I also committed some changes for some nits things I
> discovered while reviewing the code.
Ack. And thanks for those. I am not fluent yet in Python to catch any of
these issues.
>> Secondly, assuming we can come to deployment of the new britney policy,
>> I started a gobby document³ to draft an announcement to d-d-a.
[...]
> My "quick" fix would be removing the emphasis and inject a bit longer
> explanation:
> """
> There is one important note to make here on how to go about regressions
> in test cases from reverse dependencies. We recommend communication
> between the maintainers of the involved packages as one party has
> insight in what changed and the other party insight in what is being
> tested. More information is available in the britney documentation
> [britney].
> """
Ack, will take over.
> On another note: Would you be open for getting hint permissions for
> dealing with autopkgtest policy issues (at least in the initial phase of
> the deployment)?
I recon you already know the answer: yes.
I have an additional note. There is one commit to debci pending review
and deployment that I like to get landed before going live with the
official britney checking autopkgtest: the retrigger link currently only
works with a POST http request with a special debci key; the commit will
add a functional GET request option and optionally works with Debian SSO
certificates. Apart from that, my experience over the last week is very
Re: RFR: Britney - autopkgtest integration
Paul Gevers:
> Dear release team,
>
> Since Dec 2016⁰ I have been working together with multiple people
> (including nthykier) to prepare the integration of britney with
> autopkgtest results. We believe the work has reached a state where full
> deployment can be considered. Let me describe how it is designed to work
> and what I request from you.
>
>
Hi Paul,
Thanks for working on this and I am excited that we are getting ready to
deploy this. :)
> [...]
>
> Request
> ===
> First of all, I'd like to request review of the changes I made to
> britney to facilitate the integration with autopkgtest results. My work
> lives on the autopkgtest branch of the github repository of britney². I
> would appreciate it when you could have a look at the new code and
> provide feedback.
>
Ok, a few remarks:
AutopkgtestPolicy.__init__:
> self.pending_tests_file = os.path.join(self.options.state_dir, 'pending.json')
> self.results_cache_file = os.path.join(self.options.state_dir,
> 'results.cache')
The basenames ("pending.json" and "results.cache") are rather generic
and state_dir is reused between all policies. I think we should
namespace these files somehow (E.g. either put them in a subdir or name
them "autopkgtests-").
AutopkgtestPolicy.apply_policy_impl:
> # Keep track if this source package has tests of its own for
> the
> # bounty system, but only if at least one arch has something
> else than
> # running or alwaysfail
> if testsrc == source_name and r - {'RUNNING',
> 'RUNNING-ALWAYSFAIL', 'ALWAYSFAIL'}:
> src_has_own_test = True
How difficult would it be to make the bouncy only apply if the tests are
successful on all architectures (i.e. no "ALWAYSFAIL")? If it is a
trivial matter, then I would prefer to only give bonus to packages that
are a good example on all architectures.
AutopkgtestPolicy.tests_for_source:
> # Hardcode linux-meta → linux, lxc, glibc, systemd triggers until we
> get a more flexible
> # implementation: https://bugs.debian.org/779559
> if src.startswith('linux-meta'):
> for pkg in ['lxc', 'lxd', 'glibc', src.replace('linux-meta',
> 'linux'), 'systemd', 'snapd']:
> if pkg not in reported_pkgs:
> # does this have any image on this arch?
> for pkg_id in srcinfo.binaries:
> if pkg_id.architecture == arch and '-image' in
> pkg_id.package_name:
> try:
> tests.append((pkg,
> self.britney.sources['unstable'][pkg].version))
> except KeyError:
> try:
> tests.append((pkg,
> sources_info[pkg].version))
> except KeyError:
> # package not in that series? *shrug*,
> then not
> pass
> break
Is this piece still relevant? AFAICT, #779559 is fixed in stable. (I
admit having a feeling of deja vu about this - apologies if I asked
about this case earlier and simply forgot the answer).
Beyond that: I also committed some changes for some nits things I
discovered while reviewing the code.
> Secondly, assuming we can come to deployment of the new britney policy,
> I started a gobby document³ to draft an announcement to d-d-a. The
> document has already been commented on by nthykier and terceiro (CI),
> but I'd like it to be as good as possible.
>
One remark I just noted:
Cite:
> There is one important note to make here on how to go about regressions in
> test
> cases from reverse dependencies. We really recommend *communication* between
> the maintainers of the involved packages. More information is available in the
> britney documentation [britney].
We stress "communication" in that paragraph: What (I presume) we want
to say is that people to talk to the maintainers of the other party
(instead of the release team).
When I reread it today, I suddenly interpreted it as "We recommend that
you communicate" (with the implication that maintainers are not
communicating at all). I know that is not what we are trying to imply,
but some people might misread it.
My "quick" fix would be removing the emphasis and inject a bit longer
explanation:
"""
There is one important note to make here on how to go about regressions
in test cases from reverse dependencies. We recommend communication
between the maintainers of the involved packages as one party has
insight in what changed and the other party insight in what is being
tested. More information is available in the britney documentation
[britney].
"""
> [...]
>
> Looking forward to your responses,
> Paul
>
> [...]
>
Once again, thanks for your hard work. :)
On another note: Would you be open for getting hint permissions for
dealing with
Re: RFR: Britney - autopkgtest integration
Hi Raphaël On 09-04-18 17:28, Raphael Hertzog wrote: > All the information in the HTML should be in the YAML too. At least that > was the goal of Niels when he created that file. And the Debian package > tracker is now consumer of the YAML file only so everything that we want > to display to users on tracker.debian.org needs to be in the YAML file. I'll have a look at it (not highest on my list). Paul signature.asc Description: OpenPGP digital signature
Re: RFR: Britney - autopkgtest integration
Hi, On Fri, 06 Apr 2018, Paul Gevers wrote: > > Also I see the age-requirement bumped to 15 but it would be nice to > > also have somewhere the details of why we got 15 instead of the usual > > value. > > Can you explain what you find unclear in the excuses text (check the > html version please)? Do you have a proposal to make it better? > Currently it says: "Required age increased by 10 days because of > autopkgtest" or "Required age decreased by 3 days because of autopkgtest" All the information in the HTML should be in the YAML too. At least that was the goal of Niels when he created that file. And the Debian package tracker is now consumer of the YAML file only so everything that we want to display to users on tracker.debian.org needs to be in the YAML file. Cheers, -- Raphaël Hertzog ◈ Debian Developer Support Debian LTS: https://www.freexian.com/services/debian-lts.html Learn to master Debian: https://debian-handbook.info/get/
Re: RFR: Britney - autopkgtest integration
Hi Raphael On 05-04-18 14:48, Raphael Hertzog wrote: > On Sun, 01 Apr 2018, Paul Gevers wrote: >> (including nthykier) to prepare the integration of britney with >> autopkgtest results. We believe the work has reached a state where full >> deployment can be considered. Let me describe how it is designed to work >> and what I request from you. > > Huge thanks for this work! I'm looking forward to it and I plan to deploy > it for use by Kali too. Great. > Hence I would like to know a bit more how it all ties together with debci. > I have looked quickly at the code and I saw references to the AMQP > messaging queue, to a REST API on the debci side, and also to swift. > Can you quickly describe the workflow between all those components and > britney? In contrast to Ubuntu, Debian's britney2 doesn't talk to the outside world. So britney2 takes input from files and writes output to files. In the case for autopkgtest integration the following files are relevant: /debci.json -> this is a file obtained from ci.d.n with the current status via the API interface of debci (ci.d.n/documentation/API). britney1 is responsible for getting that file just before it calls britney2. /debci.input -> this is the file created by britney2 in the same format as Ubuntu uses for its AMQP messaging queue to transfer the info to ci.d.n. britney1 converts this file into the input commands suitable for the debci. Antonio and I decided that debci shouldn't care about what anybody considers as a trigger. So the only thing it cares about is which suite to use as baseline and which packages (and relevant dependencies) to take from another suite. Swift isn't used in the Debian setup, that is how Ubuntu does its thing (communicates results back to britney2). So in Debian: britney2 <-> britney1 <-> debci <-> autopktest ^^ files REST API in Ubuntu britney2 <-> debci/?? <-> autopktest ^ AMQP/swift > How does debci implement the "test package foo/unstable in testing"? > I'm asking the question because I wonder how it copes with packages > having gained a dependency on a version of another package that has not > yet migrated either. You answer a bit of this yourself later, will answer there. > Your documentation has this somewhat related paragraph: > +There are cases where it is required to have multiple packages migrate > together > +to have the test cases pass, e.g. when there was a bug in a regressing test > +case of a reverse dependency and that got fixed. In that case the test cases > +need to be triggered with both packages from the source suite in the target > +suite (again, how this is done depends on the setup). > > Can you let us know how this will be done for the official Debian setup? I haven't figured out how to do this automatically yet (and neither has Ubuntu, so apparently this isn't a big issue). In Debian for now this is less of an issue as we start with aging changes. So stuff migrates in the end anyways. For now, people with the right knowledge (= the britney key for ci.d.n) can trigger the right combination manually. > Another thing that I wonder is whether the autopkgtest policy check will > be cleared for a given version as soon as it has run or whether it might > have to run multiple times if some of its dependencies migrate in testing > before it migrated? (i.e. the test environment evolved during the aging > period) Once cleared, it remains cleared, the opposite isn't true. The current design isn't smart enough to invalidate passes again. >> Shadow britney >> == >> I have started a shadow cron job of britney on respighi to see how >> everything is behaving. I'll start filing bugs on regressions such that >> the current code already has value, but the drawback of a shadow run is >> that the information is very poorly discover-able for interested >> parties. Also, I expect people to start noticing that ci.d.n is >> generating more results, I think it would be great if we can explain >> this in one go after britney has the autopkgtest code deployed. > > Looking for such results on ci.debian.net, I found this: > https://ci.debian.net/packages/g/gnome-photos/testing/amd64/ > > Which gives some hints to my former question. It looks like > autopkgtest is receiving some supplementary option like this > one: > --add-apt-release=unstable --pin-packages=unstable=src:gnome-photos > (seen in > https://ci.debian.net/data/autopkgtest/testing/amd64/g/gnome-photos/98854/log.gz) Yes, autopktest recently gained functionality to add multiple suites to the available pool and to use apt pinning to pull in the relevant packages from the "unstable" suite. If a package has requirements that are only fulfilled by unstable, they should be pulled in from unstable. (There may be a bug discovered in Ubuntu recently about autopkgtest using the wrong resolver to do this as intended, but that is the idea). > For persons interested in seeing excuses data generated by
Re: RFR: Britney - autopkgtest integration
Hello Paul, On Sun, 01 Apr 2018, Paul Gevers wrote: > (including nthykier) to prepare the integration of britney with > autopkgtest results. We believe the work has reached a state where full > deployment can be considered. Let me describe how it is designed to work > and what I request from you. Huge thanks for this work! I'm looking forward to it and I plan to deploy it for use by Kali too. Hence I would like to know a bit more how it all ties together with debci. I have looked quickly at the code and I saw references to the AMQP messaging queue, to a REST API on the debci side, and also to swift. Can you quickly describe the workflow between all those components and britney? How does debci implement the "test package foo/unstable in testing"? I'm asking the question because I wonder how it copes with packages having gained a dependency on a version of another package that has not yet migrated either. Your documentation has this somewhat related paragraph: +There are cases where it is required to have multiple packages migrate together +to have the test cases pass, e.g. when there was a bug in a regressing test +case of a reverse dependency and that got fixed. In that case the test cases +need to be triggered with both packages from the source suite in the target +suite (again, how this is done depends on the setup). Can you let us know how this will be done for the official Debian setup? Another thing that I wonder is whether the autopkgtest policy check will be cleared for a given version as soon as it has run or whether it might have to run multiple times if some of its dependencies migrate in testing before it migrated? (i.e. the test environment evolved during the aging period) > Shadow britney > == > I have started a shadow cron job of britney on respighi to see how > everything is behaving. I'll start filing bugs on regressions such that > the current code already has value, but the drawback of a shadow run is > that the information is very poorly discover-able for interested > parties. Also, I expect people to start noticing that ci.d.n is > generating more results, I think it would be great if we can explain > this in one go after britney has the autopkgtest code deployed. Looking for such results on ci.debian.net, I found this: https://ci.debian.net/packages/g/gnome-photos/testing/amd64/ Which gives some hints to my former question. It looks like autopkgtest is receiving some supplementary option like this one: --add-apt-release=unstable --pin-packages=unstable=src:gnome-photos (seen in https://ci.debian.net/data/autopkgtest/testing/amd64/g/gnome-photos/98854/log.gz) For persons interested in seeing excuses data generated by this shadow britney, it seems to be in respighi.debian.org:/home/elbrus/data-b2/output/ BTW, the excuses.yaml does contain weird lines with "null": (in appstream entry) autopkgtest: appstream: amd64: - RUNNING - https://ci.debian.net/status/pending - https://ci.debian.net/packages/a/appstream/testing/amd64 - null - null Also I see the age-requirement bumped to 15 but it would be nice to also have somewhere the details of why we got 15 instead of the usual value. Cheers, -- Raphaël Hertzog ◈ Debian Developer Support Debian LTS: https://www.freexian.com/services/debian-lts.html Learn to master Debian: https://debian-handbook.info/get/
