Bug#896023: Bug#901897 closed by Jonathan Nieder (Bug#901897: fixed in git 1:2.18.0~rc2-2)
Paul Gevers writes ("Re: Bug#901897 closed by Jonathan Nieder
(Bug#901897: fixed in git 1:2.18.0~rc2-2)"):
> On 22-06-18 18:05, Ian Jackson wrote:
> It currently happens quite often, but often the test suite from testing
> runs happily with the package from unstable. If the (passing) results
> are of any value I can't say (E.g. transition with other source full
> uploads (not uncommon), KDE stack, new python-defaults, ...)
Right. This doesn't sound too intolerable really.
> Thanks. As said, I think currently it is wiser to aim for good than for
> perfect.
Absolutely!
Thanks,
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Bug#896023: Bug#901897 closed by Jonathan Nieder (Bug#901897: fixed in git 1:2.18.0~rc2-2)
Hi Ian, On 22-06-18 18:05, Ian Jackson wrote: > AIUI what you are promposing is, when A's tests are being run for B's > migration: > > * tell autopkgtest to run the tests from A in testing, with an apt >configuration which pins B from unstable. > > * autopkgtest runs an apt solver to install the to-be-tested >packages, including B from unstable and other things preferentially >from testing. > > * if you are lucky, the to-be-tested packages include A from testing. >So you run A/testing's tests, with A/testing installed. > > * if you are unlucky, the to-be-tested packages include A from >unstable. Discard any test results so far, and restart: >this time, tell autopkgtest to run the tests from B in testing A in unstable > I think in principle this anomaly might occur for some .deb packages > and not others. It currently happens quite often, but often the test suite from testing runs happily with the package from unstable. If the (passing) results are of any value I can't say (E.g. transition with other source full uploads (not uncommon), KDE stack, new python-defaults, ...) So sure. The whole autopkgtest setup isn't perfect. In the perfect world you would ask britney to calculate a migration. Run autopkgtest with the "new testing", when no regressions occur, you're done and the migration can happen. If there are, try again (what next?). This will slow down the whole migration a lot because of the time it takes to test and the amount of redos. So until we want to spend unlimited amounts of computation resources, we have a limited situation anyways. > And I think britney can migrate individual .debs > occasionally ? It does that all the time. > This all seems quite a lot of workaround. It would be > better to treat britney as free software that we can modify to DTRT. Yes, but it is more difficult there. Indeed, it is the Right Thing To Do. It was in my summary email¹ as my second work item. > But I'm not offering to do anh of the work so I will let you get on > with your workaroud and wish you luck! Thanks. As said, I think currently it is wiser to aim for good than for perfect. Paul ¹ https://lists.debian.org/debian-release/2018/06/msg00239.html signature.asc Description: OpenPGP digital signature
Bug#896023: Bug#901897 closed by Jonathan Nieder (Bug#901897: fixed in git 1:2.18.0~rc2-2)
Paul Gevers writes ("Re: Bug#901897 closed by Jonathan Nieder
(Bug#901897: fixed in git 1:2.18.0~rc2-2)"):
> Which option did you have in mind? I don't spot anything that isn't in
> autopkgtest either, but I didn't read carefully.
I think you would have to take more manual control and pass an essay
command line to adt-run. And you would have to know exactly what you
wanted, which I think really is the problem.
> Remember what the situation is. We start with testing and we only add a
> limited set of apt-pins. In the case we are discussing, the package we
> want to test is NOT in the pin. Then we ask autopkgtest to determine
> which test suite to pick. Naturally, it will pick the version from
> testing. Then we install the test dependencies. Due to not all
> dependencies from the packages from unstable being fulfilled in testing
> apt-get install fails and in the *fallback* we install the new version
> of the tested package, which is now out-of-sync with its own test suite.
I don't think this can be addressed without *someone* analysing the
dependencies of the proposed test case and figuring out which version
of which package will be installed for the test.
> Is there an option in adt-run that would either (a) pick the right
> version to start with or (b) will go back and try to install the test
> dependencies for its unstable test suite?
You would have to go back and actually use the unstable version of the
test suite. Effectively, starting again from scratch, I think.
AIUI what you are promposing is, when A's tests are being run for B's
migration:
* tell autopkgtest to run the tests from A in testing, with an apt
configuration which pins B from unstable.
* autopkgtest runs an apt solver to install the to-be-tested
packages, including B from unstable and other things preferentially
from testing.
* if you are lucky, the to-be-tested packages include A from testing.
So you run A/testing's tests, with A/testing installed.
* if you are unlucky, the to-be-tested packages include A from
unstable. Discard any test results so far, and restart:
this time, tell autopkgtest to run the tests from B in testing
I think in principle this anomaly might occur for some .deb packages
and not others. And I think britney can migrate individual .debs
occasionally ? This all seems quite a lot of workaround. It would be
better to treat britney as free software that we can modify to DTRT.
But I'm not offering to do anh of the work so I will let you get on
with your workaroud and wish you luck!
Regards,
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Bug#896023: Bug#901897 closed by Jonathan Nieder (Bug#901897: fixed in git 1:2.18.0~rc2-2)
Hi Ian,
On 22-06-18 15:25, Ian Jackson wrote:
> Ian Jackson writes ("Re: Bug#901897 closed by Jonathan Nieder
> (Bug#901897: fixed in git 1:2.18.0~rc2-2)"):
>> I think it _is_ optional. At least, it was in adt-run. I think
>> autopkgtest has much of the same logic. It's a question of putting
>> together the right rune to specify what you want.
>
> I am wrong. autopkgtest has a completely different command line
> parser to adt-run.
>
> https://manpages.debian.org/unstable/autopkgtest/autopkgtest.1.en.html
> https://manpages.debian.org/stretch/autopkgtest/adt-run.1.en.html
>
> adt-run's command line is a bit abstruse but I think it is more
> powerful. Unfortunately adt-run was deleted.
Which option did you have in mind? I don't spot anything that isn't in
autopkgtest either, but I didn't read carefully.
Remember what the situation is. We start with testing and we only add a
limited set of apt-pins. In the case we are discussing, the package we
want to test is NOT in the pin. Then we ask autopkgtest to determine
which test suite to pick. Naturally, it will pick the version from
testing. Then we install the test dependencies. Due to not all
dependencies from the packages from unstable being fulfilled in testing
apt-get install fails and in the *fallback* we install the new version
of the tested package, which is now out-of-sync with its own test suite.
Is there an option in adt-run that would either (a) pick the right
version to start with or (b) will go back and try to install the test
dependencies for its unstable test suite?
It is this option (b) that I am thinking about to implement as I don't
believe autopkgtest should be able to do (a).
Paul
NB: juliank is currently implementing a new native apt solver that
should be able to solve the issues that we have for which we wanted the
aspcud solver. If that lands, I think I will not spend my time on fixing
britney right now, but instead just implement this (b) option.
signature.asc
Description: OpenPGP digital signature
Bug#896023: Bug#901897 closed by Jonathan Nieder (Bug#901897: fixed in git 1:2.18.0~rc2-2)
Ian Jackson writes ("Re: Bug#901897 closed by Jonathan Nieder
(Bug#901897: fixed in git 1:2.18.0~rc2-2)"):
> I think it _is_ optional. At least, it was in adt-run. I think
> autopkgtest has much of the same logic. It's a question of putting
> together the right rune to specify what you want.
I am wrong. autopkgtest has a completely different command line
parser to adt-run.
https://manpages.debian.org/unstable/autopkgtest/autopkgtest.1.en.html
https://manpages.debian.org/stretch/autopkgtest/adt-run.1.en.html
adt-run's command line is a bit abstruse but I think it is more
powerful. Unfortunately adt-run was deleted.
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Bug#896023: Bug#901897 closed by Jonathan Nieder (Bug#901897: fixed in git 1:2.18.0~rc2-2)
Paul Gevers writes ("Re: Bug#901897 closed by Jonathan Nieder
(Bug#901897: fixed in git 1:2.18.0~rc2-2)"):
> On 22-06-18 15:02, Ian Jackson wrote:
> > I don't know exactly what autopkgtest command line rune you are using
> > but certainly it was designed to be able to run the test suite of one
> > version against a different version.
>
> I manually just told autopkgtest to take two packages from unstable
> instead of one. So you are saying that the current behavior is designed
> behavior. Thinking about it I can see why you say that, but it is
> annoying in the Debian and Ubuntu infrastructure while I see no benefit
> from it for them. Maybe this should be optional then and be turned on
> for Debian and Ubuntu?
I think it _is_ optional. At least, it was in adt-run. I think
autopkgtest has much of the same logic. It's a question of putting
together the right rune to specify what you want.
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Bug#896023: Bug#901897 closed by Jonathan Nieder (Bug#901897: fixed in git 1:2.18.0~rc2-2)
Hi Ian,
On 22-06-18 15:02, Ian Jackson wrote:
> Paul Gevers writes ("Re: Bug#901897 closed by Jonathan Nieder
> (Bug#901897: fixed in git 1:2.18.0~rc2-2)"):
>> On 22-06-18 12:45, Ian Jackson wrote:
>>> But that is appropriate, because the new git Breaks earlier versions
>>> of dgit. AIUI britney will therefore not migrate the new git until
>>> the new dgit is migrated, which is correct.
>>
>> Ack. That is why I allow myself to intervene in such a situation.
>
> If you had done nothing, I guess after dgit was migrated, git would
> get a retest and then things would pass ? So there would be a small
> delay to the git migration.
Because of the inherited Urgency: high, autopkgtest results are
irrelevant for git migration.
>> This is very much not dgit specific. Today I had to do that for about 5
>> packages (coincidence, last two months I have done it about 15-20 times
>> or so). Autopkgtest doesn't detect that the installed version of the
>> package that it is testing is not the one that it was asked to test. It
>> must, and then start over again in some form with the test suite from
>> the right version.
>
> This sounds quite annoying. I think the right fix is for britney to
> tell ci.d.n a set of packages which might migrate. ISTR discussing
> that on debian-ci recently.
Good point. Let me consider that during prioritization of my next actions.
> I don't know exactly what autopkgtest command line rune you are using
> but certainly it was designed to be able to run the test suite of one
> version against a different version.
I manually just told autopkgtest to take two packages from unstable
instead of one. So you are saying that the current behavior is designed
behavior. Thinking about it I can see why you say that, but it is
annoying in the Debian and Ubuntu infrastructure while I see no benefit
from it for them. Maybe this should be optional then and be turned on
for Debian and Ubuntu?
Paul
signature.asc
Description: OpenPGP digital signature
Bug#896023: Bug#901897 closed by Jonathan Nieder (Bug#901897: fixed in git 1:2.18.0~rc2-2)
Paul Gevers writes ("Re: Bug#901897 closed by Jonathan Nieder
(Bug#901897: fixed in git 1:2.18.0~rc2-2)"):
> On 22-06-18 12:45, Ian Jackson wrote:
> > to the bad run which prompted your intervention, and the good run
> > which resulted ?
>
> [2] for the bad one. You already found the good one: [1].
Ah, thanks.
> > But that is appropriate, because the new git Breaks earlier versions
> > of dgit. AIUI britney will therefore not migrate the new git until
> > the new dgit is migrated, which is correct.
>
> Ack. That is why I allow myself to intervene in such a situation.
If you had done nothing, I guess after dgit was migrated, git would
get a retest and then things would pass ? So there would be a small
delay to the git migration.
> This is very much not dgit specific. Today I had to do that for about 5
> packages (coincidence, last two months I have done it about 15-20 times
> or so). Autopkgtest doesn't detect that the installed version of the
> package that it is testing is not the one that it was asked to test. It
> must, and then start over again in some form with the test suite from
> the right version.
This sounds quite annoying. I think the right fix is for britney to
tell ci.d.n a set of packages which might migrate. ISTR discussing
that on debian-ci recently.
I don't know exactly what autopkgtest command line rune you are using
but certainly it was designed to be able to run the test suite of one
version against a different version.
Regards,
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Bug#896023: Bug#901897 closed by Jonathan Nieder (Bug#901897: fixed in git 1:2.18.0~rc2-2)
Hi Ian,
[Dropping bug 901897 as that has nothing to do with it; adding 896023 as
it is worth explaining it there if that bug isn't clear yet.]
On 22-06-18 12:45, Ian Jackson wrote:
> Paul Gevers writes ("Re: Bug#901897 closed by Jonathan Nieder
> (Bug#901897: fixed in git 1:2.18.0~rc2-2)"):
>> And I just helped the test for git a bit as due to bug 896023 in
>> autopkgtest it didn't use the right autopkgtest from dgit.
>
> Thanks.
>
> It's not clear to me exactly what needed fixing. Can you send me urls
> to the bad run which prompted your intervention, and the good run
> which resulted ?
[2] for the bad one. You already found the good one: [1].
> Looking here
> https://qa.debian.org/excuses.php?package=git
> I see a migration test for git 1:2.18.0-1 which used dgit 5.1,
> despite the fact that dgit 5.1 is not in testing yet.
That was because of my intervention AND in [1] and because of the
appropriate Breaks in both [1] and [2].
> But that is appropriate, because the new git Breaks earlier versions
> of dgit. AIUI britney will therefore not migrate the new git until
> the new dgit is migrated, which is correct.
Ack. That is why I allow myself to intervene in such a situation.
> Looking at the test log[1] it used the test suite from 5.1, too.
True, but ONLY because I told it so (you'll see that used 4.4 (as debci
reports) in [2]).
> I don't know how other packages do things, but in general, with dgit
> at least, mixing and matching program versions and test versions may
> work some of the time: particularly, newer programs will usually pass
> older tests. Older programs will typically fail newer tests since the
> newer tests will look for new features.
This is very much not dgit specific. Today I had to do that for about 5
packages (coincidence, last two months I have done it about 15-20 times
or so). Autopkgtest doesn't detect that the installed version of the
package that it is testing is not the one that it was asked to test. It
must, and then start over again in some form with the test suite from
the right version.
> In this particular case: dgit 4.4's test suite will always fail with
> the new git. dgit 5.1's test suite will properly test whether the
> tested dgit and git are compatible: so it will only fail with new git
> and old dgit.pass with old git; and with new git:
>git dgit test suitedgitresult
> 1:2.17.1-14.4, 5.0, 5.1 4.4, 5.0, 5.1 PASS
> 1:2.18.0-1<= 5.0any FAIL
> 1:2.18.0-15.1 <= 5.0 FAIL
> 1:2.18.0-15.1 5.1 PASS
>
> AFAICT from the migration information from britney from yesterday and
> today, all is fine except for git's an artificially inflated urgency.
Agreed.
Paul
> [1]
> https://ci.debian.net/data/autopkgtest/testing/amd64/d/dgit/491815/log.gz[2]
https://ci.debian.net/data/autopkgtest/testing/amd64/d/dgit/491341/log.gz
signature.asc
Description: OpenPGP digital signature
