Re: [systemd-devel] New "ubuntu-ci" integration tests are being added to PRs

[Lennart Poettering]

On Thu, 18.02.16 09:01, Martin Pitt (martin.p...@ubuntu.com) wrote:

> Hello all,
> 
> you might already have noticed, but from now on PRs will not only
> trigger the semaphore checks (which are essentially a "make
> distcheck"), but also trigger more comprehensive integration tests on
> Ubuntu's autopkgtest infrastructure. These build actual binary .deb
> packages, install them in an actual OS, and cover things like
> networkd, bootchart, that crucial services like NetworkManager or
> window manager come up, timedatectl and friends, cryptsetup,
> systemd-sysv-install (with various combinations of SysV+systemd
> units), and the "boot-smoke" test where the whole thing has to boot
> successfully 20 times [2].
> 
> This new test can be seen at e. g.
> https://github.com/systemd/systemd/pull/2641 or /2650 which now also
> have a second "ubuntu-ci" test.
> 
> I now wrote all the glue between github and autopkgtest.ubuntu.com,
> running into gory details like firewall issues, three different ways
> to authenticate, and my own hilarious incompetence when it comes to
> web programming (but I'm learning :) ). Thanks to Daniel for his great
> help with kickstarting me into GitHub webhooks!
> 
> However, an awful lot of the runs currently fail with a linker error.
> I filed [2] and will investigate.
> 
> So please don't put too much attention to these results yet. I want to
> to enable them to see how the testing and communication holds up in
> practice, but before this we definitively need to sort out [2] first.

Excellent work! Thanks a lot to Daniel and you for setting this up! Thanks!

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] New "ubuntu-ci" integration tests are being added to PRs

[Martin Pitt]

Martin Pitt [2016-02-18  9:01 +0100]:
> However, an awful lot of the runs currently fail with a linker error.
> I filed [2] and will investigate.

Thanks to Filipe that got fixed quickly (it was a recent bug in the
build system indeed). Fun thing is that I didn't actually intend to do
that "--disable-all" variant build, this was an unintended side effect
of GitHub modifying the test request URL. (This got worked around
now).

So all of the glue is in place, and these work now. I went through the
PRs that got opened since we enabled the new tests, and they all have
(mostly) successful runs now.

Two tests (journal getting a log and backtrace of a bash crash, and
networkd handling a hotplugged IPv6 interface) fail very often when
running against upstream for some reason; I disabled these two for now
as we don't want to have all PRs have red flags because of known
issues which already existed before (i. e. they are not regressions
from that PR). I'll investigate these.

I'll watch incoming over the next days, and sort out the remaining
fallout (I'm sure there will be something :) ).

This test is currently being triggered on x86_64 only (amd64 in
Debianspeak). In principle we can also trigger them on i386, or more
exotically, IBM System Z (aka S/390x). We don't have QEMU on the
latter and thus can only run a relatively small subset of integration
tests (those that can run in containers). But it would at least give
us the build and unit test, and a little bit of real-life testing. Is
there any interest in enabling those? (Not right away, but maybe in a
few weeks when things settle down with x86_84).

Thanks,

Martin

-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)


signature.asc
Description: Digital signature
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] New "ubuntu-ci" integration tests are being added to PRs

[Michael Biebl]

Great stuff!

Thanks for working on that, Martin and Daniel.

Regards,
Michael

2016-02-18 9:01 GMT+01:00 Martin Pitt :
> Hello all,
>
> you might already have noticed, but from now on PRs will not only
> trigger the semaphore checks (which are essentially a "make
> distcheck"), but also trigger more comprehensive integration tests on
> Ubuntu's autopkgtest infrastructure. These build actual binary .deb
> packages, install them in an actual OS, and cover things like
> networkd, bootchart, that crucial services like NetworkManager or
> window manager come up, timedatectl and friends, cryptsetup,
> systemd-sysv-install (with various combinations of SysV+systemd
> units), and the "boot-smoke" test where the whole thing has to boot
> successfully 20 times [2].
>
> This new test can be seen at e. g.
> https://github.com/systemd/systemd/pull/2641 or /2650 which now also
> have a second "ubuntu-ci" test.
>
> I now wrote all the glue between github and autopkgtest.ubuntu.com,
> running into gory details like firewall issues, three different ways
> to authenticate, and my own hilarious incompetence when it comes to
> web programming (but I'm learning :) ). Thanks to Daniel for his great
> help with kickstarting me into GitHub webhooks!
>
> However, an awful lot of the runs currently fail with a linker error.
> I filed [2] and will investigate.
>
> So please don't put too much attention to these results yet. I want to
> to enable them to see how the testing and communication holds up in
> practice, but before this we definitively need to sort out [2] first.
>
> Please bear with me!
>
> Thanks,
>
> Martin
>
>
> [1] http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/tree/debian/tests
> [2] https://github.com/systemd/systemd/issues/2651
> --
> Martin Pitt| http://www.piware.de
> Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
>
> ___
> systemd-devel mailing list
> systemd-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
>



-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] New "ubuntu-ci" integration tests are being added to PRs

[Martin Pitt]

Jóhann B. Guðmundsson [2016-02-18 11:35 +]:
> Arguably not if upstream test do not detect these things then they should be
> fixed to do so

The upstream tests are mostly unit tests which cover isolated logic.
Of course having more coverage there would be great, but that's not
the point here. systemd is not an isolated project, but it lives in an
operating system, controlling services like dbus, gdm, NetworkManager, etc.

So saying "fix them to do so" in an email is easy, but did you
actually think about *how* this should look like? We do have some QEMU
tests upstream which cover a little bit of "does this VM boot", but
not much else -- there is no integration testing with other
components. So for doing proper integration testing you need to take
an actual OS, with actual packages, and check that they still work. If
that's Fedora or Debian or whatnot is not that important in the end,
and ideally we test them all.

But it's certainly not practical for the upstream systemd tests to
pull and build glib, X/wayland, gdm, NetworkManager, evemu-tools, etc.
and construct their own synthetic OS into a VM. That would be a
huge maintenance burden, take awfully long, and this synthetic
mini-distro isn't actually being used by any any real person.

So, is it possible that these tests will fail because of a downstream
issue? Absolutely, yes, and I'm sure they will do that at some point.
I'm looking at those failures, and they are a problem that needs
fixing anyway (and if it's a downstream issue, I'm no less on the hook
for that than now). But experience has shown that almost all test
failures which aren't due to actual regressions are due to either bugs
in the tests or infrastructure failure -- and both of these will apply
equally to so-called "upstream tests".

So integrating these into PRs instead of running them every other day
on master makes it much simpler and cheaper to identify the cause of a
regrssion. You can also always compare against other PRs or a
test against master.

Martin
-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] New "ubuntu-ci" integration tests are being added to PRs

[Jóhann B . Guðmundsson]

On 02/18/2016 11:26 AM, Daniel Mack wrote:

On 02/18/2016 12:19 PM, Jóhann B. Guðmundsson wrote:

>On 02/18/2016 10:22 AM, Daniel Mack wrote:

>>I disagree. All sorts of testing is good for us, and if a PR is breaking
>>downstream Ubuntu, and we recognize that before merging, that's really
>>great.

>
>I'm all for more testing the better but due to downstream fragmentation

Fragmentation? Martin does not apply any patches downstream for his
tests, so I don't see where this fear is coming from?



Irrelevant to Martin.

I meant this as a fragmentation from a broad perspective as in the linux 
ecosystem in whole which in turn reflects itself in upstreams like 
Lennart is about to do in #2621.


If you acknowledge thus start making an exception you must realize you 
effectively start making exceptions for *all* since there is exist no 
singularity in that act.
If the patch in that report gets accepted anyone can submit a patch that 
replaces wheel with another user/group and it would have to be accepted 
since no argument could be made against it, otherwise you would be 
acknowledging that you pick favorites and grant exceptions to only those 
favorites ( in that particular case Debian/Ubuntu would be acknowledge 
as an favorite which puts every other consumer of systemd to 
disadvantage ).


In relevance to this thread by allowing one you are allowing *all* and 
so while Martin and his integration play by the book others might not 
but as you say that can be dealt with if and then when that happens.


JBG
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] New "ubuntu-ci" integration tests are being added to PRs

[Jóhann B . Guðmundsson]

On 02/18/2016 10:43 AM, Martin Pitt wrote:

I'd actually turn it the other way around and claim that if Fedora,
Arch, etc. have downstream tests, then please trigger them too. After
all, failures of them don't block anything (right now, anyway), and
having the extra information in the PRs can only be useful?


Arguably not if upstream test do not detect these things then they 
should be fixed to do so ( which in turn should make all the none 
detected failures be downstream issues )  instead of having every 
downstream consumers if systemd comment on the same issue if fails 
locally with them.


JBG
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] New "ubuntu-ci" integration tests are being added to PRs

[Daniel Mack]

On 02/18/2016 12:19 PM, Jóhann B. Guðmundsson wrote:
> On 02/18/2016 10:22 AM, Daniel Mack wrote:
>> I disagree. All sorts of testing is good for us, and if a PR is breaking
>> downstream Ubuntu, and we recognize that before merging, that's really
>> great.
> 
> I'm all for more testing the better but due to downstream fragmentation 

Fragmentation? Martin does not apply any patches downstream for his
tests, so I don't see where this fear is coming from?

> all these have the same fundamental problem as are with multiple 
> downstream issue trackers.

But it's different for CI really.

> In this case you start slowing down integration if PR start failing in 

"if".

Let's see how things go, and assess the situation if we really face too
many false positives. As I said - we're running all our CI on Ubuntu so
far, and unrelated build breakages have rarely been a real issue. And
again - if things break, we should at least learn about the fact.


Thanks,
Daniel

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] New "ubuntu-ci" integration tests are being added to PRs

[Jóhann B . Guðmundsson]

On 02/18/2016 10:22 AM, Daniel Mack wrote:

I disagree. All sorts of testing is good for us, and if a PR is breaking
downstream Ubuntu, and we recognize that before merging, that's really
great.


I'm all for more testing the better but due to downstream fragmentation 
all these have the same fundamental problem as are with multiple 
downstream issue trackers.


In this case you start slowing down integration if PR start failing in 
one or more downstreams ( Arch,Coreos, Debian,Fedora,Mageia,Ubuntu etc 
and need to start spending time researching if the failure is downstream 
specific and during that time the PR will be in frozen state ) and you 
risk those test system(s) start reporting directly upstream  ( which can 
lead to fun things like this [1]  ) since people will quickly think "I'm 
just going to have all failures and errors reported directly upstream 
instead of having to inspect/deal with downstream myself" and those 
thing start small like with "CI test -- don't merge!"


JBG

1. https://bugzilla.redhat.com/show_bug.cgi?id=1300212
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] New "ubuntu-ci" integration tests are being added to PRs

[Daniel Mack]

On 02/18/2016 11:08 AM, Jóhann B. Guðmundsson wrote:
> On 02/18/2016 08:01 AM, Martin Pitt wrote:
>> So please don't put too much attention to these results yet. I want to
>> to enable them to see how the testing and communication holds up in
>> practice, but before this we definitively need to sort out [2] first.
> 
> Will failed tests or false positives start auto creating issues on github?

No.

> If so is that really the smart thing do to here?

No.

> The only test system that should be hooked into upstream should just be 
> upstreams own not some downstreams right.

I disagree. All sorts of testing is good for us, and if a PR is breaking
downstream Ubuntu, and we recognize that before merging, that's really
great.

Also note that Semaphore is already based on Ubuntu all the time, and
the trouble it saved us so far is priceless.

We have not (yet) configured the master branch so it requires all CI
test to succeed prior to merging, so maintainers can still ignore such
information, if they have good reasons.


Thanks,
Daniel

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] New "ubuntu-ci" integration tests are being added to PRs

[Jóhann B . Guðmundsson]

On 02/18/2016 08:01 AM, Martin Pitt wrote:

So please don't put too much attention to these results yet. I want to
to enable them to see how the testing and communication holds up in
practice, but before this we definitively need to sort out [2] first.


Will failed tests or false positives start auto creating issues on github?

If so is that really the smart thing do to here?

The only test system that should be hooked into upstream should just be 
upstreams own not some downstreams right.


JBG
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

[systemd-devel] New "ubuntu-ci" integration tests are being added to PRs

[Martin Pitt]

Hello all,

you might already have noticed, but from now on PRs will not only
trigger the semaphore checks (which are essentially a "make
distcheck"), but also trigger more comprehensive integration tests on
Ubuntu's autopkgtest infrastructure. These build actual binary .deb
packages, install them in an actual OS, and cover things like
networkd, bootchart, that crucial services like NetworkManager or
window manager come up, timedatectl and friends, cryptsetup,
systemd-sysv-install (with various combinations of SysV+systemd
units), and the "boot-smoke" test where the whole thing has to boot
successfully 20 times [2].

This new test can be seen at e. g.
https://github.com/systemd/systemd/pull/2641 or /2650 which now also
have a second "ubuntu-ci" test.

I now wrote all the glue between github and autopkgtest.ubuntu.com,
running into gory details like firewall issues, three different ways
to authenticate, and my own hilarious incompetence when it comes to
web programming (but I'm learning :) ). Thanks to Daniel for his great
help with kickstarting me into GitHub webhooks!

However, an awful lot of the runs currently fail with a linker error.
I filed [2] and will investigate.

So please don't put too much attention to these results yet. I want to
to enable them to see how the testing and communication holds up in
practice, but before this we definitively need to sort out [2] first.

Please bear with me!

Thanks,

Martin


[1] http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/tree/debian/tests
[2] https://github.com/systemd/systemd/issues/2651
-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)


signature.asc
Description: Digital signature
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel