Re: Announcement/RFC: jhbuild continuous integration testing -- mystery of recent flood of failures solved
On 18/02/13 22:34, Martin Pitt wrote: Please note that there is no system D-BUS and no default session D-BUS running. If you need those, then the tests should start dbus-launch or use GTestDBus. dbus-launch is not particularly suitable for regression tests: if you use it, you have to kill the resulting dbus-daemon yourself when you're finished with it. If not using GTestDBus, please use with-session-bus.sh from telepathy-glib, or review https://bugs.freedesktop.org/show_bug.cgi?id=39196 so I can add dbus-run-session(1) to dbus. smcv ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Announcement/RFC: jhbuild continuous integration testing -- mystery of recent flood of failures solved
On Tue, Feb 19, 2013 at 10:13 PM, Simon McVittie
simon.mcvit...@collabora.co.uk wrote:
On 18/02/13 22:34, Martin Pitt wrote:
Please note that there is no system D-BUS and no default session D-BUS
running. If you need those, then the tests should start dbus-launch or
use GTestDBus.
dbus-launch is not particularly suitable for regression tests: if you
use it, you have to kill the resulting dbus-daemon yourself when you're
finished with it. If not using GTestDBus, please use with-session-bus.sh
from telepathy-glib, or review
https://bugs.freedesktop.org/show_bug.cgi?id=39196 so I can add
dbus-run-session(1) to dbus.
This is a very interesting topic (and brings to mind Colin's ideas about
installed tests).
Here's some ideas worth consideration IMHO...
o Unit Tests with GTestDBus
GTestDBus is IMO ideal for regression testing (or in-tree unit tests),
I made a short write-up on this not long ago in my blog[0].
The idea here is that you want to be absolutely sure that you
are testing isolated modules and services that are still in-tree,
you want to test ideally your services alone without clouding
your results with installed services. If your service relies on
system installed services, you would ideally want to control
which specific installed services get to run in your controlled
D-Bus environment sandbox.
I.e. if you have services in /usr/share/dbus-1, you dont want those
mixed in to your sandboxed build path, colliding with services in
/opt/devel/share/dbus-1/.
You also probably want to control cases where fallbacks can be
implemented, if your service/client can run without a complimentary
service... you want to test a case where your client has access to
an installed service vs. a case where a fallback is used instead.
o Now that we are talking about a build server and building 'all-of-gnome'
it becomes interesting to know if a service installed by some dependency
effects any modules which depend on that service in a negative way.
For this case (why I thought about Colin's ideas on installed tests), it
suddenly becomes interesting to have tests which do not use GTestDBus
in a controlled environment, but instead to test the system as a whole,
running only services from ${build_prefix}/share/dbus-1 but certainly
avoiding anything from /usr/share/dbus-1/ (or at least properly prioritizing
the build prefix services over the system installed ones, if we can't avoid
using those).
o If we have these two theories of testing D-Bus services and clients which
depend on them, we probably want to reuse the unit testing code as much
as possible.
Perhaps some additional features added to GTestDBus could allow us
to run the same tests in both contexts (i.e. the installed test context
with a full gnome environment vs. the isolated in-tree context).
Cheers,
-Tristan
[0]:
http://blogs.gnome.org/tvb/2012/12/20/isolated-unit-testing-of-d-bus-services/
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Announcement/RFC: jhbuild continuous integration testing -- mystery of recent flood of failures solved
On 19/02/13 13:54, Tristan Van Berkom wrote:
On Tue, Feb 19, 2013 at 10:13 PM, Simon McVittie
simon.mcvit...@collabora.co.uk wrote:
GTestDBus is IMO ideal for regression testing (or in-tree unit tests),
I made a short write-up on this not long ago in my blog[0].
GTestDBus combines two things: setting the D-Bus service activation path
to a private one, and running a new D-Bus session.
It does some of what self-contained regression tests should do, but by
no means all of it - I think regression tests should also consider
setting XDG_RUNTIME_DIR, XDG_DATA_*, XDG_CONFIG_*, DISPLAY (if used),
and HOME.
with-session-bus.sh (and dbus-run-session, if reviewed) specifically
only does the new D-Bus session part, although you can supply a D-Bus
configuration file via --config if you want to change the service
activation path too, and the rest can be done via env(1).
For this case (why I thought about Colin's ideas on installed tests), it
suddenly becomes interesting to have tests which do not use GTestDBus
in a controlled environment, but instead to test the system as a whole,
running only services from ${build_prefix}/share/dbus-1 but certainly
avoiding anything from /usr/share/dbus-1/ (or at least properly
prioritizing
the build prefix services over the system installed ones, if we can't
avoid
using those).
This should still use a new D-Bus session (and probably XDG_RUNTIME_DIR
and HOME) for the tests, but could set XDG_DATA_*, XDG_CONFIG_* to look
in ${build_prefix} before /usr.
Perhaps some additional features added to GTestDBus could allow us
to run the same tests in both contexts (i.e. the installed test context
with a full gnome environment vs. the isolated in-tree context).
In my experience, regression tests for D-Bus components usually need
controllable mock versions of various D-Bus services: only a small
subset of these tests (those that test normal behaviour, and aren't
too picky about implementation details) will work with the real version
of those services.
S
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Announcement/RFC: jhbuild continuous integration testing -- mystery of recent flood of failures solved
Hello Travis,
Travis Reitter [2013-02-17 23:04 -0800]:
(/home/ubuntu/gnome/checkout/folks/tests/eds/.libs/lt-persona-store-tests:100517):
CRITICAL **: backend.vala:140: Error calling StartServiceByName for
org.gnome.evolution.dataserver.Sources1:
GDBus.Error:org.freedesktop.DBus.Error.Spawn.ChildExited: Process
/home/ubuntu/gnome/packages/libexec/evolution-source-registry exited with
status 1
But it also looks like it could be related to the structure of the
buildbot environment. Is there any way I could debug this further?
This might or might not provide a first clue:
$ xvfb-run jhbuild run
/home/ubuntu/gnome/packages/libexec/evolution-source-registry
Migrating mail accounts from GConf...
Migrating addressbook sources from GConf...
Migrating calendar sources from GConf...
Migrating task list sources from GConf...
Migrating memo list sources from GConf...
Registering EGoogleBackendFactory ('google')
Registering EOwncloudBackendFactory ('owncloud')
Registering EYahooBackendFactory ('yahoo')
Registering ECollectionBackendFactory ('none')
/home/ubuntu/.config/evolution/sources: Unable to find default local directory
monitor type
... and then aborts.
Please note that there is no system D-BUS and no default session D-BUS
running. If you need those, then the tests should start dbus-launch or
use GTestDBus.
I'm happy to log into the build box and run debugging commands for you
or Matthew. Probably best to catch me on IRC? I'm pitti on both
GNOME (#gnome-hackers) and Freenode (#ubuntu-qa or #ubuntu-devel).
Thanks!
Martin
--
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
signature.asc
Description: Digital signature
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Announcement/RFC: jhbuild continuous integration testing -- mystery of recent flood of failures solved
On Fri, 2013-02-15 at 11:54 +0100, Martin Pitt wrote:
Martin Pitt [2013-02-12 7:43 +0100]:
https://jenkins.qa.ubuntu.com/view/Raring/view/JHBuild%20Gnome/
Right now there are 151 successes (blue), 5 modules fail to build
(red), and 4 modules build but fail in make check (yellow). It's
been like that for a week or two now, so I'd say we are doing
reasonably well for now.
You may have seen the sudden large increase of test failures on Feb
13, we are back at ~ 40 make check failures. It turns out that this
isn't due to a regression in our test environment after all, as I
initially suspected, but because of a bug fix. Until Feb 13, make
check wasn't run on a build which previously succeeded already, due
to an accidentally dropped --force jhbuild option. It has been put
back.
So the current failures have been there all along. Sorry for the
confusion!
Speaking of test failures - is it possible to get more details? All I
can seem to turn up for the folks module test failures are:
make check failed for folks
Check log file for details
But I can't seem to find the corresponding log from here:
https://jenkins.qa.ubuntu.com/view/Raring/view/JHBuild%
20Gnome/job/jhbuild-amd64-folks/lastSuccessfulBuild/testReport/junit/folks/test/make_check/
Also, most modules say 1 of 1 tests {failed, succeeded}. I guess this
is a side effect of make check being somewhat monolithic. Is there any
way modules can increase the granularity reported for tools like
Jenkins?
Thanks,
-Travis
smime.p7s
Description: S/MIME cryptographic signature
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Announcement/RFC: jhbuild continuous integration testing -- mystery of recent flood of failures solved
Hello Travis,
Travis Reitter [2013-02-17 22:24 -0800]:
Speaking of test failures - is it possible to get more details? All I
can seem to turn up for the folks module test failures are:
make check failed for folks
Check log file for details
But I can't seem to find the corresponding log from here:
https://jenkins.qa.ubuntu.com/view/Raring/view/JHBuild%
20Gnome/job/jhbuild-amd64-folks/lastSuccessfulBuild/testReport/junit/folks/test/make_check/
On the main page, click on folks, which leads you to
https://jenkins.qa.ubuntu.com/view/Raring/view/JHBuild%20Gnome/job/jhbuild-amd64-folks/
There, click on the topmost build in the left column, or use one of
the permalinks like last build:
https://jenkins.qa.ubuntu.com/view/Raring/view/JHBuild%20Gnome/job/jhbuild-amd64-folks/lastBuild/
There you see the complete log file.
Also, most modules say 1 of 1 tests {failed, succeeded}. I guess this
is a side effect of make check being somewhat monolithic. Is there any
way modules can increase the granularity reported for tools like
Jenkins?
I think that ought to be possible by integrating gtester-report and
add a new target such as make check-report or make gtester-report
(I think glib already does that), which we then could call from
jhbuild and (1) export as an artifact, and (2) present as test results
instead of the monolithic pass/fail one.
Martin
--
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
signature.asc
Description: Digital signature
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Announcement/RFC: jhbuild continuous integration testing -- mystery of recent flood of failures solved
On Mon, 2013-02-18 at 07:39 +0100, Martin Pitt wrote: Hello Travis, Travis Reitter [2013-02-17 22:24 -0800]: Speaking of test failures - is it possible to get more details? All I can seem to turn up for the folks module test failures are: make check failed for folks Check log file for details But I can't seem to find the corresponding log from here: https://jenkins.qa.ubuntu.com/view/Raring/view/JHBuild% 20Gnome/job/jhbuild-amd64-folks/lastSuccessfulBuild/testReport/junit/folks/test/make_check/ On the main page, click on folks, which leads you to https://jenkins.qa.ubuntu.com/view/Raring/view/JHBuild%20Gnome/job/jhbuild-amd64-folks/ There, click on the topmost build in the left column, or use one of the permalinks like last build: https://jenkins.qa.ubuntu.com/view/Raring/view/JHBuild%20Gnome/job/jhbuild-amd64-folks/lastBuild/ There you see the complete log file. This specific problem may be due to e-d-s (its process is segfaulting) for some reason I've never seen before: make[3]: Entering directory `/home/ubuntu/gnome/checkout/folks/tests/eds' /PersonaStoreTests/persona store tests: ** (/home/ubuntu/gnome/checkout/folks/tests/eds/.libs/lt-persona-store-tests:100517): CRITICAL **: backend.vala:140: Error calling StartServiceByName for org.gnome.evolution.dataserver.Sources1: GDBus.Error:org.freedesktop.DBus.Error.Spawn.ChildExited: Process /home/ubuntu/gnome/packages/libexec/evolution-source-registry exited with status 1 Trace/breakpoint trap (core dumped) FAIL: persona-store-tests But it also looks like it could be related to the structure of the buildbot environment. Is there any way I could debug this further? Matthew, any ideas? Thanks, -Travis smime.p7s Description: S/MIME cryptographic signature ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
