On 2 Sep 2015, at 2:48, Paul Jakma wrote:
On Sat, 22 Aug 2015, Martin Winter wrote:
On 20 Aug 2015, at 6:13, Paul Jakma wrote:
So what you're asking is to push a commit at a time, and wait till
the CI has run before pushing the next?
Yes, when possible. Or at least just bundle related commits together.
My goal is still to have some automatic feedback on test results back
to list (or maintainer) for each run, but at this time the result
still needs too much manual investigation to make sure it’s not a
false alarm if things break.
This could be tricky. We have to couple the pushing of commits to the
running of the CI tool on those commits. Hmm. So the commiter would
need to:
1. Push the next minimum set of commits that are needed to build
(ideally that'd be 1 commit)
2. Check the CI tool and wait till it has run
3. Go to 1
?
Yes, painful… Trying to figure out something easy.
My struggle at this time is that I’m not that clear what you are
pushing next. Each patch gets discussed
on it’s own on the list and then you pick a few of them for your next
push. I see you giving some heads up
but, not very clear on which ones.
So how about (if you plan to push multiples at the same time), you would
create a branch, apply it all there,
then announce the branch and if no objections show up within a week or
two, then you merge it into the mainline?
If you manage to give the branch a distinct name (i.e. proposed-XXXXXX
or something), then I could even trigger
automatically based on it. Could be even another git altogether if you
prefer.
This is the CI at https://ci1.netdef.org/browse/QUAGGA-QMASTER right?
Yes and no… this is the ci at ci1.netdef.org, but there are many more
(not yet) public plans.
Eg, I have the full compliance run and some basic patchwork plan
implemented as well. My plan
for the patchwork plan was to potentially pick up all new patchworks
automatically, test the build
on all the OS and do some selected compliance runs. Currently struggling
to finish this as most of
OSPF and BGP is broken (btw: your OSPF patch looks promising, but still
running the tests). So they
currently all fail miserably.
Better would be to remember the last head the CI ran on, then after
more commits are pulled in, go through each commit and do the CI on
each? E.g. if abc123 was the last commit ID, then get the list of
more recent commits, checkout and run the CI in that with something
like:
for COMMIT in $(git rev-list abc123..); do
git checkout $COMMIT && runci() || exit ...
done
Unfortunately not without rewriting the whole CI system (as fas as I
know). Atlassian Bamboo (which I use)
and Jenkins work in a way that they get triggered by git server
telling them that new commits are available
or by polling a git server for changes.
And they don't support doing something akin to 'git bisect' to find
the commit? Surely others must have run into this?
Hmm, seems so:
https://issues.jenkins-ci.org/browse/JENKINS-12972
Jenkins is more feature-rich, but after long attempts I gave up on it
for not getting it reliable to work.
(Each bug I found got fixed, but then each fixed version had new bugs.)
https://jira.atlassian.com/browse/SRCTREE-1125
This is for SourceTree, which is just a graphical git client. I’m
using Atlassian Bamboo which is their
Jenkins like CI system
I could probably trying to get something like this implemented with
enough time, but then I rather concentrate
my work on catching the bugs even earlier (i.e. with patchwork).
Also keep in mind that many crashes are not that clear cut and not
always failing or passing. For the larger runs
(every few months) I run each test at least 4 times to make sure I have
consistent results. And each test suite
(= each protocol) usually takes approx 24hrs of runtime for a full pass.
Splitting it into smaller batches
is possible, but then I miss issues because of bad state from previous
tests (i.e. like the case when the
ospf router-id gets changed)
Also, your suggestion would break builds in the rare case where
multiple commits need to be tested together to build a working system
(not sure if and how often this might happen).
Yeah, ideally every commit should be 'golden' and produce a buildable
and workign tree. The ideal and reality have a tendency to disagree
every now and then though, despite best efforts. ;)
Understood. And it would be good to find these non-golden commits as
soon as they happen…
Anyway, just trying to find the best solution while keeping it as simple
as possible for maintainers
pushing the commits and people (like me testing it).
- Martin Winter
[email protected]
_______________________________________________
Quagga-dev mailing list
[email protected]
https://lists.quagga.net/mailman/listinfo/quagga-dev
