I think that to find out quickly which change introduces the issue, it
would be nice if there was an easy way to have Jenkins run the “long
suite” for a specific commit. That way we could identify the
problematic commit in the standardized environment and without relying
on developer setup.
On 13 Jan 2016, at 16:20, Mike Carey wrote:
Agreed. I'd vote for making sure that the broken daily test report be
accompanied by a list of the changes that occurred since the last
successful run - and that it be the responsibility of all folks with
changes on that list to gang up and ensure that the problem is
identified and resolved in a timely fashion. We should probably spell
out what that looks like, but it seems eminently reasonable....
On 1/13/16 4:03 PM, Chris Hillery wrote:
+1 for Ian's initial proposal. This kind of pre-commit validation is
always
a balancing act between being comprehensive and being timely, and I
think
sticking with the distinction between "mvn test" and "mvn verify" is
a
clean and simple way to divide the tests.
Given that pushing the Gerrit head to ASF is manual anyway, it
wouldn't be
TOO much headache to also enact Ildar's suggestion - basically,
whoever
does the ASF push would need to wait to ensure each commit passes the
corresponding daily tests. However, IMHO at least this is probably
not
necessary. If a bug is uncovered in the daily tests, we'll need to
push an
additional commit to fix it, and that additional commit (plus the
original
buggy commit) will still end up on ASF anyway. And I worry a bit
about
anything that would cause the ASF Github mirror to stay out of sync
with
Gerrit for very long.
Ceej
aka Chris Hillery
On Wed, Jan 13, 2016 at 3:42 PM, Ildar Absalyamov <
[email protected]> wrote:
It would be really nice to break Jenkins test build into two stage
process: ‘mvn test’ on each patch set submission and ‘mvn
verify’ after the
patch received +2 and got merged into Gerrit, BUT before it got
merged to
ASF repo. Although I am not sure what should be done in a case when
the
latter build does not pass, since it’s already merged into Gerrit
and we
cannot revert that.
On Jan 13, 2016, at 15:34, Ian Maxon <[email protected]> wrote:
Hi all,
I'd like to field the idea of changing the criteria we use to allow
Jenkins
to give +1 verified on patch sets. Right now, it runs 'mvn verify',
which
runs darn near about every test we have. While this was somewhat OK
in
the
past, I think now, and going forward, it takes a long time, and it
will
only take longer as we make the integration tests more thorough.
Additionally, I'd like to expand and re-enable the automated
vagrant-based
cluster integration tests, and unfortunately there isn't a simple
way to
make these work inside a docker container. I can get it to work by
hand,
but to make it automated I would have to fork or submit a patch to
the
Jenkins Docker cloud plugin, and it might not be a small or
particularly
clean/easy patch either.
I'd like to change the goal that's run in the automatic tests on
every
commit to 'mvn test' rather than 'mvn verify'. This will still run
all of
the ExecutionTest(s), and the optimizer tests, and so on. It will
stop
the
recovery tests and asterix installer tests from being run every
time,
however.
However these tests are still certainly valuable, so I'd run them
on a
timer, perhaps daily, so that we would still catch bugs that these
tests
reveal rather promptly. Due to the fact we only commit a single
digit
number of times to master each day generally, the granularity of
which
commit will have introduced a bug will hopefully be rather low.
Thoughts/comments?
Thanks,
-Ian
Best regards,
Ildar
