On 04/01/2016 11:00 AM, Cleber Rosa wrote:
MOTIVATION
==========
The idea of adding hooks to be run by Avocado before and after tests
is general enough, and may be used by the community in unpredictable
ways. And that is good.
For this team, the initial motivation was to be able to bring back an
Autotest feature that some of our users are missing: the ability to
set the system-wide "kernel core pattern" configuration for tests.
Having a pre-test hook would allow "/proc/sys/kernel/core_pattern" to
be read, saved and modified to point to the test results directory.
Having a post-test hook would allow "/proc/sys/kernel/core_pattern" to
be reverted back to its original state.
Other currently core features such as sysinfo collection, could be
re-implemented as pre/post test hooks.
GENERAL DESIGN POINTS
=====================
These are the most important design decisions to be acknowledged or
questioned. Please reply with either ACK or your questions/suggestions.
1) Hooks are implemented as plugin classes, based on a given defined
interface, in the same way current "CLICmd" and "CLI" interfaces allow
plugin writers to extend Avocado and give it new commands and command
line options.
Given the discussion is way ahead and we already have a PR in place,
maybe this is too late, but I'd like to give my 2 cents here. I like the
plugins approach, but I don't think that's the best way to implement the
hooks mechanism. From the experience with oVirt, I'm used to think in
hooks as a directory where I put some executable(s) that will be
executed in a given moment. Each directory corresponds to a moment. The
order that the executables are executed is simple given by their names.
The hooking api itself can and should be a plugin, but user should not
have to bother with the plugin stuff.
Example: directories '~/avocado/before_tests/' and
'~/avocado/after_tests/' are the first candidates to cover this RFC. In
the future, if we have a request for it, we can have '/before_test/',
'/after_test/', '/after_failed_test/' and so on.
For those executables inside the directories, if using python and
importing the avocado hooking module, then they can access avocado job
and/or tests information, probably the same information you're already
considering to expose in your current proposal.
2) The hooks are executed by the *runner*, and not by the test
process. The goal is not interfere with the test itself. The pre and
post code that runs before and after the test should not *directly*
change the test behavior and outcome. Of course, the test environment
can be changed in a way (say having packages removed) that a test may
fail because of hook actions.
I don't see a problem in change the test directly. It can be dangerous,
of course, but user is expected to know what he's doing when using
hooks. In oVirt hooks we are able, for example, to edit everything we
want in the VM xml before using the xml to create the VM in libvirt.
Anyway, since we don't have a request or use case for this currently,
ACK for the 'not interfere with the test itself'.
3) Test execution time should not be changed by pre and post hooks. If
a pre-test hook takes "n" seconds to run, "n" should not be added to
the test run time.
Ack.
4) Job run time: right now, Avocado times a Job based on the sum of
individual test run times. With pre and post test hooks, this can be
very different from job "wall clock" times. My instinct is to change
that, so that a Job run time is the job "wall clock" time. I'm unsure
if we should add yet another time measure, that is, the sum of
individual test run time. This is also bound to be broken when
parallel run of tests is implemented.
Ack.
5) The pre test hook is given the test "early status". Information
such as the test tagged name, the fact that it has not yet started to
run and the test results directory are all part of the early status.
Ack.
6) Because of point #5, the test is instantiated on the test process,
its early state is sent, but the test execution itself is held until
the runner finishes running the pre-test hooks.
Ack.
7) The post test hook is given the last test status, which is also
used by the runner to identify test success, failure, etc.
With the advent of 'summary' being populated on the runner, 'summary'
might be more useful to the post test hook.
Thanks,
- Cleber.
_______________________________________________
Avocado-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/avocado-devel
