On Fri, Apr 01, 2016 at 11:00:39AM -0300, Cleber Rosa wrote:
MOTIVATION
==========
The idea of adding hooks to be run by Avocado before and after tests
is general enough, and may be used by the community in unpredictable
ways. And that is good.
For this team, the initial motivation was to be able to bring back an
Autotest feature that some of our users are missing: the ability to
set the system-wide "kernel core pattern" configuration for tests.
Having a pre-test hook would allow "/proc/sys/kernel/core_pattern" to
be read, saved and modified to point to the test results directory.
Having a post-test hook would allow "/proc/sys/kernel/core_pattern" to
be reverted back to its original state.
Other currently core features such as sysinfo collection, could be
re-implemented as pre/post test hooks.
GENERAL DESIGN POINTS
=====================
These are the most important design decisions to be acknowledged or
questioned. Please reply with either ACK or your
questions/suggestions.
I have some questions (hope you don't mind).
What are the outputs of pre- and post-test hooks?
Are there limits to the actions that are permitted in pre- and
post-test hooks? Of course, the primary use-case of the pre-test hook
is to set up the environment for the test--so environment changes are
permitted--and the use-case for a matching post-hook is to restore the
environment. About the only operation I can imagine NOT being
permitted is to abort (kill itself, or kill its controlling parent
process).
Can a pre-test hook return a status that causes the test execution to
be skipped? I can imagine this being done for another use-case:
validate the test environment (e.g., check to see if required hardware
is present).
Can a post-test hook alter the result (status) of the test?
Has there been any thought given to having pre- and post-job hooks?
For example, setting the kernel core pattern is something I would want
to do globally, for all tests in a job. It would be faster and more
convenient to do this just once. But I admit this is a +1 optimization
and need not be considered now. (It also complicates things when tests
run on multiple machines.)
Can there be multiple hooks for a given test? If so, how does one
define the order in which they are executed? Since there are pre- and
post-test hooks, there are really two orders to consider.
I found myself wanting to make an assumption so I better ask: must
hooks come in pairs (for every pre-test there must be a post-test, and
vice-versa)?
That's all for now.
-Jeff
1) Hooks are implemented as plugin classes, based on a given defined
interface, in the same way current "CLICmd" and "CLI" interfaces allow
plugin writers to extend Avocado and give it new commands and command
line options.
2) The hooks are executed by the *runner*, and not by the test
process. The goal is not interfere with the test itself. The pre and
post code that runs before and after the test should not *directly*
change the test behavior and outcome. Of course, the test environment
can be changed in a way (say having packages removed) that a test may
fail because of hook actions.
3) Test execution time should not be changed by pre and post hooks. If
a pre-test hook takes "n" seconds to run, "n" should not be added to
the test run time.
4) Job run time: right now, Avocado times a Job based on the sum of
individual test run times. With pre and post test hooks, this can be
very different from job "wall clock" times. My instinct is to change
that, so that a Job run time is the job "wall clock" time. I'm unsure
if we should add yet another time measure, that is, the sum of
individual test run time. This is also bound to be broken when
parallel run of tests is implemented.
5) The pre test hook is given the test "early status". Information
such as the test tagged name, the fact that it has not yet started to
run and the test results directory are all part of the early status.
6) Because of point #5, the test is instantiated on the test process,
its early state is sent, but the test execution itself is held until
the runner finishes running the pre-test hooks.
7) The post test hook is given the last test status, which is also
used by the runner to identify test success, failure, etc.
Thanks,
- Cleber.
--
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
_______________________________________________
Avocado-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/avocado-devel
_______________________________________________
Avocado-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/avocado-devel
