On 09/07/12 18:58, Dan Price wrote:
Hi folks,
I've been working on a small project to:
- Extend the nasty depot to be more nasty and more configurable.
- Enable the test suite to launch nasty depots.
- Extend the test suite with the ability to run nasty-depot based stress
tests.
- File and fix bugs in the core code identified by this testing.
I spoke to Brock at some length about whether or not the new test cases
in cli/t_pkg_nasty.py should be in the 'cli' suite or perhaps in a new
suite such as 'stress' which is not run by default. We're going to
start with the tests living in 'cli' but they could move pretty easily if
that doesn't work out well in practice. Let's not assume this is set in
stone.
The new tests run 10 iterations of each nasty test by default; there
are two test classes. This stuff will run in parallel as usual with
other tests. But if you are running serially, the new tests add ~180
seconds to the suite runtime. We can tune that impact down by dropping
the default number of iterations if it's a big concern. I do want to
make sure we at least run through this once on each test run in order to
make sure it doesn't rot.
The number of test iterations is configurable by environment variable,
which should enable trivial integration with hudson. For example,
you can do:
NASTY_ITERS=500 python src/tests/run.py -j 8 -o t_pkg_nasty
And you'll get 500 iterations of the testing.
With the bugfixes applied, I have been able to run this stuff cleanly up
to ~10000 iterations (I did this by adding more parallelism by hand) and
it has been stable.
There is much opportunity to improve this work-- we need to add testing
(and associated bugfixes) for other areas not covered by this work.
7192182 pkgrecv traceback trying to setup request headers on a refetch
7192183 nasty depot testing should be part of core testing
7192447 transport.py has bad exception handling in do_search
7193408 variable name incorrect in error path when getting a p5i
7193416 client can traceback if manifest response is empty
7195048 contents -g can traceback if a temporary publisher has problems
7195051 catalog.attrs code needs improved armor against bogus responses
7195083 p5i.parse should discriminate "" versus None
https://cr.opensolaris.org/action/browse/pkg/dp/nasty-testing/
-dp
catalog.py:
I'm not thrilled by this change only because it seems like this is a
list that's likely to fall out of date as other changes are made to the
module. Now, maybe we just count on our testing w/ the nasty depot to
catch that, but it would be nice if this was "someplace" else. I don't
have great ideas, but how would this one sound:
We define the list of "created", "last-modified", ... as a class
property of CatalogAttrs. Then, we add an api test that walks through
each of the items in the list and asserts that there's a one-to-mapping
with __get_<foo> functions (and similarly __set_<foo> functions).
Alternative, we could assert that a property with that name has been
defined for the class. I'm open to suggestions but I would like to
remove a bit of the magical nature this list currently has.
General question about the solution to 7193416. Given that an empty
manifest can never be a valid published manifest, what happens now when
the client gets an empty manifest? Do we fail with an appropriate error
(presumably about the manifest not being valid/matching the hash)?
server/depot.py:
I'm a little confused about the need to have two moving parts for
setting the amount of nastiness. I can call need_nasty_* which changes
the max roll, but I can also set the NASTY_LEVEL in the initializer of
the class. But if (for example) I set my nasty level to 10000, then
there'd be no difference between need_nasty_occasionally and
need_nasty_infrequently. Maybe those should use a multiple of
NASTY_LEVEL to set maxroll? But if we did that... then setting
NASTY_LEVEL would really be sort of irrelevant I think? So maybe that's
really my question, why do we need NASTY_LEVEL as a parameter?
Also, a total bikeshedding point: I don't find that occasionally,
infrequently, and rarely have some sort of consistent and implicit
ordering among them. Maybe these could be litle, normal, a lot? Or
minimal, average, most/above-average? Or level_1, level_2, level_3? Note
a huge deal, but it'd be nice not to have to look at pydoc/code to
figure out which one I should use to increase (or decrease) the amount
of nastiness.
line 1762: Why not let it go negative? (Or alternatively, treat the
number like it's an unsigned int and wrap it around to being a huge
positive number)
line 1890: Why'd we get rid of this test?
2093: Is this something that we should revisit once 7195051 is fixed?
Would it make sense to either file a separate bug to track that or
include that info as part of 7195051.
t_catalog:
Just curious. Is there a reason to prefer print >>f, <foo> as opposed to
f.write(<foo>)?
Other than that, lgtm.
Brock
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
