On Thu, Jul 24, 2008 at 02:03:13PM -0700, Dan Price wrote: > So will you be checking in a baseline? [Hopefully? :)]
Yes :) > I was wondering if perhaps the baseline should be in some sort of > text-tool-processable format? (I was thinking one line per test). I > was thinking that people might want to be able to review changes which > have been made to it by using 'diff'. I know that may be more of a pain > than a 'pkl'. OK, sounds good. I was planning to do that but just figured I'd get the stuff up for review before going further. > For human consumption, I'm wondering if it would be > helpful just to show the complete path to the test in > order: > > api.t_catalog.TestCatalog.testcatalogfmris2 pass > > I also have a preference for eliminating "..." sequences, as > I personally find them more distracting than whitespace. Maybe > it's just me. > > Another option would be to do more of a tabular form: > > api.t_action.TestActions: > test_action_errors pass > ... So this is possible but would require some surgery. We don't currently have that level of granularity, but we could if we made each "class" into its own suite, and then ran that. If people feel strongly about it'll certainly go that route, otherwise changing to the suggestion before that (or using Danek's suggestion) is pretty easy. Here's what the output looks like now: api.t_version.py TestVersion.testversionsuccessor9 pass Is it reasonable to left justify the test name to 60 columns? That gives a good whitespace separation for most tests. > In pkg5unittest.startTest(), make sure to flush the stream... that way > if the test hangs, we know what test we're running. There > may be other places where a flush is also useful, but I'll leave > it to you to find them. Done. > > ... a failing test that normally passes and you just broke ... > > > > testrolllogfiles (api.t_catalog.TestUpdateLog) ... FAIL > > So as we talked about yesterday-- there are four cases. Can > we get example output for each? > > Result Baseline Output > Pass Pass ?? > Pass Fail ?? > Fail Pass ?? > Fail Fail ?? > > > FAILED (failures=1) So I just made that change; does this look ok? (baseline result being in parens) api.t_version.py TestVersion.testversionsuccessor9 pass api.t_version.py TestVersion.testversionsuccessor9 pass (fail) api.t_version.py TestVersion.testversionsuccessor9 FAIL (pass) api.t_version.py TestVersion.testversionsuccessor9 FAIL > Would it be possible to update this "(failures=1)" output to reflect > the four cases listed above? Do you have a suggestion on what that would look like? I'm thinking: (pass=2, fail=2, baseline_mismatch=2) Thanks for reviewing, I'll post an updated webrev after the de-pkl'ing. Thanks, Brad _______________________________________________ pkg-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
