Shawn Walker wrote:
Brock Pytlik wrote:
Shawn Walker wrote:
Brock Pytlik wrote:
This webrev gives additional savings for the time to run the test
suite. It drops the total running time by about 140 seconds on my
machine.
Webrev:
http://cr.opensolaris.org/~bpytlik/ips-8534-v1/
Bug:
8534 pkgsend should optionally send packages in persistent depot tests
http://defect.opensolaris.org/bz/show_bug.cgi?id=8534
I'm concerned that changing everything to run an api install means
that we won't sufficiently test the cli.
In particular, it doesn't feel right to change t_pkg_install to use
the api rather than the pkg cli.
Maybe we need a t_pkg_api_install and a t_pkg_install?
Cheers,
I think the right approach is to have a small set of test cases which
exercise the command line options to make sure they're parsed
correctly, and compare the output of those tests to what's expected
to check that the command is displaying the right things. Other than
that, I think everything should be moved to using the api where
possible due to the speedup it provides (and the more exact error
reporting and checking it allows). Looking at the tests I changed in
t_pkg_install, I don't see any place really where we're testing the
command line parsing nor where we're checking the output of the
commands.
The counterargument to that though is ensuring that the cli handles
failure cases gracefully. In addition, with some subcommands, such as
list, set-pubslisher, etc. the amount of work the api performs is very
small, and we really are testing the cli itself.
It is likely true that many of the cases you changed are fine just
using the api, but I remain concerned about failure case testing.
Cheers,
Well, what I did (or tried to do? or meant to do?) with the search
suites was to leave tests which checked failure modes, ie, which checked
what happened when an expected exception was propagated out to the
client code. (See tests for 8098, 7835, and others in t_pkg_search).
Unless I made a mistake in my change of the test suite, none of the
places where I shifted to api calls should throw an exception, so they
weren't testing an exception escaping to the surface anyway. So I'm
still not understanding the concern here regarding my conversion. If
you're saying we need more tests to exercise specific exception paths,
I'm certainly not arguing that, but that's not this bug, and isn't this
wad's job to fix.
Brock
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
