Brock Pytlik wrote:
Shawn Walker wrote:
Brock Pytlik wrote:
This webrev gives additional savings for the time to run the test
suite. It drops the total running time by about 140 seconds on my
machine.
Webrev:
http://cr.opensolaris.org/~bpytlik/ips-8534-v1/
Bug:
8534 pkgsend should optionally send packages in persistent depot tests
http://defect.opensolaris.org/bz/show_bug.cgi?id=8534
I'm concerned that changing everything to run an api install means
that we won't sufficiently test the cli.
In particular, it doesn't feel right to change t_pkg_install to use
the api rather than the pkg cli.
Maybe we need a t_pkg_api_install and a t_pkg_install?
Cheers,
I think the right approach is to have a small set of test cases which
exercise the command line options to make sure they're parsed correctly,
and compare the output of those tests to what's expected to check that
the command is displaying the right things. Other than that, I think
everything should be moved to using the api where possible due to the
speedup it provides (and the more exact error reporting and checking it
allows). Looking at the tests I changed in t_pkg_install, I don't see
any place really where we're testing the command line parsing nor where
we're checking the output of the commands.
The counterargument to that though is ensuring that the cli handles
failure cases gracefully. In addition, with some subcommands, such as
list, set-pubslisher, etc. the amount of work the api performs is very
small, and we really are testing the cli itself.
It is likely true that many of the cases you changed are fine just using
the api, but I remain concerned about failure case testing.
Cheers,
--
Shawn Walker
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
