So here are my thoughts and questions:
Overall question: Does this framework work with the SUNWldtp approach
that's used to test the packagemanager or will only CLI tests work with
this framework?
General nit: We've been using 8 space indentation, I think these files
should probably follow the same formatting.
General comment:
Having each test override the superclass tearDown just to pass doesn't
seem useful, especially that's exactly what the superclass is doing (I
think).
src/tests/vbox/README:
72-78: I think these steps should be: "Install the <name> package" if
this isn't going to be integrated into our gate
170-193: I think more explanation is needed here. Why would I copy a vdi
into the system? What's it's mean to "register it for use"? Could I copy
it in without registering it for use? How can I delete a VDI if I've
already deleted the virtual machine? Can a VDI be associated with more
than one Virtual Machine?
t_vtf_console:
92: Why is 30 the magical number of characters to look for "/etc/hosts" in?
t_vtf_pkg:
I'm not sure I get the point of these tests. Are they just to ensure
that the basic infrastructure is working? Since there are separate proto
and non-proto tests, should these make sure that the proto (or
non-proto) version is actually being run? I'm also not sure why/how the
proto would be used in this framework? Is the idea to sometimes use the
old version of pkg and sometimes use the current workspace version?
Speaking of which, I probably just missed it when looking at the client
setup, but how does the proto area for the current workspace get plunked
down on each client? Can I control which clients get it? (So I can test
a b123 server talking with a gate client, or the other way around for
example.)
t_vtf_teardown:
Shouldn't there be some kind of check to see that the guests (and host)
are in fact torndown?
t_pkg_image_update:
Would it be worth looking at parallelizing the multiple client image
update? I'm just thinking that each of those steps could well be fairly
slow, especially the reboot. Maybe we're only anticipating having 1 or 2
guests, but if we had 10 or 20 to image-update, this seems like an
unnecessary bottleneck, or does doing one of these operations tax most
systems to the max anyway?
Also, I'm concerned that setUp is only happening once per test class. I
know we've designed our tests to allow persistent depots, but this seems
to be making persistent clients the default as well. For example, if I
wrote a test called test_zzz, would the clients it dealt with be the
pre-image updated ones or the post-image-updated ones?
75: More of a nit, but won't the exit code for installing SUNWipkg vary
depending on whether or not it's been backpublished to the version the
client's currently running? Doesn't this need to account for that?
vtfutils:
I'm surprised that tearDown simply passes. There's not cleanup that
needs to be done after a test? Shouldn't it at least be removing the
test directory it setup?
351-355: Shouldn't we always raise the traceback exception if one's
encountered, no matter what check is set to? I can see situations where
we might not want to check error codes, possibly... though that seems
dubious as well, but I can't think of times where a traceback should be
ignored.
Thanks,
Brock
Jim Walker wrote:
Have a look. All comments welcome.
http://cr.opensolaris.org/~jwalker/vboxvtf/
And, if possible try it out. I would like
a couple people to try it before it goes
back. I can provide help and vdi images
as needed.
I'm planning a bit more cleaning and
will probably deliver pexpect as a
separate package. It's a very nice pure
python implementation of expect, which
made the framework much easier to
create. Plus it gives us a good low
level interface when we need it.
The main vboxvtf.py module is designed as a
plugin API so it can be used in other test
suites like install.
Besides additional test cases, vtfutils.py
can grow as needed to support more ips commands
and utility routines.
Cheers,
Jim
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
