Greetings, Andrew,
--On Thursday, December 22, 2005 12:11 PM -0500 Andrew Funk
<[EMAIL PROTECTED]> wrote:
Hi Philip,
I agree this sounds like a good plan. Here are some initial comments:
[1] This sounds like what I've been using QMTest for with the benchmark
testing. For classroom testing at least, CLI sensor integration with
Hackystat may be the way to go. I know Viral is planning to use QMTest
with a class, so it's something to keep in mind. It may end up that we
use a QMTest database in some cases to facilitate running lots of tests,
but in all cases we rely on the CLI sensor to capture the program
output, in the interests of standardization.
I think there's a conceptual difference between QMTest and the proposed CLI
sensor, and least from the Hackystat point of view. Hackystat is designed
to collect multiple kinds of both process and product data, in order to
enable analyses that operate across these multiple data types. To do this,
we distinguish between 'sensors' (tools that monitor the product or process
and collect information) and 'sensor data types' (schemas that represent a
specific kind of process or product data). It is possible for a given
sensor to collect several different kinds of sensor data, and, of course,
it is possible for the same kind of sensor data to be collected by several
different sensors. The Hackystat User Guide
<http://hackystat.ics.hawaii.edu/hackystat/docbook/ch01.html> has more
details about this.
So, from our point of view, one could build a sensor for QMTest that
collects UnitTest sensor data, similar to the sensors that we have already
developed for other test frameworks (such as JUnit and CPPUnit).
The CLI sensor tool suite, on the other hand, is a more flexible kind of
sensor, in that it could emit sensor data of many different types. For
example, instead of building a specific sensor for QMTest, we could
alternatively imagine building into the CLI sensor tool suite the ability
to recognize when QMTest has been invoked, and categorize that command and
its results as UnitTest sensor data. But there are many other kinds of
things people do at the command line other than unit testing, particularly
in the HPC domain, and that's what will be interesting to explore with this
sensor.
I am very excited about the development of infrastructure to better monitor
what HPC developers are actually doing, because I think there are some very
interesting analyses that will be available to us once we have a good
picture of both process and product and how they interact over the course
of development.
Cheers,
Philip
