Greetings, Jim, [I am cc'ing the hackystat-dev-l mailing list so they can chime in with comments.]
I have been using Hackystat on a project for about 3 months and am impressed with the concept. I have found that the information is helpful in guiding the project and for spotting problems early. Currently I am not watching a full set of data, but enough to gain insight on the project.
That's great news. Frankly, it's still quite challenging for people outside the Hackystat developer community to get the system installed and generating useful data, so I'm happy to hear you've had some success. I am going to be spending a substantial amount of time over the next 6 to 9 months working on issues related to improved technology transfer with Hackystat. One of the results should be released in the next few days: a tool called HackyInstaller that vastly simplifies client-side sensor installation and maintenance. I am also working on substantial improvements to the documentation over the next year. So, please be patient; help is on the way. :-)
* Our shop uses IBM/Rational ClearCase as our SCM repository. Hackystat has CVS support the JPL implementation has Harvest. Do you know of anybody who has made the sensors for ClearCase? (We are also planning to implement ClearQuest rather than Jira -- know of a sensor?)
I'm not aware of one, but I'll keep my eyes open for it.
* Jupiter has difficulty reporting and summarizing data collected from multiple days. I provided code review input to a single review over two days. When running the review summary report the numbers reflect only the last days issues, not the sum of the two days. Is this a known bug?
Hmm, that's strange. I've posted it to Jira and we'll take a look: <http://hackydev.ics.hawaii.edu:8080/browse/HACK-327>
* Jupiter (somewhat related) doesn't seem to use the Team layer to access the review files -- should it? We have had comments/issues thrown away when the review files were read-only. This also causes some problems with keeping the bookmarks in sync with the code.
"Team layer" sounds like a ClearCase concept. We've never used Jupiter with anything other than CVS, so I don't exactly know what this would mean.
There definitely is a problem with keeping bookmarks in sync with the code. Currently, either people have to freeze the code under review during the course of review, or else tag a snapshot of the codebase and refer to that.
Both aren't the greatest solution, but the general problem would require much tighter integration with the underlying CM system than we can do at this point in time.
* I would like to track issues from the automated code inspection tools (PMD, FindBugs, CheckStyle, etc). I don't see a sensor or SDT related to this type of data. Am I missing something here?
Other than the sensors for these tools? :-) You're right, there are no sensors yet for PMD, FindBugs, Checkstyle. As for the sensor data type, one could either try to use Issue, or ReviewIssue, or else come up with a new one entirely.
This brings up a representational question that we've been grappling with for a while and as yet have no satisfactory answer for. We currently distinguish between two types of "defects"---defects reported through a bug tracker are collected with the "Issue" sensor data type, while defects reported through a review tool like Jupiter are collected with the "ReviewIssue" sensor data type. Defects reported through Checkstyle, FindBugs, etc. might constitute a third category of defect. At some point, one wonders whether these distinctions are artificial---is there really any reason to represent these various flavors of defect with distinct sensor data types? Maybe _everything_ is just an "Issue"? On the other hand, there is a process-level difference in the way defects reported with Jira vs. those reported with Jupiter vs. those reported with Checkstyle are managed, and that can have an impact on the underlying representation and analysis of those issues. For example, Jira issues have a disposition (open, resolved, closed), ReviewIssues have a phase (Individual, Group), Checkstyle issues have none of the above. Attempting to glom all of those things into a single Super Issue might create a very muddy representation with lots of conditionalized fields.
So, it's an open research question from my perspective, and our approach so far has been to simply forge ahead as best we can, creating SDTs and analyses for the opportunity and need arises, and figuring that at some point, we will gather enough experience and data to understand what the appropriate "unified defect theory" would be. Your insights and suggestions are certainly welcomed on this or any other issue related to Hackystat.
* We didn't like the coverage analysis of JBlanket -- it only looks at method level coverage. We are using Emma since it looks at lines and logical blocks in addition to method calls. I have a hackystat sensor that takes Emma output and sends it to Hackystat using block level coverage detail.
That's great! I hadn't heard of Emma, and it seems like a neat open source alternative to JBlanket. If you would like at some point to contribute your sensor code to the project so that everyone can benefit, let me know; we'd love to have it.
* I would like to have the unit test time information such that I can graph time much like I watch test failures. This would give an early, but not reliable, indication of negative performance impact if it were joined with unit test count. Does this sound like a worthy metric to chase?
Absolutely. We have an explicit "Perf" sensor data type and a sensor for our in-house load testing framework, but I agree that simple unit test time data could serve as an early warning signal for performance degradation.
One way to proceed is to create a reduction function where you can specify one or more unit tests whose elapsed time you want to monitor (the canaries). The reduction function allows you to generate a set of telemetry streams, one per unit test, which would allow you to see if things are changing (positively or negatively).
Cedric: want to take a quick look at this and see if you could write this reduction function in a day or two? Seems like a generally useful facility.
Thank you in advance for your thoughts on these items, and thanks for making these cool
projects available.
Jim Christenson
Cheers, Philip
