Greetings, all,
Aaron sent Takuya and I some interesting thoughts based upon the latest Hackystat review that I would like to hear other people's thoughts on. I also provide my feedback below:
--On Wednesday, September 22, 2004 2:29 AM -1000 Aaron Kagawa <[EMAIL PROTECTED]> wrote:
Hi All,
A couple comments about Review using Jupiter:
- First off, I don't believe that the Jupiter Sensor is working properly. I don't seem to be sending any data and I don't see anything in the logs. I believe I have the latest version of the sensor and have all the settings configured correctly. Has anyone successfully sent Review data to the server?
I struggled with this myself today and can't get any data sent to the server, even after installing last night's build on the public server and installing the corresponding sensors in Eclipse. The problem appears to be on the client-side---there are no error messages on the public server console, for example. Takuya, can you check this out?
- Review Issues are product metrics much like FileMetrics. So, I'm wondering if it would be easier to create an ant based sensor for Review Issues, instead of trying to capture the Issues using Jupiter. An ant based sensor makes more sense if it is easier.
This actually seems harder to me than the current situation, in that we now have to run Ant to get the issue data sent to the server. I don't see any real cost to the current approach.
Actually, I'm also thinking that we could generate a report based on the Review Issues. I think this would be useful to be able to see all the outstanding issues that have not been fixed. In addition, this report could help the education of developers, as they can refer to issues that are associated with code similar to what they are writing.
I agree that this would be a very interesting analysis report to provide. For example, you could list the number of non-resolved issues by priority. There could also be a Reduction function so that you can get Telemetry regarding the total number of review issues that are open and how that changes over time.
- ReviewId in the ReviewIssue SDT - It seems odd that the Review Id is so random. Wouldn't it be better to use <ReviewId>-<ReviewerId>-<created timestamp> or something like that? If the Review Id is truly random at some point you will have a duplicate Id depending how good your random generator is. But, I guess in the end it really doesn't matter.
A more nicely structured reviewID would seem to be better.
- reviewId in the ReviewAcitivty SDT The information provided in the SDT help page says "reviewId - The unique reviewID for the review entry, eg takuyay". I believe you meant "reviewId - The unique reviewID for the review entry, eg SelectionInterval1"
Yes.
- Consider doing a usability study within your thesis. Some readers would probably doubt your findings if the Jupiter Interface hasn't been proven to conduct reviews efficiently.
I agree.
Goals of collecting review data (this is just what I'm thinking)
- The long term goal of the Review Metrics are to be able to evaluate our review process and the effectiveness of our review. I think that we should be able to bring up Hackystat before a review meeting and discuss the effectiveness of the review based on the number and severity of review issues generated. There are many statistical information that can be derived. For example, is one hour enough preparation time? Or is it the most effective?
This is a GREAT idea. I had never thought of using Hackystat as a way of checking to see, for example, whether or not it was even appropriate to do the review meeting. For example, you might decide to not do the review meeting if there are no critical or major issues uncovered during preparation. You might decide to delay the group meeting if preparation time was not sufficient.
- It would also be interesting to test the age old belief that reviews decrease the number of defects. We can do this easily if we can associate a defect to a particular class and checking if that class has been reviewed.
I don't think this is easy at all to do. There are all sorts of conflicting independent variables, including the complexity of the code and the skill of the author. You'll need a very large sample size to factor this stuff out.
- One of the major goals of conducting reviews is to spread knowledge. I would claim that is rather important of our review process. How can we evaluate that? And, how do we build an infrastructure or process that helps promote spreading knowledge?
Excellent thoughts! Anyone have any ideas?
- Another major goal is to catch defects early in the development process. However, I would claim that for our situation we don't really adopt that trend. Projects that follow that trend tend to be larger systems where testing is expensive. I believe that most of the defects are caught in our Unit Test cases (of course that is if we have good test cases). Our review process seems more like confirmation and I would claim that we would have less critical defects (defects that cause the program to function incorrectly) than typical software projects. I would also claim that when we review code in CSDL, we need to pay more attention to the Unit Tests; are we testing the program correctly, effectively, and thoroughly?
More good ideas here. For example, if we are paying attention to Unit Tests, then it would be reasonable to expect that coverage would go up after the rework following a review. Coverage certainly shouldn't go _down_ after the rework following a review! We might also expect that test case failures related to the module under review should go down after the rework. All of these are hypotheses that could be empirically tested.
Jupiter is working great.. Now the focus shifts on how to use the data that we get from it to understand process, productivity and quality of reviews and how it affects the overall process, productivity, and quality of the software product.
I couldn't have put it better.
thanks, aaron
Thank you!
Cheers, Philip
