Really great thoughts, Aaron! Here's some feedback...
--On Saturday, September 11, 2004 2:02 AM -1000 Aaron Kagawa <[EMAIL PROTECTED]> wrote:
Hey Guys,
While working on my thesis on quality, I began to think about Cedric's Telemetry Thesis and came up with these ideas/comments.
Like I said before, Hackystat has probably proven that it can do Measure Validation (metrics collection).
Well, validation is a bit different from collection. To validate, say, Active Time, we need to measure developer activity using some other approach (such as video taping or watching them over their shoulder) and then use this data to confirm that Active Time does measure developer activity in the way we would predict.
For our theses to be interesting, we have to focus on the Measure Application (except for Takuya's thesis, because he should be concerned with Measure Validation for obvious reasons). So, putting that in the context of Telemetry, the general research question is, What can Software Telemetry be usefully applied to?
There seems to be two broad areas that most people talk about when discussing software measurement; quality and process. In my view they are both distinct entities and at the same time they should and could be related. For example, a project could be very efficient (good process) process but have really poor quality. Likewise, a project could have very good quality but have a really inefficient process. Of course, these two are hopefully related to each other (and they probably are in real life but I haven't read anything that concretely says that if you improve a process then that absolutely means that quality is improved or vice versa). So, the million dollar question is, can we relate these two using Hackystat and what is the optimum balance between the two to have a "good" product? If we can figure that out, we are going to be filthy rich. But that is VERY hard. And I don't think that we should strive for that because, we have NOT proven that we even can measure quality or the process with Hackystat (this is the unproven Measure Application thing again).
I find it easier to think in terms of three concepts:
* Quality. In general, conformance to user requirements. Hard to operationalize as a single measure, but it's clear that 'high' quality would be good and 'low' quality would be bad.
* Productivity. In general, the rate at which development proceeds. Again hard to operationalize as a single measure, but it's again clear that 'high' productivity would be good and 'low' productivity would be bad.
* Process. The tools and techniques you use that result in a certain level of quality and productivity. Some people view software development processes as making trade-offs between quality and productivity, others (Quality Is Free) don't. At any rate, unlike Quality and Productivity, it's not clear that 'high' process would be good and 'low' process would be bad. That's why I don't think one should think in terms of Quality and Process, but rather in terms of Quality and Productivity as modulated via Process.
Ok, getting back to Software Telemetry. I would claim that Software Telemetry is best suited for measuring the process and NOT quality. On the other hand, I'm hoping that my "thing" is best suited for measuring quality and NOT the process. For example, I think it would be hard to claim that we have a high quality system based on the telemetry streams. Likewise, it would be hard to claim that we have a good process based on an absence of "in need of review" declarations.
Not exactly. The basic idea of software telemetry is to detect _change_. So, while software telemetry might not be able to unequivocably state that one's quality is High, it might be able to state with a fair amount of confidence that quality appears to be Decreasing. That, to me, is part of what makes it cool; it finesses the issue of assigning an absolute value to the state of a system, and adds value instead by enabling one to talk about whether things are getting higher or lower.
That's one reason why I want to see the world in terms of productivity, quality, and process. I think Telemetry can evolve to the point where it can provide indicators about whether quality is going up or down and whether productivity is going up or down. I don't think it will ever make sense or be interesting to try to talk about whether process is going up or down.
If you agree with me so far, here are some things that Cedric could think about (which are similar to what I'm trying to do with quality): 1) define what is a good process, then operationalizing that in terms of Hackystat 2) define what is a bad process, then operationalizing that in terms of Hackystat
I think the value judgement (good vs. bad) is probably a matter of personal decision, but Telemetry could provide some interesting help for a manager trying to assess their process. Assume that you define process in terms of what it does w.r.t productivity and quality. Then telemetry could reveal that the current process:
- keeps quality and productivity constant - is improving quality and decreasing productivity - is improving productivity and decreasing quality - is decreasing both productivity and quality - is increasing both productivity and quality
Interestingly, I would claim that none except for the first process has the potential to be a stable state. All of the others will either resolve to the first (potentially) stable state or else to one of the other four (definitely) unstable states.
- consider scenes that show our active time distributions, ie. coding time, test coding time, review time, defect fixing time. if we spend all of our time coding and none on the others then in my view that indicates a bad process
I claim that this depends upon the time scale that you're looking at. Maybe it's fine simply hack for a week, for example.
3) figure out a "process fix". if things are going bad how does a software manager fix the process. (in the case of quality, review is a good fix).
Yes, that's interesting.
4) can the telemetry streams identify when we applied a "process fix"?
Good point. Do we need to be able to manually 'annotate' the telemetry stream?
- by the way, i tried to think of what it means to have a good process and i can't do a very good job! i bet if you ask 10 other programmers they won't have the slightest idea either, yet everyone talks about it.
Exactly. That's why I think it's more useful to think in terms of optimizing quality and productivity rather than process, and to think in terms of changes in productivity and quality rather than what constitutes an absolute level of high productivity and quality.
i think the problem is that it is a subjective term (just like quality!).
It's very context-sensitive. What might be a great process for Hackystat is certainly not be a great process for MDS.
- My plan is to be able to recommend Reviews (aka Quality Fixes) once a week, Cedric could recommend Process Fixes in the same manner.
That would be excellent.
- If Cedric can measure "Process" and I can measure "Quality" then maybe one day another CSDLer can put the two together. actually, because we are doing this at the same time we should be able to both increase our process and quality. For example, one plausible conclusion that telemetry can give us is that we need to conduct more reviews and by doing them we are potentially increasing our quality (which I would be measuring).
OK, I would claim that what you are doing and what Cedric is doing is really not so different after all. Cedric is attempting via telemetry not to come up with absolute values for productivity and quality (and/or other things), but with the ability to detect change in those concepts.
You are not attempting to come up with absolute values for Quality for the files in a software system, but rather a way to order them such that the Quality of the top 30% (say) is very likely to be significantly higher than the Quality of the bottom 30%. At the end of the day, that's really all a manager needs to know to make some useful decisions--if they decide that they want to focus resources on quality improvement, then you've helped them by identifying a substantial fraction of the system that would benefit from having resources committed to them in this way (as well as another substantial fraction of the system that would not.)
In both cases, it's a relative thing. Cedric's is relative over time, yours is relative over structure.
Hopefully, I explained my thoughts well enough for others to understand (especially because i wrote all of this as I was off to bed, i apologize if i was just rambling). Anyway, what do people think?
Great thoughts, Aaron. You should do more thinking right before bed!
Cheers, Philip
