Hey Guys,
While working on my thesis on quality, I began to think about Cedric's Telemetry Thesis and came up with these ideas/comments.
Like I said before, Hackystat has probably proven that it can do Measure Validation (metrics collection). For our theses to be interesting, we have to focus on the Measure Application (except for Takuya's thesis, because he should be concerned with Measure Validation for obvious reasons). So, putting that in the context of Telemetry, the general research question is, What can Software Telemetry be usefully applied to?
There seems to be two broad areas that most people talk about when discussing software measurement; quality and process. In my view they are both distinct entities and at the same time they should and could be related. For example, a project could be very efficient (good process) process but have really poor quality. Likewise, a project could have very good quality but have a really inefficient process. Of course, these two are hopefully related to each other (and they probably are in real life but I haven't read anything that concretely says that if you improve a process then that absolutely means that quality is improved or vice versa). So, the million dollar question is, can we relate these two using Hackystat and what is the optimum balance between the two to have a "good" product? If we can figure that out, we are going to be filthy rich. But that is VERY hard. And I don't think that we should strive for that because, we have NOT proven that we even can measure quality or the process with Hackystat (this is the unproven Measure Application thing again).
Ok, getting back to Software Telemetry. I would claim that Software Telemetry is best suited for measuring the process and NOT quality. On the other hand, I'm hoping that my "thing" is best suited for measuring quality and NOT the process. For example, I think it would be hard to claim that we have a high quality system based on the telemetry streams. Likewise, it would be hard to claim that we have a good process based on an absence of "in need of review" declarations.
If you agree with me so far, here are some things that Cedric could think about (which are similar to what I'm trying to do with quality):
1) define what is a good process, then operationalizing that in terms of Hackystat
2) define what is a bad process, then operationalizing that in terms of Hackystat
- consider scenes that show our active time distributions, ie. coding time, test coding time, review time, defect fixing time. if we spend all of our time coding and none on the others then in my view that indicates a bad process
3) figure out a "process fix". if things are going bad how does a software manager fix the process. (in the case of quality, review is a good fix).
4) can the telemetry streams identify when we applied a "process fix"?
- by the way, i tried to think of what it means to have a good process and i can't do a very good job! i bet if you ask 10 other programmers they won't have the slightest idea either, yet everyone talks about it. i think the problem is that it is a subjective term (just like quality!).
- My plan is to be able to recommend Reviews (aka Quality Fixes) once a week, Cedric could recommend Process Fixes in the same manner.
- If Cedric can measure "Process" and I can measure "Quality" then maybe one day another CSDLer can put the two together. actually, because we are doing this at the same time we should be able to both increase our process and quality. For example, one plausible conclusion that telemetry can give us is that we need to conduct more reviews and by doing them we are potentially increasing our quality (which I would be measuring).
Hopefully, I explained my thoughts well enough for others to understand (especially because i wrote all of this as I was off to bed, i apologize if i was just rambling). Anyway, what do people think?
thanks, aaron
ps.. i just realized that, process by definition has to do with how things are changing, which makes telemetry a perfect fit. Also, when people think about quality it usually is the quality measurement at the current time which should mean that telemetry doesn't help very much (haha. just like stocks, most of the quantitative data is based on the companies current information, in addition it is a common mistake to buy stocks solely based on the price-over-time trend). therefore, i think this further supports my comments above.
