Greetings, hackers,
As you know, we submitted a paper to IEEE Software on Hackystat and Software Project Telemetry, and I am delighted to announce that the paper was accepted for publication. Even better, the Editor in Chief has requested that the revised version be longer than our original submission so we can include more details!
I think that the comments on the paper, both positive and negative, are quite instructive and so I'd like to provide a summary for everyone. The review form included a set of questions, so I will list each question, followed by selected comments from the five reviewers of the paper in response to the question. (I've omitted some of the more minor comments and those of a clerical nature.)
---------------------------------------------- * Please summarize what you view as the key point(s) of the manuscript.
Collection of data is unobtrusive, Data collection is automatic, Display of metrics is flexible and allows grouping of metrics.
The use of predictive models for metrics-based project management is a highly regarded practice but it's difficult and fraught with uncertainty. Telemetry provides an in-process approach that may complement or supplant a model-based system. Manager who currently avoid metrics because of the modeling and collection challenges may find that telemetry enables significant data collection and useful analysis with little effort.
Key point: software project telemetry allows in-process metrics and project corrective action. Importance of content: sensor data may allow richer, more accurate data collection and automated analyses to improve the software development process and product.
The key idea in this manuscript is to collect software metrics on projects in real time, as opposed to at certain infrequent "tollgates", such as initial release to software test organization, release to customer, etc.. This technique can therefore be easily applied to a lightweight process style, such as XP. The idea has merit, but the issue of what metrics to focus on both in collection and analysis, and how to derive useful data from the large amount of time stamped data, needs further research for this to become really significant (the authors recognize this in future directions sections). There may be a related data storage issue, but I think that is a straightforward technical problem fairly easy to resolve.
---------------------------------------------- * What do you see as this manuscript's contribution to the literature in this field?
The automatic collection of data and the real-time display and decision making capability based on the metrics generated.
It describes the most comprehensive implementation of realtime metrics collection and analysis that I have seen. It may be a real eye-opener for some development managers who would otherwise never be able to justify incorporating a meaningful metrics collection regime. I think it's a great contribution.
Describes a current project with an unusually thorough application of automated metrics collection.
The contribution is in demonstrating a continous real time method for collecting software metrics.
Currently metrics in software engineering play an important role. A contribution to this topic might be important for the readers, because it could raise interesting discussions. The key point of this contribution is the approach to gather data concerning the project status in time by means of Software Project Telemetry.
---------------------------------------------- * What do you see as the strongest aspect of this manuscript?
Properties of Software Project Telemetry.
It makes the concepts very accessible. Tremendous value is added by having the real, actively used implementation available online with which readers may experiment.
Describes a current implementation rather than merely a theoretical concept. May be of value in commercial software development.
Does a good job presenting the motivations for the research project, and giving a somewhat detailed overview of the design and implemention. Providing some examples of how this was used in practice is also useful, though more detail may be needed here.
----------------------------------------------- * What do you see as the weakest aspect of this manuscript?
The paper does not discuss the use of Software Project Telemetry by processes such as CMM, CMMI, and Agile methods. Such discussion would help the acceptance of Software Project Telemetry by organization using CMM, CMMI, and Agile methods.
It has no compelling examples of how telemetry has been used to facilitate real improvements in a project. The reader may be quite taken by the possiblities, but a bit of solid empirical validation would strengthen the case for telemetry.
This manuscript is too short. Generally in computer science / software engineering academic researchers seem closer to practitioners then most academic fields. There needs to be more detail presented in this paper in 2 areas to be useful to practioners: 1. A summary of the effort involved in implementing the telemetry system into their project, from embedding the sensors to data presentation and analysis. 2. Detail on what the most effective metrics have been, and how they have affected the project.
What I find weak about the manuscript is that quite a lot of important points are missing. There is, beside a passing mention concerning time measurement and its susceptibility to be abused, no discussion concerning the social aspects that might be important if one installs such a system which can constantly control the developers. It is not described, which data are important to meet project management decisions and the sensor data types that are collected are only mentioned - the way they are presented often stays dubious. What I missed painfully was a discussion concerning the interpretation of the data and how misinterpretations could be prevented.
------------------------------------------------ * Other comments (each reviewer separated by '=============='
Overall a good subject for the increasing demand for software project managers to be on top of projects. The use of Software Project Telemetry allows real-time decision making during tight project schedules.
===========
The manuscript is very relevant. It presents concepts that are particularly interesting at a time when many development managers from more traditional process backgrounds are testing the agile waters.
The manuscript is technically sound, with a practitioner's perspective that is atypically strong for academically rooted material.
The length is fine, although I would appreciate more! I would really like to see some thoughts about how telemetry can be integrated with or complement predictive models.
Overall it's quite good. I would accept it as written despite wanting more detail.
The room for improvement is mainly in addressing the relationship of telemetry to traditional metrics methods, and in providing some examples of real improvements realized through the application of telemetry. The two scenarios presented in Figure 2 and Lessons Learned are interesting but neither has a fully satisfying resolution.
A comment on essential property 3: While I do not disagree with the intent, as a longtime engineer and manager I would argue that broad availability is simply a matter of policy, not a defining characteristic of telemetry. Should the data and analysis tools be available to developers? I would personally argue strongly for this in a healthy environment. But I wouldn't say that another manager who wishes to control access more tightly isn't doing "software project telemetry".
The point about telemetry supporting decentralized project management is well taken, although it could be noted that the scientific method implied by hypothesis testing may be seriously compromised if too many theorists are operating concurrently.
As noted in Lessons Learned, telemetry can't measure "effort" in its broadest sense. I suspect that there are serious limits to measurements of effort in a narrow sense as well. A team employing a methodology in which individual ownership is encouraged (or a team with a good configuration management system) may have members who check out and fitfully edit files for hours or days. Variations in work style would seem to make meaningful measurements of "effort" applied to specific work products prone to error.
I enjoyed this article very much and look forward to playing with the software.
==============================
The visual interpretation of metrics displays, even on 9 simultaneous views, is not shown to provide any particular benefit. The paucity of results make the claims for the Hackystat monitors, and project telemetry in general, less convincing. The "Future Directions" section of the paper acknowledges these limitations.
The assertion that software project telemetry can overcome the difficulties of applying historical metrics to current projects is unconvincing. If not based on previous project experience, how would the development managers know which sensor data types to collect and which show significant correlations to desired results? Further, the number of developers involved, and the number of projects for which telemetry has been applied, should be stated. Have there been any advances in metrics collection between projects?
Thus, while software project telemetry may indeed represent an advance in software project metrics, it is not clearly shown that it represents "a new approach to software project management."
Unlike telemetry of automated systems requiring real-time intervention and process correction, software development performed by humans is performed on a different time scale (daily, in this example). The use of the term "telemetry" conveys an unsupported aura of timeliness.
In view of the complexity, time, and expert human labor required for setting up, monitoring, interpreting, and applying Hackystat results, it is quite a stretch to claim that it "contrasts with software metrics data that requires human intervention or developer effort to collect." Further, the paper should indicate the operational system overhead needed for data collection, or state that telemetry data collection never slowed software development processing.
=====
I like that Eclipse is used, many companies are moving to use this.
The authors are not clearly presenting the primary reasons metrics are not more widely used in software project management. There are 3 major reasons I can think of: 1. Cost of implementing metrics collections. 2. Relevance of the metrics, and deciding which are important. I think this is the major issue addressed in this paper, since they collect lots of data and can determine the relevent data steams during the project. 3. The relation between cost and relevance.
I think a section before lessons learned detailing some cases where specific telemetry data was used to improve the project would be good. I realize there are instances of this mentioned in the paper, but I think a detailed section on seeing a correlation in the telemetry data, taking action based on that correlation, and seeing the improved resu;ts would be interesting.
====
As metrics still play an important role in research but are often neglected in practice it seems to be important to raise a discussion going beyond the research community; the tool oriented approach that has been chosen by the authors of the manuscript might be a good idea to bring the approach nearer to industry.
Altogether I think it is an interesting topic and the authors could certainly say interesting experiences they made in this context, but as presented in the manuscript at hand I am not convinced that it will be of great interest for the readers. The authors should consider a major revision to improve the already mentioned aspects: Experiences during use, social aspects, support for interpretation and so on.
We arranged similar experiments to gather data during the development of requirements specifications but we could not find neither meaningful development patterns nor evidences concerning the project status that we could not capture classically by looking into the specifications. It would be interesting, if here we have another case - but unfortunately the manuscript remains vague about these aspects.
