--On Tuesday, November 09, 2004 2:26 PM +0000 Ulla Becker <[EMAIL PROTECTED]> wrote:
Hello Philip,
There is one more question I would like to ask related to the PSP Analyses you perform with the data stored on the server.
As I read in your documents, you changed the original PSP metrics so that automated data gathering is possible. As far as I can see, the analyses you perform on the server are pretty much related to time aspects. So you can see, for instance, how much time developers spent editing files. But what conclusions are you finally drawing from all these analyses? The original PSP includes the "Project Plan Summary", which summarises the conclusions drawn from the gathered data, so that you can see where improvement is necessary. How can you determine improvement necessities in the end of a project or during a project?
Hi Ulla,
Great question, which is going to provoke a long answer. :-)
First, it's useful to know a bit of background. I received a review copy of "A Discipline for Software Engineering" when it was first published, and it completely changed the way I taught software engineering. I have the highest regard for that book as a ground-breaking advancement regarding metrics collection and analysis for that time period, and for Watts Humphrey for possessing the experience, intellectual rigor, and creativity to put all of the pieces together.
I then taught the course, practiced the PSP, and performed a variety of experiments on it for the next few years. On one hand, I had very good outcomes in my courses using the PSP: the student data showed all the "right" trends, and I even had one student achieve 100% yield on one project--the Holy Grail of the PSP. On the other hand, post-classroom adoption of the PSP was pathetic, even when we later augmented it with substantial tool support in the form of the Leap toolkit.
After analyzing our experiences in the classroom and post-classroom environments, I came to view two fundamental assumptions in the PSP as invalid under many/most contexts:
1. The PSP requirements for effort and defect collection are appropriate for real-world development and the resulting analytic benefits justify the overhead on developers for collection of this information, even when tool support is available.
2. A historical database of completed PSP projects can form a useful repository for future project planning.
The Hackystat Project, in one sense, can be viewed as an exploration of the implications of the assumption that (a) software engineering metrics collection cannot require intrusion on the day-to-day activities of developers, and (b) the collected metrics should be useful not as data points in a project repository, but rather as information that can be applied directly to the _current_ project.
So. I would like to gently correct your reference to "the PSP Analyses you perform with the data stored on the server". In fact, Hackystat does not support any "PSP analyses". We can't support these analyses because, for example, the PSP definition of the "effort" metric is fundamentally different and incompatible with the Hackystat "Active Time" metric, as well as because the PSP requirements for defect collection are impossible for us to accomplish with our automated mechanisms.
What we support instead is the notion of "Software Project Telemetry". The basic idea is that once you begin automatically gathering enough product and process data at regular intervals (such as active time, build, unit test invocation, size, complexity, coverage, commits, churn, review issues, defects), you can begin to track trends in these values over the course of the project you are currently working on, and use these values to become aware of anomalous events in development as well as more subtle trends over time. When you say,
"As far as I can see, the analyses you perform on the server are pretty much related to time aspects", that is true in the sense that we are interested in trends over time. However, it is not true in the sense that collecting only time-related metrics such as Active Time is typically sufficient to support interesting process/product observations. We are finding that you need to obtain a 'critical mass' of different process/product metrics before Hackystat analyses start to be interesting and useful.
To learn more about software telemetry in general, you might want to read the following paper:
Improving Software Development Management through Software Project Telemetry, Philip M. Johnson and Hongbing Kou and Michael Paulding and Qin Zhang and Aaron Kagawa and Takuya Yamashita, Submitted to the IEEE Software special issue on Software Project Management, August, 2004 <http://csdl.ics.hawaii.edu/techreports/04-11/04-11.pdf>
There is also a powerpoint presentation I recently created for my software engineering class, which has screen shots of some current telemetry-based analyses:
Telemetry-based analysis of software engineering product and process data. <http://csdl.ics.hawaii.edu/~johnson/413f04/12.telemetry.ppt>
For more information about how our research path diverged from the PSP/TSP:
Beyond the Personal Software Process: Metrics collection and analysis for the differently disciplined, Philip M. Johnson and Hongbing Kou and Joy M. Agustin and Christopher Chan and Carleton A. Moore and Jitender Miglani and Shenyan Zhen and William E. Doane, Proceedings of the 2003 International Conference on Software Engineering, Portland, Oregon, May, 2003. <http://csdl.ics.hawaii.edu/techreports/02-07/02-07.pdf>
You can't even ask them to push a button: Toward ubiquitous, developer-centric, empirical software engineering, Philip M. Johnson, The NSF Workshop for New Visions for Software Design and Productivity: Research and Applications, Nashville, TN, December, 2001 <http://csdl.ics.hawaii.edu/techreports/01-12/01-12.pdf>
I hope this helps, and I look forward to your next question. :-)
Cheers, Philip
