Hi Mel,
I note that if you're trying to measure the activity of the TOS community,
you've identified some reasonable measures. And yes, these appear to be
easily achievable measures (in other words, low-hanging fruit where the data
is relatively easy to obtain with low effort).
However, if you're actually trying to measure how TOS and/or POSSE have
impacted faculty (which I suspect is of more interest to Max and Red Hat),
note that the last three measures are FOSS measures, not academic measures.
(And the first measure could be problematic if faculty don't identify
themselves as TOS folks at an event.)
I'm highlighting a difference in perspective. That academic measures of
success are very different than FOSS measures and since faculty strive to
attain the academic measures (because that is what they're rewarded for),
then they're likely to put those before FOSS measures. And lack of evidence
of TOS/POSSE impact using a FOSS measure does not mean that there isn't
impact, but instead the need for a different measure. And those measures are
likely to be more difficult to identify and to gain data for. More subtle I
think. I wouldn't want the subtlety of the measure to give the impression
that TOS/POSSE isn't having an impact because I think it is having an
impact. A large and very positive one.
This leads me to also ask the question "what defines success"? I'm waxing
philosophical here, but I continue to be impressed with the immensity of the
task of culture change and that it may take years before we fully realize
the impact of faculty involving students in FOSS projects. I've been
involved in open source for five years now and I still consider that I have
lots to learn about how things work.
And this discussion is intended to encourage you as I think you're doing
great things! Just trying to shed some light on what is going on.
Heidi
-----Original Message-----
From: tos-boun...@teachingopensource.org
[mailto:tos-boun...@teachingopensource.org] On Behalf Of Mel Chua
Sent: Wednesday, April 06, 2011 4:35 PM
To: TOS
Subject: Re: [TOS] Help put Research Minion Mel to work - what do you want
to measure?
> Well... "valuable" w.r.t. measurement usually implies that you know
> what the question is. Collecting data without knowing the question is
> a problem. The spirit of your questions seem to be "what is the
> activity/pulse of this community?" If that's the question, then that
> might make it easier to think of what to measure.
I think that's accurate - "is the TOS community alive and helping
professors teach open source classes?" Max and I pow-wowed for a bit
after lunch today and I decided I was going to limit myself to measuring
5 things, every week:
1. TOS presence at events - that's
http://mchua.fedorapeople.org/tos-scorecard/ambassadors.png, except
expanding it to be all TOS community members rather than just students.
2. TOS publications - that's
http://mchua.fedorapeople.org/tos-scorecard/publications.png, basically
any paper any TOS community member writes about TOS.
3. TOS Planet activity - # of total posts, # of individuals posting, #
of individual classes represented by these posts.
4. TOS list activity - # of total messages, # of individuals posting, #
of subscribers (which I don't think I have permission to view, actually
- this is something the infra team would have to decide to give me
access to, else I'll drop that metric).
5. POSSE module attendance - how many folks show up each week to the
online classes we're teaching? (Just going for sheer numbers right now;
once we get a gauge on what types of folks show up - profs, students,
devs, etc - we might also start tracking proportions.)
Lots of open questions about how precisely to get accurate numberss on
these; need criteria, etc and there are some things to address like
"boy, it's hard to count Planet posts when Planet is down!" but it gives
me a solid target to aim for.
I'll be running these numbers for the first time for this week's stats,
and we'll see how hard they are to generate.
> If you genuinely thought that the study of FOSS communities was going
> to play a role in your dissertation work, then a dashboard-like
> project could be a very powerful way to get research data. However,
> you can likely pursue a dissertation with three case studies just as
> well as a firehose of incomprehensible commit logs... it all depends
> on your question. But, in the meantime, if you designed an amazing,
> best-of-breed, FOSS participation dashboard... yeah, I'd use it. :D
I'm hoping that I'll work on this stuff and a few years from now look up
and go "...wait, 70% of my dissertation data has already been collected
by shell scripts over the past 3 years, I just need to do a bunch of
analysis and writing!" I'd love to have a ton of quantitative data to
get some general trends across classes and institutions and help us zone
in on more interesting places to look for case studies... but that's
wishful thinking from someone who's not yet learned to do proper
research and does not have a properly-formed research question, and I
realize all this is potentially a ton of work and I need to figure out a
smallish bite-sized bit that will actually let me graduate before I retire.
I'll start very small, with metrics that I personally care about and can
manually capture in a very short amount of time every week, then see how
that goes before I even think about the automation. Not getting all
starry-eyed about a groundbreaking new research platform right now or
anything... just trying to consistently measure *something* and see if
it gets any insights, and then going from there.
--Mel
_______________________________________________
tos mailing list
tos@teachingopensource.org
http://teachingopensource.org/mailman/listinfo/tos
_______________________________________________
tos mailing list
tos@teachingopensource.org
http://teachingopensource.org/mailman/listinfo/tos
