In the 5 year history of GSoC we have only ever had two students appeal
against a mid-term or full-term evaluation, one of which was this year.
Google take these appeals very seriously and the new Google lead on GSoC
has the energy to follow up in some detail. So, here are the lessons
learned from the examination of this appeal. I welcome any other
suggestions that people might have for improving our process.
Summary
=======
Mentors and the project community must take ownership of the appeal and
the decision, Apache admins cannot (and should not) police all mentor
decisions. I'm pleased to report that this is exactly what happened in
this case.
Apache admins should take responsibility for ensuring the process we
follow is fair and balanced, that any appeal is taken seriously and that
processes are modified if appropriate.
In this case I do not believe our process, the mentors or the project
community are at fault. I do however suggest a minor tweak to help avoid
these cases in the future.
Recommendation
-------------
In this case the lesson learned for our process is that we need to
clearly communicate our expectations with respect to the students time
commitments. We need to not only ask and evaluate "how much time will
you put into GSoC" but also make it explicit in our communications with
the student that putting in less time than this is likely to result in
failure.
Action
------
Our mentee "roles and expectations" section of the documentation already
included "The mentee is expected to dedicate a pre-agreed amount of time
each week to the project", I have added "failure to
commit the agreed time is likely to result in failure on the programme".
These roles and responsibilities should be shared with all applicants to
the mentoring programme via our GSoC documentation. They should also be
sent to all successful students.
Detail
======
The student claimed that they had been interacting well with the
community. This was confirmed by the mentor. However, the mentor and
project community felt that progress made was not sufficient and there
was a strong suspicion that the student was working as well as engaging
with GSoC in their spare time.
Google engineers examined the code submitted by the student and although
they did not object to the mentors decision they did wonder whether the
project was of an achievable scope.
The mentor and project community responded with plenty of detail,
including information about previous work that that paved the way for
the student, that satisfied myself as an admin.
My position in this case
------------------------
Three years ago I asked Leslie Hawthorn about the significantly lower
level of passes here in the ASF when compared to other organisations (if
memory serves me right it is around 73% against 85%). I also asked our
membership for their opinion.
The ASF membership felt that we might have higher standards here in the
ASF. Having met a great many mentors in other organisations I believe
that (on average) this is probably true.
I won't repeat what Leslie said since it was a private conversation,
however her points did not make me significantly alter my report back to
the ASF.
The question then, and now, is whether or not our standards are too high.
I've been an admin since the very first year of GSoC (boy that was
chaos). In all that time, I've been hands on with something like 250
students. We've only ever had two appeals. One of which was a very clear
case (very little effort). The other, this one, is less clear.
I would like to remind us all that the projects PMC feel that this
student was trying to hold a day job down whilst also doing GSoC. We
must remember that part of our evaluation is how many hours the student
expects to spend on GSoC. In general if they are not close to full time
they will not be selected.
With that in mind I see a small number of possible reasons for this failure:
a) The project was not reasonably scoped. In this case I do not believe
this to be a problem and thus nothing needs changing in our process
b) ASF expectations are unreasonably high. I don't believe this to be
the case (see discussion above). As an ex-computer science lecturer I'd
say our expectations are about right, or even too low, since we are
after the A grade students.
c) The student does not have the skills required to complete a
reasonably scoped project - in which case our evaluation process is
flawed. I don't think this is the case, see discussion above and
remember our process has been refined and improved three times since
those conversations.
d) the student did not have sufficient support from their mentor - this
was not part of the students complaint (quite the opposite in fact) and
thus not relevant in this case.
e) The student misled the evaluation team with respect to time available
during the application process - in which case we need to not only ask
"how much time will you put into GSoC" but also make it explicit that
putting in less than this is likely to result in failure. The
justification for this is that ASF expectations are high and therefore
nothing but a full commitment to the project will result in success.
My conclusion is therefore that our standards should remain high.
Lowering them reduces the value to our mentors and to future students.
Instead we should (even more) clearly communicate our high expectations
to applicants at the earliest possible stage.
Ross
