On Thu, Feb 17, 2011 at 6:37 AM, Thomas Green <green...@ntlworld.com> wrote:
> How someone doing something on understanding these programs? I always found
> that experiments on understanding programs were much easier than ones on
> writing programs. Far less inter-subject variability.
At the expense, of course, of systematic variability between
conditions. How were you thinking of controlling for that?

> Now that you have
> suitable materials, it might be possible for someone to have a go at that?
Anyone who wants them is welcome to my materials, protocols and
intermediate results. I might even be open to a more active
collaboration. However, I have since run off and started doing "hard"
implementation work on STM systems, so I am unlikely to be pushing
this work any further on my own.

> As for the high variability and the apparent bi-modality: Keith Stenning and
> others have argued that successful problem solving depends heavily on the
> external representation adopted for the problem. He and Richard Cox did a
> big study on students doing scholastic aptitude problems, which are well
> normed I understand, and showed very convincingly that only students who
> chose an appropriate representation solved a given problem. I can't remember
> whether they were able to get several problems per student and show that
> same students were perfectly successful on other occasions when they did
> choose an appropriate representation.
I'm not sure how you'd faithfully observe the external representation
of a program - heaven knows it would help my teaching if I had some
systematic way of extracting it from the students and dissecting it!
Is there prior work on this?

> So I would suggest, as a possible explanation, that the two humps may not be
> because some of your participants were in any way inferior, but that they
> unluckily set off on the wrong track and never recovered.
> Did you debrief them about strategies, or collect their working notes, or
> anything like that? If so, you could check that out.
I collected their working notes - of which they kept virtually none -
and 1-minute-interval snapshots of their source tree so as to see how
their programs developed. (Compiler and program invocations were also
recorded, along with their output. There's also a video of the
participant and a screen recording of each session.) Crude measures
like graphing "lines added/removed", "LOC touched/minute", etc over
the experimental period didn't reveal anything. After a while I felt I
was grubbing around in the tiny details looking for something
signal-shaped because all the more obvious metrics were showing
garbage, and decided to give up and write up the null finding. As I
say, I'm happy to share the data if someone wants to dive deeper.


> On 16 Feb 2011, at 18:46, Meredydd Luff wrote:
>> Hi Russel,
>> I did a pilot study along those lines a couple of years ago,
>> attempting to compare performance with the Actor model, Transactional
>> Memory and traditional mutex locking. (Not quite the precise
>> distinctions you're looking for, but at least in a similar ballpark.)
>> 17 subjects attempted a simple unstructured grid problem in an
>> afternoon - measuring time taken, NCLOC, and subjective responses on a
>> questionnaire afterwards.
>> The upshot was those metrics turned out to be pretty much completely
>> uncorrelated (r^2 < 0.07 in all cases), and the only results to reach
>> anything close to statistical significance on any metric were (1) that
>> people completed tasks faster on the second trial, and (2) that people
>> claimed to prefer the novel paradigms to mutex locking in the
>> after-task questionnaires. The former is crushingly obvious, and the
>> latter is subject to a number of obvious biases.
>> (I also noticed an odd bifurication in finishing times - subjects
>> either finished the task in half the time allotted or didn't manage it
>> at all, but this is irrelevant to your question :) )
>> The paper is entitled "Empirically Investigating Parallel Programming
>> Paradigms: A Null Result", and is available online at:
>> http://ecs.victoria.ac.nz/twiki/pub/Events/PLATEAU/2009Programme/plateau09-luff.pdf
>> To my knowledge, at the time of writing, there was no work published
>> more recent or applicable to your query than the supercomputing
>> studies I cited in the Related Work section. (There has been a more
>> recent study of TM vs SMTL in an undergraduate-assignment setting - I
>> don't have the reference but could grub it up for you if you wanted.)
>> Meredydd
>> On Wed, Feb 16, 2011 at 4:10 PM, Russel Winder <rus...@russel.org.uk>
>> wrote:
>>> Prompted by various discussions elsewhere, I am on the search for recent
>>> experimental results and/or people doing or about to do experiments.
>>> The questions all relate to the models of parallel software:
>>> shared-memory multithreading, Actor Model, Dataflow Model, Communicating
>>> Sequential Processes (CSP), data parallelism.
>>> Question 1 is:  is synchronous message passing easier for programmers to
>>> work with than asynchronous message passing.
>>> Question 2 is:  are the case classes of Actor Model easier for
>>> programmers to work with than the select statement of Dataflow Model and
>>> CSP.
>>> There are more but those are the two "biggies".
>>> There is a lot of people who know precious little about psychology using
>>> advocacy research out there building up various "known facts" about what
>>> is and is not better from a cognitive perspective.  Some of them base
>>> this on observed anecdotal evidence which gives it some legitimacy, some
>>> of them are peddling their own beliefs.
>>> So real experimental evidence from people who know what they are doing
>>> would be most welcome.
>>> Thanks.
>>> --
>>> Russel.
>>> =============================================================================
>>> Dr Russel Winder      t: +44 20 7585 2200   voip:
>>> sip:russel.win...@ekiga.net
>>> 41 Buckmaster Road    m: +44 7770 465 077   xmpp: rus...@russel.org.uk
>>> London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
>>> --
>>> The Open University is incorporated by Royal Charter (RC 000391), an
>>> exempt charity in England & Wales and a charity registered in Scotland (SC
>>> 038302).
> 73 Huntington Rd, York YO31 8RL
> 01904-673675
> http://homepage.ntlworld.com/greenery/

Reply via email to