On Thu, Feb 17, 2011 at 6:37 AM, Thomas Green <green...@ntlworld.com> wrote: > How someone doing something on understanding these programs? I always found > that experiments on understanding programs were much easier than ones on > writing programs. Far less inter-subject variability. At the expense, of course, of systematic variability between conditions. How were you thinking of controlling for that?
> Now that you have > suitable materials, it might be possible for someone to have a go at that? Anyone who wants them is welcome to my materials, protocols and intermediate results. I might even be open to a more active collaboration. However, I have since run off and started doing "hard" implementation work on STM systems, so I am unlikely to be pushing this work any further on my own. > As for the high variability and the apparent bi-modality: Keith Stenning and > others have argued that successful problem solving depends heavily on the > external representation adopted for the problem. He and Richard Cox did a > big study on students doing scholastic aptitude problems, which are well > normed I understand, and showed very convincingly that only students who > chose an appropriate representation solved a given problem. I can't remember > whether they were able to get several problems per student and show that > same students were perfectly successful on other occasions when they did > choose an appropriate representation. I'm not sure how you'd faithfully observe the external representation of a program - heaven knows it would help my teaching if I had some systematic way of extracting it from the students and dissecting it! Is there prior work on this? > So I would suggest, as a possible explanation, that the two humps may not be > because some of your participants were in any way inferior, but that they > unluckily set off on the wrong track and never recovered. > > Did you debrief them about strategies, or collect their working notes, or > anything like that? If so, you could check that out. I collected their working notes - of which they kept virtually none - and 1-minute-interval snapshots of their source tree so as to see how their programs developed. (Compiler and program invocations were also recorded, along with their output. There's also a video of the participant and a screen recording of each session.) Crude measures like graphing "lines added/removed", "LOC touched/minute", etc over the experimental period didn't reveal anything. After a while I felt I was grubbing around in the tiny details looking for something signal-shaped because all the more obvious metrics were showing garbage, and decided to give up and write up the null finding. As I say, I'm happy to share the data if someone wants to dive deeper. Meredydd > > On 16 Feb 2011, at 18:46, Meredydd Luff wrote: > >> Hi Russel, >> >> I did a pilot study along those lines a couple of years ago, >> attempting to compare performance with the Actor model, Transactional >> Memory and traditional mutex locking. (Not quite the precise >> distinctions you're looking for, but at least in a similar ballpark.) >> 17 subjects attempted a simple unstructured grid problem in an >> afternoon - measuring time taken, NCLOC, and subjective responses on a >> questionnaire afterwards. >> >> The upshot was those metrics turned out to be pretty much completely >> uncorrelated (r^2 < 0.07 in all cases), and the only results to reach >> anything close to statistical significance on any metric were (1) that >> people completed tasks faster on the second trial, and (2) that people >> claimed to prefer the novel paradigms to mutex locking in the >> after-task questionnaires. The former is crushingly obvious, and the >> latter is subject to a number of obvious biases. >> >> (I also noticed an odd bifurication in finishing times - subjects >> either finished the task in half the time allotted or didn't manage it >> at all, but this is irrelevant to your question :) ) >> >> The paper is entitled "Empirically Investigating Parallel Programming >> Paradigms: A Null Result", and is available online at: >> >> http://ecs.victoria.ac.nz/twiki/pub/Events/PLATEAU/2009Programme/plateau09-luff.pdf >> >> To my knowledge, at the time of writing, there was no work published >> more recent or applicable to your query than the supercomputing >> studies I cited in the Related Work section. (There has been a more >> recent study of TM vs SMTL in an undergraduate-assignment setting - I >> don't have the reference but could grub it up for you if you wanted.) >> >> Meredydd >> >> On Wed, Feb 16, 2011 at 4:10 PM, Russel Winder <rus...@russel.org.uk> >> wrote: >>> >>> Prompted by various discussions elsewhere, I am on the search for recent >>> experimental results and/or people doing or about to do experiments. >>> The questions all relate to the models of parallel software: >>> shared-memory multithreading, Actor Model, Dataflow Model, Communicating >>> Sequential Processes (CSP), data parallelism. >>> >>> Question 1 is: is synchronous message passing easier for programmers to >>> work with than asynchronous message passing. >>> >>> Question 2 is: are the case classes of Actor Model easier for >>> programmers to work with than the select statement of Dataflow Model and >>> CSP. >>> >>> There are more but those are the two "biggies". >>> >>> There is a lot of people who know precious little about psychology using >>> advocacy research out there building up various "known facts" about what >>> is and is not better from a cognitive perspective. Some of them base >>> this on observed anecdotal evidence which gives it some legitimacy, some >>> of them are peddling their own beliefs. >>> >>> So real experimental evidence from people who know what they are doing >>> would be most welcome. >>> >>> Thanks. >>> >>> -- >>> Russel. >>> >>> ============================================================================= >>> Dr Russel Winder t: +44 20 7585 2200 voip: >>> sip:russel.win...@ekiga.net >>> 41 Buckmaster Road m: +44 7770 465 077 xmpp: rus...@russel.org.uk >>> London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder >>> >>> >>> -- >>> The Open University is incorporated by Royal Charter (RC 000391), an >>> exempt charity in England & Wales and a charity registered in Scotland (SC >>> 038302). >>> >>> >> > > 73 Huntington Rd, York YO31 8RL > 01904-673675 > http://homepage.ntlworld.com/greenery/ > > > >
