Re: URGENT: Testing Inclination to Programming
On 2011-03-21 13:41, Enda Dunican wrote: I have been reading this discussion with interest. Marian's message provoked further thought. In recent months (as part of my involvement in sports coaching), I have read a number of books that put forward the idea that rather than being the cause of something, talent/ability is actually the product of something, namely deep/purposeful/deliberate practice. When we see an individual who appears to have a natural talent or aptitude for something e.g. music, programming, sport we only see the tip of the iceberg and only when we study the individual more closely do we find out the true reasons behind their 'talent'. These books use the stories of great sports stars (Tiger Woods, Jack Nicklaus), compsers (Mozart) and authors (Bronte sisters) to illustrate their assertion. Just to throw in another reference, for a good overview of these topics I recommend Ericsson, Charness, Feltovich, and Hoffman (eds), "The Cambridge Handbook of Expertise and Expert Performance", Cambridge University Press, 2006. Especially chapter 38. -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).
RE: URGENT: Testing Inclination to Programming
Dear All, I have been reading this discussion with interest. Marian's message provoked further thought. In recent months (as part of my involvement in sports coaching), I have read a number of books that put forward the idea that rather than being the cause of something, talent/ability is actually the product of something, namely deep/purposeful/deliberate practice. When we see an individual who appears to have a natural talent or aptitude for something e.g. music, programming, sport we only see the tip of the iceberg and only when we study the individual more closely do we find out the true reasons behind their 'talent'. These books use the stories of great sports stars (Tiger Woods, Jack Nicklaus), compsers (Mozart) and authors (Bronte sisters) to illustrate their assertion. In addition, they also rely heavily on work conducted by eminent American psychologists Carol Dweck and Anders Ericcson. For example an interesting study that emerges out of this work would be to test if expert programmers exhibit a growth versus a fixed mindset (Dewck expirement) see: http://mindsetonline.com/ If I had lots of time I would assemble all of my readings into a coherent paper for PPIG and we could spend a while debating/discussing it. Regards, Enda Dr. Enda Dunican Lecturer in Computing, Dept. of Computing and Networking, Institute of Technology Carlow, Kilkenny Road, Carlow, Ireland. ITel: 1-353-(0)59-9175525 5 Email: enda.duni...@itcarlow.ie -Original Message- From: Alan Blackwell [mailto:alan.blackw...@cl.cam.ac.uk] Sent: 21 March 2011 11:59 To: Stefano Federici Cc: Richard O'Keefe; Thomas Green; PPIG Listserve; alan.blackw...@cl.cam.ac.uk Subject: Re: URGENT: Testing Inclination to Programming I may have missed it, but I don't think I saw a reference to the series of studies by Beckwith and Burnett on self-efficacy as a significant factor leading to gender differences in early programming experiences. If your experimental population includes a mix of males and females, you may find that this is a significant effect. I would strongly recommend recruiting balanced numbers of males and females, and carrying out analyses that consider interaction of self-efficacy and gender. A typical study in this area, which shows *opposite* effects of an experimental intervention for males and females, is this one: http://portal.acm.org/citation.cfm?doid=1124772.1124808 (Perhaps needless to say to you, but for the benefit of other readers - if you carry out an experiment in which the manipulation has opposite effects for two halves of the sample, and don't take this into account during analysis, the overall result will be highly inconclusive, resulting in large variance and small mean difference). Alan > Dear All, > I went through one of the suggested papers about self-efficacy > (Self-efficacy and mental models in learning to program, Ramalingam et > al, 2004). Unfortunately I'm at present totally unable to understand > the final results (path analysis of the model): > > post Self-Efficacy (R2 = .44) ==23*==> Performance - Grade (R2 = .30) > Mental Model (R2 = .05) ==.40*==> Performance - Grade (R2 = .30) > > The paper says that "both what student know, as represented by their > internal mental model, and what they believe about themselves, as > represented by their self-efficacy, affect their performance in the > course." > > Is there a naive way of rephrasing the 23* and .40* weights on the > arrows from "post Self-Efficacy" to "Performance - Grade" and from > "Mental Model" to "Performance - Grade"? I mean, in terms of > percentages, meaningfulness or other. > > Thanks in advance for all the help you keep giving me > > Stefano > > > > Stefano Federici > - > Università degli Studi di Cagliari > Facoltà di Scienze della Formazione > Dipartimento di Scienze Pedagogiche e Filosofiche > Via Is Mirrionis 1, 09123 Cagliari, Italia > - > Cell: +39 349 818 1955 Tel.: +39 070 675 7815 > Fax: +39 070 675 7113 > > > > -- > The Open University is incorporated by Royal Charter (RC 000391), an exempt > charity in England & Wales and a charity registered in Scotland (SC 038302). > > -- Alan Blackwell Reader in Interdisciplinary Design, University of Cambridge Further details from www.cl.cam.ac.uk/~afb21/
Re: URGENT: Testing Inclination to Programming
I have a much simpler method to test inclination top programming, although it needs to be standardised. Nevertheless it is worth a try with programmers with proven inclination and a control group. It is a joke: Tell them or show them this: A vet falls ill. he goes to see his family doctor. Vet: Hi Joe: GP: Hi Alan. What's the matter with you? Vet: you figure out. Regards, Ferenc "Music can't exist without notes and intervals. Conversation is the same. "Shut up and listen" has always been good advice to follow" Geoffrey Hamilton, Ph.D.(hon) - Original Message > From: Alan Blackwell > To: Stefano Federici > Cc: Richard O'Keefe ; Thomas Green > ; >PPIG Listserve ; alan.blackw...@cl.cam.ac.uk > Sent: Monday, 21 March, 2011 7:59:15 > Subject: Re: URGENT: Testing Inclination to Programming > > I may have missed it, but I don't think I saw a reference to the > series of studies by Beckwith and Burnett on self-efficacy as a > significant factor leading to gender differences in early > programming experiences. > > If your experimental population includes a mix of males and > females, you may find that this is a significant effect. I would > strongly recommend recruiting balanced numbers of males and > females, and carrying out analyses that consider interaction of > self-efficacy and gender. > > A typical study in this area, which shows *opposite* effects of > an experimental intervention for males and females, is this one: > http://portal.acm.org/citation.cfm?doid=1124772.1124808 > > (Perhaps needless to say to you, but for the benefit of other > readers - if you carry out an experiment in which the > manipulation has opposite effects for two halves of the sample, > and don't take this into account during analysis, the overall > result will be highly inconclusive, resulting in large variance > and small mean difference). > > Alan > > > Dear All, > > I went through one of the suggested papers about self-efficacy > > (Self-efficacy and mental models in learning to program, Ramalingam et > > al, 2004). Unfortunately I'm at present totally unable to understand > > the final results (path analysis of the model): > > > > post Self-Efficacy (R2 = .44) ==23*==> Performance - Grade (R2 = .30) > > Mental Model (R2 = .05) ==.40*==> Performance - Grade (R2 = .30) > > > > The paper says that "both what student know, as represented by their > > internal mental model, and what they believe about themselves, as > > represented by their self-efficacy, affect their performance in the > > course." > > > > Is there a naive way of rephrasing the 23* and .40* weights on the > > arrows from "post Self-Efficacy" to "Performance - Grade" and from > > "Mental Model" to "Performance - Grade"? I mean, in terms of > > percentages, meaningfulness or other. > > > > Thanks in advance for all the help you keep giving me > > > > Stefano > > > > > > > > Stefano Federici > > - > > Università degli Studi di Cagliari > > Facoltà di Scienze della Formazione > > Dipartimento di Scienze Pedagogiche e Filosofiche > > Via Is Mirrionis 1, 09123 Cagliari, Italia > > - > > Cell: +39 349 818 1955 Tel.: +39 070 675 7815 > > Fax: +39 070 675 7113 > > > > > > > > -- > > The Open University is incorporated by Royal Charter (RC 000391), an exempt >charity in England & Wales and a charity registered in Scotland (SC 038302). > > > > > > -- > Alan Blackwell > Reader in Interdisciplinary Design, University of Cambridge > Further details from www.cl.cam.ac.uk/~afb21/ > >
Re: URGENT: Testing Inclination to Programming
I may have missed it, but I don't think I saw a reference to the series of studies by Beckwith and Burnett on self-efficacy as a significant factor leading to gender differences in early programming experiences. If your experimental population includes a mix of males and females, you may find that this is a significant effect. I would strongly recommend recruiting balanced numbers of males and females, and carrying out analyses that consider interaction of self-efficacy and gender. A typical study in this area, which shows *opposite* effects of an experimental intervention for males and females, is this one: http://portal.acm.org/citation.cfm?doid=1124772.1124808 (Perhaps needless to say to you, but for the benefit of other readers - if you carry out an experiment in which the manipulation has opposite effects for two halves of the sample, and don't take this into account during analysis, the overall result will be highly inconclusive, resulting in large variance and small mean difference). Alan > Dear All, > I went through one of the suggested papers about self-efficacy > (Self-efficacy and mental models in learning to program, Ramalingam et > al, 2004). Unfortunately I'm at present totally unable to understand > the final results (path analysis of the model): > > post Self-Efficacy (R2 = .44) ==23*==> Performance - Grade (R2 = .30) > Mental Model (R2 = .05) ==.40*==> Performance - Grade (R2 = .30) > > The paper says that "both what student know, as represented by their > internal mental model, and what they believe about themselves, as > represented by their self-efficacy, affect their performance in the > course." > > Is there a naive way of rephrasing the 23* and .40* weights on the > arrows from "post Self-Efficacy" to "Performance - Grade" and from > "Mental Model" to "Performance - Grade"? I mean, in terms of > percentages, meaningfulness or other. > > Thanks in advance for all the help you keep giving me > > Stefano > > > > Stefano Federici > - > Università degli Studi di Cagliari > Facoltà di Scienze della Formazione > Dipartimento di Scienze Pedagogiche e Filosofiche > Via Is Mirrionis 1, 09123 Cagliari, Italia > - > Cell: +39 349 818 1955 Tel.: +39 070 675 7815 > Fax: +39 070 675 7113 > > > > -- > The Open University is incorporated by Royal Charter (RC 000391), an exempt > charity in England & Wales and a charity registered in Scotland (SC 038302). > > -- Alan Blackwell Reader in Interdisciplinary Design, University of Cambridge Further details from www.cl.cam.ac.uk/~afb21/
RE: URGENT: Testing Inclination to Programming
That's a great question--really does sound fascinating, Stefano! I'm eager to hear about your results! My best recommendation is to take a few subjects and watch them use MiniC then traditional C. A good think-aloud protocol may give you a lot of insight into what's going on, e.g., when faced with C, do they explicitly call upon their experience with MiniC? Good luck! Mark -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).
Re: URGENT: Testing Inclination to Programming
Dear All, I went through one of the suggested papers about self-efficacy (Self-efficacy and mental models in learning to program, Ramalingam et al, 2004). Unfortunately I'm at present totally unable to understand the final results (path analysis of the model): post Self-Efficacy (R2 = .44) ==23*==> Performance - Grade (R2 = .30) Mental Model (R2 = .05) ==.40*==> Performance - Grade (R2 = .30) The paper says that "both what student know, as represented by their internal mental model, and what they believe about themselves, as represented by their self-efficacy, affect their performance in the course." Is there a naive way of rephrasing the 23* and .40* weights on the arrows from "post Self-Efficacy" to "Performance - Grade" and from "Mental Model" to "Performance - Grade"? I mean, in terms of percentages, meaningfulness or other. Thanks in advance for all the help you keep giving me Stefano Stefano Federici - Università degli Studi di Cagliari Facoltà di Scienze della Formazione Dipartimento di Scienze Pedagogiche e Filosofiche Via Is Mirrionis 1, 09123 Cagliari, Italia - Cell: +39 349 818 1955 Tel.: +39 070 675 7815 Fax: +39 070 675 7113 -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).
Re: URGENT: Testing Inclination to Programming
But consider that the run of participations you're about to make could just be a pilot. I understand. To clarify my test, I'm not going to compare Scratch/BYOB with my tool. I'm going to compare miniC (a minimal C implementation built on BYOB) vs regular C environments. I want to test if students that learned C by using miniC: - do less syntactic mistakes - remember to declare their variables more often - use sequence/loop/conditionals in a more consistent way HCI designers regularly deal with these kinds of issues all the time I did it myself several times. What I'm not sure is if they are using the correct instruments to get correct, unbiased data from their tests. Do they look only at what is commonly seen as a programming base-skill (logic/math)? Or do they take into account also other -maybe apparently unrelated- but important dimensions? Which are their most recent studies on this subject? Did they publish their test-bed (as Dehani-Bornat did)? After all, from these tests they should claim that their work is successful. Are they doing it in the right way? Shouldn't we have a common, clearly-understood test-bed on which this kind of experimentation should be performed? This doesn't mean that the test-bed should be unupdateable, but at least important part of it should. Otherwise our tests wouldn't be comparable as they should. I'm I dreaming too much? Thanks again stefano Citando "Guzdial, Mark" : I'm listening, but have little to add. I just point back to Marian's list -- those are the issues, Stefano. Gather what data you can and hope that your two groups are roughly equivalent on the variables you're hoping to control for. A bigger issue (again, reflecting back to Marian's list) is the design of the task. I'm guessing that this is Mini-Logo vs. Scratch? Or MiniLogo vs. BYOB-Scratch vs. Scratch? In any case, there's going to likely be an interaction between the task and the environment -- some tasks are impossible in Scratch that the others can do, so a task that all three can do is likely to NOT take advantage of whatever makes MiniLogo and BYOB-Scratch unique. It's hard to do these kinds of tests. Even after this experiment, some think-aloud protocols with the environments that you most want to compare might give you the most insightful results about the real impact of the tool. Stefano, computer scientists do this, too. In particular, HCI designers regularly deal with these kinds of issues all the time. I'm a professor in a School of Interactive Computing in a College of Computing -- my colleagues and I do work with human subjects regularly. I think it's terrific for Programming Language designers to care about user testing and experiments with human participants! But consider that the run of participations you're about to make could just be a pilot. Make your mistakes now, and the next iteration will be publishable and head-turning. Cheers, Mark On Mar 19, 2011, at 6:50 AM, Thomas Green wrote: On 19 Mar 2011, at 09:55, Stefano Federici wrote: what I claim is the easiest programming environment ever designed so far). Er, yes. You might need to restrict what you mean by 'programming' . I regard using spreadsheets as programming. But Scratch is very good at its job, to be sure. I take it then that you're trying to out-do Scratch. Does this sound reasonable? Yes, it's probably the best you can do. I think the worst threat to generalisability is probably the risk of 'experimenter effect', where the students do better in the group that you want to do better. I don't know how to minimise that risk. If Sally Fincher or Mark Guzdial is listening, or anyone else with a good knowledge of these issues, I hope they'll join in. Good luck! Make sure to tell us how it goes. Thomas 73 Huntington Rd, York YO31 8RL 01904-673675 http://homepage.ntlworld.com/greenery/ -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302). Stefano Federici - Università degli Studi di Cagliari Facoltà di Scienze della Formazione Dipartimento di Scienze Pedagogiche e Filosofiche Via Is Mirrionis 1, 09123 Cagliari, Italia - Cell: +39 349 818 1955 Tel.: +39 070 675 7815 Fax: +39 070 675 7113
Re: URGENT: Testing Inclination to Programming
I'm listening, but have little to add. I just point back to Marian's list -- those are the issues, Stefano. Gather what data you can and hope that your two groups are roughly equivalent on the variables you're hoping to control for. A bigger issue (again, reflecting back to Marian's list) is the design of the task. I'm guessing that this is Mini-Logo vs. Scratch? Or MiniLogo vs. BYOB-Scratch vs. Scratch? In any case, there's going to likely be an interaction between the task and the environment -- some tasks are impossible in Scratch that the others can do, so a task that all three can do is likely to NOT take advantage of whatever makes MiniLogo and BYOB-Scratch unique. It's hard to do these kinds of tests. Even after this experiment, some think-aloud protocols with the environments that you most want to compare might give you the most insightful results about the real impact of the tool. Stefano, computer scientists do this, too. In particular, HCI designers regularly deal with these kinds of issues all the time. I'm a professor in a School of Interactive Computing in a College of Computing -- my colleagues and I do work with human subjects regularly. I think it's terrific for Programming Language designers to care about user testing and experiments with human participants! But consider that the run of participations you're about to make could just be a pilot. Make your mistakes now, and the next iteration will be publishable and head-turning. Cheers, Mark On Mar 19, 2011, at 6:50 AM, Thomas Green wrote: > > On 19 Mar 2011, at 09:55, Stefano Federici wrote: > >> what I claim is the easiest programming environment ever designed so far). > > > Er, yes. You might need to restrict what you mean by 'programming' . I > regard using spreadsheets as programming. But Scratch is very good at its > job, to be sure. > > I take it then that you're trying to out-do Scratch. > >> Does this sound reasonable? > > Yes, it's probably the best you can do. I think the worst threat to > generalisability is probably the risk of 'experimenter effect', where the > students do better in the group that you want to do better. I don't know how > to minimise that risk. If Sally Fincher or Mark Guzdial is listening, or > anyone else with a good knowledge of these issues, I hope they'll join in. > > Good luck! Make sure to tell us how it goes. > > Thomas > > > > > 73 Huntington Rd, York YO31 8RL > 01904-673675 > http://homepage.ntlworld.com/greenery/ > > > > > -- > The Open University is incorporated by Royal Charter (RC 000391), an exempt > charity in England & Wales and a charity registered in Scotland (SC 038302). >
Re: URGENT: Testing Inclination to Programming
Thanks a lot again Short version: Look at the Rountree et al work for a summary of previous studies. Have a look at one of the BRACE papers. I will certainly do. Read a good book on experiment design, and take into account Thomas's advice about pilot studies. It's worth putting the effort into planning, so that you don't end up investing effort in gathering data that tells you nothing. For this first experiment, I won't be able to do this (I'm running out of time, unfortunately). That is why, having so little time, I appreciated a lot Richard's suggestion, as it self-contained a full test about "consistency" choose a set of tasks that tap deeper cognitive skills such as planning, articulation, reading and following instructions, and systematic mapping. I would really like to. But I'm a computer scientist, so I have no prior knowledge on how these skills can be tested. An example (hopefully an extensive one) would be of real help. Motivation/belief does seem to have an impact, so you might want to include something on self-efficacy. You might see which instruments Susan Wiedenbeck and her group used in their studies that looked at the impact of self-efficacy. I will. But even in her "Self-Efficacy and Mental Models in Learning to Program" I couldn't find any extensive exemplification of which are the questions I should ask. Stefano -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).
Re: URGENT: Testing Inclination to Programming
Thanks again. But, as I said before, 298 pages are really too much for me right now. If you could simply summarize his main ideas so that I can design a sound test to group my students, I would really appreciate it. Thanks in advance Stefano Citando Marian Petre : You might also have a look at David Greathead's doctoral dissertation (2009) "An Investigation into the Influence of Student Personality Type and Other Factors on a Code Comprehension and Design Task in Java." http://homepages.cs.ncl.ac.uk/david.greathead/formal/GreatheadThesisDoubleSpaced.pdf Marian -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302). Stefano Federici - Università degli Studi di Cagliari Facoltà di Scienze della Formazione Dipartimento di Scienze Pedagogiche e Filosofiche Via Is Mirrionis 1, 09123 Cagliari, Italia - Cell: +39 349 818 1955 Tel.: +39 070 675 7815 Fax: +39 070 675 7113
Re: URGENT: Testing Inclination to Programming
Thanks! So many references but... if I can certainly start going through them, being you the experts I would really appreciate a "start from this" indication. If not, being my present need very urgent, I risk to take into account only a small fraction of these works, maybe not the most relevant to my present interest (grouping students in two similar groups with respect to their aptitude to learn computer programming). Thanks Stefano Stefano Federici - Università degli Studi di Cagliari Facoltà di Scienze della Formazione Dipartimento di Scienze Pedagogiche e Filosofiche Via Is Mirrionis 1, 09123 Cagliari, Italia - Cell: +39 349 818 1955 Tel.: +39 070 675 7815 Fax: +39 070 675 7113 Dear all, Please, please let's not re-invent the wheel -- or perhaps reiterate our own ignorance. We actually know very little about true indicators of programming aptitude. There are some correlations with spatial reasoning, and some with accurate articulation of process in one's native language. The point is that programming is not 'one thing' - it's a complex, composite interaction of skills. There have been a number of relevant studies (of varying quality) in the past decade alone, not to mention the proprietary instruments developed by personnel services in industry. Rountree, Rountree & Robins did a good literature review a few years ago. Here are a few relevant pointers (there is almost certainly more recent work, too): Rountree, N., Rountree, J., Robins, A. & Hannah, R. Interacting factors that predict success and failure in a CS1 course. SIGCSE Bulletin, 36(4), 101 - 104 (2004) Nathan Rountree, Janet Rountree and Anthony Robins Predictors of Success and Failure in a CS1 Course (2002) SIGCSE Bulletin vol. 34, no. 4. Vikki Fix, Susan Wiedenbeck, Jean Scholtz (1993) Mental representations of programs by novices and experts. Proceedings of the SIGCHI conference on Human factors in computing systems. M. McCracken, V. Almstrum, D. Diaz, M. Guzdial, D. Hagan, Y.B.-D. Kolikant, C. Laxer, L. Thomas, I. Utting, and T. Wilusz. (2001) A multinational, multi-institutional study of assessment of programming skills of first-year CS students. Proceedings of ITiCSE. B. Cantwell Wilson & S. Shrock (2001) Contributing to Success in an Introductory Computer Science Course: A Study of Twelve Factors SIGCSE Symposium Graham Daniel & Kevin Cox (2003) Computing Courses: Testing for Student Aptitude Web Tools Newsletter http://webtools.cityu.edu.hk/news/newsletter/aptitude.htm Mayer, R. E. (1989) The psychology of how novices learn computer programming. In E. Soloway & J. C. Spohrer (Eds.) Studying the novice programmer (pp 129-159) Hillsdale, NJ Lawrence Elbaum. Matt Roddan (2002) The Determinants of Student Failure and Attrition in First Year Computer Science http://www.psy.gla.ac.uk/~steve/localed/roddenpsy.pdf As part of the BRACE project (http://www.cs.otago.ac.nz/brace/), a whole collection of CS Ed researchers looked into this and conducted a multi-institution study. (The list above is from the reading list for BRACE) We used four diagnostic tasks: i) The Biggs Study Process Questionnaire (Biggs et al, 2001). The revised questionnaire assesses deep and surface approaches to learning in a given context. ii) The Paper Folding Test (VZ-2) is from the ETS Kit of Referenced Tests for Cognitive Factors (Ekstrom et al, 1976). The test is designed to measure visualisation and spatial reasoning. iii & iv) The description of a phone book search and a sketch-map giving directions across campus: two common-place examples to convey programming concepts and make them relevant to students (drawing on the work of Paul Curzon, 2002). The tasks assess students? ability to articulate a simple and familiar search and decision strategy accurately. Here are pointers to some of the resultant publications: Simon, Cutts, Q., Fincher, S., Haden, P., Robins, A., Sutton, K., Baker, B., Box, I., de Raadt, M., Hamer, J., Hamilton, M., Lister, R., Petre, M., Tolhurst, D., Tutty, J. (2006) The ability to articulate strategy as a predictor of programming skill. Australian Computer Science Communications, 28(5):181-188. ISSN 1445-1336. Simon, Fincher, S., Robins, A., Baker, B., Box, I., Cutts, Q., de Raadt, M., Haden, P., Hamer, J., Hamilton, M., Lister, R., Petre, M., Sutton, K., Tolhurst, D., Tutty, J. (2006). Predictors of success in a first programming course. Australian Computer Science Communications, 28(5):189-196. ISSN 1445-1336. de Raadt, M., Hamilton, M., Lister, R., Tutty, J., Box, I., Cutts, Q., Fincher, S., Haden, P., Petre, M., Robins, A., Simon, Sutton, K., Tolhurst, D., Baker, B., Hame
Re: URGENT: Testing Inclination to Programming
You might also have a look at David Greathead's doctoral dissertation (2009) "An Investigation into the Influence of Student Personality Type and Other Factors on a Code Comprehension and Design Task in Java." http://homepages.cs.ncl.ac.uk/david.greathead/formal/GreatheadThesisDoubleSpaced.pdf Marian -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).
Re: URGENT: Testing Inclination to Programming
Dear all, Please, please let's not re-invent the wheel -- or perhaps reiterate our own ignorance. We actually know very little about true indicators of programming aptitude. There are some correlations with spatial reasoning, and some with accurate articulation of process in one's native language. The point is that programming is not 'one thing' - it's a complex, composite interaction of skills. There have been a number of relevant studies (of varying quality) in the past decade alone, not to mention the proprietary instruments developed by personnel services in industry. Rountree, Rountree & Robins did a good literature review a few years ago. Here are a few relevant pointers (there is almost certainly more recent work, too): Rountree, N., Rountree, J., Robins, A. & Hannah, R. Interacting factors that predict success and failure in a CS1 course. SIGCSE Bulletin, 36(4), 101 - 104 (2004) Nathan Rountree, Janet Rountree and Anthony Robins Predictors of Success and Failure in a CS1 Course (2002) SIGCSE Bulletin vol. 34, no. 4. Vikki Fix, Susan Wiedenbeck, Jean Scholtz (1993) Mental representations of programs by novices and experts. Proceedings of the SIGCHI conference on Human factors in computing systems. M. McCracken, V. Almstrum, D. Diaz, M. Guzdial, D. Hagan, Y.B.-D. Kolikant, C. Laxer, L. Thomas, I. Utting, and T. Wilusz. (2001) A multinational, multi-institutional study of assessment of programming skills of first-year CS students. Proceedings of ITiCSE. B. Cantwell Wilson & S. Shrock (2001) Contributing to Success in an Introductory Computer Science Course: A Study of Twelve Factors SIGCSE Symposium Graham Daniel & Kevin Cox (2003) Computing Courses: Testing for Student Aptitude Web Tools Newsletter http://webtools.cityu.edu.hk/news/newsletter/aptitude.htm Mayer, R. E. (1989) The psychology of how novices learn computer programming. In E. Soloway & J. C. Spohrer (Eds.) Studying the novice programmer (pp 129-159) Hillsdale, NJ Lawrence Elbaum. Matt Roddan (2002) The Determinants of Student Failure and Attrition in First Year Computer Science http://www.psy.gla.ac.uk/~steve/localed/roddenpsy.pdf As part of the BRACE project (http://www.cs.otago.ac.nz/brace/), a whole collection of CS Ed researchers looked into this and conducted a multi-institution study. (The list above is from the reading list for BRACE) We used four diagnostic tasks: i) The Biggs Study Process Questionnaire (Biggs et al, 2001). The revised questionnaire assesses deep and surface approaches to learning in a given context. ii) The Paper Folding Test (VZ-2) is from the ETS Kit of Referenced Tests for Cognitive Factors (Ekstrom et al, 1976). The test is designed to measure visualisation and spatial reasoning. iii & iv) The description of a phone book search and a sketch-map giving directions across campus: two common-place examples to convey programming concepts and make them relevant to students (drawing on the work of Paul Curzon, 2002). The tasks assess students’ ability to articulate a simple and familiar search and decision strategy accurately. Here are pointers to some of the resultant publications: Simon, Cutts, Q., Fincher, S., Haden, P., Robins, A., Sutton, K., Baker, B., Box, I., de Raadt, M., Hamer, J., Hamilton, M., Lister, R., Petre, M., Tolhurst, D., Tutty, J. (2006) The ability to articulate strategy as a predictor of programming skill. Australian Computer Science Communications, 28(5):181-188. ISSN 1445-1336. Simon, Fincher, S., Robins, A., Baker, B., Box, I., Cutts, Q., de Raadt, M., Haden, P., Hamer, J., Hamilton, M., Lister, R., Petre, M., Sutton, K., Tolhurst, D., Tutty, J. (2006). Predictors of success in a first programming course. Australian Computer Science Communications, 28(5):189-196. ISSN 1445-1336. de Raadt, M., Hamilton, M., Lister, R., Tutty, J., Box, I., Cutts, Q., Fincher, S., Haden, P., Petre, M., Robins, A., Simon, Sutton, K., Tolhurst, D., Baker, B., Hamer, J. (2006). Do map drawing styles
Re: URGENT: Testing Inclination to Programming
I regard using spreadsheets as programming. Sure, I seen it the same way. Programming is creating general "rules" that will apply to different cases/situations. Is organized and logical thinking. the worst threat to generalisability is probably the risk of 'experimenter effect', where the students do better in the group that you want to do better. As a collegue of mine suggested, I will have students take their final test in a double-blind situation. Furthermore, the final test will include the same evalutation tests that students used to take in the past years. So, no test question will be specifically designed to match the explanations/usage of the new tool. I don't know if this is enough, or if other strategies can be succefully applied. Stefano Stefano Federici - Università degli Studi di Cagliari Facoltà di Scienze della Formazione Dipartimento di Scienze Pedagogiche e Filosofiche Via Is Mirrionis 1, 09123 Cagliari, Italia - Cell: +39 349 818 1955 Tel.: +39 070 675 7815 Fax: +39 070 675 7113 -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).
Re: URGENT: Testing Inclination to Programming
On 19 Mar 2011, at 09:55, Stefano Federici wrote: what I claim is the easiest programming environment ever designed so far). Er, yes. You might need to restrict what you mean by 'programming' . I regard using spreadsheets as programming. But Scratch is very good at its job, to be sure. I take it then that you're trying to out-do Scratch. Does this sound reasonable? Yes, it's probably the best you can do. I think the worst threat to generalisability is probably the risk of 'experimenter effect', where the students do better in the group that you want to do better. I don't know how to minimise that risk. If Sally Fincher or Mark Guzdial is listening, or anyone else with a good knowledge of these issues, I hope they'll join in. Good luck! Make sure to tell us how it goes. Thomas 73 Huntington Rd, York YO31 8RL 01904-673675 http://homepage.ntlworld.com/greenery/ -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).
Re: URGENT: Testing Inclination to Programming
1) If you're trying to set up balanced groups for a study, then you only need to know about factors that will give a sizeable noise level if they are not balanced across groups. That's what I thought you wanted to do, am I right? Yes, this is my main goal. how good is the interface? If there's something horrible in it, then the interference from that will drown every other effect. The experiment is exactly on the interface: a visual interface based on Scratch by MIT (what I claim is the easiest programming environment ever designed so far). good experimenters used to run at least two pilot studies before starting the main study, to ensure that all the shallow problems were ironed out. This is an excellent way of setting up the experiment. I will try to follow it for sure in the future. For now, due to the urgent set up, I will have to skip this step. Also, how good are the instructions? Make SURE that people can understand them. Get people to read them and explain them back to you. Anything they find hard, REWRITE IT. Instructions are given in four 2-hours seminars (we are simulating a minicourse on the subject, based on the new tool). Students are expected to take part to all seminars to be taken into account in the evalutation. The control group will have the exact same lessons by using the usual programming tool used in the past years. Does this sound reasonable? Stefano Stefano Federici - Università degli Studi di Cagliari Facoltà di Scienze della Formazione Dipartimento di Scienze Pedagogiche e Filosofiche Via Is Mirrionis 1, 09123 Cagliari, Italia - Cell: +39 349 818 1955 Tel.: +39 070 675 7815 Fax: +39 070 675 7113 -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).
Re: URGENT: Testing Inclination to Programming
Re "self-efficacy", read Unskilled and Unaware of it: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments Kruger and Dunning Psychology, 2009, 1, 30-46 People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities. This suggests that "people who believe in themselves" probably do WORSE than ones that don't. There's room for doubt about people who really ARE good at something, but people who are just beginning a subject are not likely to be good at estimating how good they are at it yet. (Grammar and logic are at least somewhat relevant to programming.) -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).
Re: URGENT: Testing Inclination to Programming
On 18 Mar 2011, at 17:28, Richard Bornat wrote: > Those who use the wrong model or no model can't program, whether or not they > say they can I meant, of course, can't program _yet_. Richard Bornat -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).
Re: URGENT: Testing Inclination to Programming
On 18 Mar 2011, at 14:40, Stefano Federici wrote: > Now I have an urgent need to know which (if any) are the specific tests that > would allow me to group students that have an inclination to programming from > students that have so such an inclination, and students that already know how > to program a computer from students that have never tried to program a > computer. > > I can imagine that I have to test their mathematical and logical skills. I > can check if they have previous knowledge about programming and programming > languages. But should I check if they have good knowledge about specific > areas of linguistics, such as, e.g., syntax? Are there other domains that may > be relevent for an individual so to understand in advance if they have an > inclination to computer programming? Hmm. I think that the ability to form and use mental models of program execution is very important in learning to program. The best survey of Saeed Dehnadi's results is in "Meta-analysis of the effect of consistency on success in early learning of programming" (Dehnadi, Bornat, Adams; PPIG 2009) which you can download from http://www.ppig.org/papers/21st-dehnadi.pdf If you ask students if they have learnt to program, and they use the conventional mental models of assignment and sequence in Saeed's test, then they can probably already program. Those who use the wrong model or no model can't program, whether or not they say they can. Lots of people seem to be able to guess the right model (there are clues in the questions), so just using the right model would capture more than those who have learnt to program. We don't yet understand what Saeed's result means, although 20% exam failures in those who 'pass' his test versus 55% exam failures in those who 'fail' suggests that it has something to do with the question. Richard Bornat -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).
Re: URGENT: Testing Inclination to Programming
How deeply do you want to go into this? 1) If you're trying to set up balanced groups for a study, then you only need to know about factors that will give a sizeable noise level if they are not balanced across groups. That's what I thought you wanted to do, am I right? 2) If you want to know what the state of knowledge is about factors that might, possibly, have some relationship to learning to program, even if only a small one, then it's a whole different question. I would recommend looking at work by Mark Eisenstadt on 'everyday programming' (or some similar title), at a big review by John Pane, and at a whole heap of material on Logo. But that's a big big review issue. If you're sticking with (1), you can stop worrying so much. Some few years ago Jarinee Chatatrichart found that of a very large number of possible factors that she studied, the only one with a significant contribution was whether people had used Lego blocks when they were little. And even that didn't have much effect. Much better to worry about whether you've designed the experiment right. For example, how good is the interface? If there's something horrible in it, then the interference from that will drown every other effect. Really good experimenters, like Patricia Wright, used to run at least two pilot studies before starting the main study, to ensure that all the shallow problems were ironed out. Also, how good are the instructions? Make SURE that people can understand them. Get people to read them and explain them back to you. Anything they find hard, REWRITE IT. So that's I recommend you to do. Run two people in each condition of your study, then TALK TO THEM and ask what they found hard. Then FIX IT. Then do it again until they stop complaining about little things that you hadn't intended to be problems. Thomas On 18 Mar 2011, at 16:45, Stefano Federici wrote: I see. But don't you think that, among those people that don't know anything about programming, someone being very good at punctuation could perform better at programming? I'm thinking to the classical Logo example to draw a square: 73 Huntington Rd, York YO31 8RL 01904-673675 http://homepage.ntlworld.com/greenery/ -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).
Re: URGENT: Testing Inclination to Programming
YES: Mathematical and logical skills, yes. But I really don't think you need to do more than ask them what educational qualifications they have. Being them Italian students at the university level, having an educational qualification is not often a good indicator of what skills they have (sadly) Previous programming experience - the best criterion might be number of languages known, but if you're working at a lower level then your suggestions are fine - loops and conditionals and function definitions. Yes, most of them have no knowledge at all about programming NO: linguistics, no (plenty of good computer scientists whose native language is English aren't good at spelling and punctuation) I see. But don't you think that, among those people that don't know anything about programming, someone being very good at punctuation could perform better at programming? I'm thinking to the classical Logo example to draw a square: repeat 4 [forward 100 right 90] that is, when rephrased in plain english: repeat 4 times move 100 steps and turn right. Interpreting this with the right punctuation "repeat 4 times: move 100 steps and turn right" or with the wrong one "repeat 4 times: move 100 steps. Then turn right" makes a difference. I noticed that some students are better at grasping this difference, other students need more time (or don't grasp the difference at all). Can this be evaluated in a pre-test? other domains - nothing good found so far. There is very little that anyone can do that will predict specific success at learning to program, other than general tests of ability. ALSO: One other thing I would recommend is a test of self-efficacy. Someone else must know more about testing that than I do. Self-efficacy means, roughly, believing in your own ability; there's evidence that people who believe in themselves do better than people who don't. I hadn't thought of it. But I can imagine that it can be influential. Do you have references to research on this topic? DISCLAIMER There are people on this list who know a whole lot more about this than I do. If they speak up, believe what they say, not what I said. Thomas Green On 18 Mar 2011, at 14:40, Stefano Federici wrote: Dear collegues, I want to thank you the list for the precious suggestions about the evaluation of programming environments. Now I have an urgent need to know which (if any) are the specific tests that would allow me to group students that have an inclination to programming from students that have so such an inclination, and students that already know how to program a computer from students that have never tried to program a computer. I can imagine that I have to test their mathematical and logical skills. I can check if they have previous knowledge about programming and programming languages. But should I check if they have good knowledge about specific areas of linguistics, such as, e.g., syntax? Are there other domains that may be relevent for an individual so to understand in advance if they have an inclination to computer programming? Going in more detail: - which are the mathematical and logical skills I have to evaluate? I can imagine fractions, series, simple equations, geometrical analogies, problem solving, truth tables - which are the programming knowledge I should pre-evaluate? I can imagine variables, loops, conditionals, function definition - maybe even the following skills in linguistics can be of help: phrase understanding, knowledge of punctuation Are there other domains/skills I could/should check? Which are the specific tests that can allow me to evaluate their relevant skills? Thanks a lot in advance Stefano Federici - Università degli Studi di Cagliari Facoltà di Scienze della Formazione Dipartimento di Scienze Pedagogiche e Filosofiche Via Is Mirrionis 1, 09123 Cagliari, Italia - Cell: +39 349 818 1955 Tel.: +39 070 675 7815 Fax: +39 070 675 7113 -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302). 73 Huntington Rd, York YO31 8RL 01904-673675 http://homepage.ntlworld.com/greenery/ Stefano Federici - Università degli Studi di Cagliari Facoltà di Scienze della Formazione Dipartimento di Scienze Pedagogiche e Filosofiche Via Is Mirrionis 1, 09123 Cagliari, Italia - Cell: +39 349 818 1955 Tel.: +39 070 675 7815 Fax: +39 070 675 7113
Re: URGENT: Testing Inclination to Programming
Are there other domains that may be relevent for an individual so to understand in advance if they have an inclination to computer programming? Possibly relevant: "Mental models, Consistency and Programming Aptitude" (Bornat, Dehnadi and Simon 2008). It's an investigation into whether students who are good at being consistent are better at learning to program than those who are not consistent. Regards, Becky -- Rebecca Yates http://staff.lero.ie/ryates/ Lero - The Irish Software Engineering Research Centre University of Limerick -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302).
Re: URGENT: Testing Inclination to Programming
Hi there!!! This is great material about Self-efficacy theory, also self-efficacy to programming 1. Bandura, A., ?Self-efficacy: Toward a unifying theory of behavioral change?, Psychological Review, Vol. 84, No. 2, 1977, pp. 191-215. 2. Karsten R. & Roth R.M. /Computer self/-/efficacy: a practical indicator of student computer competency in introductory IS courses/. Informing Science. Vol1, No. 3, 1998, pp. 61-68. 3. Ramalingam, V. and Wiedenbeck, S., ?Development and Validation of Scores on a Computer Programming Self-Efficacy Scale and Group Analyses of Novice Programmer Self-Efficacy?, /Journal of Educational Computing Research,/ Vol. 19, No.4, 1998, pp. 367-81. 4. Askar, P. and DaVenport, D.,?An Investigation of Factors Related to Self-Efficacy for JAVA Programming Among Engineering Students?, /The Turkish Online Journal of Educational Technology ? TOJET,/ Vol. 8, Issue 1, Article 3, Jan. 2009. Scheila Martins Citando Thomas Green : > Dear Stefano > > Since nobody else has replied yet, and you say it's urgent: > > YES: > > Mathematical and logical skills, yes. But I really don't think you > need to do more than ask them what educational qualifications they > have. > > Previous programming experience - the best criterion might be number > of languages known, but if you're working at a lower level then your > suggestions are fine - loops and conditionals and function > definitions. > > NO: > linguistics, no (plenty of good computer scientists whose native > language is English aren't good at spelling and punctuation) > > other domains - nothing good found so far. > > There is very little that anyone can do that will predict specific > success at learning to program, other than general tests of ability. > > ALSO: > > One other thing I would recommend is a test of self-efficacy. Someone > else must know more about testing that than I do. Self-efficacy > means, roughly, believing in your own ability; there's evidence that > people who believe in themselves do better than people who don't. > > DISCLAIMER > > There are people on this list who know a whole lot more about this > than I do. If they speak up, believe what they say, not what I said. > > Thomas Green > > > > On 18 Mar 2011, at 14:40, Stefano Federici wrote: > >> Dear collegues, >> I want to thank you the list for the precious suggestions about the >> evaluation of programming environments. >> >> Now I have an urgent need to know which (if any) are the specific >> tests that would allow me to group students that have an inclination >> to programming from students that have so such an inclination, and >> students that already know how to program a computer from students >> that have never tried to program a computer. >> >> I can imagine that I have to test their mathematical and logical >> skills. I can check if they have previous knowledge about >> programming and programming languages. But should I check if they >> have good knowledge about specific areas of linguistics, such as, >> e.g., syntax? Are there other domains that may be relevent for an >> individual so to understand in advance if they have an inclination >> to computer programming? >> >> Going in more detail: >> - which are the mathematical and logical skills I have to evaluate? >> I can imagine fractions, series, simple equations, geometrical >> analogies, problem solving, truth tables >> - which are the programming knowledge I should pre-evaluate? I can >> imagine variables, loops, conditionals, function definition >> - maybe even the following skills in linguistics can be of help: >> phrase understanding, knowledge of punctuation >> >> Are there other domains/skills I could/should check? Which are the >> specific tests that can allow me to evaluate their relevant skills? >> >> Thanks a lot in advance >> >> >> Stefano Federici >> - >> Università degli Studi di Cagliari >> Facoltà di Scienze della Formazione >> Dipartimento di Scienze Pedagogiche e Filosofiche >> Via Is Mirrionis 1, 09123 Cagliari, Italia >> - >> Cell: +39 349 818 1955 Tel.: +39 070 675 7815 >> Fax: +39 070 675 7113 >> >> >> >> -- >> The Open University is incorporated by Royal Charter (RC 000391), an >> exempt charity in England & Wales and a charity registered in >> Scotland (SC 038302). >> > > 73 Huntington Rd, York YO31 8RL > 01904-673675 > http://homepage.ntlworld.com/greenery/ > > > >
Re: URGENT: Testing Inclination to Programming
Dear Stefano Since nobody else has replied yet, and you say it's urgent: YES: Mathematical and logical skills, yes. But I really don't think you need to do more than ask them what educational qualifications they have. Previous programming experience - the best criterion might be number of languages known, but if you're working at a lower level then your suggestions are fine - loops and conditionals and function definitions. NO: linguistics, no (plenty of good computer scientists whose native language is English aren't good at spelling and punctuation) other domains - nothing good found so far. There is very little that anyone can do that will predict specific success at learning to program, other than general tests of ability. ALSO: One other thing I would recommend is a test of self-efficacy. Someone else must know more about testing that than I do. Self-efficacy means, roughly, believing in your own ability; there's evidence that people who believe in themselves do better than people who don't. DISCLAIMER There are people on this list who know a whole lot more about this than I do. If they speak up, believe what they say, not what I said. Thomas Green On 18 Mar 2011, at 14:40, Stefano Federici wrote: Dear collegues, I want to thank you the list for the precious suggestions about the evaluation of programming environments. Now I have an urgent need to know which (if any) are the specific tests that would allow me to group students that have an inclination to programming from students that have so such an inclination, and students that already know how to program a computer from students that have never tried to program a computer. I can imagine that I have to test their mathematical and logical skills. I can check if they have previous knowledge about programming and programming languages. But should I check if they have good knowledge about specific areas of linguistics, such as, e.g., syntax? Are there other domains that may be relevent for an individual so to understand in advance if they have an inclination to computer programming? Going in more detail: - which are the mathematical and logical skills I have to evaluate? I can imagine fractions, series, simple equations, geometrical analogies, problem solving, truth tables - which are the programming knowledge I should pre-evaluate? I can imagine variables, loops, conditionals, function definition - maybe even the following skills in linguistics can be of help: phrase understanding, knowledge of punctuation Are there other domains/skills I could/should check? Which are the specific tests that can allow me to evaluate their relevant skills? Thanks a lot in advance Stefano Federici - Università degli Studi di Cagliari Facoltà di Scienze della Formazione Dipartimento di Scienze Pedagogiche e Filosofiche Via Is Mirrionis 1, 09123 Cagliari, Italia - Cell: +39 349 818 1955 Tel.: +39 070 675 7815 Fax: +39 070 675 7113 -- The Open University is incorporated by Royal Charter (RC 000391), an exempt charity in England & Wales and a charity registered in Scotland (SC 038302). 73 Huntington Rd, York YO31 8RL 01904-673675 http://homepage.ntlworld.com/greenery/
