Hi Andy, Yes, it is regression, so that explains it.
Here is the script and data that produced the output: https://dl.dropbox.com/u/74279156/accuracy.zip Thanks, Zach On 13 August 2012 16:21, Andreas Mueller <[email protected]> wrote: > Hi Zach. > If this is related to your previous problems, let me just > answer 1: the values depend on what error score is used. > If your problem is a regression problem, the standard score is r2, > which can become negative. > > That the CV values vary so much is really a bit odd. > Could you post a gist with your script? > > Cheers, > Andy > > On 08/13/2012 10:49 PM, Zach Bastick wrote: >> Changing the cv parameter (number of iterations) in cross_val_score() >> really changes the returned scores. Increasing the CV doesn't >> neccessarily mean that the returned scores stabalise. Instead, they get >> worse, and only get better later. I have included the output of >> increasing the CV below. >> >> My questions are: >> 1. How is it possible that the accuracy of a prediction is sometimes >> negative? Isn't the lowest accuracy 0.00? >> 2. The default value of CV (if none is specified is 3). However, when >> there is so much varience in the output of different CV values, how do >> you know which one is accurate? Which one should you choose? >> 3. Why do the accuracy and std become so variable as the CV increases, >> maax out around the middle (CV 21), and only then stabalise again ? >> >> 53 cases, 6596 features: >> CV: 2 Accuracy: 0.40 (+/- 0.00) >> CV: 3 Accuracy: 0.33 (+/- 0.10) >> CV: 4 Accuracy: 0.45 (+/- 0.04) >> CV: 5 Accuracy: 0.40 (+/- 0.05) >> CV: 6 Accuracy: 0.25 (+/- 0.19) >> CV: 7 Accuracy: -3.55 (+/- 4.77) >> CV: 8 Accuracy: 0.26 (+/- 0.17) >> CV: 9 Accuracy: 0.27 (+/- 0.21) >> CV: 10 Accuracy: -6.12 (+/- 9.69) >> CV: 11 Accuracy: -2.48 (+/- 4.52) >> CV: 12 Accuracy: -2.28 (+/- 4.34) >> CV: 13 Accuracy: -7.13 (+/- 9.51) >> CV: 14 Accuracy: -41.77 (+/- 71.35) >> CV: 15 Accuracy: -38.97 (+/- 69.13) >> CV: 16 Accuracy: -40.49 (+/- 67.01) >> CV: 17 Accuracy: -48.56 (+/- 66.93) >> CV: 18 Accuracy: -98.55 (+/- 96.74) >> CV: 19 Accuracy: -99.74 (+/- 94.20) >> CV: 20 Accuracy: -95.06 (+/- 92.38) >> CV: 21 Accuracy: -100.46 (+/- 90.95) >> CV: 22 Accuracy: -95.88 (+/- 89.48) >> CV: 23 Accuracy: -91.69 (+/- 88.06) >> CV: 24 Accuracy: -90.05 (+/- 86.31) >> CV: 25 Accuracy: -93.43 (+/- 84.81) >> CV: 26 Accuracy: -95.33 (+/- 83.39) >> CV: 27 Accuracy: 0.01 (+/- 0.04) >> CV: 28 Accuracy: 0.01 (+/- 0.04) >> CV: 29 Accuracy: 0.01 (+/- 0.03) >> CV: 30 Accuracy: 0.01 (+/- 0.03) >> CV: 31 Accuracy: 0.01 (+/- 0.03) >> CV: 32 Accuracy: 0.01 (+/- 0.03) >> CV: 33 Accuracy: 0.01 (+/- 0.02) >> CV: 34 Accuracy: 0.01 (+/- 0.02) >> CV: 35 Accuracy: 0.00 (+/- 0.01) >> CV: 36 Accuracy: 0.00 (+/- 0.01) >> CV: 37 Accuracy: 0.01 (+/- 0.02) >> CV: 38 Accuracy: 0.01 (+/- 0.02) >> CV: 39 Accuracy: 0.01 (+/- 0.03) >> CV: 40 Accuracy: 0.01 (+/- 0.03) >> CV: 41 Accuracy: 0.01 (+/- 0.02) >> CV: 42 Accuracy: 0.01 (+/- 0.02) >> CV: 43 Accuracy: -0.00 (+/- 0.01) >> CV: 44 Accuracy: -0.02 (+/- 0.05) >> CV: 45 Accuracy: -0.01 (+/- 0.05) >> CV: 46 Accuracy: -1.40 (+/- 4.68) >> CV: 47 Accuracy: -1.14 (+/- 3.87) >> CV: 48 Accuracy: -0.97 (+/- 3.31) >> CV: 49 Accuracy: -1.35 (+/- 4.69) >> CV: 50 Accuracy: -1.11 (+/- 3.90) >> CV: 51 Accuracy: -1.11 (+/- 3.93) >> CV: 52 Accuracy: 0.00 (+/- 0.00) >> CV: 53 Accuracy: 0.00 (+/- 0.00) >> >> >> Zach >> >> ------------------------------------------------------------------------------ >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. Discussions >> will include endpoint security, mobile security and the latest in malware >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> _______________________________________________ >> Scikit-learn-general mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Scikit-learn-general mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
