I agree with Patil that consistent below-chance accuracy is a sign that something is not working properly.
I collected some thoughts in http://mvpa.blogspot.com/2013/04/below-chance-classification-accuracy.html (and a few other posts tagged "below-chance"). Also, be careful with terminology; I assume by "leave-one-run-out cross-validation on 4 acquisitions" you mean each person completed four scanning runs (each with the same fMRI acquisition parameters)? And a t-test can be fine for a quick significance test, but it should be done at the group level, testing if the subjects' accuracies are above chance (i.e., each person contributing one number to the t-test), not on the cross-validation folds within each person. Jo On 11/26/2018 7:05 AM, Raúl Hernández wrote: > I also consider that option, but when I try the very same thing with a > different, region (not related to the task). I get accuracies of 50%. So > that makes me think that it is affected by the task, but I don't know > what to think of it. > > Regards > > On Mon, Nov 26, 2018 at 1:34 PM Kaustubh Patil <kaustubh.pa...@gmail.com > <mailto:kaustubh.pa...@gmail.com>> wrote: > > I suspect that there might be something wrong in the code/how the > data is handled. > > If you 30% accuracy then that would mean that you will get 70% if > you use a simple rule to predict the "other class" after your > classifier. This is a sign that something is not right in data > handling/evaluation. > > Best > > On Mon, Nov 26, 2018 at 1:27 PM Raúl Hernández <r...@lafuentelab.org > <mailto:r...@lafuentelab.org>> wrote: > > No, it is balanced. It has the same number of observations for > each class. > > On Mon, Nov 26, 2018 at 12:52 PM Kaustubh Patil > <kaustubh.pa...@gmail.com <mailto:kaustubh.pa...@gmail.com>> wrote: > > Just for clarification. > > Is that data imbalanced, i.e. many more observations from > one class? > > Best, > Kaustubh > > On Mon, Nov 26, 2018 at 12:50 PM Raúl Hernández > <r...@lafuentelab.org <mailto:r...@lafuentelab.org>> wrote: > > Dear PyMVPA community, > > I'm doing classification in ROI's, I'm performing a > simple 2 way classification using LSVM, and a > leave-one-run-out cross-validation on 4 acquisitions. On > some ROI's, I get a good accuracy for the number of > participants (60%), but in others I get consistently bad > accuracy (30%). To test whether the performance is above > chance, I use a one sample t test (I know that it is not > the best test for this type of data, I just use it as > quick overview). When I test the bad accuracies, those > are also significant. > > What does it mean a consistently bad accuracy? > > Regards, > > Raul > _______________________________________________ > Pkg-ExpPsy-PyMVPA mailing list > Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net > <mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net> > > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > _______________________________________________ > Pkg-ExpPsy-PyMVPA mailing list > Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net > <mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net> > > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > _______________________________________________ > Pkg-ExpPsy-PyMVPA mailing list > Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net > <mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net> > > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > _______________________________________________ > Pkg-ExpPsy-PyMVPA mailing list > Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net > <mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net> > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > > _______________________________________________ > Pkg-ExpPsy-PyMVPA mailing list > Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > _______________________________________________ Pkg-ExpPsy-PyMVPA mailing list Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa