Hey,
On Wed, Mar 19, 2014 at 10:44 PM, Wouter De Baene
<wouter.deba...@ugent.be>wrote:
> Hi all,
>
> I’m running my first pymvpa-analysis after attending a very nice pymvpa
> workshop in Delmenhorst.
>
Glad you liked it!
> Although there are a few related questions in the archive, I don’t seem to
> get this working.
> I’m setting up a searchlight analysis with Nfold cross-validation. I'm
> trying to look at the confusion matrix for each cross-validation for each
> searchlight. This information seems to be lost when using searchlights.
> As I understood there were some plans to aggregate the output per
> searchlight into the output of the total results. Is there any news about
> this? OR could I get to this information in another way for now?
>
The easiest way to aggregate this kind of non-scalar information within a
searchlight is to
get the confusion matrix to come out of the CrossValidation as a vector
(flattened matrix).
For the first part of a possible solution we have "Confusion", a node that
turns raw targets
predictions into a confusion table. Here is a sketch: no error computing in
the classifier, return
raw predictions, use Confusion as a post-processing node to yield matrices.
CrossValidation(some_classifier,
some_partitioner,
errorfx=None,
postproc=Confusion())
Now the last bit is to flatten the confusion matrix into a vector. You
could write a simple Node
implementation that does that. Here is some pseudo code:
class MyFlatMatrix(Node):
def _call(ds):
return Dataset(ds.samples.ravel())
and now use this class as a post-processing node for Confusion itself.
CrossValidation(some_classifier,
some_partitioner,
errorfx=None,
postproc=Confusion(postproc=MyFlatMatrix()))
stick this into a searchlight and it should harvest what you want. This is
all written as a sketch and wasn't tested...
Instead of a custom Node class you could also use a ChainMapper comprised
of a FlattenMapper and a TransposeMapper
to achieve the same thing....
Good luck!
Michael
--
Michael Hanke
http://mih.voxindeserto.de
_______________________________________________
Pkg-ExpPsy-PyMVPA mailing list
Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
