Dear Xiangzhen, in the end I did not try to implement permutation-test for each crossv in each searchlight because this was rather impractical in terms of computation time. As a 30 time permutation (which was my first thought) is quite low, and with 500 permutation it took too much time, I decided to use other methods of ensuring the validity of the results. E.g., I shuffled the labels, and I looked in a control region (which shouldn't have predictive value) if this resulted in significant predictions. Good luck, Best regards, Nynke
---------------------------------------------------- Nynke van der Laan, MSc PhD-candidate Image Sciences Institute University Medical Center Utrecht Heidelberglaan 100 Post: Rm Q0S.459/Office: Rm E.02.557 3584 CX Utrecht The Netherlands Phone: +31 88 75 58562 http://www.isi.uu.nl/People/?nynke http://www.selfregulationlab.nl/index.php/nynke-van-der-laan On Thu, Apr 12, 2012 at 6:29 PM, Yaroslav Halchenko <[email protected]> wrote: > Hi guys, > > sorry for not following up with speedy replies ... > > Searchlight is just a Measure so its chance distribution (per each > searchlight) and corresponding p-values could be computed the same > fashion as any other measure, e.g. like in > http://www.pymvpa.org/examples/permutation_test.html > done for CrossValidation > > and e.g. look into this unittest > https://github.com/PyMVPA/PyMVPA/blob/HEAD/mvpa2/tests/test_searchlight.py#L236 > > hope this helps > > cheers, > > On Fri, 13 Apr 2012, Xiangzhen Kong wrote: > >> Hi Nynke, >> Could you help me about the question that you have asked? >> �How could I perform the permutation test for each >> searchlight�and let the�searchlight�output also the p values for >> testing against the null distribution?� >> I want to do the same thing and have you solved this? >> Best! >> Xiangzhen > >> On Tue, Aug 2, 2011 at 8:00 PM, >> <[1][email protected]> wrote: > >> Send Pkg-ExpPsy-PyMVPA mailing list submissions to >> � � � �[2][email protected] > >> To subscribe or unsubscribe via the World Wide Web, visit >> � � � >> >> �[3]http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > >> or, via email, send a message with subject or body 'help' to >> � � � �[4][email protected] > >> You can reach the person managing the list at >> � � � �[5][email protected] > >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of Pkg-ExpPsy-PyMVPA digest..." > >> Today's Topics: > >> � 1. Searchlight + cross validation & significance (permutation >> � � �test) (Nynke van der Laan) >> � 2. Re: Searchlight + cross validation & significance >> � � �(permutation test) (Francisco Pereira) >> � 3. Re: Searchlight + cross validation & significance >> � � �(permutation test) (J.A. Etzel) >> � 4. Searchlight + cross validation & significance (permutation >> � � �test) (Nynke van der Laan) > >> ---------------------------------------------------------------------- > >> Message: 1 >> Date: Mon, 1 Aug 2011 16:34:03 +0200 >> From: Nynke van der Laan <[6][email protected]> >> To: [7][email protected] >> Subject: [pymvpa] Searchlight + cross validation & significance >> � � � �(permutation � �test) >> Message-ID: >> � � � >> �<CADHpHBOUFyc4nfboeYjBOngHry=[8][email protected]> >> Content-Type: text/plain; charset=ISO-8859-1 > >> Hello all, > >> I have a question about the following: I am performing a searchlight >> analysis (v04) with a crossvalidation (leave one subject out, 20 >> samples per subject, 2 labels (10 samples label A, 10 samples label >> B)). >> I have adapted the example scripts to generate a map with the mean >> classification error over de NFold crossvalidation. Now, in addition I >> would like to be able to say something about the significance of the >> accuracies that the analysis yielded. >> In the module reference >> >> ([9]http://v04.pymvpa.org/modref/mvpa.measures.searchlight.html#module-measures-searchlight) >> I noticed the null_prob and null_T are available variables. Null_prob >> is enabled as default. However, the analysis now only results in the >> searchlight errormap. How can I acces the null_prob information? Is >> this averaged over the NFolds? > >> cv = CrossValidatedTransferError( >> � � � � TransferError(LinearCSVMC()), >> � � � � NFoldSplitter()) >> sl = Searchlight(cv, radius=6) >> sl_map = sl(dataset_total) > >> An additional question I have is whether there is any easy way to >> combine a searchlight analysis as described above with a permutation >> test. In a previous discussion I read about the NullhypothesisTest >> >> ([10]http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2008q2/000031.html) >> but this does not seem to be present in the v.04 version anymore (?). >> I predict that this analysis will be very costly in time as the >> permutation is done in each searchlight, but I wouldn't know any >> good/valid alternatives. Does anyone else have a suggestion? > >> Thanks a lot! > >> Best, >> Nynke > >> ------------------------------ > >> Message: 2 >> Date: Mon, 1 Aug 2011 10:47:35 -0400 >> From: Francisco Pereira <[11][email protected]> >> To: Development and support pf PyMVPA >> � � � �<[12][email protected]> >> Subject: Re: [pymvpa] Searchlight + cross validation & significance >> � � � �(permutation test) >> Message-ID: >> � � � >> >> �<[13]cafdubx0hlplfkftoqkzcg7vizjuq0ecai7hhnpkhxxptwt8...@mail.gmail.com> >> Content-Type: text/plain; charset=ISO-8859-1 > >> On Mon, Aug 1, 2011 at 10:34 AM, Nynke van der Laan >> <[14][email protected]> wrote: > >> > An additional question I have is whether there is any easy way to >> > combine a searchlight analysis as described above with a permutation >> test. > >> For a particular classifier (GNB) it's possible to do this fast >> enough, if I recall correctly this is the PyMVPA implementation > >> [15]http://www.pymvpa.org/generated/mvpa.measures.gnbsearchlight.html > >> cheers, >> Francisco > >> ------------------------------ > >> Message: 3 >> Date: Mon, 01 Aug 2011 09:58:19 -0500 >> From: "J.A. Etzel" <[16][email protected]> >> To: [17][email protected] >> Subject: Re: [pymvpa] Searchlight + cross validation & significance >> � � � �(permutation test) >> Message-ID: <[18][email protected]> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed > >> My usual strategy is to split up the permutations and run the jobs on a >> cluster; with luck your university will have access to one. Then you can >> have dozens of computers working on the job at the same time, for >> example one node doing permutations 1 to 5, another 6 to 10, etc. > >> Jo > >> On 8/1/2011 9:47 AM, Francisco Pereira wrote: >> > On Mon, Aug 1, 2011 at 10:34 AM, Nynke van der Laan >> > <[19][email protected]> �wrote: > >> >> An additional question I have is whether there is any easy way to >> >> combine a searchlight analysis as described above with a permutation >> test. > >> > For a particular classifier (GNB) it's possible to do this fast >> > enough, if I recall correctly this is the PyMVPA implementation > >> > [20]http://www.pymvpa.org/generated/mvpa.measures.gnbsearchlight.html > >> > cheers, >> > Francisco > >> > _______________________________________________ >> > Pkg-ExpPsy-PyMVPA mailing list >> > [21][email protected] > >> >> [22]http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > >> ------------------------------ > >> Message: 4 >> Date: Tue, 2 Aug 2011 11:55:21 +0200 >> From: Nynke van der Laan <[23][email protected]> >> To: [24][email protected] >> Subject: [pymvpa] Searchlight + cross validation & significance >> � � � �(permutation � �test) >> Message-ID: >> � � � >> >> �<CADHpHBPh=[25][email protected]> >> Content-Type: text/plain; charset=ISO-8859-1 > >> Thanks for your replies! >> Francisco, great suggestion, I will absolutely consider that >> classifier for my next study. For the present study/additional >> analysis though, I would like to stick with the v4.0 version and the >> linear SVM as this showed good results (and I preferably don't switch >> to a different version in between analyses/revisions). >> Jo, using multiple computers is indeed a good option, > >> However, in the documentation I read that if there is a strong >> assumptiom of the data being normally/gaussian distributed >> ([26]http://v04.pymvpa.org/examples/permutation_test.html), 30 >> permutations would also suffice. As my data indeed approaches a normal >> distribution, I would like to try this first. But, unfortunately I did >> not manage to implement this into the searchlight. In the example, it >> is clearly described how this permutation test would be done in a >> 'normal' analysis, but how exactly to combine this with the >> searchlight is unclear to me. I assume the null_dist = MCNullDist... >> should be included in the TransferError in the script below, but the >> searchlight now only returns the error scores for each voxel and no >> other results. How could I perform the permutation test for each >> searchlight and let the searchlight output also the p values for >> testing against the null distribution? > >> cv = CrossValidatedTransferError( >> � � � � TransferError(LinearCSVMC()), >> � � � � NFoldSplitter()) >> sl = Searchlight(cv, radius=6) >> sl_map = sl(dataset_total) > >> ------------------------------ > >> _______________________________________________ >> Pkg-ExpPsy-PyMVPA mailing list >> [27][email protected] >> >> [28]http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > >> End of Pkg-ExpPsy-PyMVPA Digest, Vol 42, Issue 1 >> ************************************************ > -- > =------------------------------------------------------------------= > Keep in touch www.onerussian.com > Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic _______________________________________________ Pkg-ExpPsy-PyMVPA mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
