> why do you want separate permutations per each center_id? > null_dist here than mixes random results across all the searchlights if > I see it correctly. IMHO it is better to estimate that distribution per > each searchlight center (what would happen anyways if you didn't do this > manual "for i in searchlights")
I actually don't want separate permutations for each center_id. That was an oversight. But I do want to run the searchlights separately from each other (so I can parallelize them), which was what I was trying to do. I think I'll preseed the RNG the same across all of the searchlights, and that should do what I want. > once again I feel that trying to keep target labeling in test split > might be complicating things for your more than being of any value (do > you get notably different significance results in comparison to simple > permutations in all runs?). I was a bit uncomfortable with the idea of doing the simple permutation, but re-reading the Stelzer et al (2013) paper where they introduce the cluster thresholding method (which we're planning on using), it looks like that's what they do as well. I haven't had a chance to compare them yet, but I felt like there might be some theoretical issues. Since there doesn't seem to be any consensus (and there's some argument in favor of the simple permutations), I'll start with that way. Thanks, Bill On Fri, Aug 14, 2015 at 11:18 PM, Yaroslav Halchenko <[email protected]> wrote: > > > On Mon, 20 Jul 2015, Bill Broderick wrote: > > > Hi all, > > > I feel like this should be relatively simple, but I can't figure out how to > > do it. Is it possible to get at the labels generated by > > AttributePermutator? I would like to see what the individual permutations > > look like, to make sure it's doing what I think it is, but other than > > saving the whole dataset generated by CrossValidation, I can't see a way to > > do it. > > > I'm trying to build a null distribution like the following, so I can save > > each permutation, each searchlight separately (with how long the > > permutation testing has been taking, I want to make sure there's constant > > output in case something crashes and so I can monitor its progress, so I'm > > not using MCNullDist). > > if you were just want to check "in general" on what permutations permutator > generates, and didn't have limit={'partitions':1}, count=1 you could just > > [x.targets for x in permutator.generate(ds)] > > then if you preseeded RNG the same way before testing permutator > (e.g. mvpa2.seed(index_of_subject)) you could thus collect all those > generations without running actual analysis pipeline > > > for i in searchlights: > > for j in permutations: > > permutator = > > AttributePermutator('targets',limit={'partitions':1},count=1) > > nf = > > NFoldPartitioner(attr=partition_attr,cvtype=leave_x_out,count=fold_num,selection_strategy=fold_select_strategy) > > null_cv = > > CrossValidation(clf,ChainNode([nf,permutator],space=nf.get_space()),enable_ca='datasets',pass_attr=[('ca.datasets','fa')]) > > sl_null = sphere_searchlight(null_cv,radius=3,center_ids=[i]) > > null_dist.append(sl_null(ds)) > > null_dist=hstack(null_dist) > > why do you want separate permutations per each center_id? > null_dist here than mixes random results across all the searchlights if > I see it correctly. IMHO it is better to estimate that distribution per > each searchlight center (what would happen anyways if you didn't do this > manual "for i in searchlights") > > > once again I feel that trying to keep target labeling in test split > might be complicating things for your more than being of any value (do > you get notably different significance results in comparison to simple > permutations in all runs?). > -- > Yaroslav O. Halchenko, Ph.D. > http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org > Research Scientist, Psychological and Brain Sciences Dept. > Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 > Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 > WWW: http://www.linkedin.com/in/yarik > > _______________________________________________ > Pkg-ExpPsy-PyMVPA mailing list > [email protected] > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa _______________________________________________ Pkg-ExpPsy-PyMVPA mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
