That worked! I assumed I would need to do something else, but just adding n_blocks=1000 brought down my memory usage more than enough. It took about 3 times as long, but I think that will be fine.
Thanks! Bill On Thu, Sep 3, 2015 at 12:28 PM, Nick Oosterhof <n.n.ooster...@googlemail.com> wrote: > >> On 03 Sep 2015, at 18:04, Bill Broderick <billb...@gmail.com> wrote: >> >> I'm trying to run group cluster thresholding using the defaults of >> GroupClusterThreshold (100000 bootstraps) and I'm running into memory >> issues. In the documentation, it looks like either n_proc (to split >> the load across several nodes on our cluster) or n_blocks would help, >> but it's not clear to me how to use these parameters. > > Peak memory usage is in the order of (n_bootstrap * n_features / n_blocks), > where n_features is the number of features of the dataset. > For example, if you set n_blocks=1000, then memory consumption will be > reduced by about a factor of 1,000 compared to n_blocks=1. > > I'm not sure how the Parallel module behaves, but it may be the case that > using n_proc processes will actually multiply memory demands by a factor of > n_proc. If you want to keep memory consumption low, my suggestion would be to > start with n_proc=1 and try higher values for n_blocks. > _______________________________________________ > Pkg-ExpPsy-PyMVPA mailing list > Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa _______________________________________________ Pkg-ExpPsy-PyMVPA mailing list Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
