Hi, Gaël, this was a while back (~Dec. 2014), and I haven't investigated it further, because I got it to work with "manual" workaround back then. Maybe it was due to the particular hardware architecture or the particular Python interpreter. There were a lot of updates since then, and I should run the computation again before I submit an issue. I also posted a question on SO last year; the number of upvotes may suggest that there were other people that encountered similar issues (http://stackoverflow.com/questions/27508844/scikits-gridsearch-and-python-in-general-are-not-freeing-memory <http://stackoverflow.com/questions/27508844/scikits-gridsearch-and-python-in-general-are-not-freeing-memory>).
Anyway, I will put this issue on my todo list, and hopefully, I get a chance to check whether this is still an issue or not in the next couple of days or so. I uploaded the description, example code, and dataset to github to reproduce the issue, you can find it here: https://github.com/rasbt/bugreport/tree/master/scikit-learn/gridsearch_memory <https://github.com/rasbt/bugreport/tree/master/scikit-learn/gridsearch_memory>. Best, Sebastian > On Oct 19, 2015, at 10:31 AM, Gael Varoquaux <gael.varoqu...@normalesup.org> > wrote: > >> I had similar issues when I was running code on our clusters. I think that >> software & hardware architecture may also be a factor to be considered. What >> I >> observed was that the memory usage was growing over time. I added a few lines >> to manually clear the garbage collection, which solved the issue. > > I'd really love to see these lines. Do you want to create an issue on the > joblib tracker (joblib would probably be the place) to discuss this? > > Gaël > > ------------------------------------------------------------------------------ > _______________________________________________ > Scikit-learn-general mailing list > Scikit-learn-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
_______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
