Re: [Scikit-learn-general] Memory explosion with GridSearchCV

2012-09-11 Thread Olivier Grisel
2012/9/11 Olivier Grisel : > > I will open an issue to track this. Done: https://github.com/scikit-learn/scikit-learn/issues/1137 -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel -- Live Security Virtu

Re: [Scikit-learn-general] Memory explosion with GridSearchCV

2012-09-11 Thread Olivier Grisel
2012/9/11 Olivier Grisel : > > Anyway this simulation is probably not representative of your scenario > as integers can be unboxed in the array datastructure hence the copy > while string objects cannot and the array will only store references > to the original string objects. Actually I was wrong

Re: [Scikit-learn-general] Memory explosion with GridSearchCV

2012-09-11 Thread Andreas Mueller
> Well you still have the original x in memory so you should at least > expect a doubling of the memory. The remaining memory might be > temporary stuff allocated during the conversion although that seems > weird. You can add: > > del x > import gc > gc.collect() > > Anyway this simulation is prob

Re: [Scikit-learn-general] Memory explosion with GridSearchCV

2012-09-11 Thread Olivier Grisel
2012/9/11 Christian Jauvin : > Hi, > > I'm working on a text classification problem, and the strategy I'm > currently studying is based on this example: > > http://scikit-learn.org/dev/auto_examples/grid_search_text_feature_extraction.html > > When I replace the data component by my own, I have fou

[Scikit-learn-general] Memory explosion with GridSearchCV

2012-09-10 Thread Christian Jauvin
Hi, I'm working on a text classification problem, and the strategy I'm currently studying is based on this example: http://scikit-learn.org/dev/auto_examples/grid_search_text_feature_extraction.html When I replace the data component by my own, I have found that the memory requirement explodes in