Hi,
I think some memory monitoring/warning stuff would be very helpful in general.
As far as I know, memory usage via e.g,. psutil is not supported by every OS or
machine, but we could add an optional "monitor_memory" parameter to
estimators/transformers like
SomeEstimator(..., monitor_memory=True)
and inside initialize something like
if self.monitor:
import psutil
self.process = psutil.Process()
that gets updated and checked every few iterations
cpu_total = self.process.get_cpu_percent()
mem_total = self.process.get_memory_percent()
if mem_total > 0.9:
print('warning, high memory usage')
What do you think?
Also, for the "fit" of estimators, an optional progress bar iterator could be
printed, e.g., each update step could be an epoch in an SVM or so. E.g.,
something like this: https://github.com/rasbt/pyprind
Best,
Sebastian
> On Mar 25, 2015, at 3:56 PM, Andreas Mueller <t3k...@gmail.com> wrote:
>
> It would be nice to do something else instead of crash and burn, but for the
> moment that's on the user.
>
> Well the kernel approximation should make it work. If you are after
> visualization I'd also recommend the T-SNE
> from this branch:
> https://github.com/scikit-learn/scikit-learn/pull/4025
>
>
> On 03/25/2015 03:38 PM, Stephen O'Neill wrote:
>> Hey Andy,
>>
>> Hmmm, that might be it. My machine only has 8GB of RAM - why didn't I think
>> of that? Indeed the RAM usage seems to have pretty large fluctuations for
>> the process, and when I re-run now instead of just silently dying its
>> choking up my whole computer - indicative of a RAM issue.
>>
>> Thank you very much - and sorry for the false alarm.
>>
>> Best,
>> Steve O'Neill
>>
>>
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming The Go Parallel Website,
>> sponsored
>> by Intel and developed in partnership with Slashdot Media, is your hub for
>> all
>> things parallel software development, from weekly thought leadership blogs to
>> news, videos, case studies, tutorials and more. Take a look and join the
>> conversation now.
>> http://goparallel.sourceforge.net/
>>
>>
>> _______________________________________________
>> Scikit-learn-general mailing list
>>
>> Scikit-learn-general@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for all
> things parallel software development, from weekly thought leadership blogs to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
