Implementing this directly in the estimators seems very messy.
If we had decent logging, we could try that. Unfortunately we don't.
Pretty printing could also be achieved via a logging mechanism, so that
people could define it themselves.
I don't think it is something we necessary want to provide.
You don't want to pass a callback to a python function to the inner loop
of libsvm, I think (there is a priority queue of pairs to pick, no
epochs, right?)
On 03/25/2015 04:40 PM, Sebastian Raschka wrote:
> Hi,
>
> I think some memory monitoring/warning stuff would be very helpful in
> general. As far as I know, memory usage via e.g,. psutil is not supported by
> every OS or machine, but we could add an optional "monitor_memory" parameter
> to estimators/transformers like
>
> SomeEstimator(..., monitor_memory=True)
>
> and inside initialize something like
>
> if self.monitor:
> import psutil
> self.process = psutil.Process()
>
> that gets updated and checked every few iterations
>
> cpu_total = self.process.get_cpu_percent()
> mem_total = self.process.get_memory_percent()
>
> if mem_total > 0.9:
> print('warning, high memory usage')
>
> What do you think?
>
> Also, for the "fit" of estimators, an optional progress bar iterator could be
> printed, e.g., each update step could be an epoch in an SVM or so. E.g.,
> something like this: https://github.com/rasbt/pyprind
>
> Best,
> Sebastian
>
>
>> On Mar 25, 2015, at 3:56 PM, Andreas Mueller <t3k...@gmail.com> wrote:
>>
>> It would be nice to do something else instead of crash and burn, but for the
>> moment that's on the user.
>>
>> Well the kernel approximation should make it work. If you are after
>> visualization I'd also recommend the T-SNE
>> from this branch:
>> https://github.com/scikit-learn/scikit-learn/pull/4025
>>
>>
>> On 03/25/2015 03:38 PM, Stephen O'Neill wrote:
>>> Hey Andy,
>>>
>>> Hmmm, that might be it. My machine only has 8GB of RAM - why didn't I
>>> think of that? Indeed the RAM usage seems to have pretty large
>>> fluctuations for the process, and when I re-run now instead of just
>>> silently dying its choking up my whole computer - indicative of a RAM issue.
>>>
>>> Thank you very much - and sorry for the false alarm.
>>>
>>> Best,
>>> Steve O'Neill
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Dive into the World of Parallel Programming The Go Parallel Website,
>>> sponsored
>>> by Intel and developed in partnership with Slashdot Media, is your hub for
>>> all
>>> things parallel software development, from weekly thought leadership blogs
>>> to
>>> news, videos, case studies, tutorials and more. Take a look and join the
>>> conversation now.
>>> http://goparallel.sourceforge.net/
>>>
>>>
>>> _______________________________________________
>>> Scikit-learn-general mailing list
>>>
>>> Scikit-learn-general@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming The Go Parallel Website,
>> sponsored
>> by Intel and developed in partnership with Slashdot Media, is your hub for
>> all
>> things parallel software development, from weekly thought leadership blogs to
>> news, videos, case studies, tutorials and more. Take a look and join the
>> conversation now. http://goparallel.sourceforge.net/
>> _______________________________________________
>> Scikit-learn-general mailing list
>> Scikit-learn-general@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for all
> things parallel software development, from weekly thought leadership blogs to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
