Thanks. Yes it does appear that liblinear uses only a 64 bit dense format,
so this memory usage is normal/caused by the implementation of liblinear.
You may want to update the documentation hosted at this site:
http://scikit-learn.sourceforge.net/modules/svm.html#
It has a section on "avoiding data copy" which only says that the data
should be C contiguous.
It looks like there's a different implementation of libsvm that uses a dense
format so I'll look into using that.
On Thu, Oct 6, 2011 at 11:12 PM, Olivier Grisel <[email protected]>wrote:
> If your data is really dense, then you should try to use the
> SGDClassifier model instead of LinearSVC. It has an implementation for
> dense numpy arrays hence will use twice as less memory as a sparse
> representation.
>
> However I am pretty sure that it will force a copy of your data to be
> double precision (64bit). If you install cython you can patch the
> source code to force single precision instead.
>
> We might want to add support for single precision for SGDClassifier
> and other models in the future although this is not planned yet.
>
> --
> Olivier
> http://twitter.com/ogrisel - http://github.com/ogrisel
>
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2dcopy2
> _______________________________________________
> Scikit-learn-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2dcopy2
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
