This is strange indeed - since you said you're doing text classification I
suppose X is sparse? which format (csr, csc) and dtype (float64,32) are you
using?
The coef matrix is allocated before the sub processes are forked so you
will need (n_jobs + 1) * 12 gb just for the coefs.
The systemerror is quite strange though... I would expect a memory error...
Lars, do you have any thoughts on this?
best,
Peter
Am 13.08.2013 22:10 schrieb "A" <4rk....@gmail.com>:
> > >
> > > I have 64G of memory, so I do not think memory is the issue in this
> case.
> > If the features are dense, the n_classes many coefficients of n_features
> > are 12gb (if I haven't miss-calculated).
> - Correct, it occupies about 12.5G
> > If they are for some reason all replicated for all cores, you would get
> > into trouble.
> - Note that same is the case with njobs=2,3,4...., just to clarify that
> without using all cores, even if structures are replicated for cores,
> the max available should be enough in this case atleast(n_jobs=2,3,4),
> correct?
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
