Re: [Scikit-learn-general] Weird memory error

On 08/04/2015 01:49 PM, Maria Gorinova wrote: On 4 August 2015 at 18:25, Ronnie Ghose > wrote: are you able to make a np.ones stand alone of that size? Yes, I can create a np.ones array of size 100 000 000 approximatelly. On 4 August 2015 at 18:26, Andreas

Re: [Scikit-learn-general] Weird memory error

On 4 August 2015 at 18:25, Ronnie Ghose wrote: > are you able to make a np.ones stand alone of that size? > Yes, I can create a np.ones array of size 100 000 000 approximatelly. On 4 August 2015 at 18:26, Andreas Mueller wrote: > That array would take about 700mb of ram. Do you have that much

Re: [Scikit-learn-general] Weird memory error

That array would take about 700mb of ram. Do you have that much available? Btw, you could work around this issue probably by using HashingVectorizer instead of CountVectorizer. On 08/04/2015 01:20 PM, Maria Gorinova wrote: Hi Andy, Thanks, I updated to 0.16.1, but the problem persists. len(j_

Re: [Scikit-learn-general] Weird memory error

are you able to make a np.ones stand alone of that size? On Tue, Aug 4, 2015 at 10:20 AM, Maria Gorinova wrote: > Hi Andy, > > Thanks, I updated to 0.16.1, but the problem persists. > len(j_indices) is 68 356 000 when running for range(0,2000) and exactly > half of that when running for range(0,

Re: [Scikit-learn-general] Weird memory error

Hi Andy, Thanks, I updated to 0.16.1, but the problem persists. len(j_indices) is 68 356 000 when running for range(0,2000) and exactly half of that when running for range(0,1000). Sebastian, thank you for the suggestion, but again, the issue doesn't seem to be that the process is using too much

Re: [Scikit-learn-general] Weird memory error

Thanks Maria. What I was asking was that you could use the debugger to see what len(j_indices) is when it crashes. I'm not sure if there were improvements to this code since 0.15.2 but I'd encourage you to upgrade to 0.16.1 anyhow. Cheers, Andy On 08/04/2015 11:56 AM, Maria Gorinova wrote:

Re: [Scikit-learn-general] Weird memory error

Hm, I have never used Python on Windows but I have heard from many people that it is way buggier than the Posix equivalent; maybe it's just a quirk of the garbage collector? Maybe you could try to add the following lines: gc.collect() len(gc.get_objects()) inside your for-loop and give it anot

Re: [Scikit-learn-general] Weird memory error

Hi Andreas, Thank you for the reply. The error also happens if I load different files, yes, but here I am actually loading the SAME file "a.txt". Which I did, just to demonstrate how awkward the error is... I don't know what len(j_indices) is, that's in sklearn\feature_extraction\text.py as shown

Re: [Scikit-learn-general] Weird memory error

Just to make sure, you are actually loading different files, not the same file over and over again, right? It seems an odd place for a memory error. Which version of scikit-learn are you using? What is ``len(j_indices)``? On 08/04/2015 10:18 AM, Maria Gorinova wrote: Hello, (I think I might

[Scikit-learn-general] Weird memory error

Hello, (I think I might have sent this to the wrong address the first time, so I'm sending it again) I have been trying to find my way around a weird memory error for days now. If I'm doing something wrong and this question is completely dumb, I'm sorry for spamming the maillist. But I'm desperat