What I meant was if you set the number of max iterations to like 80,
does it run through?
On 10/08/2015 06:29 PM, Peter Rickwood wrote:
Yes, I can get up to 80-100 trees/iterations and everything works
normally (but slow due to thrashing) before the OS kills it.
I'll try and look into it with the profiler you suggest and if I find
anything will get back to the list.
It is of course possible I'm doing something else on the side which is
stupid and is causing my problem... I'll look into that too :-)
cheers
Message: 6
Date: Thu, 8 Oct 2015 18:16:55 -0400
From: Andreas Mueller <t3k...@gmail.com <mailto:t3k...@gmail.com>>
Subject: Re: [Scikit-learn-general] memory use of sklearn GBM
implementation
To: scikit-learn-general@lists.sourceforge.net
<mailto:scikit-learn-general@lists.sourceforge.net>
Message-ID: <5616eb57.9070...@gmail.com
<mailto:5616eb57.9070...@gmail.com>>
Content-Type: text/plain; charset="windows-1252"
Hm, that does sound a bit odd.
Maybe the memory_profiler will shed light on it?
https://pypi.python.org/pypi/memory_profiler
So if you use less than 100 trees it runs through?
Andy
On 10/08/2015 06:12 PM, Peter Rickwood wrote:
>
>
> Hello all,
>
> I'm puzzled by the memory use of sklearns GBM implementation. It
takes
> up all available memory and is forced to terminate by the OS, and I
> cant think of why it is using as much memory as it does.
>
> Here is the siituation:
>
> I have modest data set of size ~ 4GB (1800 columns, 550000 rows, all
> read in to a float32 matrix)
>
>
> I can read this in and start training a GBM with no memory
issues, but
> the memory use climbs rapidly as I add more estimators to the GBM.
> Once I get to about 100 trees it is using ~50GB of memory, which
kills
> my laptop.
>
> I dont understand why this is happening. Each tree is shallow (depth
> 3) so shouldn't take up much memory. The only way I can
understand the
> behaviour is if the data is somehow getting copied and stored
for each
> instance of the tree.
>
> What am I missing?
>
>
>
> Thanks in advance
>
>
>
> Peter
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
------------------------------------------------------------------------------
>
>
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
<mailto:Scikit-learn-general@lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
-------------- next part --------------
An HTML attachment was scrubbed...
------------------------------
------------------------------------------------------------------------------
------------------------------
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
<mailto:Scikit-learn-general@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
End of Scikit-learn-general Digest, Vol 69, Issue 10
****************************************************
------------------------------------------------------------------------------
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
