Yes, I can get up to 80-100 trees/iterations and everything works normally (but slow due to thrashing) before the OS kills it.
I'll try and look into it with the profiler you suggest and if I find anything will get back to the list. It is of course possible I'm doing something else on the side which is stupid and is causing my problem... I'll look into that too :-) cheers > Message: 6 > Date: Thu, 8 Oct 2015 18:16:55 -0400 > From: Andreas Mueller <t3k...@gmail.com> > Subject: Re: [Scikit-learn-general] memory use of sklearn GBM > implementation > To: scikit-learn-general@lists.sourceforge.net > Message-ID: <5616eb57.9070...@gmail.com> > Content-Type: text/plain; charset="windows-1252" > > Hm, that does sound a bit odd. > Maybe the memory_profiler will shed light on it? > https://pypi.python.org/pypi/memory_profiler > > So if you use less than 100 trees it runs through? > > Andy > > > On 10/08/2015 06:12 PM, Peter Rickwood wrote: > > > > > > Hello all, > > > > I'm puzzled by the memory use of sklearns GBM implementation. It takes > > up all available memory and is forced to terminate by the OS, and I > > cant think of why it is using as much memory as it does. > > > > Here is the siituation: > > > > I have modest data set of size ~ 4GB (1800 columns, 550000 rows, all > > read in to a float32 matrix) > > > > > > I can read this in and start training a GBM with no memory issues, but > > the memory use climbs rapidly as I add more estimators to the GBM. > > Once I get to about 100 trees it is using ~50GB of memory, which kills > > my laptop. > > > > I dont understand why this is happening. Each tree is shallow (depth > > 3) so shouldn't take up much memory. The only way I can understand the > > behaviour is if the data is somehow getting copied and stored for each > > instance of the tree. > > > > What am I missing? > > > > > > > > Thanks in advance > > > > > > > > Peter > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > > _______________________________________________ > > Scikit-learn-general mailing list > > Scikit-learn-general@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general > > -------------- next part -------------- > An HTML attachment was scrubbed... > > ------------------------------ > > > ------------------------------------------------------------------------------ > > > ------------------------------ > > _______________________________________________ > Scikit-learn-general mailing list > Scikit-learn-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general > > > End of Scikit-learn-general Digest, Vol 69, Issue 10 > **************************************************** >
------------------------------------------------------------------------------
_______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
