Hi, Waseem, I think lowering the value of n_jobs would help; as far as I know, each process get a copy of the data? Just stumbled upon spark-sklearn a few days ago, maybe that could help as well:
https://databricks.com/blog/2016/02/08/auto-scaling-scikit-learn-with-spark.html When I understand correctly, the data is still copied, but here, each node gets a copy instead of one machine with many copies. > On Feb 12, 2016, at 11:35 AM, muhammad waseem <m.waseem.ah...@gmail.com> > wrote: > > Hi, > > I am trying to fit my model using regression trees but the problem is, it > consumes a lot of RAM, which makes my code unresponsive. By looking at > different forums and platforms, I think this is a common problem. I was > wondering, how you free up memory or what are the best ways to run the > fitting process/cross-validation without running out of memory? This problem > is mostly with all regression trees (I think with other ML algorithms as > well). Shall I try to run without n_job=-1 and use some other value (e.g. > n_jobs=10) in cross_validation? > > Thanks > Kindest Regards > Waseem > ------------------------------------------------------------------------------ > Site24x7 APM Insight: Get Deep Visibility into Application Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________ > Scikit-learn-general mailing list > Scikit-learn-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
