Hi Sebastian and Manoj,
@Manoj: What should be the value of max_nbytes parameter and will this
affect the results and time it takes to run cross_validation, grid_search
etc?
@Sebastian: Will the Spark implication will also improve the memory use or
just the CPU?
Thanks
Kindest Regards
On Fri, Feb 12, 2016 at 5:29 PM, muhammad waseem <m.waseem.ah...@gmail.com>
wrote:
> Hi Sebastian and Manoj,
> @Manoj: What should be the value of max_nbytes parameter and will this
> affect the results and time it takes to run cross_validation, grid_search
> etc?
>
> Thanks
> Kindest Regards
> Waseem
>
> On Fri, Feb 12, 2016 at 4:42 PM, Sebastian Raschka <se.rasc...@gmail.com>
> wrote:
>
>> Hi, Waseem,
>> I think lowering the value of n_jobs would help; as far as I know, each
>> process get a copy of the data? Just stumbled upon spark-sklearn a few days
>> ago, maybe that could help as well:
>>
>>
>> https://databricks.com/blog/2016/02/08/auto-scaling-scikit-learn-with-spark.html
>>
>> When I understand correctly, the data is still copied, but here, each
>> node gets a copy instead of one machine with many copies.
>>
>>
>>
>>
>> > On Feb 12, 2016, at 11:35 AM, muhammad waseem <m.waseem.ah...@gmail.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > I am trying to fit my model using regression trees but the problem is,
>> it consumes a lot of RAM, which makes my code unresponsive. By looking at
>> different forums and platforms, I think this is a common problem. I was
>> wondering, how you free up memory or what are the best ways to run the
>> fitting process/cross-validation without running out of memory? This
>> problem is mostly with all regression trees (I think with other ML
>> algorithms as well). Shall I try to run without n_job=-1 and use some other
>> value (e.g. n_jobs=10) in cross_validation?
>> >
>> > Thanks
>> > Kindest Regards
>> > Waseem
>> >
>> ------------------------------------------------------------------------------
>> > Site24x7 APM Insight: Get Deep Visibility into Application Performance
>> > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>> > Monitor end-to-end web transactions and take corrective actions now
>> > Troubleshoot faster and improve end-user experience. Signup Now!
>> >
>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________
>> > Scikit-learn-general mailing list
>> > Scikit-learn-general@lists.sourceforge.net
>> > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>> Monitor end-to-end web transactions and take corrective actions now
>> Troubleshoot faster and improve end-user experience. Signup Now!
>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>> _______________________________________________
>> Scikit-learn-general mailing list
>> Scikit-learn-general@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>
>
>
>
--
Dr Muhammad Waseem Ahmad
Research Associate,
BRE Center for Sustainable Construction,
School of Engineering,
Cardiff University,
Cardiff, UK.
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
