Hi,
On Tue, 2017-11-21 at 23:47 +0800, Lijie Xu wrote:
> Hi Ravi,
>
> The suggestions in the blog are quite useful. This SVM application
> really caches some training data in memory. To lower the memory
> consumption, many methods can be used including changing the storage
> level, lowering the e
Hi Ravi,
The suggestions in the blog are quite useful. This SVM application really
caches some training data in memory. To lower the memory consumption, many
methods can be used including changing the storage level, lowering the
execution memory threshold, and improving the parallelism. I'm doing
Did you go thru this blog:
https://performancetestexpert.wordpress.com/2017/03/16/important-configuration-parameters-for-tuning-apache-spark-job/
If you have limitation on the hardware availability especially RAM,, one
important suggestion is "Storage level has been changed to
‘Disk_Only’:Before
Hi,
On Tue, 2017-11-21 at 21:48 +0800, Lijie Xu wrote:
> Hi Thomas,
>
[...]
> > > I want to know whether my guess is right ...
> >
> > Very likely. This is a long-standing issue (actually I have once
> > investigated about it like 10 years ago on a different regional
> > collector), and given yo
Hi Thomas,
Sorry for the late reply and thanks for your nice interpretation.
I added some comments and questions inline.
Thanks,
Lijie
On Sat, Nov 18, 2017 at 11:37 PM, Thomas Schatzl
wrote:
> Hi,
>
> On Sat, 2017-11-18 at 20:53 +0800, Lijie Xu wrote:
> > Hi All,
> > I recently encountered an
Hi,
On Sat, 2017-11-18 at 20:53 +0800, Lijie Xu wrote:
> Hi All,
> I recently encountered an OOM error in a Spark application using G1
> collector. This application launches multiple JVM instances to
> process the large data. Each JVM has 6.5GB heap size and uses G1
> collector. A JVM instance thr
Hi,
On Sat, 2017-11-18 at 20:53 +0800, Lijie Xu wrote:
> Hi All,
> I recently encountered an OOM error in a Spark application using G1
> collector. This application launches multiple JVM instances to
> process the large data. Each JVM has 6.5GB heap size and uses G1
> collector. A JVM instance thr
