Perhaps your job does not use more than 9g. Even though the dashboard shows
64g the process only uses whats needed and grows to 64g max.
On Tue, Sep 16, 2014 at 5:40 PM, francisco <ftanudj...@nextag.com> wrote:
> Hi, I'm a Spark newbie.
>
> We had installed spark-1.0.2-bin-cdh4 on a 'super machine' with 256gb
> memory
> and 48 cores.
>
> Tried to allocate a task with 64gb memory but for whatever reason Spark is
> only using around 9gb max.
>
> Submitted spark job with the following command:
> "
> /bin/spark-submit -class SimpleApp --master local[16] --executor-memory 64G
> /var/tmp/simple-project_2.10-1.0.jar /data/lucene/ns.gz
> "
>
> When I run 'top' command I see only 9gb of memory is used by the spark
> process
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 3047005 fran 30 10 8785m 703m 18m S 112.9 0.3 48:19.63 java
>
>
> Any idea why this is happening? I've also tried to set the memory
> programatically using
> " new SparkConf().set("spark.executor.memory", "64g") " but that also
> didn't
> do anything.
>
> Is there some limitation when running in 'local' mode?
>
> Thanks.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Memory-under-utilization-tp14396.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
