Re: High virtual memory consumption on spark-submit client.

2016-05-13 Thread jone
no, i have set master to yarn-cluster. when the sparkpi.running,the result of  free -t as follow [running]mqq@10.205.3.29:/data/home/hive/conf$ free -t total   used   free shared    buffers cached Mem:  32740732   32105684 635048  0 683332   28863456

Re: High virtual memory consumption on spark-submit client.

2016-05-12 Thread Mich Talebzadeh
Is this a standalone set up single host where executor runs inside the driver? also run *free -t* To see the virtual memory usage which is basically swap space free -t total used free sharedbuffers cached Mem: 24546308 24268760 277548 0

Re: High virtual memory consumption on spark-submit client.

2016-05-12 Thread Jone Zhang
mich, Do you want this == [running]mqq@10.205.3.29:/data/home/hive/conf$ ps aux | grep SparkPi mqq 20070 3.6 0.8 10445048 267028 pts/16 Sl+ 13:09 0:11 /data/home/jdk/bin/java -Dlog4j.configuration=file:

Re: High virtual memory consumption on spark-submit client.

2016-05-12 Thread Harsh J
How many CPU cores are on that machine? Read http://qr.ae/8Uv3Xq You can also confirm the above by running the pmap utility on your process and most of the virtual memory would be under 'anon'. On Fri, 13 May 2016 09:11 jone, wrote: > The virtual memory is 9G When i run org.apache.spark.example

Re: High virtual memory consumption on spark-submit client.

2016-05-12 Thread Mich Talebzadeh
can you please do the following: jps|grep SparkSubmit| and send the output of ps aux|grep pid top -p PID and the output of free HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

High virtual memory consumption on spark-submit client.

2016-05-12 Thread jone
The virtual memory is 9G When i run org.apache.spark.examples.SparkPi under yarn-cluster model,which using default configurations.   PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND