u've asked total cores to be 2 + 1 for driver(since you are running in cluster mode, so it's running on one of the slaves) change total cores to be 3*2 change submit mode to be client - you'll have full utilization (btw it's not advisable to use all cores of slave...since there is OS processes and other processes...)
On 20 November 2015 at 02:02, Andy Davidson <a...@santacruzintegration.com> wrote: > I am having a heck of a time figuring out how to utilize my cluster > effectively. I am using the stand alone cluster manager. I have a master > and 3 slaves. Each machine has 2 cores. > > I am trying to run a streaming app in cluster mode and pyspark at the same > time. > > t1) On my console I see > > * Alive Workers: 3 > * Cores in use: 6 Total, 0 Used > * Memory in use: 18.8 GB Total, 0.0 B Used > * Applications: 0 Running, 15 Completed > * Drivers: 0 Running, 2 Completed > * Status: ALIVE > > t2) I start my streaming app > > $SPARK_ROOT/bin/spark-submit \ > --class "com.pws.spark.streaming.IngestDriver" \ > --master $MASTER_URL \ > --total-executor-cores 2 \ > --deploy-mode cluster \ > $jarPath --clusterMode $* > > t3) on my console I see > > * Alive Workers: 3 > * Cores in use: 6 Total, 3 Used > * Memory in use: 18.8 GB Total, 13.0 GB Used > * Applications: 1 Running, 15 Completed > * Drivers: 1 Running, 2 Completed > * Status: ALIVE > > Looks like pyspark should be able to use the 3 remaining cores and 5.8 GB > of memory > > t4) I start pyspark > > export PYSPARK_PYTHON=python3.4 > export PYSPARK_DRIVER_PYTHON=python3.4 > export IPYTHON_OPTS="notebook --no-browser --port=7000 > --log-level=WARN" > > $SPARK_ROOT/bin/pyspark --master $MASTER_URL > --total-executor-cores 3 > --executor-memory 2g > > t5) on my console I see > > * Alive Workers: 3 > * Cores in use: 6 Total, 4 Used > * Memory in use: 18.8 GB Total, 15.0 GB Used > * Applications: 2 Running, 18 Completed > * Drivers: 1 Running, 2 Completed > * Status: ALIVE > > > I have 2 unused cores and a lot of memory left over. My pyspark > application is going getting 1 core. If streaming app is not running > pyspark would be assigned 2 cores each on a different worker. I have tried > using various combinations of --executor-cores and --total-executor-cores. > Any idea how to get pyspark to use more cores and memory? > > > Kind regards > > Andy > > P.s. Using different values I have wound up with pyspark status == > ³waiting² I think this is because there are not enough cores available? > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
