I figured out for spark-shell by passing the --master option. However, still
troubleshooting for launching sql queries. My current command is like that:
./bin/beeline -u jdbc:hive2://ms0220:10000 -n `whoami` -p ignored -f
tpch_query10.sql
On Thursday, March 12, 2015 10:37 AM, Grandl Robert
<[email protected]> wrote:
Hi guys,
I have a stupid question, but I am not sure how to get out of it.
I deployed spark 1.2.1 on a cluster of 30 nodes. Looking at master:8088 I can
see all the workers I have created so far. (I start the cluster with
sbin/start-all.sh)
However, when running a Spark SQL query or even spark-shell, I cannot see any
job executing at master webUI, but the jobs are able to finish. I suspect they
are executing locally on the master, but I don't understand why/how and why not
on slave machines.
My conf/spark-env.sh is as following:export SPARK_MASTER_IP="ms0220"
export
SPARK_CLASSPATH=$SPARK_CLASSPATH:/users/rgrandl/software/spark-1.2.1-bin-hadoop2.4/lib/snappy-java-1.0.4.1.jar
export SPARK_LOCAL_DIRS="/users/rgrandl/software/data/spark/local"
export SPARK_WORKER_MEMORY="52000M"
export SPARK_WORKER_INSTANCES="2"
export SPARK_WORKER_CORES="2"
export SPARK_WORKER_DIR="/users/rgrandl/software/data/spark/worker"
export SPARK_DAEMON_MEMORY="5200M"
#export SPARK_DAEMON_JAVA_OPTS="4800M"
While conf/slaves is populated with the list of machines used for workers. I
have to mention that spark-env.sh and slaves files are deployed on all
machines.
Thank you,Robert
