Can you try adding this to your spark-env file and sync to all hosts export MASTER="spark://hadoop-pg-5.cluster:7077"
On Sat, Apr 12, 2014 at 6:50 PM, ge ko <[email protected]> wrote: > Hi, > > I'm starting using Spark and have installed Spark within CDH5 using > ClouderaManager. > I set up one master (hadoop-pg-5) and 3 workers (hadoop-pg-7[-8,-9]). > Master WebUI looks good, all workers seem to be registered. > > If I open "spark-shell" and try to execute the wordcount example, the > execution hangs at the step "reduceByKey" and prints the Warning > "" > 14/04/11 21:29:47 WARN TaskSchedulerImpl: Initial job has not accepted any > resources; check your cluster UI to ensure that workers are registered and > have sufficient memory > 14/04/11 21:30:02 WARN TaskSchedulerImpl: Initial job has not accepted any > resources; check your cluster UI to ensure that workers are registered and > have sufficient memory > "" > again and again. In the Web-UI the task is in state "WAITING". > > Some googling just told me to check networking/DNS stuff between master > and workers, but "host" "ping" and "telnet" are working in both ways => > on worker hadoop-pg-7: > ---------------------- > [root@hadoop-pg-7 ~]# host hadoop-pg-5 > hadoop-pg-5.cluster has address 10.147.210.5 > > [root@hadoop-pg-7 ~]# host hadoop-pg-5.cluster > hadoop-pg-5.cluster has address 10.147.210.5 > > [root@hadoop-pg-7 ~]# telnet hadoop-pg-5.cluster 7077 > Trying 10.147.210.5... > Connected to hadoop-pg-5.cluster. > Escape character is '^]'. > > > on master hadoop-pg-5: > ---------------------- > [root@hadoop-pg-5 ~]# host hadoop-pg-7 > hadoop-pg-7.cluster has address 10.147.210.7 > [root@hadoop-pg-5 ~]# host hadoop-pg-7.cluster > hadoop-pg-7.cluster has address 10.147.210.7 > > [root@hadoop-pg-5 ~]# ping -c 1 hadoop-pg-7.cluster > PING hadoop-pg-7.cluster (10.147.210.7) 56(84) bytes of data. > 64 bytes from hadoop-pg-7.cluster (10.147.210.7): icmp_seq=1 ttl=64 > time=0.878 ms > > --- hadoop-pg-7.cluster ping statistics --- > 1 packets transmitted, 1 received, 0% packet loss, time 2ms > rtt min/avg/max/mdev = 0.878/0.878/0.878/0.000 ms > > [root@hadoop-pg-5 ~]# telnet hadoop-pg-7.cluster 7078 > Trying 10.147.210.7... > Connected to hadoop-pg-7.cluster. > Escape character is '^]'. > > This is the content of spark-env.sh on all hosts: > ## > # Generated by Cloudera Manager and should not be modified directly > ## > export > SPARK_HOME=/opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/spark > export STANDALONE_SPARK_MASTER_HOST=hadoop-pg-5.cluster > export SPARK_MASTER_PORT=7077 > export > DEFAULT_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop > ### Path of Spark assembly jar in HDFS > export SPARK_JAR_HDFS_PATH=/user/spark/share/lib/spark-assembly.jar > ### Let's run everything with JVM runtime, instead of Scala > export SPARK_LAUNCH_WITH_SCALA=0 > export SPARK_LIBRARY_PATH=${SPARK_HOME}/lib > export SCALA_LIBRARY_PATH=${SPARK_HOME}/lib > export SPARK_MASTER_IP=$STANDALONE_SPARK_MASTER_HOST > export HADOOP_HOME=${HADOOP_HOME:-$DEFAULT_HADOOP_HOME} > if [ -n "$HADOOP_HOME" ]; then > export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:${HADOOP_HOME}/lib/native > fi > > What am I missing, or doing wrong? > > Any help appreciated, br Gerd >
