Hi,
I'm starting using Spark and have installed Spark within CDH5 using
ClouderaManager.
I set up one master (hadoop-pg-5) and 3 workers (hadoop-pg-7[-8,-9]).
Master WebUI looks good, all workers seem to be registered.
If I open "spark-shell" and try to execute the wordcount example, the
execution hangs at the step "reduceByKey" and prints the Warning
""
14/04/11 21:29:47 WARN TaskSchedulerImpl: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and
have sufficient memory
14/04/11 21:30:02 WARN TaskSchedulerImpl: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and
have sufficient memory
""
again and again. In the Web-UI the task is in state "WAITING".
Some googling just told me to check networking/DNS stuff between master and
workers, but "host" "ping" and "telnet" are working in both ways =>
on worker hadoop-pg-7:
----------------------
[root@hadoop-pg-7 ~]# host hadoop-pg-5
hadoop-pg-5.cluster has address 10.147.210.5
[root@hadoop-pg-7 ~]# host hadoop-pg-5.cluster
hadoop-pg-5.cluster has address 10.147.210.5
[root@hadoop-pg-7 ~]# telnet hadoop-pg-5.cluster 7077
Trying 10.147.210.5...
Connected to hadoop-pg-5.cluster.
Escape character is '^]'.
on master hadoop-pg-5:
----------------------
[root@hadoop-pg-5 ~]# host hadoop-pg-7
hadoop-pg-7.cluster has address 10.147.210.7
[root@hadoop-pg-5 ~]# host hadoop-pg-7.cluster
hadoop-pg-7.cluster has address 10.147.210.7
[root@hadoop-pg-5 ~]# ping -c 1 hadoop-pg-7.cluster
PING hadoop-pg-7.cluster (10.147.210.7) 56(84) bytes of data.
64 bytes from hadoop-pg-7.cluster (10.147.210.7): icmp_seq=1 ttl=64
time=0.878 ms
--- hadoop-pg-7.cluster ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 2ms
rtt min/avg/max/mdev = 0.878/0.878/0.878/0.000 ms
[root@hadoop-pg-5 ~]# telnet hadoop-pg-7.cluster 7078
Trying 10.147.210.7...
Connected to hadoop-pg-7.cluster.
Escape character is '^]'.
This is the content of spark-env.sh on all hosts:
##
# Generated by Cloudera Manager and should not be modified directly
##
export SPARK_HOME=/opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/spark
export STANDALONE_SPARK_MASTER_HOST=hadoop-pg-5.cluster
export SPARK_MASTER_PORT=7077
export
DEFAULT_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop
### Path of Spark assembly jar in HDFS
export SPARK_JAR_HDFS_PATH=/user/spark/share/lib/spark-assembly.jar
### Let's run everything with JVM runtime, instead of Scala
export SPARK_LAUNCH_WITH_SCALA=0
export SPARK_LIBRARY_PATH=${SPARK_HOME}/lib
export SCALA_LIBRARY_PATH=${SPARK_HOME}/lib
export SPARK_MASTER_IP=$STANDALONE_SPARK_MASTER_HOST
export HADOOP_HOME=${HADOOP_HOME:-$DEFAULT_HADOOP_HOME}
if [ -n "$HADOOP_HOME" ]; then
export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:${HADOOP_HOME}/lib/native
fi
What am I missing, or doing wrong?
Any help appreciated, br Gerd
