Ok, so I found the issue . My initial sbt assembly was not done using the SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 ...
So I did sbt/sbt clean followed by SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 sbt/sbt assembly on each of the node (I am not doing the make anymore). Then I am starting the master and worker nodes. Now I can get proper webUI with all nodes showing and I get connectivity to HDFS Thanks Matei for your help On Wed, Jan 22, 2014 at 2:13 PM, Matei Zaharia <[email protected]>wrote: > Did you do sbt publish-local before by any chance? Maybe it’s picking up > another build of Spark from somewhere. In that case delete that build from > your local .m2 and .ivy2 directories. > > Matei > > On Jan 22, 2014, at 12:14 PM, Manoj Samel <[email protected]> > wrote: > > Did this on a fresh install - still same error. > > > On Wed, Jan 22, 2014 at 10:46 AM, Matei Zaharia > <[email protected]>wrote: > >> Try doing a sbt clean before rebuilding. >> >> Matei >> >> On Jan 22, 2014, at 10:22 AM, Manoj Samel <[email protected]> >> wrote: >> >> See thread below. Reposted as compilation error thread >> >> ---------- Forwarded message ---------- >> From: Manoj Samel <[email protected]> >> Date: Wed, Jan 22, 2014 at 10:20 AM >> Subject: Re: make-distribution.sh error >> org.apache.hadoop#hadoop-client;2.0.0: not found >> To: [email protected] >> >> >> I think I found the way to go beyond error "unresolved dependency: >> org.apache.hadoop#hadoop-client;2.0.0: not found" >> >> On my machine, the command "hadoop version" gives "2.0.0-cdh4.5.0" , not >> "2.0.0". >> >> After I changed the SPARK_HADOOP_VERSION to 2.0.0-cdh4.5.0, it is >> proceeding beyond that point. >> >> However, now it fails with multiple compilation errors in >> /.../streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala >> >> [error] >> /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:605: >> type mismatch; >> [error] found : org.apache.spark.streaming.DStream[(K, (V, >> com.google.common.base.Optional[W]))] >> [error] required: >> org.apache.spark.streaming.api.java.JavaPairDStream[K,(V, >> com.google.common.base.Optional[W])] >> [error] Note: implicit method fromPairDStream is not applicable here >> because it comes after the application point and it lacks an explicit >> result type >> [error] joinResult.mapValues{case (v, w) => (v, >> JavaUtils.optionToOptional(w))} >> [error] ^ >> [error] >> /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:617: >> type mismatch; >> [error] found : org.apache.spark.streaming.DStream[(K, >> (com.google.common.base.Optional[V], W))] >> [error] required: >> org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V], >> W)] >> [error] Note: implicit method fromPairDStream is not applicable here >> because it comes after the application point and it lacks an explicit >> result type >> [error] joinResult.mapValues{case (v, w) => >> (JavaUtils.optionToOptional(v), w)} >> [error] ^ >> [error] >> /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:632: >> type mismatch; >> [error] found : org.apache.spark.streaming.DStream[(K, >> (com.google.common.base.Optional[V], W))] >> [error] required: >> org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V], >> W)] >> [error] Note: implicit method fromPairDStream is not applicable here >> because it comes after the application point and it lacks an explicit >> result type >> [error] joinResult.mapValues{case (v, w) => >> (JavaUtils.optionToOptional(v), w)} >> [error] ^ >> [error] >> /data/spark/spark-0.8.1-incubating/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:647: >> type mismatch; >> [error] found : org.apache.spark.streaming.DStream[(K, >> (com.google.common.base.Optional[V], W))] >> [error] required: >> org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V], >> W)] >> [error] Note: implicit method fromPairDStream is not applicable here >> because it comes after the application point and it lacks an explicit >> result type >> [error] joinResult.mapValues{case (v, w) => >> (JavaUtils.optionToOptional(v), w)} >> [error] ^ >> [error] 43 errors found >> [error] (streaming/compile:compile) Compilation failed >> >> >> >> >> On Wed, Jan 22, 2014 at 10:09 AM, Manoj Samel >> <[email protected]>wrote: >> >>> Hi, >>> >>> On my cluster I have CDH 4.5 (Hadoop 2.0.0) installed. I installed spark >>> 0.8.1 and tried to run make-distribution.sh after >>> setting SPARK_HADOOP_VERSION=2.0.0 >>> >>> I get >>> >>> [warn] >>> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.0.0/hadoop-client-2.0.0.pom >>> [info] Resolving org.apache.derby#derby;10.4.2.0 ... >>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>> [warn] :: UNRESOLVED DEPENDENCIES :: >>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>> [warn] :: org.apache.hadoop#hadoop-client;2.0.0: not found >>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>> sbt.ResolveException: unresolved dependency: >>> org.apache.hadoop#hadoop-client;2.0.0: not found >>> at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:214) >>> at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122) >>> at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121) >>> at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:117) >>> at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:117) >>> >>> Any thoughts? >>> >> >> >> >> > >
