i compiled against cdh 4.3.0 and been using it with 4.5.0 without using. i believe all cdh 4.x are wire compatible
On Fri, Dec 13, 2013 at 1:37 AM, Debasish Das <[email protected]>wrote: > Hi, > > I could compile Spark with CDH 4.2.0 but when I tried to access hdfs it > failed. > > I looked for the old post on Spark user group and found that Spark should > be compiled with the exact hadoop client version of the cluster. > > Our cluster is at CDH 4.5.0. I put the following configs for the > compilation on the master branch: > > export SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 > export SPARK_YARN=true > > I also tried to see if I can build against the client only > > export SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.5.0 > export SPARK_YARN=false > > I am getting 43 compilation errors from spark-streaming project. > > I have attached few msgs. > > [error] > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:51: > type mismatch; > [error] found : org.apache.spark.streaming.DStream[(K, V)] > [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > [error] Note: implicit method fromPairDStream is not applicable here > because it comes after the application point and it lacks an explicit > result type > [error] dstream.filter((x => f(x).booleanValue())) > [error] ^ > [error] > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:54: > type mismatch; > [error] found : org.apache.spark.streaming.DStream[(K, V)] > [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > [error] Note: implicit method fromPairDStream is not applicable here > because it comes after the application point and it lacks an explicit > result type > [error] def cache(): JavaPairDStream[K, V] = dstream.cache() > [error] ^ > [error] > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:57: > type mismatch; > [error] found : org.apache.spark.streaming.DStream[(K, V)] > [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > [error] Note: implicit method fromPairDStream is not applicable here > because it comes after the application point and it lacks an explicit > result type > [error] def persist(): JavaPairDStream[K, V] = dstream.persist() > [error] ^ > [error] > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:60: > type mismatch; > [error] found : org.apache.spark.streaming.DStream[(K, V)] > [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > [error] Note: implicit method fromPairDStream is not applicable here > because it comes after the application point and it lacks an explicit > result type > [error] def persist(storageLevel: StorageLevel): JavaPairDStream[K, V] = > dstream.persist(storageLevel) > [error] > ^ > [error] > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:66: > type mismatch; > [error] found : org.apache.spark.streaming.DStream[(K, V)] > [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > [error] Note: implicit method fromPairDStream is not applicable here > because it comes after the application point and it lacks an explicit > result type > [error] def repartition(numPartitions: Int): JavaPairDStream[K, V] = > dstream.repartition(numPartitions) > [error] > ^ > [error] > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:83: > type mismatch; > [error] found : org.apache.spark.streaming.DStream[(K, V)] > [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > [error] Note: implicit method fromPairDStream is not applicable here > because it comes after the application point and it lacks an explicit > result type > [error] dstream.window(windowDuration) > [error] ^ > > Note that the project compiled fine with CDH 4.2.0 but I could not access > our HDFS data. > > Thanks. > Deb > > -- > You received this message because you are subscribed to the Google Groups > "Spark Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Spark Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
