Yes, did you see the PR for SPARK-2808? https://github.com/apache/spark/pull/3631/files
It requires more than just changing the version. On Tue, Feb 10, 2015 at 3:11 PM, Ted Yu <[email protected]> wrote: > Compiling Spark master branch against Kafka 0.8.2, I got: > > [WARNING] > /home/hbase/spark/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala:62: > no valid targets for annotation on value ssc_ - it is discarded unused. > You may specify targets with meta-annotations, e.g. @(transient @param) > [WARNING] @transient ssc_ : StreamingContext, > [WARNING] ^ > [ERROR] > /home/hbase/spark/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaCluster.scala:273: > type mismatch; > found : > Map[kafka.common.TopicAndPartition,kafka.common.OffsetMetadataAndError] > required: > scala.collection.immutable.Map[kafka.common.TopicAndPartition,kafka.common.OffsetAndMetadata] > [ERROR] val req = OffsetCommitRequest(groupId, metadata) > [ERROR] ^ > [ERROR] > /home/hbase/spark/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaCluster.scala:278: > value requestInfo is not a member of kafka.api. > OffsetCommitResponse > [ERROR] val respMap = resp.requestInfo > [ERROR] ^ > > FYI > > On Tue, Feb 10, 2015 at 3:23 AM, Sean Owen <[email protected]> wrote: >> >> I believe Kafka 0.8.2 is imminent for master / 1.4.0 -- see SPARK-2808. >> Anecdotally I have used Spark 1.2 with Kafka 0.8.2 without problem already. >> >> On Feb 10, 2015 10:53 AM, "critikaled" <[email protected]> wrote: >>> >>> When can we expect the latest kafka and scala 2.11 support in spark >>> streaming? >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-Version-Update-0-8-2-status-tp21573.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
