Re: where I can find spark-streaming-kafka for spark2.0
Thank you,I can't find spark-streaming-kafka_2.10 jar for spark2 from maven center. so I try the version 1.6.2,it not work ,it need class org.apache.spark.Logging, which can't find in spark2. so I build spark-streaming-kafka_2.10 jar for spark2 from the source code. it's work now. 2016-07-26 2:12 GMT+08:00 Cody Koeninger : > For 2.0, the kafka dstream support is in two separate subprojects > depending on which version of Kafka you are using > > spark-streaming-kafka-0-10 > or > spark-streaming-kafka-0-8 > > corresponding to brokers that are version 0.10+ or 0.8+ > > On Mon, Jul 25, 2016 at 12:29 PM, Reynold Xin wrote: > > The presentation at Spark Summit SF was probably referring to Structured > > Streaming. The existing Spark Streaming (dstream) in Spark 2.0 has the > same > > production stability level as Spark 1.6. There is also Kafka 0.10 > support in > > dstream. > > > > On July 25, 2016 at 10:26:49 AM, Andy Davidson > > (a...@santacruzintegration.com) wrote: > > > > Hi Kevin > > > > Just a heads up at the recent spark summit in S.F. There was a > presentation > > about streaming in 2.0. They said that streaming was not going to > production > > ready in 2.0. > > > > I am not sure if the older 1.6.x version will be supported. My project > will > > not be able to upgrade with streaming support. We also use kafka > > > > Andy > > > > From: Marco Mistroni > > Date: Monday, July 25, 2016 at 2:33 AM > > To: kevin > > Cc: "user @spark" , "dev.spark" > > > > Subject: Re: where I can find spark-streaming-kafka for spark2.0 > > > > Hi Kevin > > you should not need to rebuild everything. > > Instead, i believe you should launch spark-submit by specifying the kafka > > jar file in your --packages... i had to follow same when integrating > spark > > streaming with flume > > > > have you checked this link ? > > https://spark.apache.org/docs/latest/streaming-kafka-integration.html > > > > > > hth > > > > > > > > On Mon, Jul 25, 2016 at 10:20 AM, kevin wrote: > >> > >> I have compile it from source code > >> > >> 2016-07-25 12:05 GMT+08:00 kevin : > >>> > >>> hi,all : > >>> I try to run example > org.apache.spark.examples.streaming.KafkaWordCount , > >>> I got error : > >>> Exception in thread "main" java.lang.NoClassDefFoundError: > >>> org/apache/spark/streaming/kafka/KafkaUtils$ > >>> at > >>> > org.apache.spark.examples.streaming.KafkaWordCount$.main(KafkaWordCount.scala:57) > >>> at > >>> > org.apache.spark.examples.streaming.KafkaWordCount.main(KafkaWordCount.scala) > >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>> at > >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>> at > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>> at java.lang.reflect.Method.invoke(Method.java:498) > >>> at > >>> > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:724) > >>> at > >>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) > >>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) > >>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) > >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > >>> Caused by: java.lang.ClassNotFoundException: > >>> org.apache.spark.streaming.kafka.KafkaUtils$ > >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > >>> ... 11 more > >>> > >>> so where I can find spark-streaming-kafka for spark2.0 > >> > >> > > >
Re: where I can find spark-streaming-kafka for spark2.0
For 2.0, the kafka dstream support is in two separate subprojects depending on which version of Kafka you are using spark-streaming-kafka-0-10 or spark-streaming-kafka-0-8 corresponding to brokers that are version 0.10+ or 0.8+ On Mon, Jul 25, 2016 at 12:29 PM, Reynold Xin wrote: > The presentation at Spark Summit SF was probably referring to Structured > Streaming. The existing Spark Streaming (dstream) in Spark 2.0 has the same > production stability level as Spark 1.6. There is also Kafka 0.10 support in > dstream. > > On July 25, 2016 at 10:26:49 AM, Andy Davidson > (a...@santacruzintegration.com) wrote: > > Hi Kevin > > Just a heads up at the recent spark summit in S.F. There was a presentation > about streaming in 2.0. They said that streaming was not going to production > ready in 2.0. > > I am not sure if the older 1.6.x version will be supported. My project will > not be able to upgrade with streaming support. We also use kafka > > Andy > > From: Marco Mistroni > Date: Monday, July 25, 2016 at 2:33 AM > To: kevin > Cc: "user @spark" , "dev.spark" > > Subject: Re: where I can find spark-streaming-kafka for spark2.0 > > Hi Kevin > you should not need to rebuild everything. > Instead, i believe you should launch spark-submit by specifying the kafka > jar file in your --packages... i had to follow same when integrating spark > streaming with flume > > have you checked this link ? > https://spark.apache.org/docs/latest/streaming-kafka-integration.html > > > hth > > > > On Mon, Jul 25, 2016 at 10:20 AM, kevin wrote: >> >> I have compile it from source code >> >> 2016-07-25 12:05 GMT+08:00 kevin : >>> >>> hi,all : >>> I try to run example org.apache.spark.examples.streaming.KafkaWordCount , >>> I got error : >>> Exception in thread "main" java.lang.NoClassDefFoundError: >>> org/apache/spark/streaming/kafka/KafkaUtils$ >>> at >>> org.apache.spark.examples.streaming.KafkaWordCount$.main(KafkaWordCount.scala:57) >>> at >>> org.apache.spark.examples.streaming.KafkaWordCount.main(KafkaWordCount.scala) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> at >>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:724) >>> at >>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) >>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) >>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> Caused by: java.lang.ClassNotFoundException: >>> org.apache.spark.streaming.kafka.KafkaUtils$ >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>> ... 11 more >>> >>> so where I can find spark-streaming-kafka for spark2.0 >> >> > - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: where I can find spark-streaming-kafka for spark2.0
Hi Kevin Just a heads up at the recent spark summit in S.F. There was a presentation about streaming in 2.0. They said that streaming was not going to production ready in 2.0. I am not sure if the older 1.6.x version will be supported. My project will not be able to upgrade with streaming support. We also use kafka Andy From: Marco Mistroni Date: Monday, July 25, 2016 at 2:33 AM To: kevin Cc: "user @spark" , "dev.spark" Subject: Re: where I can find spark-streaming-kafka for spark2.0 > Hi Kevin > you should not need to rebuild everything. > Instead, i believe you should launch spark-submit by specifying the kafka jar > file in your --packages... i had to follow same when integrating spark > streaming with flume > > have you checked this link ? > https://spark.apache.org/docs/latest/streaming-kafka-integration.html > > > hth > > > > On Mon, Jul 25, 2016 at 10:20 AM, kevin wrote: >> I have compile it from source code >> >> 2016-07-25 12:05 GMT+08:00 kevin : >>> hi,all : >>> I try to run example org.apache.spark.examples.streaming.KafkaWordCount , I >>> got error : >>> Exception in thread "main" java.lang.NoClassDefFoundError: >>> org/apache/spark/streaming/kafka/KafkaUtils$ >>> at >>> org.apache.spark.examples.streaming.KafkaWordCount$.main(KafkaWordCount.scal >>> a:57) >>> at >>> org.apache.spark.examples.streaming.KafkaWordCount.main(KafkaWordCount.scala>>> ) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62>>> ) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl >>> .java:43) >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> at >>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$ru >>> nMain(SparkSubmit.scala:724) >>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) >>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) >>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> Caused by: java.lang.ClassNotFoundException: >>> org.apache.spark.streaming.kafka.KafkaUtils$ >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>> ... 11 more >>> >>> so where I can find spark-streaming-kafka for spark2.0 >> >
Re: where I can find spark-streaming-kafka for spark2.0
The presentation at Spark Summit SF was probably referring to Structured Streaming. The existing Spark Streaming (dstream) in Spark 2.0 has the same production stability level as Spark 1.6. There is also Kafka 0.10 support in dstream. On July 25, 2016 at 10:26:49 AM, Andy Davidson ( a...@santacruzintegration.com) wrote: Hi Kevin Just a heads up at the recent spark summit in S.F. There was a presentation about streaming in 2.0. They said that streaming was not going to production ready in 2.0. I am not sure if the older 1.6.x version will be supported. My project will not be able to upgrade with streaming support. We also use kafka Andy From: Marco Mistroni Date: Monday, July 25, 2016 at 2:33 AM To: kevin Cc: "user @spark" , "dev.spark" Subject: Re: where I can find spark-streaming-kafka for spark2.0 Hi Kevin you should not need to rebuild everything. Instead, i believe you should launch spark-submit by specifying the kafka jar file in your --packages... i had to follow same when integrating spark streaming with flume have you checked this link ? https://spark.apache.org/docs/latest/streaming-kafka-integration.html hth On Mon, Jul 25, 2016 at 10:20 AM, kevin wrote: > I have compile it from source code > > 2016-07-25 12:05 GMT+08:00 kevin : > >> hi,all : >> I try to run example org.apache.spark.examples.streaming.KafkaWordCount , >> I got error : >> Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/spark/streaming/kafka/KafkaUtils$ >> at >> org.apache.spark.examples.streaming.KafkaWordCount$.main(KafkaWordCount.scala:57) >> at >> org.apache.spark.examples.streaming.KafkaWordCount.main(KafkaWordCount.scala) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at >> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:724) >> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.spark.streaming.kafka.KafkaUtils$ >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> ... 11 more >> >> so where I can find spark-streaming-kafka for spark2.0 >> > >
Re: where I can find spark-streaming-kafka for spark2.0
Hi Kevin you should not need to rebuild everything. Instead, i believe you should launch spark-submit by specifying the kafka jar file in your --packages... i had to follow same when integrating spark streaming with flume have you checked this link ? https://spark.apache.org/docs/latest/streaming-kafka-integration.html hth On Mon, Jul 25, 2016 at 10:20 AM, kevin wrote: > I have compile it from source code > > 2016-07-25 12:05 GMT+08:00 kevin : > >> hi,all : >> I try to run example org.apache.spark.examples.streaming.KafkaWordCount , >> I got error : >> Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/spark/streaming/kafka/KafkaUtils$ >> at >> org.apache.spark.examples.streaming.KafkaWordCount$.main(KafkaWordCount.scala:57) >> at >> org.apache.spark.examples.streaming.KafkaWordCount.main(KafkaWordCount.scala) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at >> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:724) >> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.spark.streaming.kafka.KafkaUtils$ >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> ... 11 more >> >> so where I can find spark-streaming-kafka for spark2.0 >> > >
Re: where I can find spark-streaming-kafka for spark2.0
I have compile it from source code 2016-07-25 12:05 GMT+08:00 kevin : > hi,all : > I try to run example org.apache.spark.examples.streaming.KafkaWordCount , > I got error : > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/spark/streaming/kafka/KafkaUtils$ > at > org.apache.spark.examples.streaming.KafkaWordCount$.main(KafkaWordCount.scala:57) > at > org.apache.spark.examples.streaming.KafkaWordCount.main(KafkaWordCount.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:724) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: > org.apache.spark.streaming.kafka.KafkaUtils$ > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 11 more > > so where I can find spark-streaming-kafka for spark2.0 >