Hello Akhil,
I chanced my Kafka dependency to 2.10 (which is the version of kafka that
was on 10.0.1.232). I am getting a slightly different error, but at the
same place as the previous error (pasted below).
FYI, when I make these changes to the pom file, I do "mvn clean package"
then cp the new jar files from the repository to my lib of jar files which
is a argument in my spark-submit script which is in my original post.
Thanks again for the time and help...much appreciated.
14/12/29 07:56:00 INFO ReceiverSupervisorImpl: Starting receiver
14/12/29 07:56:00 INFO KafkaReceiver: Starting Kafka Consumer Stream with
group: c1
14/12/29 07:56:00 INFO KafkaReceiver: Connecting to Zookeeper:
10.0.1.232:2181
14/12/29 07:56:00 INFO BlockGenerator: Started block pushing thread
14/12/29 07:56:00 INFO VerifiableProperties: Verifying properties
14/12/29 07:56:00 INFO VerifiableProperties: Property group.id is
overridden to c1
14/12/29 07:56:00 INFO VerifiableProperties: Property zookeeper.connect is
overridden to 10.0.1.232:2181
14/12/29 07:56:00 INFO VerifiableProperties: Property
zookeeper.connection.timeout.ms is overridden to 10000
14/12/29 07:56:00 INFO ReceiverSupervisorImpl: Stopping receiver with
message: Error starting receiver 0: java.lang.NoClassDefFoundError:
com/yammer/metrics/Metrics
14/12/29 07:56:00 INFO ReceiverSupervisorImpl: Called receiver onStop
14/12/29 07:56:00 INFO ReceiverSupervisorImpl: Deregistering receiver 0
14/12/29 07:56:00 ERROR ReceiverTracker: Deregistered receiver for stream
0: Error starting receiver 0 - java.lang.NoClassDefFoundError:
com/yammer/metrics/Metrics
at
kafka.metrics.KafkaMetricsGroup$class.newMeter(KafkaMetricsGroup.scala:51)
at
kafka.consumer.ZookeeperConsumerConnector.newMeter(ZookeeperConsumerConnector.scala:83)
at
kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:107)
at
kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:142)
at kafka.consumer.Consumer$.create(ConsumerConnector.scala:89)
at
org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:97)
at
org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121)
at
org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106)
at
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264)
at
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257)
at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
at org.apache.spark.scheduler.Task.run(Task.scala:54)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:180)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.ClassNotFoundException: com.yammer.metrics.Metrics
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 18 more
Suhas Shekar
University of California, Los Angeles
B.A. Economics, Specialization in Computing 2014
On Sun, Dec 28, 2014 at 11:52 PM, Suhas Shekar <suhsheka...@gmail.com>
wrote:
> I made both versions 1.1.1 and I got the same error. I then tried making
> both 1.1.0 as that is the version of my Spark Core, but I got the same
> error.
>
> I noticed my Kafka dependency is for scala 2.9.2, while my spark streaming
> kafka dependency is 2.10.x...I will try changing that next, but don't think
> that will solve the error as I dont think the application had got to level
> yet.
>
> Please let me know of any possible next steps.
>
> Thank you again for the time and the help!
>
>
>
> Suhas Shekar
>
> University of California, Los Angeles
> B.A. Economics, Specialization in Computing 2014
>
> On Sun, Dec 28, 2014 at 11:31 PM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> Just looked at the pom file that you are using, why are you having
>> different versions in it?
>>
>> <dependency>
>> <groupId>org.apache.spark</groupId>
>> <artifactId>spark-streaming-kafka_2.10</artifactId>
>> <version>*1.1.1*</version>
>> </dependency>
>> <dependency>
>> <groupId>org.apache.spark</groupId>
>> <artifactId>spark-streaming_2.10</artifactId>
>> <version>*1.0.2*</version>
>> </dependency>
>>
>> can you make both the versions the same?
>>
>>
>> Thanks
>> Best Regards
>>
>> On Mon, Dec 29, 2014 at 12:44 PM, Suhas Shekar <suhsheka...@gmail.com>
>> wrote:
>>
>>> 1) Could you please clarify on what you mean by checking the Scala
>>> version is correct? In my pom.xml file it is 2.10.4 (which is the same as
>>> when I start spark-shell).
>>>
>>> 2) The spark master URL is definitely correct as I have run other apps
>>> with the same script that use Spark (like a word count with a local file)
>>>
>>> Thank you for the help!
>>>
>>>
>>>
>>>
>>> Suhas Shekar
>>>
>>> University of California, Los Angeles
>>> B.A. Economics, Specialization in Computing 2014
>>>
>>> On Sun, Dec 28, 2014 at 11:04 PM, Akhil Das <ak...@sigmoidanalytics.com>
>>> wrote:
>>>
>>>> Make sure you verify the following:
>>>>
>>>> - Scala version : I think the correct version would be 2.10.x
>>>> - SparkMasterURL: Be sure that you copied the one displayed on the
>>>> webui's top left corner (running on port 8080)
>>>>
>>>> Thanks
>>>> Best Regards
>>>>
>>>> On Mon, Dec 29, 2014 at 12:26 PM, suhshekar52 <suhsheka...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello Everyone,
>>>>>
>>>>> Thank you for the time and the help :).
>>>>>
>>>>> My goal here is to get this program working:
>>>>>
>>>>> https://github.com/apache/spark/blob/master/examples/scala-2.10/src/main/java/org/apache/spark/examples/streaming/JavaKafkaWordCount.java
>>>>>
>>>>> The only lines I do not have from the example are lines 62-67. pom.xml
>>>>> <
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n20879/pom.xml
>>>>> >
>>>>>
>>>>> Background: Have ec2 instances running. The standalone spark is
>>>>> running on
>>>>> top of Cloudera Manager 5.2.
>>>>>
>>>>> Pom file is attached and the same for both clusters.
>>>>> pom.xml
>>>>> <
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n20879/pom.xml
>>>>> >
>>>>>
>>>>> Here are a few different approaches I have taken and the issues I run
>>>>> into:
>>>>>
>>>>> *Standalone Mode*
>>>>>
>>>>> 1) Use spark-submit script to run:
>>>>>
>>>>>
>>>>> /opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.1.p0.12/lib/spark/bin/spark-submit
>>>>> --class SimpleApp --master spark://10.0.1.230:7077 --jars $(echo
>>>>> /home/ec2-user/sparkApps/SimpleApp/lib/*.jar | tr ' ' ',')
>>>>> /home/ec2-user/sparkApps/SimpleApp/target/simple-project-1.0.jar
>>>>>
>>>>> Interesting...I was getting an error like this: Initial job has not
>>>>> accepted
>>>>> any resources; check your cluster UI
>>>>>
>>>>> Now, when I run, it prints out the 3 Hello world statements in my code:
>>>>> KafkaJavaConsumer.txt
>>>>> <
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n20879/KafkaJavaConsumer.txt
>>>>> >
>>>>>
>>>>> and then it seems to try to start the Kafka Stream, but fails:
>>>>>
>>>>> 14/12/29 05:58:05 INFO KafkaReceiver: Starting Kafka Consumer Stream
>>>>> with
>>>>> group: c1
>>>>> 14/12/29 05:58:05 INFO ReceiverTracker: Registered receiver for stream
>>>>> 0
>>>>> from akka://sparkDriver
>>>>> 14/12/29 05:58:05 INFO KafkaReceiver: Connecting to Zookeeper:
>>>>> 10.0.1.232:2181
>>>>> 14/12/29 05:58:05 INFO BlockGenerator: Started block pushing thread
>>>>> 14/12/29 05:58:05 INFO ReceiverSupervisorImpl: Stopping receiver with
>>>>> message: Error starting receiver 0: java.lang.NoClassDefFoundError:
>>>>> scala/reflect/ClassManifest
>>>>> 14/12/29 05:58:05 INFO ReceiverSupervisorImpl: Called receiver onStop
>>>>> 14/12/29 05:58:05 INFO ReceiverSupervisorImpl: Deregistering receiver 0
>>>>> ^C14/12/29 05:58:05 ERROR ReceiverTracker: Deregistered receiver for
>>>>> stream
>>>>> 0: Error starting receiver 0 - java.lang.NoClassDefFoundError:
>>>>> scala/reflect/ClassManifest
>>>>> at
>>>>> kafka.utils.Log4jController$.<init>(Log4jController.scala:29)
>>>>> at kafka.utils.Log4jController$.<clinit>(Log4jController.scala)
>>>>> at kafka.utils.Logging$class.$init$(Logging.scala:29)
>>>>> at
>>>>> kafka.utils.VerifiableProperties.<init>(VerifiableProperties.scala:26)
>>>>> at
>>>>> kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:94)
>>>>> at
>>>>>
>>>>> org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:96)
>>>>> at
>>>>>
>>>>> org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121)
>>>>> at
>>>>>
>>>>> org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106)
>>>>> at
>>>>>
>>>>> org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264)
>>>>> at
>>>>>
>>>>> org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257)
>>>>> at
>>>>>
>>>>> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
>>>>> at
>>>>>
>>>>> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
>>>>> at
>>>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:54)
>>>>> at
>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:180)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>> at java.lang.Thread.run(Thread.java:722)
>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>> scala.reflect.ClassManifest
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>>>>> ... 18 more
>>>>>
>>>>> 14/12/29 05:58:05 INFO ReceiverSupervisorImpl: Stopped receiver 0
>>>>> 14/12/29 05:58:05 INFO BlockGenerator: Stopping BlockGenerator
>>>>>
>>>>> I ran into a couple other Class not found errors, and was able to
>>>>> solve them
>>>>> by adding dependencies on the pom file, but have not found such a
>>>>> solution
>>>>> to this error.
>>>>>
>>>>> On the Kafka side of things, I am simply typing in messages as soon as
>>>>> I
>>>>> start the Java app on another console. Is this okay?
>>>>>
>>>>> I have not set up an advertised host on the kafka side as I was able to
>>>>> still receive messages from other consoles by setting up a consumer to
>>>>> listen to the private ip:port. Is this okay?
>>>>>
>>>>> Lastly, is there command, like --from-beginning for a consumer in the
>>>>> java
>>>>> application to get messages from the beginning?
>>>>>
>>>>> Thanks a lot for the help and happy holidays!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Setting-up-Simple-Kafka-Consumer-via-Spark-Java-app-tp20879.html
>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>> Nabble.com.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>>>
>>>>>
>>>>
>>>
>>
>
