[
https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865350#comment-15865350
]
Piotr Nestorow commented on SPARK-19579:
----------------------------------------
Are you stating that the Apache Spark Python API support is to be discontinued
??!!
> spark-submit fails to run Kafka Stream python script
> ----------------------------------------------------
>
> Key: SPARK-19579
> URL: https://issues.apache.org/jira/browse/SPARK-19579
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 2.1.0
> Environment: Linux Ubuntu 16.10 64bit
> Reporter: Piotr Nestorow
>
> Kafka Stream python script is executed but it fails with:
> TypeError: 'JavaPackage' object is not callable
> The Spark Kafka streaming jar is provided:
> spark-streaming-kafka-0-10_2.11-2.1.0.jar
> Kafka version: kafka_2.11-0.10.1.1
> In the conf/spark-defaults.conf:
> spark.jars.packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.1.0
> Also at runtime:
> bin/spark-submit --jars
> /usr/local/spark/jars/spark-streaming-kafka-0-10_2.11-2.1.0.jar
> --driver-class-path
> /usr/local/spark/jars/spark-streaming-kafka-0-10_2.11-2.1.0.jar
> /home/sysveradmin/work/Programs/ApacheSpark/ex_kafka_stream.py
> localhost:2181 example_topic
> Also the Spark Kafka streaming example:
> 'examples/src/main/python/streaming/kafka_wordcount.py'
> can be used to check this problem.
> Nothing helps...
> Detailed output:
> _________________
> Spark Streaming's Kafka libraries not found in class path. Try one of the
> following.
> 1. Include the Kafka library and its dependencies with in the
> spark-submit command as
> $ bin/spark-submit --packages
> org.apache.spark:spark-streaming-kafka-0-8:2.1.0 ...
> 2. Download the JAR of the artifact from Maven Central
> http://search.maven.org/,
> Group Id = org.apache.spark, Artifact Id =
> spark-streaming-kafka-0-8-assembly, Version = 2.1.0.
> Then, include the jar in the spark-submit command as
> $ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ...
> ________________________________________________________________________________________________
> Traceback (most recent call last):
> File "/home/sysveradmin/work/Programs/ApacheSpark/ex_kafka_stream.py", line
> 18, in <module>
> kvs = KafkaUtils.createStream(ssc, zkQuorum, "spark-streaming-consumer",
> {topic: 1})
> File "/usr/local/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py",
> line 69, in createStream
> File "/usr/local/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py",
> line 195, in _get_helper
> TypeError: 'JavaPackage' object is not callable
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
