I write code to connect kafka with spark using python and I run code on
jupyer
my code
import os
#os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars
/home/hadoop/Desktop/spark-program/kafka/spark-streaming-kafka-0-8-assembly_2.10-2.0.0-preview.jar
pyspark-shell'
os.environ['PYSPARK_SUBMIT_ARGS'] = "--packages
org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.0 pyspark-shell"
os.environ['PYSPARK_SUBMIT_ARGS'] = "--packages
org.apache.spark:spark-streaming-kafka-0-8_2.11:2.3.0 pyspark-shell"
import pyspark
from pyspark.streaming.kafka import KafkaUtils
from pyspark.streaming import StreamingContext
#sc = SparkContext()
ssc = StreamingContext(sc,1)
broker = "iotmsgs"
directKafkaStream = KafkaUtils.createDirectStream(ssc, ["test1"],
{"metadata.broker.list": broker})
directKafkaStream.pprint()
ssc.start()
error display
Spark Streaming's Kafka libraries not found in class path. Try one of the
following.
1. Include the Kafka library and its dependencies with in the
spark-submit command as
$ bin/spark-submit --packages
org.apache.spark:spark-streaming-kafka-0-8:2.3.0 ...
2. Download the JAR of the artifact from Maven Central
http://search.maven.org/,
Group Id = org.apache.spark, Artifact Id =
spark-streaming-kafka-0-8-assembly, Version = 2.3.0.
Then, include the jar in the spark-submit command as
$ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ...
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
