I am trying to read stored tweets in a file and send it to Kafka using Spark
python.
The code is very simple but it does not work.
The spark job runs correctly but nothing sent to Kafka
Here is the code
/#!/usr/bin/python
# -*- coding: utf-8 -*-
from pyspark import SparkContext, SparkConf
from kafka import KafkaProducer
import sys
reload(sys)
sys.setdefaultencoding('utf8')
topic = "testTopic"
if __name__ == "__main__":
conf = SparkConf().setAppName("TestSparkFromPython")
sc = SparkContext(conf=conf)
producer = KafkaProducer(bootstrap_servers="10.62.54.111:9092")
tweets =
sc.textFile("/home/fanooos/Desktop/historical_scripts/output/1/activities_201603270430_201603270440.json")
tweetsCollection = tweets.collect()
for tweet in tweetsCollection:
producer.send('testTopic', value=bytes(tweet))/
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Sending-events-to-Kafka-from-spark-job-tp26622.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
