[
https://issues.apache.org/jira/browse/SPARK-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386070#comment-14386070
]
q79969786 commented on SPARK-6594:
----------------------------------
1. Create kafka topic as follow:
$kafka-topics.sh --create --zookeeper
zkServer1:2181,zkServer2:2181,zkServer3:2181 --replication-factor 1
--partitions 5 --topic ORDER
2. I'm use Java API to process data as follow:
SparkConf sparkConf = new SparkConf().setAppName("TestOrder");
sparkConf.set("spark.cleaner.ttl", "600");
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new
Duration(1000));
Map<String, Integer> topicorder = new HashMap<String, Integer>();
topicorder.put("order", 5);
JavaPairReceiverInputDStream<String, String> jPRIDSOrder =
KafkaUtils.createStream(jssc, ’zkServer1:2181,zkServer2:2181,zkServer3:2181‘,
’test-consumer-group‘, topicorder);
jPRIDSOrder.map(new Function<Tuple2<String, String>, String>() {
@Override
public String call(Tuple2<String, String> tuple2) {
return tuple2._2();
}
}).print();
3. Submit this application as follow:
spark-submit --class com.bigdata.TestOrder --master spark://SPKMASTER:19002
/home/bigdata/test-spark.jar TestOrder
4. It will shows five warnings as follows when submit application:
15/03/29 21:23:03 WARN ZookeeperConsumerConnector:
[test-consumer-group_work1-1427462582342-5714642d], No broker partitions
consumed by consumer thread test-consumer-group_work1-1427462582342-5714642d-0
for topic ORDER
......
> Spark Streaming can't receive data from kafka
> ---------------------------------------------
>
> Key: SPARK-6594
> URL: https://issues.apache.org/jira/browse/SPARK-6594
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.2.1
> Environment: kafka_2.10-0.8.1.1 + Spark-1.2.1
> Reporter: q79969786
>
> I use KafkaUtils to receive data from Kafka In my Spark streaming application
> as follows:
> Map<String, Integer> topicorder = new HashMap<String, Integer>();
> topicorder.put("order", Integer.valueOf(readThread));
> JavaPairReceiverInputDStream<String, String> jPRIDSOrder =
> KafkaUtils.createStream(jssc, zkQuorum, group, topicorder);
> It worked well at fist, but after I submit this application several times,
> Spark streaming can‘t receive data anymore(Kafka works well).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
