[
https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331007#comment-15331007
]
Jinxia Liu commented on SPARK-12177:
------------------------------------
[[email protected]] thanks for contributing the connector for kafka0.9 and
kafka0.10.
I used your kafka0.10 connector and ran into some problems, would you mind
looking at them?
1. when build using "mvn clean package", there is error about not passing the
test case in DirectKafkaStreamSuite:
offset recovery *** FAILED ***
The code passed to eventually never returned normally. Attempted 196 times
over 10.031047939 seconds. Last failure message: 55 did not equal 210.
(DirectKafkaStreamSuite.scala:337)
2. another problem is(with kafka0.9 connector as well), can we add a wrapper,
something like CreateDirectKafkaStream in kafka0.8 connector, to wrap up the
DirectKafkaStream constructor?
The benefit is that user does not need to know the kafka consumer APIs, in
order to use the connector.
E.g.: the kafka consumer in the connector gets assigned a collection of
TopicPartition, in most cases, all the partitions for given topic, if no
wrapper, user needs to exploit the kafka consumer API to first retrieve the
partitionInfo. Using the wrapper, user only needs to provide the topics, and
such info can be passed to consumer inside the wrapper without the users
knowledge.
> Update KafkaDStreams to new Kafka 0.10 Consumer API
> ---------------------------------------------------
>
> Key: SPARK-12177
> URL: https://issues.apache.org/jira/browse/SPARK-12177
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.6.0
> Reporter: Nikita Tarasenko
> Labels: consumer, kafka
>
> Kafka 0.9 already released and it introduce new consumer API that not
> compatible with old one. So, I added new consumer api. I made separate
> classes in package org.apache.spark.streaming.kafka.v09 with changed API. I
> didn't remove old classes for more backward compatibility. User will not need
> to change his old spark applications when he uprgade to new Spark version.
> Please rewiew my changes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
