[
https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174415#comment-15174415
]
Cody Koeninger commented on SPARK-12177:
----------------------------------------
Mansi are you talking about performance improvements from caching the new
consumer, or from caching the old simple consumer? I had a branch of direct
stream that cached the old simple consumer, but in my testing it didn't make
enough of a difference to be worth the added complexity.
Regarding throwing out messages beyond the untilOffset, I agree, that's why I'm
saying the iterator would need to be redesigned. I don't think it needs to
communicate back to the driver, it needs to cache those messages locally, then
we can (hopefully) use getPreferredLocations to encourage future requests for
that partition to be scheduled on that executor.
> Update KafkaDStreams to new Kafka 0.9 Consumer API
> --------------------------------------------------
>
> Key: SPARK-12177
> URL: https://issues.apache.org/jira/browse/SPARK-12177
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.6.0
> Reporter: Nikita Tarasenko
> Labels: consumer, kafka
>
> Kafka 0.9 already released and it introduce new consumer API that not
> compatible with old one. So, I added new consumer api. I made separate
> classes in package org.apache.spark.streaming.kafka.v09 with changed API. I
> didn't remove old classes for more backward compatibility. User will not need
> to change his old spark applications when he uprgade to new Spark version.
> Please rewiew my changes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
