Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/14026#discussion_r70158520
--- Diff:
external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
---
@@ -79,8 +81,71 @@ private case class Subscribe[K, V](
def onStart(currentOffsets: ju.Map[TopicPartition, jl.Long]):
Consumer[K, V] = {
val consumer = new KafkaConsumer[K, V](kafkaParams)
consumer.subscribe(topics)
- if (currentOffsets.isEmpty) {
- offsets.asScala.foreach { case (topicPartition, offset) =>
+ val toSeek = if (currentOffsets.isEmpty) {
+ offsets
+ } else {
+ currentOffsets
--- End diff --
Basically, yeah.
On Jul 8, 2016 7:38 PM, "Tathagata Das" <notificati...@github.com> wrote:
> In
>
external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
> <https://github.com/apache/spark/pull/14026#discussion_r70158281>:
>
> > @@ -79,8 +81,71 @@ private case class Subscribe[K, V](
> > def onStart(currentOffsets: ju.Map[TopicPartition, jl.Long]):
Consumer[K, V] = {
> > val consumer = new KafkaConsumer[K, V](kafkaParams)
> > consumer.subscribe(topics)
> > - if (currentOffsets.isEmpty) {
> > - offsets.asScala.foreach { case (topicPartition, offset) =>
> > + val toSeek = if (currentOffsets.isEmpty) {
> > + offsets
> > + } else {
> > + currentOffsets
>
> guess it wasnt designed for rapidly seeking back and forth. so to confirm
> my understanding, these are the following calls in Subscribe
>
> - poll - to get the initial assignments. handles none case of no
> committed offset
> - seek - set the position to whatever is known from user-provided
> offsets or current offsets, so that later regular polls in the DStream
can
> proceed with as little difficulty as possible.
>
> right?
>
> â
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
>
<https://github.com/apache/spark/pull/14026/files/796045ff3ad53afeb56cdddb69c4770090f7c168#r70158281>,
> or mute the thread
>
<https://github.com/notifications/unsubscribe/AAGAB8pP18jSrIgoWLnugKDNUnTrlRuwks5qTu2FgaJpZM4JDmrK>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
