Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/22042#discussion_r209476712
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -347,9 +391,12 @@ private[kafka010] case class InternalKafkaConsumer(
}
private def poll(pollTimeoutMs: Long): Unit = {
+ offsetBeforePoll = consumer.position(topicPartition)
--- End diff --
This variable `offsetBeforePoll` seems to be only used to identify whether
data was actually fetched in a poll and nothing else. Rather than define
another var (there are already many that already confusing), why not just
return a boolean from poll which is true or false depending on whether poll
moved anything.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
