HeartSaVioR commented on PR #38898:
URL: https://github.com/apache/spark/pull/38898#issuecomment-1341979338
I'm trying to understand the case - if my understanding is correct, the new
test is just to trigger the same behavior rather than reproducing actual
problem, right? In the new test, recognizing all topic partitions as new one
and process all records in next microbatch is arguably NOT a wrong behavior for
me, hence I really would like to understand the actual problem.
According to the JIRA description, the actual problem is that Kafka can
"transiently" give no topic partition as assignment when it performs
reassignment among consumers, specifically here:
```
consumer.poll(0)
val partitions = consumer.assignment()
```
which we expect Kafka to assign topic partitions to this consumer
accordingly after calling poll.
Do I understand correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
