Github user QuentinAmbard commented on the issue:
https://github.com/apache/spark/pull/21917
> By failed, you mean returned an empty collection after timing out, even
though records should be available? You don't. You also don't know that it
isn't just lost because kafka skipped a message. AFAIK from the information you
have from a kafka consumer, once you start allowing gaps in offsets, you don't
know.
Ok that's interesting, my understanding was that if you successfully poll
and get results you are 100% sure that you don't lose anything. Do you have
more details on that? Why would kafka skip a record while consuming?
> Have you tested comparing the results of consumer.endOffsets for
consumers with different isolation levels?
endOffsets returns the last offset (same as seekToEnd). But you're right
that the easiest solution for us would be to have something like
seekToLastRecord method instead. Maybe something we could also ask ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
