Github user QuentinAmbard commented on the issue:
https://github.com/apache/spark/pull/21917
I'm not sure to understand your point. The cause of the gap doesn't matter,
we just want to stop on an existing offset to be able to poll it. It can be
because of a transaction marker, a transaction abort or even just a temporary
poll failure it's not relevant in this case.
The driver is smart enough to be able to restart from any Offset, even in
the middle of a transaction (abort or not)
The issue with gap at the end is that you can't know if it's a gap or if
the poll failed.
For example SeekToEnd gives you 5 but the last record you get is 3 and
there is no way to know if 4 is missing or just an offset gap.
How could we fix that in a different way?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
