Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21676
The offset log contains the watermark value that is going to be used in the
batch corresponding to that offset. For example, "checkpoint/offsets/10" will
contain the watermark value to be used for batch 10. The problem is that when
batch 10 completes and new watermark values is computed, it is not saved in a
persistent location until batch 11 is planned and "offsets/11" is written out.
In trigger.once, this never happens as the query is terminated as soon as batch
10 completes. So the new watermark value is not saved. If the query running in
trigger.once mode right from the beginning, that is batch 0, then no new
watermark value is ever written, and so the watermark shows up always as 0.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
