HyukjinKwon commented on PR #59942: URL: https://github.com/apache/spark/pull/59942#issuecomment-6522052208
On the two questions about the `latest_offset` call that only serves the progress, following the [earlier discussion](https://github.com/apache/spark/pull/59942#discussion_r4418390279): 1. Done in 0b94e5b: `ColumnarMicroBatchStream.latestOffset` only asks with the limit when the latest offset without a limit is not the start, as Spark compares offsets. So an idle poll makes a single call, as without a limit, and so does the last check of a run of `Trigger.AvailableNow`, once it reached its end; only the micro-batches that read data make two. The header (Streaming reads), the guide and `ColumnarStreamReader` say so. The test library now records the start, the limit and the end of the run of each call (`limits_path`), and the tests check the exact calls for the default trigger, for `Trigger.AvailableNow`, also after a restart, and for `Trigger.Once`. 2. The two calls stay, without an ABI change. The latest offset without a limit is what Spark asks every source for, also without admission control, and the progress reports it, so reporting the limited end, or no latest offset, would lose the backlog that it shows. A library for which that offset is costly, such as one that lists an object store, does not have to page through everything after `start` each time: the limited call comes right after it with the same `start`, so it can reuse that listing, and across micro-batches it can keep what it listed and only list what is new. If a library still needs both offsets from a single call, a sink for the latest offset without the limit can be added within version 1 as a new capability: Spark only uses a capability with the data sources that declare it (Compatibility), so the order of the calls stays as documented for the others, and the choice does not have to be made before 4.4.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
