binarybrain-009 opened a new pull request, #18494: URL: https://github.com/apache/iceberg/pull/18494
When running a structured streaming read with an unbounded read limit such as `Trigger.AvailableNow` or `ReadLimit.allAvailable()` and the default synchronous micro-batch planner (`streaming-async-micro-batch-planning-enabled=false`), `SyncSparkMicroBatchPlanner.latestOffset` previously unpacked `maxFiles` and `maxRows` to `Integer.MAX_VALUE` and walked every manifest and file scan task from the starting offset up to the latest snapshot. Because neither limit can trip during an unbounded run, this single-threaded driver walk performs redundant manifest I/O across the entire snapshot gap just to return the latest valid snapshot's added-file count, and `planFiles` immediately re-plans the same range afterwards. This change updates `SyncSparkMicroBatchPlanner.latestOffset` across Spark 3.5, 4.0, 4.1, and 4.2 to short-circuit when `limit instanceof ReadAllAvailable`, mirroring the existing behavior in `AsyncSparkMicroBatchPlanner.latestOffset`. When a `Trigger.AvailableNow` cap has already been computed, `latestOffsetForUnbounded` uses that pre-computed offset directly; otherwise it walks only the snapshot metadata chain via `nextValidSnapshot` (skipping rewrite and configured delete/overwrite snapshots) and constructs the end offset from `MicroBatchUtils.addedFilesCount(table(), lastValidSnapshot)`, returning `null` if the offset did not advance from `startingOffset`. This refreshes and extends the changes from stale PR #16941 to include Spark 4.2, and adds `TestStructuredStreamingRead3#testLatestOffsetForUnboundedLimitAdvancesToHeadSnapshot` across all supported Spark versions to verify that both the synchronous and asynchronous planners advance directly to the head snapshot and its added-file count under `ReadLimit.allAvailable()`. Fixes #16940. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
