huan233usc opened a new pull request, #18092: URL: https://github.com/apache/iceberg/pull/18092
Spark 4.2 currently embeds initial-offset checkpoint persistence inside `SparkMicroBatchStream`. Extract it into a package-private `StreamingInitialOffsetStore` so other streaming sources can reuse the same checkpoint handling. The existing stream supplies its starting-offset calculation lazily, after checking for a persisted offset. The checkpoint location (`offsets/0`), JSON format, and use of Hadoop FileIO remain unchanged. A regression test verifies that reopening a checkpoint restores the saved offset without invoking the initializer again. This is the first preparatory refactor split from #17953 and can merge independently. Validation: `TestStreamingInitialOffsetStore`, `TestStreamingCheckpointHadoopIO`, and the checkpoint-resume and timestamp-start cases in `TestStructuredStreamingRead3`; Spark 4.2 Spotless check. --- **AI Disclosure** - Model: GPT-6 - Platform/Tool: Codex - Human Oversight: unreviewed - Prompt Summary: Extract the initial-offset store refactor from #17953 into a standalone prerequisite PR, including its existing streaming integration and regression test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
