ericyuan915 commented on PR #19376: URL: https://github.com/apache/hudi/pull/19376#issuecomment-5113729239
@danny0405 The offset approach doesn't carry over, unfortunately: DefaultHoodieSplitProvider.state() persists only pendingSplits, all stamped UNASSIGNED — assigned and completed splits aren't in the checkpoint at all. Streaming has one monotonic watermark that fully describes progress; a bounded read has a shrinking remainder set with no completion record, so there's no offset to resume from without adding a completed-split ledger. On warn-only — my worry is that it's near-invisible for a bounded read. Since createBatchHoodieSplits() is skipped when state is non-null, warning and continuing reads neither the old range nor the new one: it reads whatever fragment of the old range hadn't been dispatched at checkpoint time, then signals no-more-splits and the job exits successfully. A WARN line on a green batch job is hard to distinguish from the silent behavior this is meant to catch, whereas a streaming job at least keeps running with the new config. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
