danny0405 commented on PR #20516: URL: https://github.com/apache/hudi/pull/20516#issuecomment-6648646913
Addressed the five numbered items in [this review](https://github.com/apache/hudi/pull/20516#pullrequestreview-5777467852) in fb2e9b48cc6: 1. Assign the source inside `withSQLConf` instead of relying on its return value. All five failed GitHub checks showed the same Spark 3.5 `getBatch is not a member of Unit` compilation error. 2. Hoist first-write, checkpoint, unread-write, and requested-instant lookups outside the cleaner loop. Skip old cleans in the normal case. Retain older cleans when a late-completing write may still need their history, or the initial requested-time bound is unknown; filtering only on completion time would miss that case. The regression includes an old clean with unreadable metadata, proving it is skipped for a newer checkpoint. 3. Document the conservative `KEEP_LATEST_FILE_VERSIONS` limitation beside the config and in SOURCE_SAFETY.md, and name it in the error. 4. Name `hoodie.keep.min.commits` / `hoodie.keep.max.commits` in both config and recovery documentation. 5. Document that #20378's non-clustering truncate replacecommit is expected to require a downstream rebuild. Validation: Spark 3.5 / Scala 2.12 / JDK 11 reactor install, test compilation and style checks passed. Full `TestStreamingSource`: **20 passed, 0 failed, 2 canceled** by the existing Spark >= 4.1 shredded-variant assumption. `git diff --check` passed. GitHub CI needs to run on the new commit; the earlier Spark 4.1 results are not a rerun of this follow-up. On the separate recovery-strategy proposal: this commit retains the opt-in boolean and existing default. I suggest settling the default/API contract separately before adding automatic recovery. A snapshot batch can duplicate append rows or leave stale upsert keys, and `READ_LATEST` deliberately loses the gap, so neither replaces coordinated table/checkpoint recovery. A move to default `FAIL` for all structured-streaming readers also needs release-note coverage (or an explicitly defined SDP-only gate). That design question remains open; this commit does not implement the three-strategy API. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
