danny0405 commented on PR #20516:
URL: https://github.com/apache/hudi/pull/20516#issuecomment-6648646913

   Addressed the five numbered items in [this 
review](https://github.com/apache/hudi/pull/20516#pullrequestreview-5777467852) 
in fb2e9b48cc6:
   
   1. Assign the source inside `withSQLConf` instead of relying on its return 
value. All five failed GitHub checks showed the same Spark 3.5 `getBatch is not 
a member of Unit` compilation error.
   2. Hoist first-write, checkpoint, unread-write, and requested-instant 
lookups outside the cleaner loop. Skip old cleans in the normal case. Retain 
older cleans when a late-completing write may still need their history, or the 
initial requested-time bound is unknown; filtering only on completion time 
would miss that case. The regression includes an old clean with unreadable 
metadata, proving it is skipped for a newer checkpoint.
   3. Document the conservative `KEEP_LATEST_FILE_VERSIONS` limitation beside 
the config and in SOURCE_SAFETY.md, and name it in the error.
   4. Name `hoodie.keep.min.commits` / `hoodie.keep.max.commits` in both config 
and recovery documentation.
   5. Document that #20378's non-clustering truncate replacecommit is expected 
to require a downstream rebuild.
   
   Validation: Spark 3.5 / Scala 2.12 / JDK 11 reactor install, test 
compilation and style checks passed. Full `TestStreamingSource`: **20 passed, 0 
failed, 2 canceled** by the existing Spark >= 4.1 shredded-variant assumption. 
`git diff --check` passed. GitHub CI needs to run on the new commit; the 
earlier Spark 4.1 results are not a rerun of this follow-up.
   
   On the separate recovery-strategy proposal: this commit retains the opt-in 
boolean and existing default. I suggest settling the default/API contract 
separately before adding automatic recovery. A snapshot batch can duplicate 
append rows or leave stale upsert keys, and `READ_LATEST` deliberately loses 
the gap, so neither replaces coordinated table/checkpoint recovery. A move to 
default `FAIL` for all structured-streaming readers also needs release-note 
coverage (or an explicitly defined SDP-only gate). That design question remains 
open; this commit does not implement the three-strategy API.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to