Davis-Zhang-Onehouse opened a new pull request, #20709:
URL: https://github.com/apache/hudi/pull/20709

   ### Describe the issue this Pull Request addresses
   
   issue: #20708
   Part of #20683, RFC-118 (#20684).
   Stacked on #20707: review only the last commit. Until the PRs below it 
merge, this PR also shows their commits.
   
   Rollback of out-of-line blob files by markers, copy failure classification, 
the inline ceiling and Parquet settings for large binary columns. A failed blob 
round no longer leaves unreferenced copies: each out-of-line file gets a marker 
of the round's instant, and the round is moved to INFLIGHT before the copy so a 
rollback deletes them. The copy retry and failure classes are removed again by 
follow-up sub-issue 37 of the plan; marker rollback stays for the embedding 
table's stored inputs.
   
   ### Summary and Changelog
   
   - New `hoodie.write.out.of.line.markers.enable` (default true) in 
`HoodieWriteConfig`: each out-of-line file under `.hoodie/.blob/` gets a 
timeline-server CREATE marker of the write instant, and a rollback of a failed 
write deletes it. When on, the writer requires timeline-server markers and 
marker-based rollback. `HoodieTable` leaves managed blob paths out of the 
data-file write stats.
   - `StreamSync` / `HoodieStreamerConfig`: the round's instant can go INFLIGHT 
before the transformer runs, so files a transformer writes follow the marker 
protocol and are rolled back with a failed round. Off unless a source contract 
asks for it.
   - Blob copy: retry settings, copy failure classes for source and 
row-building failures, GCS access-denial handling and handle release.
   - Inline ceiling for blob rows and Parquet settings for large binary 
columns: `HoodieBaseParquetWriter` honours per-column 
`parquet.column.statistics.enabled#<col>` (Parquet 1.14+, refused on older 
Parquet) and `parquet.enable.dictionary#<col>`, explicit page-size-check 
cadence (skipped with a warning before Parquet 1.12.3), and an internal size 
check on every record. `HoodieInternalRowFileWriterFactory` applies the 
configured Parquet config injector on the row-writer path, as the record writer 
already does.
   - 33 test classes, including failed-copy cleanup, the early-instant contract 
and Parquet page bounds.
   
   Reader, projection and listener changes for BLOB reads are in #20688, not 
here. No code was copied from other projects.
   
   ### Impact
   
   One new advanced write config (`hoodie.write.out.of.line.markers.enable`, 
default true) that only affects writers creating out-of-line files. 
`HoodieBaseParquetWriter` gains opt-in per-column keys; writers that do not set 
them keep their current behaviour. Row-writer bulk inserts and clustering now 
honour a configured Parquet config injector.
   
   ### Risk Level
   
   high: touches `StreamSync`, `HoodieTable`, `HoodieWriteConfig` and the 
shared Parquet writers. The new behaviour is opt-in or limited to out-of-line 
files; default paths are unchanged. Verified on JDK 11 / Spark 3.5 / Scala 2.12 
with 309 tests across hudi-hadoop-common, hudi-spark and hudi-utilities (none 
skipped), module compilation, checkstyle and RAT. A failed-copy cleanup test 
fails without the marker rollback and passes with it.
   
   ### Documentation Update
   
   The new write config and the blob settings carry their config documentation. 
RFC-118 describes rollback; a website page follows once the chain lands.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] feat(ingest): RFC-118: ... Davis-Zhang-Onehouse (via GitHub) via commits

Reply via email to