danny0405 opened a new pull request, #18776: URL: https://github.com/apache/hudi/pull/18776
### Describe the issue this Pull Request addresses Several writer construction and write-handle failure paths can leave underlying file writers or output streams open when an exception is thrown before the normal close path runs. This affects create, append, merge, binary copy, stream writer, HFile bootstrap index, Parquet utility, and Spark helper code paths. This PR makes those failure paths close the relevant writer resources, preserving the original failure while adding close failures as suppressed exceptions where applicable. It does not change storage format, public APIs, or write configuration defaults. ### Summary and Changelog This change improves resource cleanup for file writers and output streams when write initialization or write/close logic fails, and consolidates repeated close-with-suppression logic in client write handles. #### Commit 1: fix: close the file writers properly in fail cases (`0f4784f2c99`) - Close `HoodieFileWriter`/log writer resources when `BaseCreateHandle`, `HoodieAppendHandle`, and `HoodieWriteMergeHandle` encounter fail-fast write or close errors. - Move `HoodieSortedMergeHandle` pending-record writing into `writeIncomingRecords()` so the base merge close path can handle writer cleanup consistently. - Close `HoodieBinaryCopyHandle` copier on binary-copy failures. - Close output streams if Parquet/HFile stream writer construction fails in Spark, Flink, Hadoop, and HFile writer paths. - Use try-with-resources in `ParquetUtils.serializeRecordsToLogBlock`. - Ensure `SparkHelpers` closes `HoodieAvroParquetWriter` in a `finally` block. - Harden `HFileBootstrapIndexWriter` begin/close handling so partially initialized writers and streams are closed. - Harden `HoodieParquetBinaryCopyBase` initialization, close, and column writer cleanup paths. - Add tests for create-handle writer cleanup and Parquet binary-copy close behavior. #### Commit 2: add common utils (`a7da5bc4d5`) - Add `org.apache.hudi.util.AutoClosableUtils` in `hudi-client-common` for closing `AutoCloseable` resources while preserving suppressed exceptions. - Reuse the utility in `BaseCreateHandle`, `HoodieAppendHandle`, and `HoodieWriteMergeHandle`. - Update `testFileWriterClosedWhenDoWriteFails` to explicitly run with `hoodie.write.ignore.failed=false`, matching the fail-fast path under test. ### Impact No public API, configuration, storage format, or compatibility changes are introduced. The user-visible behavior is safer cleanup on exceptional writer paths, reducing leaked file writers/output streams and preserving close failures as suppressed exceptions where applicable. There is no expected performance impact on successful write paths beyond small helper calls during close. ### Risk Level medium The changes touch core write-handle close paths and multiple writer initialization paths, so lifecycle regressions are possible if a writer has unusual close semantics. The risk is mitigated by keeping normal close behavior intact, nulling writer references after close attempts, and adding targeted tests for failure cleanup. Validation run: - `git diff --check` - `mvn -pl hudi-client/hudi-client-common -am -DskipTests -DskipITs -Dcheckstyle.skip -Dspotbugs.skip compile` - `mvn -pl hudi-client/hudi-client-common -am -DskipITs -Dcheckstyle.skip -Dspotbugs.skip -Dtest=TestHoodieCreateHandle#testFileWriterClosedWhenDoWriteFails -Dsurefire.failIfNoSpecifiedTests=false test` - `mvn -pl hudi-hadoop-common -DskipITs -Dcheckstyle.skip -Dspotbugs.skip -Dtest=TestHoodieParquetBinaryCopyBaseSchemaEvolution test` ### Documentation Update none This is an internal resource-cleanup fix and utility refactor with no user-facing configuration, API, or behavior that requires documentation updates. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
