cshuo opened a new pull request, #20396: URL: https://github.com/apache/hudi/pull/20396
### Describe the issue this Pull Request addresses Log compaction reads deletes through the engine record context. On Spark, rewriting string- or decimal-ordered deletes into inline logs fails because `HoodieEmptyRecord` exposes engine values to the Java-based delete serializer. For native logs, using a different record context to construct delete rows can send Spark ordering values to an Avro writer and lose deletes. This PR addresses the scenarios covered by [apache/hudi#20366](https://github.com/apache/hudi/pull/20366) through reader/writer context alignment and lazy Java ordering conversion. **Dependency:** [apache/hudi#20373](https://github.com/apache/hudi/pull/20373) is required for complete native delete support: converting comparable ordering values to their stored form, unwrapping Spark strings, and mapping null components. Those changes are not duplicated here. ### Summary and Changelog - Use the file group reader's record context to construct and write native log compaction deletes; retain the existing data writer selection. - Implement `HoodieEmptyRecord.getOrderingValueAsJava` using shared, stateless Spark and Flink converter singletons, while retaining engine values for comparisons. - Preserve converters across key copies and serialization. Reuse existing conversion utilities, including Flink schema- and timezone-aware conversion. - Add focused coverage to existing test classes, including the six log compaction regression cases from apache/hudi#20366. ### Impact - Fixes Spark inline log compaction for string and decimal ordering on table versions 8 and 9, and native decimal delete compaction on version 10 for both AVRO and SPARK mergers. - Avoids allocating a converter per empty record or retaining reader context in each record. - Adds Java ordering conversion for Flink empty records, including nested decimal and timestamp fields. ### Risk Level Medium: this changes native delete writer context selection and the Kryo representation of `HoodieEmptyRecord` (including an additional byte when no converter is present). Complete native string/composite-ordering support depends on apache/hudi#20373. Validation: - Re-ran all six unmodified apache/hudi#20366 E2E cases on this commit with Spark 3.5 / Java 11: 6 passed, 0 failures, 0 skipped. - Focused Spark, Flink, common, and serialization tests passed after test consolidation (37 tests total). - Java Checkstyle, Scala Scalastyle, and `git diff --check` passed. ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
