wombatu-kun opened a new pull request, #20373: URL: https://github.com/apache/hudi/pull/20373
### Describe the issue this Pull Request addresses Closes #20372 On table version 10 with the Spark record type, a delete with a string ordering field fails the write, and native log compaction silently drops such deletes. The native delete log writer put the comparable `HoodieUTF8String` wrapper into the delete row, which the Spark Parquet writer rejects. With several ordering fields, a null field failed the same way because the writer stored the default ordering value instead of NULL. ### Summary and Changelog - `HoodieNativeLogFormatWriter` converts each ordering value with `RecordContext.convertValueToEngineType` before it builds the delete row, as it already did for the record key. A null field in a multi-field ordering is written as NULL, as a single null field already was. - `BaseSparkInternalRecordContext.convertValueToEngineType` unwraps `HoodieUTF8String` to the raw `UTF8String`, and the `RecordContext` javadoc now says the method accepts the comparable form. - `TestMORDataSource` deletes a record on a version 10 table, runs native log compaction, and checks that an older update leaves the key deleted while a newer one brings it back. The rows cover both record types, string and decimal ordering, two ordering fields with a null second value, and Avro-typed writes compacted by a Spark-typed job. - `TestHoodieNativeLogFormatWriter` and `TestBaseSparkInternalRowReaderContext` cover the conversion of each ordering value and the unwrap. ### Impact Spark-typed writes and log compaction on version 10 tables, the master default, keep string-ordered deletes again. Values that engines already store as they are, such as Spark decimals and longs, are written unchanged. With the Avro record type, a `String` becomes an Avro `Utf8`, which the Avro writers already accept for the record key. Not covered: with the Avro record type, native log compaction still loses deletes with a decimal ordering field, because it reads them back as Spark `Decimal`s. That needs a separate fix on top of #20366. ### Risk Level low Only the values written into native delete logs change, and version 10 is not released yet. Verified: on master the new test fails at the delete write with `ClassCastException: Spark3HoodieUTF8String cannot be cast to UTF8String`, and the Avro-written row loses its delete in log compaction. Removing the unwrap, either writer conversion, or the NULL mapping brings a failure back. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
