ravjot28 opened a new pull request, #19872: URL: https://github.com/apache/hudi/pull/19872
### Describe the issue this Pull Request addresses An Avro payload decoded using a projected schema can lose or misassign values when the schema used for decoding differs from the schema that encoded the record. Repeated projections, metadata-declaring schemas, nested renames, and serialization round trips expose this problem in merge paths. This is the first prerequisite for the standalone Java work in [discussion #19798](https://github.com/apache/hudi/discussions/19798) and the [dev-list proposal](https://www.mail-archive.com/dev%40hudi.apache.org/msg06536.html). ### Summary and Changelog - Retain the actual writer schema and resolve named projections, aliases, defaults, and supported legacy positional renames without shifting unrelated fields. - Preserve the writer schema through Java serialization and default Kryo serialization. Keep the pre-change Java serialization ID and accept legacy streams. - Select the original compact payload representation in Spark's Kryo registrar. Keep the existing Spark size assertion unchanged. - Add projection/evolution tests, independent serializer and graph-reset tests, tombstones, and golden legacy Java streams. The change uses existing Hudi Avro rewrite utilities. No third-party source or dependency is added. ### Impact Shared payload decoding affects every engine using these payloads. Java serialization retains the legacy class identity. Spark's registered Kryo wire representation remains unchanged. **Compatibility change:** default non-Spark Kryo writers now emit a schema-bearing representation with a negative-length discriminator. Updated readers accept both representations; old readers cannot read the new one. `BaseAvroPayload.useLegacyKryoFormat(kryo)` selects the old representation for transports supplying the writer schema separately. That selection sacrifices self-contained schema retention. Mixed-version users must upgrade readers first or explicitly retain the legacy transport mode; arbitrary mixed-version compatibility is not claimed. No new table storage layout/version is introduced. ### Risk Level high This affects shared merge behavior and an in-memory serialization format. The isolated payload branch builds and passes 13 payload tests and three Spark registrar tests on Java 11. Earlier validation of the combined implementation also passed the unchanged Spark record serialization tests and selected Hive evolution tests. The [combined fork CI](https://github.com/ravjot28/hudi/actions/runs/34168606289) passed at `e701bebf6c`; it is not CI evidence for this newly split commit. The split PR must run its own CI. ### Documentation Update A companion change for the `asf-site` branch documents payload transport compatibility in `website/docs/java-client.md`, alongside the Java-client usage guide. It will be submitted separately and linked before merge. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
