ravjot28 opened a new pull request, #19872: URL: https://github.com/apache/hudi/pull/19872
### Describe the issue this Pull Request addresses An Avro payload decoded with a projected schema can lose or misassign values when the requested schema differs from the schema that encoded the record. Repeated projections, metadata-declaring schemas, nested renames, and serialization round trips expose this in merge paths. This is a prerequisite for the standalone Java work in [discussion #19798](https://github.com/apache/hudi/discussions/19798) and the [dev-list proposal](https://www.mail-archive.com/dev%40hudi.apache.org/msg06536.html). ### Summary and Changelog - Retain the writer schema and resolve projections, aliases, defaults, and supported legacy positional renames without shifting unrelated fields. - Restrict positional renames to compatible source fields that are not already claimed by another reader field or alias. Incompatible drop/add replacements receive their reader default. - Avoid inferring a legacy writer schema from a prefix read that leaves unread record bytes. Later full-schema reads can still recover trailing values. - Preserve the writer schema through Java serialization and default Kryo serialization. Cache compressed schema strings when smaller, keeping each serialized payload independently readable across threads and random-access spills. - Preserve the original Java serialVersionUID and accept golden legacy streams. Keep Spark's registered compact Kryo representation and its existing size assertions unchanged. - Add regression coverage for the review cases, compressed/uncompressed transport, tombstones, and Spark Kryo without the Hudi registrar. Split recursive reconciliation into named helpers. This uses Hudi's Avro rewrite utilities, Apache Avro's compatibility checker, and JDK gzip/Base64. No dependency or third-party source is added. ### Impact Shared payload decoding affects every engine using these payloads. No new table storage layout or table version is introduced. **Breaking transport change (the `!` in the title):** default non-Spark Kryo writers emit a schema-bearing representation with a negative-length discriminator. The schema string contains JSON or `gzip:` followed by Base64-encoded compressed JSON. Updated readers accept legacy non-negative-length payloads, uncompressed schema-bearing payloads, and compressed schema-bearing payloads. Pre-change readers cannot read the schema-bearing representation; readers from an earlier revision of this PR that only understand JSON cannot read its compressed form. `BaseAvroPayload.useLegacyKryoFormat(kryo)` retains the old representation for transports supplying the writer schema separately. The Spark Hudi registrar selects that mode. Mixed-version users must upgrade readers first or explicitly retain legacy transport; arbitrary mixed-version compatibility is not claimed. Legacy bytes remain schema-free: consuming all bytes avoids caching a truncated prefix, but does not prove historical field names/types. Compression reduces repeated schema overhead without introducing stream-local IDs. A sparse 128-column size probe changed from 8,016 to 729 bytes per serialized payload (146 bytes in legacy mode). This is a size measurement, not an end-to-end performance claim; compression and schema parsing are cached with a 1,024-entry bound. ### Risk Level high Shared merge behavior and the default in-memory Kryo representation change. Review regressions were reproduced against the previous PR head before the fixes. Local validation in the Hudi Java 11 CI image passed 70 focused cases with zero failures, errors, skips or retry events: 26 payload cases, four shared serializer cases, 32 existing disk-spill cases, five Spark registrar/default-serializer cases, and the three unchanged Spark record-serialization/size cases. The dependency reactor builds successfully; Checkstyle passes for both changed modules. Unrelated ScalaTest suites were excluded from this targeted run. This is not a full repository test run. The previous head `a8530e348f` passed [Azure CI](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16927); that result does not validate this revision. ### Documentation Update The transport boundary and migration requirements are documented above. The prepared companion change for the `asf-site` branch will document this in `website/docs/java-client.md` and be linked before merge. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
