sankalpsthakur opened a new pull request, #20398: URL: https://github.com/apache/hudi/pull/20398
### Describe the issue this Pull Request addresses Closes #19833 A BLOB column puts a record of four fields (`reference.external_path`, `offset`, `length`, `managed`) into the `InternalSchema`. Those four fields were given the ids 0..3, the same ids the first table columns have. `InternalSchema` keeps its lookups in flat id maps, so the nested blob fields shadow the table's own columns. For a schema like `(id INT, payload BLOB)`, `HoodieSchemaUtils.asNullable` came back with the first column renamed `external_path` and typed as a string; with a required BLOB it threw "Cannot update nullability for column 'payload' because it does not exist in the schema". ### Summary and Changelog Table columns keep their own ids when a schema has a BLOB column. - `InternalSchemaConverter`: the four reference fields are tagged with the sentinel ids -13..-16, next to the existing -10..-12 of the blob's own children. The blob-shape check of the reverse conversion moves into one helper that the new `upgradeLegacyBlobRecord` shares. - `SerDeHelper`: releases 1.2.0 and 1.2.1 persisted blob schemas with the reference ids 0..3. Field equality includes the id, so such a schema would no longer equal the one converted from the same HoodieSchema. The parser now re-tags a blob record that has exactly the legacy ids; schemas that already carry the new ids and every other record are returned as is. - Tests: `TestHoodieSchemaUtils` (`asNullable` with a nullable and with a required BLOB), `TestInternalSchemaConverter` (each column found under its own id, nested fields found by name, round trip including the persisted JSON), `TestSerDeHelper` (legacy ids upgraded on read). I did not add a "no duplicate ids" check to `buildIdToField`: two BLOB (or two VARIANT) columns share their sentinel child ids today, so a check would reject tables that work. That sharing is unchanged here, and it has a consequence of its own: on a MOR schema-on-read table with two BLOB columns, a query that projects a nested field of one of them still fails with "cannot prune col" (before this change the same table failed earlier, on this bug). Giving those ids a per-column range is a separate change. AI disclosure: this change and its tests were drafted with an AI coding assistant (Claude Code) and checked by the submitter; no code was copied from elsewhere. ### Impact No public API change beyond one new static helper, `InternalSchemaConverter.upgradeLegacyBlobRecord`. Schemas written from now on persist -13..-16 for the reference fields; a reader detects a blob by the -10/-11/-12 ids of its children, so the nested ids do not change how the type is read back. Schema-on-read tables with a BLOB column stop resolving table columns to the blob's nested fields. ### Risk Level low. The converter change touches only the ids of the four reference fields, and the parser change does nothing unless a record has the exact blob layout with the ids 0..3. The four new tests fail without the change and pass with it, and the full hudi-common unit suite (2610 tests) passes with it: https://github.com/sankalpsthakur/hudi/actions/runs/39588286403. Selected BLOB, InternalSchema and schema-evolution suites on Spark 3.5, Spark 4.2, Flink 2.2 and the hadoop-mr and client modules give the same result with and without the change (1359 passed, 0 failed, 13 skipped by the tests themselves): https://github.com/sankalpsthakur/hudi/actions/runs/39597036903. Downgrade: a writer without this change fails when it reconciles against a schema persisted with the new ids ("cannot modify hudi meta col", checked on the base commit on fork CI). In the two setups I tried (ALTER TABLE then update and insert; reconcile-schema then the first insert), a BLOB table with a persisted InternalSchema already fails on the base writer on its own, so I did not find a working setup that this breaks. Bulk insert, insert overwrite and the streamer paths were not tried. Going back to a release without this change after writing with it should not be expected to work. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
