voonhous opened a new issue, #19833:
URL: https://github.com/apache/hudi/issues/19833
### Bug Description
`InternalSchemaConverter.buildBlobInternalRecordType()`
(`InternalSchemaConverter.java:70-87`, since #18538) gives the BLOB sub-fields
sentinel ids `-10/-11/-12`, but the nested `reference` record's four fields get
ids `0`, `1`, `2`, `3`. `InternalSchema.buildIdToField` is a single flat map
over the whole tree, filled child-first with last-put-wins
(`InternalSchemaBuilder.java:150-156`), so every table that has a BLOB column
also has `reference.external_path` registered under id `0`, `reference.offset`
under `1`, and so on -- the same ids the table's own leading columns carry.
Any id-based InternalSchema operation on such a table resolves the wrong
field. Through `HoodieSchemaUtils.asNullable`
(`HoodieSchemaUtils.java:284-288`), which marks the still-required top-level
columns nullable with a `ColumnUpdateChange`:
```java
HoodieSchema schema = HoodieSchema.createRecord("r", null, null, false,
Arrays.asList(
HoodieSchemaField.of("id", HoodieSchema.create(HoodieSchemaType.INT),
null, null),
HoodieSchemaField.of("b",
HoodieSchema.createNullable(HoodieSchema.createBlob()), null, null)));
HoodieSchemaUtils.asNullable(schema).toAvroSchema().toString();
//
{"type":"record","name":"r","fields":[{"name":"external_path","type":["null","string"],"default":null},{"name":"b",...
```
The `id` column comes back renamed to `external_path`: when the changed
schema is rebuilt, field id 0 is looked up in the flat id map and resolves to
the blob's nested `reference.external_path`, which was put last. With a
required BLOB column instead (`HoodieSchema.createBlob()` without the nullable
wrapper) the same call throws:
```
org.apache.hudi.exception.SchemaCompatibilityException: Cannot update
nullability for column 'b' because it does not exist in the schema
at
org.apache.hudi.common.schema.internal.action.TableChanges$ColumnUpdateChange.updateColumnNullability(TableChanges.java:192)
at
org.apache.hudi.common.schema.HoodieSchemaUtils.lambda$asNullable$3(HoodieSchemaUtils.java:287)
```
Reachable from Flink clustering: `HoodieSchemaConverter.convertToSchema`
emits `HoodieSchema.createBlob()` for a BLOB-shaped `RowType`
(`hudi-flink-client/.../HoodieSchemaConverter.java:221-224`) and
`ClusteringOperator.open()` passes that schema to `asNullable`
(`ClusteringOperator.java:170-177`), so a Flink table with a BLOB column and a
NOT NULL key gets a reader schema whose key column is renamed (nullable BLOB)
or fails to open (required BLOB). The other InternalSchema entry points that
see table schemas (`InternalSchemaCache`, `FileGroupReaderSchemaHandler`,
`HoodieMergeHelper`, `BaseHoodieWriteClient` schema-on-read paths) share the
same id map and are exposed to the same collision whenever a BLOB column is
present.
Present since #18538 (`4ef56e4ebd79`). Not a regression from #19810, which
keeps the conversion unchanged (the old `AvroSchemaUtils#asNullable` path
produced the identical output; verified by running both against the schemas
above).
Fix direction: allocate the `reference` field ids from the sentinel range as
well (for example `-13..-16`) so no BLOB-internal id can collide with a table
id, and make `buildIdToField` refuse duplicate ids (or assert on them) so the
next fixed-shape type cannot reintroduce this. Tables that already persisted an
InternalSchema with the colliding ids (schema-on-read enabled, BLOB column)
need the reader to tolerate both id sets; that is the part that makes this a
separate change rather than a one-line constant edit. Tests:
`TestInternalSchemaConverter` round trip of a record with a leading required
column plus a BLOB column asserting the leading column's name survives, and
`TestHoodieSchemaUtils#asNullable` on the same shape.
### Environment
- Hudi master (`93f1f711e065`); Flink clustering on a table with a BLOB
column is the reachable path
### Logs and Stack Trace
See above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
