erajio opened a new issue, #17650:
URL: https://github.com/apache/iceberg/issues/17650
### Apache Iceberg version
1.10.1
### Query engine
Flink
### Please describe the bug 🐞
CompareSchemasVisitor.struct() computes its result as:
Result result =
fields.stream().reduce(Result::merge).orElse(Result.SCHEMA_UPDATE_NEEDED);
For a struct with zero fields, the reduce is empty, so
orElse(SCHEMA_UPDATE_NEEDED) always fires. Consequently any schema containing
an empty struct (e.g. derived from an Avro union whose branch is a zero-field
"marker" record — common in registry schemas) can never compare SAME with any
table schema — including a table created from that exact schema.
In the Dynamic Sink this is fatal: TableUpdater.findOrCreateSchema sees
SCHEMA_UPDATE_NEEDED, applies EvolveSchemaVisitor (a no-op here), commits,
re-compares — still SCHEMA_UPDATE_NEEDED — and TableMetadataCache.schema()
caches and returns NOT_FOUND, whose resolvedTableSchema() is null. The null
propagates to DynamicRecordProcessor.emit, crashing with an unrelated-looking
NPE:
Caused by: java.lang.NullPointerException
at
org.apache.iceberg.flink.sink.dynamic.DynamicSinkUtil.getEqualityFieldIds(DynamicSinkUtil.java:34)
at
org.apache.iceberg.flink.sink.dynamic.DynamicRecordProcessor.emit(DynamicRecordProcessor.java:173)
at
org.apache.iceberg.flink.sink.dynamic.DynamicRecordProcessor.collect(DynamicRecordProcessor.java:118)
The job then crash-loops on every record for that table.
Minimal reproduction
Schema schema = new Schema(
Types.NestedField.required(1, "id", Types.StringType.get()),
Types.NestedField.optional(2, "marker", Types.StructType.of())); //
empty struct
InMemoryCatalog catalog = new InMemoryCatalog();
catalog.initialize("mem", Map.of());
catalog.createNamespace(Namespace.of("db"));
Table table = catalog.createTable(TableIdentifier.of("db", "t"), schema);
CompareSchemasVisitor.Result r = CompareSchemasVisitor.visit(schema,
table.schema(), true);
// r == SCHEMA_UPDATE_NEEDED — expected SAME (identical schemas)
Expected behavior
An empty struct compared against an empty struct should be SAME.
Additionally, TableUpdater.findOrCreateSchema returning a ResolvedSchemaInfo
with a null schema after a "successful" evolution deserves a descriptive
exception rather than surfacing as an NPE three operators later.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
