fmorillo7694 commented on issue #17615:
URL: https://github.com/apache/iceberg/issues/17615#issuecomment-5476352803
We hit this independently and can confirm the report with a minimal
reproduction on 1.11.0 (Flink 2.3.0, `iceberg-flink-runtime-2.1:1.11.0`).
Schema under test — primitives plus one variant field:
```
table {
1: event_id: required string
2: event_type: optional string
3: payload: optional variant
}
```
Two findings worth adding to the report:
1. **The failure occurs even when the target table already exists with an
exactly matching schema.** `TableMetadataCache.schema()` runs
`CompareSchemasVisitor` over the full incoming schema on every record, so the
variant field is visited during comparison regardless — the job fails on the
first record, before any writer is created:
```
java.lang.UnsupportedOperationException: Unsupported type: variant
at
org.apache.iceberg.schema.SchemaWithPartnerVisitor.variant(SchemaWithPartnerVisitor.java:167)
at
org.apache.iceberg.schema.SchemaWithPartnerVisitor.visit(SchemaWithPartnerVisitor.java:111)
at
org.apache.iceberg.schema.SchemaWithPartnerVisitor.visit(SchemaWithPartnerVisitor.java:62)
at
org.apache.iceberg.schema.SchemaWithPartnerVisitor.visit(SchemaWithPartnerVisitor.java:45)
at
org.apache.iceberg.flink.sink.dynamic.CompareSchemasVisitor.visit(CompareSchemasVisitor.java:60)
at
org.apache.iceberg.flink.sink.dynamic.TableMetadataCache.schema(TableMetadataCache.java:161)
at
org.apache.iceberg.flink.sink.dynamic.TableMetadataCache.schema(TableMetadataCache.java:112)
at
org.apache.iceberg.flink.sink.dynamic.DynamicRecordProcessor.collect(DynamicRecordProcessor.java:138)
```
2. **The static sink control succeeds with the identical schema and row
data**: the same `RowData` (string fields + a `Variant` built with Flink's
`VariantBuilder`) written through `IcebergSink.forRowData` against the same
pre-created format-version-3 table commits fine and the variant value
round-trips on read-back. So the gap is isolated to the dynamic sink's
schema-resolution layer, exactly as described.
Opened #17900 with the fix retargeted to the current module layout
(v2.1/v2.2/v2.3) plus test coverage, co-authoring @waterWang from #17631.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
