Ramin Gharib created FLINK-40255:
------------------------------------
Summary: COMPILE PLAN fails for any plan containing a VARIANT type
Key: FLINK-40255
URL: https://issues.apache.org/jira/browse/FLINK-40255
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Reporter: Ramin Gharib
Assignee: Ramin Gharib
VARIANT plans and executes correctly, but serializing a plan containing it
throws. {{{}COMPILE PLAN{}}}, {{StatementSet#compilePlan}} and
{{TableEnvironment#compilePlanSql}} are unusable with VARIANT, as is plan
restore.
h3. Symptoms
A VARIANT-typed {{{}RexNode{}}}, for example the return type of {{PARSE_JSON}}
in a projection:
{code:java}
TableException: Unsupported RelDataType: VARIANT ExecNodeGraphJsonSerializer ->
RexNodeJsonSerializer#serializeCall -> RelDataTypeJsonSerializer ->
LogicalRelDataTypeConverter#toLogicalTypeNotNull{code}
A VARIANT in an {{ExecNode}} output type:
{code:java}
TableException: Unable to serialize logical type 'VARIANT'.
ExecNodeGraphJsonSerializer -> LogicalTypeJsonSerializer{code}
h3. Root cause
Two {{RelDataType}} to {{LogicalType}} converters exist and only one knows
VARIANT. {{{}o.a.f.table.planner.calcite.FlinkTypeFactory{}}}, used during
planning, handles it at {{FlinkTypeFactory.java:490-491}} and {{{}:861{}}}.
{{{}o.a.f.table.planner.typeutils.LogicalRelDataTypeConverter{}}}, used only by
plan serde, has no VARIANT handling at all. Nothing forces the two to agree.
h3. Locations on master
under {{{}flink-table/flink-table-planner/.../table/planner/{}}}:
* {{typeutils/LogicalRelDataTypeConverter.java:609}} —
{{toLogicalTypeNotNull}} hits {{default:}} and throws
* {{typeutils/LogicalRelDataTypeConverter.java:473}} — reverse direction
throws, breaks restore
* {{plan/nodes/exec/serde/LogicalTypeJsonSerializer.java:507-550}} — VARIANT
missing from {{CompactSerializationChecker}}
* {{plan/nodes/exec/serde/LogicalTypeJsonSerializer.java:253}} and
{{LogicalTypeJsonDeserializer.java:169-170}} — no {{case VARIANT}}
h3. Fix
Handle VARIANT in {{LogicalRelDataTypeConverter}} both ways and add it to
{{{}CompactSerializationChecker{}}}; that is sufficient, since the compact path
already round-trips VARIANT via {{{}LogicalTypeParser:585-586{}}}. The explicit
switch cases are cheap defensive coverage.
Add VARIANT to {{LogicalTypeJsonSerdeTest}} and
{{{}RelDataTypeJsonSerdeTest{}}}, plus a {{restore-tests}} program, all at zero
coverage today. {{VariantSemanticTest}} cannot catch this because
{{SemanticTestBase}} never serializes a plan.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
