Pritam Kumar created FLINK-40264:
------------------------------------
Summary: nested legacyTimestampMapping
Key: FLINK-40264
URL: https://issues.apache.org/jira/browse/FLINK-40264
Project: Flink
Issue Type: Bug
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 2.3.0, 2.2.0, 1.20.0
Reporter: Pritam Kumar
h2. Problem
{{AvroToRowDataConverters#createConverter}} dispatches a nested {{ROW}} to
the single argument overload:
{code:java}
case ROW:
return createRowConverter((RowType) type);
{code}
{{createRowConverter(RowType)}} defaults {{legacyTimestampMapping}} to
{{true}}, so the flag is dropped for every nesting level below the top one.
Since {{TIMESTAMP_LTZ}} is only supported by the non legacy mapping, a nested
{{TIMESTAMP_LTZ}}
column fails even when the non legacy mapping was explicitly requested:
{noformat}
java.lang.UnsupportedOperationException: Unsupported type: TIMESTAMP_LTZ(3)
NOT NULL
{noformat}
The failure happens while the deserialization schema is being constructed, so
the job never starts.
h2. Reproduction
{code:java}
DataType dataType =
ROW(
FIELD("id", INT().notNull()),
FIELD("nested", ROW(FIELD("ltz3",
TIMESTAMP_LTZ(3).notNull())).notNull()))
.notNull();
RowType rowType = (RowType) dataType.getLogicalType();
new AvroRowDataDeserializationSchema(
rowType,
InternalTypeInfo.of(rowType),
AvroEncoding.BINARY,
false /* legacyTimestampMapping */); // throws
{code}
{{AvroSchemaConverter#convertToSchema}} handles the same type correctly with
{{legacyTimestampMapping = false}}, and {{RowDataToAvroConverters}} propagates
the flag properly, so only the deserialization side is affected.
h2. Fix
Propagate the flag:
{code:java}
case ROW:
return createRowConverter((RowType) type, legacyTimestampMapping);
{code}
Strictly a widening: the only code paths whose behaviour changes are ones
that throw today.
h2. Introduced by
FLINK-33198, first released in 1.19.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
