Pritam Kumar created FLINK-41027:
------------------------------------
Summary: [format/avro] Add an option to map time-millis and
time-micros to INT and BIGINT
Key: FLINK-41027
URL: https://issues.apache.org/jira/browse/FLINK-41027
Project: Flink
Issue Type: Improvement
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Reporter: Pritam Kumar
h2. What
An option, off by default, to map the Avro logical types {{time-millis}} and
{{time-micros}} to plain numbers ({{{}INT{}}} and {{{}BIGINT{}}}) instead of to
{{TIME(3)}} and {{{}TIME(6){}}}.
h2. Why
* A Flink {{TIME}} stores milliseconds since midnight in an int, so the
microseconds of a {{time-micros}} are lost as a {{{}TIME{}}}. As a {{BIGINT}}
they are kept.
* A {{TIME(6)}} has no Avro schema: {{AvroSchemaConverter.convertToSchema}}
refuses a precision above 3. In {{avro-confluent}} with a {{{}schema{}}}, a
{{time-micros}} field therefore could not be used at all: a {{TIME(6)}} column
failed before the provided schema was looked at, and a {{BIGINT}} column did
not match it.
h2. Change
* {{AvroSchemaConverter.convertToDataType(String, boolean, boolean)}} and
{{{}convertToTypeInfo(String, boolean, boolean){}}}: with {{{}timeAsNumber{}}},
{{time-millis}} becomes {{INT}} / {{Types.INT}} and {{time-micros}} becomes
{{BIGINT}} / {{{}Types.LONG{}}}, also nested in arrays, maps and nullable
unions, with the legacy and the new timestamp mapping. The existing overloads
keep the current mapping.
* {{{}AvroToVariantDataConverters.createVariantConverter(Schema, boolean){}}}:
with {{{}timeAsNumber{}}}, {{time-millis}} stays the Avro int instead of being
converted to microseconds of day and stored as a long. {{time-micros}} is a
long either way.
* {{avro-confluent}} option {{time-mapping.as-number}} (default
{{{}false{}}}), used when the table schema is checked against the provided
{{{}schema{}}}. Documented in the English and Chinese pages.
* Related fix in {{{}RegistryAvroFormatFactory{}}}: the Avro schema of the
table was built with {{{}Optional.orElse{}}}, which runs even when a {{schema}}
is provided, so a {{TIME(6)}} column failed there. It now uses
{{{}orElseGet{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
