Pritam Kumar created FLINK-40263:
------------------------------------
Summary: Enum symbol validation
Key: FLINK-40263
URL: https://issues.apache.org/jira/browse/FLINK-40263
Project: Flink
Issue Type: Bug
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 2.4.0
Reporter: Pritam Kumar
h2. Problem
FLINK-39053 taught {{RowDataToAvroConverters}} to map a {{CHAR}}/{{VARCHAR}}
column onto an Avro {{ENUM}} field:
{code:java}
if (schema.getType() == Schema.Type.ENUM) {
return new GenericData.EnumSymbol(schema, object.toString());
}
{code}
{{GenericData.EnumSymbol}} does not validate its argument. A string that is
not one of the declared symbols therefore survives the conversion and only
fails much later, inside {{GenericDatumWriter}}:
{noformat}
org.apache.avro.AvroTypeException: Not an enum: ...
{noformat}
wrapped in {{Failed to serialize row.}} That message names neither the
column, nor the offending value, nor the symbols that would have been accepted,
which makes it hard to act on for a value that occurs once in a stream.
Separately, the enum {{"default"}} attribute (Avro 1.9+) is ignored. Avro's
own schema resolution maps an unknown symbol onto the declared default; a
schema author who sets one is explicitly asking for that, but here the job
fails instead.
h2. Proposal
Validate before constructing the symbol:
* a declared symbol converts exactly as today;
* an unknown symbol falls back to {{Schema#getEnumDefault()}} when the enum
declares one;
* otherwise fail with a message naming the value, the enum and the allowed
symbols.
h2. Note
FLINK-39053 is not in a release yet, so this can land before 2.4.0 ships and
no released behaviour changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
