Ramin Gharib created FLINK-41323:
------------------------------------
Summary: Add a binary encoding for a VARIANT column in the raw
format
Key: FLINK-41323
URL: https://issues.apache.org/jira/browse/FLINK-41323
Project: Flink
Issue Type: Improvement
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
SQL / Runtime
Reporter: Ramin Gharib
Assignee: Ramin Gharib
The {{raw}} format reads and writes a VARIANT column as a JSON document. The
sink renders the value with {{Variant#toJson}} and the source parses it like
{{PARSE_JSON}}. JSON has no type for dates, times, timestamps, bytes or UUIDs,
and it does not keep the width of a number. A round trip through the {{raw}}
format changes the type of those values:
{noformat}
VARIANT type raw sink writes raw source reads
INT 7 TINYINT
FLOAT 1.5 DECIMAL
DOUBLE 1.5 DECIMAL
DATE "2026-11-03" STRING
TIMESTAMP_LTZ "2026-11-03T12:30:00Z" STRING
BYTES "AQID" STRING
UUID "123e4567-e89b-12d3-a456-426614174000" STRING
{noformat}
Every message is also parsed on read and rendered on write.
*Proposal*
# Add the option {{raw.variant-encoding}} with the values {{json}} and
{{binary}}. {{json}} stays the default and keeps the current behavior.
# With {{binary}}, a message holds the Variant binary encoding: the metadata
followed by the value, as defined by the Parquet Variant spec. The metadata
states its own length, so no separator is needed. Iceberg bounds and Delta
stats use the same layout. Reading splits the message and wraps the two parts
without parsing. Writing concatenates them. Every Variant type survives the
round trip.
# The bytes are only checked as far as needed to split them. A malformed value
fails when it is read, as for any other VARIANT.
# {{binary}} does not support {{raw.line-delimiter}}, because the binary
encoding can contain any byte. {{raw.charset}} does not apply to it.
The docs recommend {{binary}} for correctness and performance, and {{json}} for
convenience, when people or other systems read and write the messages as JSON.
*Compatibility*
The new option defaults to {{json}}, so existing tables behave as before.
VARIANT support in the {{raw}} format is new in 2.4.0 (FLINK-40241) and not
released yet.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
