[
https://issues.apache.org/jira/browse/FLINK-41323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18138060#comment-18138060
]
Yuepeng Pan commented on FLINK-41323:
-------------------------------------
Merged into master(2.5.0) via : c59810c04f830f8d6d2d7a2fc77dcbd03b29e714
> Add a binary encoding for a VARIANT column in the raw format
> ------------------------------------------------------------
>
> Key: FLINK-41323
> URL: https://issues.apache.org/jira/browse/FLINK-41323
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Runtime
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Major
>
> The {{raw}} format reads and writes a VARIANT column as a JSON document. The
> sink renders the value with {{Variant#toJson}} and the source parses it like
> {{PARSE_JSON}}. JSON has no type for dates, times, timestamps, bytes or
> UUIDs, and it does not keep the width of a number. A round trip through the
> {{raw}} format changes the type of those values:
> {noformat}
> VARIANT type raw sink writes raw source reads
> INT 7 TINYINT
> FLOAT 1.5 DECIMAL
> DOUBLE 1.5 DECIMAL
> DATE "2026-11-03" STRING
> TIMESTAMP_LTZ "2026-11-03T12:30:00Z" STRING
> BYTES "AQID" STRING
> UUID "123e4567-e89b-12d3-a456-426614174000" STRING
> {noformat}
> Every message is also parsed on read and rendered on write.
> *Proposal*
> # Add the option {{raw.variant-encoding}} with the values {{json}} and
> {{binary}}. {{json}} stays the default and keeps the current behavior.
> # With {{binary}}, a message holds the Variant binary encoding: the metadata
> followed by the value, as defined by the Parquet Variant spec. The metadata
> states its own length, so no separator is needed. Iceberg bounds and Delta
> stats use the same layout. Reading splits the message and wraps the two parts
> without parsing. Writing concatenates them. Every Variant type survives the
> round trip.
> # The bytes are only checked as far as needed to split them. A malformed
> value fails when it is read, as for any other VARIANT.
> # {{binary}} does not support {{raw.line-delimiter}}, because the binary
> encoding can contain any byte. {{raw.charset}} does not apply to it.
> The docs recommend {{binary}} for correctness and performance, and {{json}}
> for convenience, when people or other systems read and write the messages as
> JSON.
> *Compatibility*
> The new option defaults to {{json}}, so existing tables behave as before.
> VARIANT support in the {{raw}} format is new in 2.4.0 (FLINK-40241) and not
> released yet.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
