[
https://issues.apache.org/jira/browse/FLINK-40318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramin Gharib updated FLINK-40318:
---------------------------------
Parent: FLINK-41336
Issue Type: Sub-task (was: Improvement)
> Add a validation entry point for a VARIANT binary
> -------------------------------------------------
>
> Key: FLINK-40318
> URL: https://issues.apache.org/jira/browse/FLINK-40318
> Project: Flink
> Issue Type: Sub-task
> Components: API / Core, Table SQL / API
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Minor
>
> Flink has no way to check that a variant binary is well formed before using
> it. {{BinaryVariant}} validates only the metadata version and the 16 MiB size
> limit. Everything else relies on {{checkIndex}} failing lazily during access.
> Spark's {{VariantUtil}}, which {{BinaryVariantUtil}} is derived from,
> provides {{isValidVariant(value, metadata)}} backed by a recursive
> {{validateImpl}}. It checks the metadata version, structural well formedness
> of every value, access bounds, primitive type-info validity, dictionary-id
> existence for object fields, and recurses through nested objects and arrays.
> Neither method was carried over.
> This matters once variant bytes arrive from outside the job, for example from
> a Parquet or Avro source or from restored state.
> h3. Scope
> Port {{isValidVariant}} and {{validateImpl}} as the implementation of the
> planned {{IS_VALID_VARIANT}} SQL function.
> Do not call the check automatically when a {{BinaryVariant}} is constructed
> or read. That walks every value in full, even when a query reads one field.
> Users decide where to pay for the check by calling {{IS_VALID_VARIANT}}.
> The check answers "can every part of the value be read", the same as Spark:
> * Key order and duplicate keys are not checked. Reading works without them,
> and Spark does not check them either.
> * A type id that Flink cannot read makes the value invalid. Until the new
> primitive codes 17-20 are supported, this includes a UUID, a TIME or a
> nanosecond timestamp written by another engine.
> h3. FLIP-618
> This ticket is the IS_VALID_VARIANT subtask of FLINK-41336. On top of the
> check above, it ships:
> * The SQL function IS_VALID_VARIANT(variant), which returns BOOLEAN. SQL NULL
> gives NULL.
> * Its Table API and PyFlink expressions, SQL and Table API tests, and docs.
> * Fuzz tests on corrupted values. The check never throws, and a value it
> accepts can be read in full.
> FLIP-604 added the type ids 17 to 20, so the check accepts a UUID, a TIME and
> the nanosecond timestamps.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
