Ramin Gharib created FLINK-40318:
------------------------------------
Summary: Add a validation entry point for a VARIANT binary
Key: FLINK-40318
URL: https://issues.apache.org/jira/browse/FLINK-40318
Project: Flink
Issue Type: Improvement
Components: API / Core
Reporter: Ramin Gharib
Assignee: Ramin Gharib
Flink has no way to check that a variant binary is well formed before using it.
\{{BinaryVariant}} validates only the metadata version, and everything else
relies on \{{checkIndex}} failing lazily during access.
Spark's \{{VariantUtil}}, which \{{BinaryVariantUtil}} is derived from,
provides \{{isValidVariant(value, metadata)}} backed by a recursive
\{{validateImpl}}. It checks the metadata version, structural well formedness
of every value, access bounds, primitive type-info validity, dictionary-id
existence for object fields, and recurses through nested objects and arrays.
Neither method was carried over.
This matters once variant bytes arrive from outside the job, for example from a
Parquet or Avro source or from restored state, where malformed input should be
rejected up front with a clear error instead of surfacing as an index failure
somewhere deep in a query.
h3. Scope
Port \{{isValidVariant}} and \{{validateImpl}}, and decide where to call them,
most likely when constructing a \{{BinaryVariant}} from external bytes rather
than on every access.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
