Hi everyone,
thanks Ramin for updating the FLIP. It took us quite some iterations to
get it into a consistent and high quality shape.
For context, these are items we put into consideration:
- Look at other vendors: BigQuery, Snowflake, Spark, DuckDB
- Look at existing Flink SQL built-in functions e.g. JSON_*, IS_* and
collection functions
- Look at use cases and find usability gaps
- Provide a reasonable set of building blocks
- What is required to make VARIANT in Flink a large success?
The FLIP covers a lot of function at once. Some are not described in
every detail, but we will cover this as part of PR discussions and
writing docs.
Overall, I hope we can make many users happy.
Thanks,
Timo
On 23.10.26 13:32, Ramin Gharib wrote:
Hi everyone,
After a long discussion with Timo Walther, I updated the FLIP to
change how the functions handle SQL NULL.
The functions now handle SQL NULL and variant null consistently. A
SQL NULL argument gives a SQL NULL result.
To write a variant null on purpose, the FLIP adds two functions:
- VARIANT_NULL() returns a variant null, the same as
PARSE_JSON('null').
- TO_VARIANT(x) is the same as CAST(x AS VARIANT), except for SQL
NULL. CAST(NULL AS VARIANT) returns NULL, so TO_VARIANT gives a
variant null instead.
For example, with discount = NULL:
VARIANT_SET(payload, '$.discount', discount)
-- NULL
VARIANT_SET(payload, '$.discount', TO_VARIANT(discount))
-- {"discount":null,"order_id":8,"user":"bob"}
Other changes:
- VARIANT_OBJECT, VARIANT_ARRAY and VARIANT_MERGE return NULL for a
SQL NULL argument. A NULL key in VARIANT_OBJECT is still an error.
- VARIANT_OBJECT_AGG skips rows with a SQL NULL value.
- VARIANT_DELETE and VARIANT_PICK still skip a NULL path.
- The prior-art table has a new column that names the proposed
function for each row.
One point is still open. IS_VARIANT_NULL now returns NULL for SQL
NULL. Returning FALSE instead would keep its result NOT NULL. The
wiki has an example under Open Questions. I'm happy to hear your
opinion on it.
Wiki:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-618%3A+Edit%2C+Inspect+and+Build+VARIANT+Values
Cheers,
Ramin Gharib
On Thu, Oct 22, 2026 at 5:36 PM Ramin Gharib <[email protected]> wrote:
Hi everyone,
I'd like to start a discussion on FLIP-618: Edit, Inspect and Build
VARIANT Values.
Wiki:
https://cwiki.apache.org/confluence/spaces/FLINK/pages/461570450/FLIP-618+Edit+Inspect+and+Build+VARIANT+Values
Flink can store and read VARIANT values since 2.1. SQL users still
cannot inspect or change one field and keep the rest, ask what a value
contains,
or build an object from columns without going through JSON text.
Today each of these needs a UDF that rebuilds the whole value.
The FLIP proposes built-in functions to edit (VARIANT_SET,
VARIANT_DELETE, VARIANT_MERGE, ...), inspect (IS_VARIANT_NULL,
VARIANT_KEYS, SCHEMA_OF_VARIANT, ...) and build (VARIANT_OBJECT,
VARIANT_OBJECT_AGG) VARIANT values, plus a Java API to edit a value
inside a UDF.
Looking forward to your feedback.
Best,
Ramin Gharib
