Hi Fabian,
it is very common in the SQL area that passing a NULL as input to a
function results in NULL as output. I would consider this as a bug that
we should fix asap. From my gut feeling, I would say that IS JSON
functions are not very popular. If we were talking about JSON_VALUE or
JSON_QUERY, things would be different. So +1 for the breaking change.
Cheers,
Timo
On 09.07.26 14:24, Fabian Hueske wrote:
Hi folks,
I'd like to discuss FLINK-39943 [1].
Today, Flink's IS JSON / IS NOT JSON functions are hard-coded to never
return NULL.
The recent Calcite 1.38.0 upgrade (FLINK-36602) changed Calcite's own
default to the standard-compliant behavior instead: IS JSON is a predicate
subject to three-valued logic and should evaluate to UNKNOWN (NULL) when
the input is NULL, not FALSE.
Flink currently suppresses this new Calcite default to preserve the old,
standards-incompatible semantics [2].
I've checked a few other engines and all agree with the standard here.
Oracle, PostgreSQL (16+), SQL Server (ISJSON), MySQL (JSON_VALID), and
Snowflake (IS_OBJECT/IS_ARRAY/CHECK_JSON) all return NULL/UNKNOWN for a
NULL input.
Originally, Flink implemented the correct nullable behavior when the
functions were added in Flink 1.11. For Flink 1.15, this was hotfixed [3]
to comply with Calcite's non-null semantics.
Since Calcite fixed its definition with the 1.38 release, I'm proposing
that we revert the Flink 1.15 hotfix, even though it's a breaking change.
* PRO: standards compliance, consistency with every other engine and with
Calcite's own default (less surprise porting SQL from other systems),
consistency with Flink's other JSON functions which already propagate NULL,
and removing custom codegen that fights Calcite on every future upgrade.
* CON: it's a breaking change, most notably flipping NULL IS NOT JSON from
TRUE to NULL, which can affect existing filters, views, or NOT NULL-typed
computed columns/sinks built on this predicate, and there's no
compatibility flag today to ease the transition.
This is one of several breaking side-effects of the Calcite 1.38.0
upgrade. Hence, I'd propose treating it as a deliberate, documented break
targeted at the next minor version (2.4) rather than adding a dedicated
config flag.
Curious to hear if others agree, or see a strong case for keeping the
current NOT NULL semantics.
Best,
Fabian
[1] https://issues.apache.org/jira/browse/FLINK-39943
[2]
https://github.com/apache/flink/commit/8de3293e2d#diff-53c19443d4ddaabe90d0041379f3656f3d52983ae7a0226a9b23bc87675471a8R93
[3]
https://github.com/apache/flink/commit/74146c626edd7603f00a6f0c90c7fccaf2066912
