Hi folks, I'd like to discuss FLINK-39943 [1]. Today, Flink's IS JSON / IS NOT JSON functions are hard-coded to never return NULL. The recent Calcite 1.38.0 upgrade (FLINK-36602) changed Calcite's own default to the standard-compliant behavior instead: IS JSON is a predicate subject to three-valued logic and should evaluate to UNKNOWN (NULL) when the input is NULL, not FALSE. Flink currently suppresses this new Calcite default to preserve the old, standards-incompatible semantics [2]. I've checked a few other engines and all agree with the standard here. Oracle, PostgreSQL (16+), SQL Server (ISJSON), MySQL (JSON_VALID), and Snowflake (IS_OBJECT/IS_ARRAY/CHECK_JSON) all return NULL/UNKNOWN for a NULL input.
Originally, Flink implemented the correct nullable behavior when the functions were added in Flink 1.11. For Flink 1.15, this was hotfixed [3] to comply with Calcite's non-null semantics. Since Calcite fixed its definition with the 1.38 release, I'm proposing that we revert the Flink 1.15 hotfix, even though it's a breaking change. * PRO: standards compliance, consistency with every other engine and with Calcite's own default (less surprise porting SQL from other systems), consistency with Flink's other JSON functions which already propagate NULL, and removing custom codegen that fights Calcite on every future upgrade. * CON: it's a breaking change, most notably flipping NULL IS NOT JSON from TRUE to NULL, which can affect existing filters, views, or NOT NULL-typed computed columns/sinks built on this predicate, and there's no compatibility flag today to ease the transition. This is one of several breaking side-effects of the Calcite 1.38.0 upgrade. Hence, I'd propose treating it as a deliberate, documented break targeted at the next minor version (2.4) rather than adding a dedicated config flag. Curious to hear if others agree, or see a strong case for keeping the current NOT NULL semantics. Best, Fabian [1] https://issues.apache.org/jira/browse/FLINK-39943 [2] https://github.com/apache/flink/commit/8de3293e2d#diff-53c19443d4ddaabe90d0041379f3656f3d52983ae7a0226a9b23bc87675471a8R93 [3] https://github.com/apache/flink/commit/74146c626edd7603f00a6f0c90c7fccaf2066912
