raminqaf opened a new pull request, #30048:
URL: https://github.com/apache/flink/pull/30048
## What is the purpose of the change
When a MAP, ARRAY or ROW operand is NULL, `a <> b` is NULL when it is
projected, but a WHERE condition and CASE WHEN treated it as TRUE. Rows with a
NULL collection passed a `<>` filter without an error.
```sql
-- t(id INT, m MAP<STRING, STRING>): (1, {'x': 'y'}), (2, NULL), read from a
values source
SELECT id FROM t WHERE m <> MAP['a', 'b'];
```
| Query | Before
| After |
|-----------------------------------------------------------|----------------|----------------|
| `WHERE m <> MAP['a', 'b']` | 1, 2
| 1 |
| `WHERE (m <> MAP['a', 'b']) IS NOT TRUE` | no rows
| 2 |
| `CASE WHEN m <> MAP['a', 'b'] THEN 'T' ELSE 'F' END` | (1, T), (2, T)
| (1, T), (2, F) |
| `SELECT m <> MAP['a', 'b']` | (1, true), (2,
null) | unchanged |
`<>` on these types negated the result term of the `=` comparison and kept
its null term, so a NULL operand gave result term `true`. The Calc filter, CASE
WHEN, IS [NOT] TRUE and join conditions read only the result term and rely on
it being `false` for a NULL boolean.
**Upgrade note:** queries that use `<>` on a nullable collection column now
return fewer rows. A streaming job restored from a savepoint over an update
stream can keep rows that passed the old filter, because their retractions no
longer pass it. I'll add a release note to the Jira ticket.
## Brief change log
- `wrapExpressionIfNonEq` builds `<>` with `generateNot`, which applies
three-valued NOT. This covers ARRAY, MAP, MULTISET, ROW and structured types,
which all share it.
## Verifying this change
- `ComparisonFunctionITCase`: `<>` with a NULL ARRAY, MAP and ROW operand
stays NULL when projected, and CASE WHEN takes the ELSE branch. The CASE WHEN
cases fail without the fix.
- Manually ran the queries of the ticket against a values source in batch
and streaming mode: WHERE, `IS TRUE`, `IS NOT TRUE`, `NOT (a = b)`, ARRAY, ROW
and a join condition. All return the expected rows with the fix and the wrong
ones without it.
- Existing suites pass: `MapFunctionITCase`, `StructuredFunctionsITCase`,
`CollectionFunctionsITCase`, `ImplicitConversionEqualsFunctionITCase`,
`EqualiserCodeGeneratorTest`, `ScalarOperatorsTest`, `ArrayTypeTest`,
`MapTypeTest`, `RowTypeTest`.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes, the
generated `<>` on these types adds one null check
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code 2.1.329 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]