mikebridge opened a new pull request, #45904:
URL: https://github.com/apache/superset/pull/45904
### SUMMARY
**TL;DR:** A semantic-view chart filtered with an empty IN or NOT IN
selection now returns a result, where it previously failed with "Semantic
IN/NOT IN filters cannot be empty."
- An empty IN matches no rows: a grouped chart returns no rows.
- An empty NOT IN matches every row, including rows where the value is NULL,
so the chart matches the unfiltered chart.
- An ungrouped total over an empty IN still returns one row: COUNT and
COUNT_DISTINCT are 0, and SUM, MIN, MAX and AVG are NULL.
**Background.** These are the semantics of filtering by an empty set:
membership in an empty set is FALSE, so an empty IN is FALSE and an empty NOT
IN is TRUE, even when the value is NULL. The predicate applies where the filter
sits, before aggregation, so an ungrouped `SELECT COUNT(*) ... WHERE FALSE`
returns one row containing 0. This is about the empty-set predicate, not the
literal `IN ()` syntax: many SQL dialects reject an empty list, so providers
must render a constant or an equivalent native predicate instead.
Dataset charts behave differently: they still reject an empty selection
("Filter value list cannot be empty"). This PR does not change dataset charts.
**Problem.**
- The semantic-layer host rejected both empty operators before reaching the
provider.
- The semantic cache's in-memory filter dropped NULL rows for an empty NOT
IN.
- The SDK docs said empty membership had no well-defined rendering, so
providers handled it differently.
**Changes.**
- The query mapper keeps an empty IN/NOT IN as one filter, ANDed with the
query's other filters, instead of raising.
- Cached results:
- an empty IN keeps no rows, and an empty NOT IN keeps every row, NULLs
included;
- an empty NOT IN is no longer treated as proof that NULL rows were
excluded when deciding whether a cached result can be reused;
- an ungrouped rollup of a cached result over zero matching rows now
returns one aggregate row instead of failing. Pandas applies an aggregation
function element by element on an empty series, so the function is now called
directly.
- The `Operator` docstring defines the semantics and tells providers to
render a constant or an equivalent predicate, never an empty SQL list.
- The provider conformance kit gains `test_empty_membership`, derived from
each provider's existing `identity` fixture, so providers add no fixtures. It
checks:
- grouped empty IN returns zero rows with the normal columns;
- empty NOT IN returns the identity result unchanged, NULL group included;
- ungrouped empty IN returns one row: COUNT and COUNT_DISTINCT metrics are
0, SUM, MIN, MAX and AVG are NULL, and other aggregations are not checked;
- ungrouped empty NOT IN matches the unfiltered total;
- inside the `or` fixture's OR group, an extra empty IN leaf leaves the
result unchanged and an extra empty NOT IN leaf makes the group match every row.
- The example provider renders the constants, and kit self-tests show the
witness rejects providers that drop NULLs, swap the constants, ignore the
filter or return no rows.
- `UPDATING.md` and the user, extension and provider-testing docs describe
the new behavior; they previously said empty selections were rejected.
- The provider testing guide documents the new cases. It notes that SQLite
accepts `IN ()`, so provider renderer tests should pin dialect-valid output.
Nonempty membership, including the existing NULL handling, is unchanged.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Not applicable. Before: the chart-data request fails with "Semantic IN/NOT
IN filters cannot be empty." After: the chart returns the SQL result described
above.
### TESTING INSTRUCTIONS
- `pytest tests/unit_tests/semantic_layers/null_filters_test.py
tests/unit_tests/semantic_layers/cache_policy_test.py`
- `pytest superset-core/tests/semantic_layers/test_conformance.py`
- Manually, on a semantic view with a nullable dimension, send a chart-data
request with an `IN` filter whose value list is `[]`. Grouped by that
dimension, it returns no rows; with only a COUNT metric, it returns one row of
0. Change the filter to `NOT IN` with `[]`: the result matches the unfiltered
chart, NULL group included.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in
[SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
Provider authors: the SDK conformance kit now includes
`test_empty_membership` and `test_empty_membership_in_or_group`. They need no
new fixture beyond an `identity` result with a NULL group, but a provider that
renders an empty SQL list or drops the filter will need to render a constant or
an equivalent predicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
