mikebridge opened a new pull request, #45133:
URL: https://github.com/apache/superset/pull/45133
### SUMMARY
Selecting NULL in a semantic-view filter can silently drop the missing-value
rows because the provider receives SQL-style equality or membership against
NULL. Explore and drill-by also send the `<NULL>` sentinel, which needs
normalization before type coercion.
This PR normalizes those selections in the host and adds a small provider
contract for the one case that needs an OR group:
- NULL equality and NULL-only membership become `IS NULL`; their negative
forms become `IS NOT NULL`.
- Mixed `IN ('a', NULL)` becomes a parenthesized `IN ('a') OR IS NULL`.
Providers must opt in to `OR_FILTERS` after implementing grouped filters;
otherwise the host returns a query validation error (HTTP 400) before provider
execution.
- Mixed `NOT IN ('a', NULL)` becomes `NOT IN ('a') AND IS NOT NULL`, using
existing leaf predicates.
- UI empty-string sentinels are decoded, invalid NULL comparison/LIKE
operands and empty membership are rejected, and scalar comparison collections
follow the native datasource's first-value behavior.
The SDK adds a frozen `OrFilter` containing at least two leaves from one
predicate stage. Query and group-limit filter sets remain AND sets.
`get_values` keeps its existing leaf-only contract. Main queries, row counts,
time offsets and group-limit subqueries use the capability gate.
A constant in direct semantic result-cache keys prevents reuse of
pre-normalization answers during rolling deployment. `UPDATING.md` and provider
documentation cover SDK union narrowing, guarded loading on older hosts, the
new validation behavior and cache warming. Adapters still need to implement
grouped rendering before advertising the capability; this PR does not complete
adapter adoption.
**Merge coordination:** whichever of this PR and #45100 lands second must
preserve both cache-key guarantees: the semantic-filter protocol marker **and**
the metadata cache token. Keep #45100's semantic annotation datasource resolver
and its source security/version keying; add a regression that pins both direct
semantic cache-key elements. Do not resolve the overlap by choosing either
branch's cache implementation wholesale.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
No visual UI change; browser screenshots were not captured.
| Selected values | Before | After |
| --- | --- | --- |
| NULL only | Equality/membership can return no matching rows |
Missing-value rows selected with `IS NULL` |
| `a` and NULL | NULL rows can be silently omitted | Both selected with
supported grouped filtering; explicit 400 on a provider without the capability |
| NOT IN `a` and NULL | SQL UNKNOWN can exclude all rows | Excludes `a` and
missing values; other rows remain |
### TESTING INSTRUCTIONS
Automated regression and coverage gate:
```bash
TZ=UTC python -m pytest tests/unit_tests/semantic_layers \
--cov=superset/semantic_layers --cov-branch --cov-fail-under=100
```
The tests compare NULL/a/b fact membership, decode Explore/drill-by
sentinels, reject unsupported groups before table and row-count dispatch,
retain offset and inner ranking bounds, and separate legacy result-cache keys.
They also cover duplicate inputs, integer boolean rejection, empty selections
and NULL comparison operands.
Manual, with `SEMANTIC_LAYERS` enabled and a semantic view containing NULL,
`a` and `b`:
1. In Explore, select only NULL, then exclude only NULL. Verify missing
rows, then nonmissing rows.
2. With a provider supporting `OR_FILTERS`, select `a` plus NULL and verify
both populations. Without support, verify the explicit validation error rather
than a partial answer.
3. Exclude `a` plus NULL and verify only `b` remains. Repeat with a row
count, a time comparison and a series limit.
4. Drill by a NULL value and verify it selects missing rows. Check that an
empty-string choice stays distinct from NULL.
Local validation: 880 semantic tests passed with 100% statement/branch
coverage. Complete branch-file hooks passed, including MyPy and docs lint. The
whole unit suite at the final commit reported 22,702 passed, 40 skipped and 2
xfailed, with only the two documented ARM long-double baseline failures in
unchanged test files. A docs build, browser round trip and live provider
validation were not run locally.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [x] Required feature flags: `SEMANTIC_LAYERS`
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in
[SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [x] Introduces new feature or API: optional grouped-filter SDK contract
- [ ] Removes existing feature or API
@aminghadersohi @rebenitez1802 — could you review the NULL normalization and
optional provider contract? This is a draft while adapter adoption and the
cache-key merge coordination are pending.
Generated with OpenAI Codex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
