rusackas opened a new pull request, #42598: URL: https://github.com/apache/superset/pull/42598
### SUMMARY This is a **test-only PR** opened as a TDD-style validation of issue #36304. #36304 reports that a cross-database join via the `ENABLE_SUPERSET_META_DB` feature returns "no data" once a `WHERE ... OR` (or equivalent `IN`) filter is added, even though the same query with a single equality filter returns rows. Two bot theories were floated and refuted in the issue thread (single-filter-per-column limitation); the real root cause, confirmed by tracing `superset/extensions/metadb.py`, is `SUPERSET_META_DB_LIMIT` (default 1000): `SupersetSQLiteAdapter.get_data` applies it to **each underlying table independently**, before Shillelagh/SQLite runs the in-memory join. If a joined table has more rows than the limit, only the first `SUPERSET_META_DB_LIMIT` rows are ever read from it — so a row with a genuine match on the other side of the join can be silently truncated away before the join logic even sees it, producing an incomplete or empty result with no error. Docs for this caveat were already added in #41302; this PR pins down the actual behavior with a regress ion test so a real fix can be TDD'd against it. This PR adds one regression test on `SupersetSQLiteAdapter.get_data` (via the `superset://` dialect): 1. **`test_superset_joins_with_limit_drops_matches`** — builds a 3-row table (`a=1,2,3`) and a 1-row table whose only row matches `a=3`, sets `SUPERSET_META_DB_LIMIT=2`, and joins them. The genuine match at `a=3` exists in both tables, so the join should return it — but the per-table limit truncates the 3-row table to `a=1,2` before the join runs, so the match is silently dropped. ### How to interpret CI - **CI green** → the per-table truncation is fixed (e.g. limit applied to the joined result instead of to each source table); merging closes #36304. - **CI red** → bug is still live, exactly as diagnosed. Fix belongs in `SupersetSQLiteAdapter.get_data` in `superset/extensions/metadb.py` (around the `app_limit`/`SUPERSET_META_DB_LIMIT` handling at the point each table's `get_data` request is limited) — the per-table limit needs to stop being blindly applied to tables participating in a join, or the join needs to run before per-table truncation. ### TESTING INSTRUCTIONS ```bash pytest tests/unit_tests/extensions/test_sqlalchemy.py::test_superset_joins_with_limit_drops_matches -v ``` ### ADDITIONAL INFORMATION - [x] Has associated issue: closes #36304 - [ ] Required feature flags: `ENABLE_SUPERSET_META_DB` (already required by existing tests in this file) - [ ] Changes UI - [ ] Includes DB Migration - [ ] Introduces new feature or API - [ ] Removes existing feature or API 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
