rusackas opened a new pull request, #38588: URL: https://github.com/apache/superset/pull/38588
### SUMMARY Adopts and improves the fix from #36387 (originally by @ethan-l-geotab). Fixes #36385. BigQuery queries returning huge result sets (950+ MB) crash Chrome by loading everything into browser memory at once. This PR implements memory-aware progressive fetching in `BigQueryEngineSpec.fetch_data`: 1. **Progressive fetch**: Samples an initial batch (1000 rows) to estimate row size, then extrapolates how many total rows fit within the `BQ_FETCH_MAX_MB` config limit (default 200 MB) 2. **Warning propagation**: When results are truncated, a warning is passed through Flask `g` to the query context processor, which adds it to the response payload 3. **Frontend toast**: The chart action handler displays a warning toast to the user when results were truncated 4. **Graceful fallback**: On any error in the progressive fetch, falls back to the parent `BaseEngineSpec.fetch_data` implementation **Key differences from the original PR (#36387):** - Removed the `BQ_MEMORY_LIMIT_FETCH` feature flag -- the fix is always-on - The `BQ_FETCH_MAX_MB` config constant (default 200 MB) is the only operator-level knob - Added comprehensive unit tests for the new BigQuery fetch logic and warning propagation - Applied to the renamed `chartAction.ts` (was `.js` in the original PR) **Files changed:** - `superset/db_engine_specs/bigquery.py` -- Memory-aware progressive fetch implementation - `superset/common/query_context_processor.py` -- Warning propagation via Flask `g` - `superset/config.py` -- `BQ_FETCH_MAX_MB = 200` config constant - `superset-frontend/src/components/Chart/chartAction.ts` -- Warning toast display - `superset-frontend/packages/superset-ui-core/src/query/types/QueryResponse.ts` -- `warning` field on `ChartDataResponseResult` - `tests/unit_tests/db_engine_specs/test_bigquery.py` -- 5 new test cases - `tests/unit_tests/common/test_query_context_processor.py` -- 2 new test cases ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF Before: Chrome crashes or becomes unresponsive when BigQuery returns 950+ MB of data. After: Results are truncated to fit within the configured memory budget, and a warning toast informs the user. ### TESTING INSTRUCTIONS 1. Configure a BigQuery datasource in Superset 2. Run a query that returns a very large result set (millions of rows) 3. Verify that results are truncated and a warning toast appears 4. Set `BQ_FETCH_MAX_MB` to a smaller value (e.g., 10) in `superset_config.py` to test truncation with smaller datasets 5. Verify that queries returning small result sets work normally without any warning 6. Run the new unit tests: `pytest tests/unit_tests/db_engine_specs/test_bigquery.py -k test_fetch_data -v` ### ADDITIONAL INFORMATION - [x] Has associated issue: Fixes #36385 - [ ] Required feature flags: - [x] Changes UI - [ ] Includes DB Migration - [ ] Introduces new feature or API - [ ] Removes existing feature or API Co-authored-by: ethan-l-geotab <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
