aipiJuancho opened a new pull request, #44624: URL: https://github.com/apache/superset/pull/44624
### SUMMARY When the frontend decides to stream a CSV export (CSV, not a pivot, row count ≥ `CSV_STREAMING_ROW_THRESHOLD`), it sends `expected_rows` and expects the rows to start flowing. On the backend, however, `ChartDataRestApi._get_data_response` still calls `command.execute()` first: the full query runs and every row is materialized in memory, and only then does `_send_chart_response` notice the export should be streamed, discard that result and hand the query context to `StreamingCSVExportCommand`, which **runs the same query again**. For large exports that first pass is long and completely silent (no byte is sent to the client), so load balancers and reverse proxies drop the idle connection and the download never starts. It also executes the query twice and holds the whole result in a web worker for nothing. This PR short-circuits that case: when `expected_rows` is present and ≥ `CSV_STREAMING_ROW_THRESHOLD`, the format is CSV and the result type is `FULL`, `_get_data_response` goes straight to `_create_streaming_csv_response` without executing the command first. Everything else (JSON, XLSX, post-processed results such as pivot tables, CSV below the threshold or without `expected_rows`) keeps the regular path. Design notes: - **Permissions:** the export permission check that `_send_chart_response` applies (`can_csv`, or `can_export_data` with `GRANULAR_EXPORT_CONTROLS`) is extracted into `_has_export_permission()` and used by both paths, so they cannot drift apart. Datasource access is still enforced by `command.validate()` in the endpoint and again by `StreamingCSVExportCommand.validate()`. - **Same eligibility as the frontend:** `expected_rows` is only sent by `useStreamingExport` after the dashboard/explore code decides to stream with the same threshold read from the server config, and the backend re-checks that threshold. - **Nothing is lost by skipping execution:** the streaming command builds its SQL from the same query context; dashboard filter context is applied to the query context before the command is created; the acting-user handling from #44425 lives inside `StreamingCSVExportCommand` and is unchanged. Real-world impact: we have been running this same change as a runtime patch on a 6.1.0 deployment (AWS ALB in front, idle timeout 60 s at the time). A 380–400k-row table export from a dashboard never completed — minutes of silence, then `TimeoutError` on the gunicorn write and a dropped DB connection. With the change, the same export (402,749 rows, 139.6 MB) completes in 17–25 s, and the query runs once. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF N/A (backend only). ### TESTING INSTRUCTIONS Unit tests (added to `tests/unit_tests/charts/test_chart_data_api.py`): - a large CSV export with `expected_rows` streams without calling `command.execute()`; - below the threshold, without `expected_rows`, `POST_PROCESSED` and XLSX all keep executing through the regular path; - the up-front streaming path returns 403 without the export permission, for both `can_csv` and `can_export_data` (`GRANULAR_EXPORT_CONTROLS`). The three tests covering the new behavior fail on current `master` and pass with this change; the four regular-path tests pass on both. `tests/unit_tests/charts`, `tests/unit_tests/commands/chart` and `tests/unit_tests/common` pass (743 passed, 2 xfailed). Manual: 1. Set `CSV_STREAMING_ROW_THRESHOLD` low (e.g. `1000`) and use a table chart with more rows than that. 2. From the dashboard, choose *Download → Export to .CSV*. 3. The download starts right away; the logs show a single execution of the chart query (from the streaming command) instead of two. ### ADDITIONAL INFORMATION - [ ] Has associated issue: - [ ] Required feature flags: - [ ] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [ ] Introduces new feature or API - [ ] Removes existing feature or API A note on the tests: `security_manager` is a proxy, so a plain `patch("superset.charts.data.api.security_manager")` turns `can_access` into an async mock whose truthy coroutine grants every permission. The new tests patch it with `new_callable=MagicMock` so a denied permission is actually exercised. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
