rusackas opened a new pull request, #45363: URL: https://github.com/apache/superset/pull/45363
### SUMMARY Design discussion: https://github.com/apache/superset/discussions/45345 This is a **draft / work-in-progress PR tied to an unvoted SIP idea** (hence `hold:sip`). It makes the discussion's design real enough to produce a working mechanism and a real benchmark, but is not meant to be merged as-is. **What this does:** lets `/api/v1/chart/data` serve query results as Apache Arrow IPC bytes instead of JSON, as an **opt-in, per-chart-type** mechanism: - A new `ChartMetadata.supportsArrowFormat` boolean (same pattern as the existing `parseMethod`/`enableNoResults` fields) controls whether a viz type's live chart-data requests ask for `result_format: 'arrow'` instead of `'json'`. Unset/false (the default) is identical to today's behavior. - The one call site that decides this (`exploreJSON` in `chartAction.ts`) resolves it from the viz type's registered `ChartMetadata` — there is no other path by which `result_format` can become `'arrow'` for a live render. - On the wire, Arrow bytes are parsed with `apache-arrow`'s `tableFromIPC` and converted back into the exact same `DataRecord[]` row-object shape JSON already produces, in a shim (`arrowBufferToQueryData`) that sits at the single network chokepoint (`v1ChartDataRequest`). Every plugin's `transformProps` — flagged or not — receives identical row dicts; native columnar consumption per-plugin is explicitly **not** part of this PR (that's the "later, optional" follow-up from the discussion). - Because the flag defaults to unset and nothing else can trigger Arrow, **a third-party/custom viz plugin that never sets `supportsArrowFormat` is completely unaffected** by any part of this change — same request, same response parsing, same data shape, byte-for-byte. - ~24 `plugin-chart-echarts` chart types pick this up in one edit via the shared `EchartsChartPlugin` metadata factory; ~10 other built-in plugins (table, pivot-table, ag-grid-table, word-cloud, deck.gl layers, etc.) are flipped individually. **Backend:** adds an Arrow branch to `ChartDataRestApi._send_chart_response` (`superset/charts/data/api.py`) mirroring the existing, already-shipped Arrow branch on the datasource query endpoint (`DatasourceRestApi._run`) — same mimetype, same single-query constraint. Lifts the `NoneOf([ARROW])` restriction on `superset/charts/schemas.py`'s `result_format` field that previously hard-rejected Arrow on this endpoint (that restriction's own docstring said it existed only because this branch didn't exist yet). **Known gap (flagged honestly, not hidden):** the Arrow IPC bytes carry only the DataFrame — none of the JSON response's sidecar query metadata (`cache_key`, `is_cached`, the SQL `query` text, `status`, `timing`, `annotation_data`, `warning`, applied/rejected filters) travels over Arrow today. The frontend shim fills these with inert defaults (null/false/empty) so UI chrome that reads them degrades quietly instead of throwing, but a flagged chart's cache badge, "View query" action, and timing panel will not be accurate until a follow-up extends the wire format to carry that metadata too. A second, real (and more interesting) divergence found while verifying fidelity: `_to_arrow_ipc` has no non-finite sanitization, unlike the JSON path's `df_to_records` (which maps `+/-Infinity` to `null`). So raw Arrow bytes can carry real IEEE-754 `Infinity` values that JSON never would for the same data. The frontend shim corrects for this explicitly (see `convertNumber` in `arrowToQueryData.ts`) so a flagged chart can't observe a value a JSON-format chart never could for the same query — verified empirically against real `pyarrow`-produced bytes, not asserted from memory. **Explicitly out of scope:** SQL Lab's own execute/results endpoints and grid (`superset/sql_lab.py`, `superset-frontend/src/SqlLab/`). That path has no per-plugin/`ChartMetadata` concept to gate on and is architecturally different from the chart-data endpoint — left alone deliberately, flagged here as a follow-up rather than an oversight. ### BENCHMARK Standalone script: `scripts/benchmark_arrow_vs_json.py`. No running dev server was used for these numbers — see below for why. It reuses the real production serialization code on both sides (`superset.dataframe.df_to_records` + `superset.utils.json.dumps(..., default=json_int_dttm_ser, ignore_nan=True)` for JSON; the verbatim body of `QueryContextProcessor._to_arrow_ipc` for Arrow) against real example-dataset fixtures already checked into the repo (`superset/examples/*/data.parquet`), not synthetic data. Real, measured output: ``` scenario json bytes arrow bytes arrow vs json ------------------------------------------------------------------------------------------------ mixed string+numeric (video_game_sales, 500 rows) 147,550 61,432 -58.4% datetime-heavy (usa_births_names, 500 rows) 60,434 29,656 -50.9% numeric-heavy, 328 cols (world_health, 500 rows) 4,457,177 1,420,760 -68.1% mixed string+numeric (video_game_sales, 5000 rows) 1,453,614 591,784 -59.3% datetime-heavy (usa_births_names, 5000 rows) 600,569 279,592 -53.4% numeric-heavy, 328 cols (world_health, 5000 rows) 45,166,779 13,503,192 -70.1% ``` Arrow IPC payloads come in roughly **50-70% smaller** than the equivalent JSON across these scenarios, with the gap widening on wider, more numeric-heavy result sets. A local dev server was running at the time of writing, but it belongs to an independent docker-compose stack for a different worktree (not this branch), so it isn't serving this change — redeploying someone else's running stack with this branch's code was out of scope for this PR, per the "don't spend significant time standing up a dev server for this" guidance. **The numbers above are the only real numbers in this PR; no live-HTTP before/after comparison was attempted or fabricated.** ### TESTING INSTRUCTIONS Backend: ``` pytest tests/unit_tests/charts/test_chart_data_api.py tests/unit_tests/datasource/test_query_api.py ``` New/changed coverage: `test_send_chart_response_arrow_returns_single_query_bytes`, `test_send_chart_response_arrow_rejects_multiple_queries`, `test_chart_data_schema_accepts_arrow`. Frontend (single worker): ``` npx jest --runInBand packages/superset-ui-core/test/query packages/superset-ui-core/test/chart/models \ src/explore/exploreUtils/exploreUtils.test.tsx src/components/Chart/chartActions.test.ts npx jest --runInBand plugins/plugin-chart-table plugins/plugin-chart-echarts/test/Pie plugins/plugin-chart-echarts/test/Funnel plugins/preset-chart-deckgl/src ``` New coverage: `arrowToQueryData.test.ts` (fidelity against real `pyarrow`-produced IPC bytes: strings/nulls/NaN/booleans/naive+tz-aware timestamps/dates, Int64 in- and out-of-safe-range, Decimal128, and the Infinity-nulling correction), `ChartMetadata.test.ts` (flag defaulting), `exploreUtils.test.tsx` (`supportsArrowResultFormat`), `chartActions.test.ts` (end-to-end: opted-in viz type → `result_format: 'arrow'` + `parseMethod: 'raw'` on the request → converted row dicts on the dispatched success action). Manually: this PR does not change behavior for any plugin unless `ChartMetadata.supportsArrowFormat` is set, so existing manual testing of any dashboard/chart should show zero behavior change. ### ADDITIONAL INFORMATION - [ ] Has associated issue: - [ ] Required feature flags: - [x] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [x] Introduces new feature or API - [ ] Removes existing feature or API 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
