schizophrenicmaniac opened a new pull request, #45351:
URL: https://github.com/apache/superset/pull/45351
### SUMMARY
Streaming CSV exports (the chart data and SQL Lab paths used for large
results) didn't honor the `encoding` key of `CSV_EXPORT`. Both endpoints set
`charset=<encoding>` in the Content-Type header, but the command yields `str`
chunks and Werkzeug encodes those as plain UTF-8. With the default `utf-8-sig`,
the downloaded file had no BOM, so Excel opened it as ANSI and showed umlauts
as `Müller`. Non-streaming and ZIP exports already encode with the configured
value.
This adds a small `encode_csv_stream()` helper next to the streaming base
command. It encodes the chunks with an incremental encoder, so the BOM is
written once at the start of the file rather than once per chunk. Both
`_create_streaming_csv_response` methods wrap the generator with it, using the
same encoding they already put in the header, so the header and body can't
disagree. Doing it at the response level keeps the commands yielding `str`, so
the existing command tests don't change.
The separator half of the issue is already handled on master: the streaming
path applies `sep` and `decimal` from `CSV_EXPORT` (added for #32371).
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Not applicable, backend only. Before: the streamed file starts with
`name;city` (no BOM). After: it starts with `EF BB BF` followed by `name;city`.
### TESTING INSTRUCTIONS
1. In `superset_config.py` set `CSV_EXPORT = {"encoding": "utf-8-sig",
"sep": ";"}` and set `CSV_STREAMING_ROW_THRESHOLD` low (e.g. `10`) so the
streaming path is used.
2. Run a query or open a chart whose results include non-ASCII text (e.g.
`SELECT 'Müller' AS name`) with more rows than the threshold, then download it
as CSV.
3. Check the first bytes of the file (`xxd export.csv | head -1`): they
should be `efbb bf`, and the file should use `;` as the separator.
4. Open the file in Excel by double-clicking it: the umlauts should display
correctly.
Unit tests: `pytest tests/unit_tests/commands/streaming_export
tests/unit_tests/charts/data/streaming_filename_test.py
tests/unit_tests/sqllab/api_test.py`
### ADDITIONAL INFORMATION
- [x] Has associated issue: Fixes #45347
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in
[SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
