L4XB opened a new pull request, #45613:
URL: https://github.com/apache/superset/pull/45613
### SUMMARY
If a CSV upload is not valid UTF-8, `CSVReader._read_csv` asks
`_detect_encoding` for another encoding and reads the file again. The detection
had two defects. Together they make a common kind of file, a CSV saved by Excel
on Windows (Windows-1252), fail outright or load with the wrong characters.
**1. The first readable sample decided.** `_detect_encoding` tries samples
of 1, 8, 32 and 64 KB and returned the first encoding that decoded one. In a
typical file the header and the first rows are plain ASCII, and ASCII is valid
UTF-8, so the 1 KB sample came back as `utf-8`. That is the encoding that had
just failed, so the upload stopped with `Parsing error: 'utf-8' codec can't
decode byte 0xe9 …`, even though the larger samples would have revealed the
real encoding:
```python
content = "name,city\n" + "Anna,Berlin\n" * 200 + "José,München\n"
CSVReader().file_to_dataframe(FileStorage(io.BytesIO(content.encode("cp1252"))))
# master: DatabaseUploadFailed: Parsing error: 'utf-8' codec can't decode
byte 0xe9 in position 2413
```
**2. latin-1 came before cp1252.** latin-1 decodes every byte, so `cp1252`
(and `iso-8859-1`, the same codec as latin-1) could never be chosen. A Windows
file's curly quotes, dashes and euro sign (bytes 0x80–0x9F) were stored as C1
control characters:
```python
"name,note\nShop,“Best” – 5 €\n".encode("cp1252")
# master: [['Shop', '\x93Best\x94 \x96 5 \x80']]
# this PR: [['Shop', '“Best” – 5 €']]
```
Changes:
- `ENCODING_FALLBACKS` is now `["utf-8", "cp1252", "latin-1"]`. cp1252 reads
every Latin-1 letter the same way and also covers 0x80–0x9F. latin-1 stays last
for the five bytes cp1252 leaves undefined.
- A sample that decodes as UTF-8 settles the question only when it is the
whole file. Otherwise the next, larger sample is tried. When even the largest
sample reads as UTF-8, the whole file decides.
- A single-byte encoding that decodes a sample is still accepted right away,
as before.
### TESTING INSTRUCTIONS
- `pytest tests/unit_tests/commands/databases/csv_reader_test.py`: 66 passed.
- New `test_csv_reader_encoding_detection_cp1252_punctuation` and
`test_csv_reader_encoding_detection_past_an_ascii_start` both fail on `master`.
The second one covers a file past the first sample and one past the largest
sample (about 84 KB).
- `test_csv_reader_progressive_encoding_detection` now accepts `cp1252`
alongside `latin-1`. For the bytes it uses (`café,résumé,naïve`), both decode
to the same text.
- `ruff check` / `ruff format` (0.9.7, as in `requirements/development.txt`)
are clean. `mypy --check-untyped-defs` reports nothing in `csv_reader.py`, and
`pylint` reports the same messages as on `master`.
Manual check: upload a Windows-1252 CSV whose non-ASCII characters start
after the first kilobyte (CSV upload, default options). It now loads, and `é`,
`ü`, `“”`, `–` and `€` arrive as written.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in
[SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
