L4XB commented on code in PR #45613:
URL: https://github.com/apache/superset/pull/45613#discussion_r4345945010
##########
superset/commands/database/uploaders/csv_reader.py:
##########
@@ -79,16 +82,30 @@ def _detect_encoding(file: FileStorage) -> str:
sample = file.read(sample_size)
if not sample: # Empty file or reached end
break
+ whole_file = len(sample) < sample_size
for encoding in ENCODING_FALLBACKS:
try:
sample.decode(encoding)
- file.seek(0)
- return encoding
except UnicodeDecodeError:
continue
+ # A start that is valid UTF-8 (often plain ASCII) says nothing
about
+ # the rest of the file, so it only counts once the sample is
the file
+ if whole_file or encoding != "utf-8":
+ file.seek(0)
+ return encoding
+ break
+ # Larger than every sample, and each one read as UTF-8: decide on the
whole file
+ file.seek(0)
+ content = file.read()
file.seek(0)
+ for encoding in ENCODING_FALLBACKS:
+ try:
+ content.decode(encoding)
+ except UnicodeDecodeError:
+ continue
Review Comment:
Fixed in fccbdc475. The whole-file check now runs an incremental decoder
over 64 KiB chunks (`codecs.getincrementaldecoder`). It holds at most one chunk
in memory and never reads the upload in one call.
`test_csv_reader_encoding_detection_reads_large_files_in_chunks` records every
`read` call on an 84 KB upload and asserts that each call reads at most 65,536
bytes. The test fails on the previous commit, which read the file in one call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
