codeant-ai-for-open-source[bot] commented on code in PR #45613:
URL: https://github.com/apache/superset/pull/45613#discussion_r4345953563
##########
superset/commands/database/uploaders/csv_reader.py:
##########
@@ -79,16 +82,30 @@ def _detect_encoding(file: FileStorage) -> str:
sample = file.read(sample_size)
if not sample: # Empty file or reached end
break
+ whole_file = len(sample) < sample_size
for encoding in ENCODING_FALLBACKS:
try:
sample.decode(encoding)
- file.seek(0)
- return encoding
except UnicodeDecodeError:
continue
+ # A start that is valid UTF-8 (often plain ASCII) says nothing
about
+ # the rest of the file, so it only counts once the sample is
the file
+ if whole_file or encoding != "utf-8":
+ file.seek(0)
+ return encoding
+ break
+ # Larger than every sample, and each one read as UTF-8: decide on the
whole file
+ file.seek(0)
+ content = file.read()
file.seek(0)
+ for encoding in ENCODING_FALLBACKS:
+ try:
+ content.decode(encoding)
+ except UnicodeDecodeError:
+ continue
Review Comment:
✅ **CodeAnt verified this suggestion was addressed in subsequent commits and
marked this thread resolved** as of `fccbdc4`.
The full-file validation now uses an incremental decoder and reads
fixed-size chunks, rather than buffering the entire upload in memory.
<sub>If that's not right, unresolve this thread and CodeAnt will leave it
open.</sub>
<!-- codeant-auto-resolve-reply -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
