codeant-ai-for-open-source[bot] commented on code in PR #45613:
URL: https://github.com/apache/superset/pull/45613#discussion_r4345432782
##########
superset/commands/database/uploaders/csv_reader.py:
##########
@@ -79,16 +82,30 @@ def _detect_encoding(file: FileStorage) -> str:
sample = file.read(sample_size)
if not sample: # Empty file or reached end
break
+ whole_file = len(sample) < sample_size
for encoding in ENCODING_FALLBACKS:
try:
sample.decode(encoding)
- file.seek(0)
- return encoding
except UnicodeDecodeError:
continue
+ # A start that is valid UTF-8 (often plain ASCII) says nothing
about
+ # the rest of the file, so it only counts once the sample is
the file
+ if whole_file or encoding != "utf-8":
+ file.seek(0)
+ return encoding
+ break
+ # Larger than every sample, and each one read as UTF-8: decide on the
whole file
+ file.seek(0)
+ content = file.read()
file.seek(0)
+ for encoding in ENCODING_FALLBACKS:
+ try:
+ content.decode(encoding)
+ except UnicodeDecodeError:
+ continue
Review Comment:
**Suggestion:** When the first 64 KiB is valid UTF-8 but later bytes are
not, detection buffers and decodes the entire upload; with the size limit
disabled, large uploads can exhaust worker memory.
**Assessment:** ๐ `Major` ยท ๐ `Occurrence: Rarely` ยท ๐ท๏ธ `Performance`
[](https://docs.codeant.ai/cli/resolve-pr-comments-skill)
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=cb65a25bfdb94c48840437a20e4f0027&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
[](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=cb65a25bfdb94c48840437a20e4f0027&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
<details>
<summary><b>Prompt for AI Agent ๐ค </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/commands/database/uploaders/csv_reader.py
**Line:** 100:107
**Comment:**
*Performance: When the first 64 KiB is valid UTF-8 but later bytes are
not, detection buffers and decodes the entire upload; with the size limit
disabled, large uploads can exhaust worker memory.
Validate the correctness of the flagged issue. If correct, How can I resolve
this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask
user if the user wants to fix the rest of the comments as well. if said yes,
then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a
href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F45613&comment_hash=9c2903a2d8753c51306422bc95fae3de9faf0e43c6a76d7cfc5ae8d4861667e4&reaction=like'>๐</a>
| <a
href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F45613&comment_hash=9c2903a2d8753c51306422bc95fae3de9faf0e43c6a76d7cfc5ae8d4861667e4&reaction=dislike'>๐</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
