aarmens702-hub commented on code in PR #76699:
URL: https://github.com/apache/airflow/pull/76699#discussion_r4505626945
##########
providers/common/ai/src/airflow/providers/common/ai/utils/file_analysis.py:
##########
@@ -649,7 +649,9 @@ def _read_limited_bytes(handle: io.BufferedIOBase, *, path:
ObjectStoragePath, m
def _decode_text(data: bytes) -> str:
- return data.decode("utf-8", errors="replace")
+ # utf-8-sig drops the byte order mark Excel and Windows tools write, which
json.loads
+ # refuses and which would otherwise be read as part of the first CSV
column's name.
+ return data.decode("utf-8-sig", errors="replace")
Review Comment:
Good catch, you’re right: 5.1 defaults to UTF-16, so my description only
covered -Encoding UTF8. I cherry-picked your commit (kept you as author) and
updated the comment and test name to cover both cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
