aarmens702-hub opened a new issue, #76386:
URL: https://github.com/apache/airflow/issues/76386
Trying `LLMFileAnalysisOperator` on a small CSV on main (5114080, common.ai
1.1.0, macOS arm64, Python 3.12), the model gets the sample rows with their
quotes removed. A value with a comma in it then looks like two values:
```python
from pathlib import Path
from airflow.providers.common.ai.utils.file_analysis import
build_file_analysis_request
Path("people.csv").write_text('name,city,amount\n"Smith, John","Vancouver,
BC",120\nAlice,Toronto,80\n')
request = build_file_analysis_request(
file_path="people.csv", file_conn_id=None, prompt="Total amount per
city?",
multi_modal=False, max_files=20, max_file_size_bytes=5 * 1024 * 1024,
max_total_size_bytes=20 * 1024 * 1024, max_text_chars=100_000,
sample_rows=10,
)
print(request.user_content)
```
```
Header: name, city, amount
Sample rows:
Smith, John, Vancouver, BC, 120
Alice, Toronto, 80
```
The header has 3 columns and the first row now shows 5 values, so a question
like "total amount per city" has nothing reliable to work from. A quoted value
with a line break in it is split across two lines the same way. This is the
text `execute()` sends as the prompt:
https://github.com/apache/airflow/blob/51140809d2a7299955bdba51588947eeb6f015c1/providers/common/ai/src/airflow/providers/common/ai/operators/llm_file_analysis.py#L142
`_render_csv` reads the file with `csv.reader`, which removes the quotes,
then joins each row back with `", "`:
https://github.com/apache/airflow/blob/51140809d2a7299955bdba51588947eeb6f015c1/providers/common/ai/src/airflow/providers/common/ai/utils/file_analysis.py#L485-L488
JSON, Parquet and Avro samples go through `dumps_masked`, so they keep their
structure (Parquet, for example:
[L543](https://github.com/apache/airflow/blob/51140809d2a7299955bdba51588947eeb6f015c1/providers/common/ai/src/airflow/providers/common/ai/utils/file_analysis.py#L543)).
CSV is the only one that loses it.
What I'd suggest: quote a value the way CSV does when it holds a comma, a
quote or a line break, and leave the rest alone, so the current `alpha, 1`
format stays the same:
```python
def _join_csv_values(values: list[str]) -> str:
quoted = [
'"' + value.replace('"', '""') + '"' if any(char in value for char
in ',"\r\n') else value
for value in values
]
return ", ".join(quoted)
```
The first row then reads `"Smith, John", "Vancouver, BC", 120`.
I tried this locally:
- a new test in `test_file_analysis.py` (a quoted comma, an escaped quote
and a line break) fails on main and passes with the change;
- `test_file_analysis.py`, `test_llm_file_analysis.py` and
`test_object_storage.py` pass (120 passed, 8 skipped because fastavro isn't
installed);
- running the operator itself with a `FunctionModel` that records its prompt
shows the same thing before and after.
**What should happen instead:** each CSV value should reach the model
intact, so the sample rows have the same columns as the header.
I didn't find an existing issue or PR for this. I have the fix and the test
ready and will open a PR.
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before
posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
