raghavyadav01 commented on issue #26514:
URL: https://github.com/apache/datafusion/issues/26514#issuecomment-6283564493
I looked into this. The reader and writer disagree because `escape` is
conditional on the writer side only.
**Read path** — `datafusion/datasource-csv/src/file_format.rs:579` applies
the escape unconditionally:
```rust
if let Some(escape) = self.options.escape {
format = format.with_escape(escape);
}
```
**Write path** — `datafusion/common/src/file_options/csv_writer.rs:91`
passes it to arrow's `WriterBuilder`:
```rust
if let Some(v) = &value.escape {
builder = builder.with_escape(*v)
}
```
but `arrow-csv` documents `with_escape` as "only used when `double_quote` is
disabled", and `CsvOptions::double_quote` defaults to `None`
(`common/src/config.rs:3982`), so DataFusion never calls `with_double_quote`
and arrow's default of `true` wins. The escape is accepted, plumbed through,
and then silently inert — which is why the file is written with quote-doubling
and read back with backslash escaping.
So `format.escape` on its own is only half-applied. Adding
`'format.double_quote' 'false'` alongside it makes the repro round-trip
correctly, which confirms the mechanism.
Two ways to fix, and I'd rather check which you prefer before opening a PR:
1. **Make the write path honour `escape`** — when `escape` is set and
`double_quote` is unset, default `double_quote` to `false` for writing. This
makes the documented "available when reading or writing" promise true and fixes
the round-trip. It does change the bytes written for anyone setting `escape`
today, though those writes are already unreadable with the same options.
2. **Validate instead** — reject `escape` without `double_quote = false` on
write, so the failure is loud rather than silent data loss.
I'd lean towards (1) with a round-trip sqllogictest, since the current
behaviour drops rows with no error. Happy to implement either — just let me
know which you'd prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
