kmitchener opened a new issue, #3658:
URL: https://github.com/apache/arrow-datafusion/issues/3658
**Describe the bug**
When inferring the schema, the complete CSV will be read into memory even if
you leave it at the default 1000 rows to infer from.
**To Reproduce**
```rust
let df = ctx.read_csv("./test/", CsvReadOptions::new()).await?;
```
Happens here:
https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/datasource/file_format/csv.rs#L109
**Expected behavior**
It should read in only as much data as it needs for the given row count to
infer data from.
**Additional context**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
