yangbaechu opened a new issue, #10382:
URL: https://github.com/apache/arrow-rs/issues/10382
### Describe the bug
When an `arrow_csv::Reader` is configured with
`ReaderBuilder::with_projection`, `RecordBatchReader::schema()` returns the
original unprojected schema, while the yielded `RecordBatch` uses the projected
schema.
Therefore, the schema advertised by the reader does not match the schema of
its batches.
### To Reproduce
```rust
use std::io::Cursor;
use std::sync::Arc;
use arrow_array::RecordBatchReader;
use arrow_csv::ReaderBuilder;
use arrow_schema::{DataType, Field, Schema};
#[test]
fn projected_reader_schema_matches_batch_schema() {
let schema = Arc::new(Schema::new(vec![
Field::new("a", DataType::Int32, false),
Field::new("b", DataType::Int32, false),
]));
let mut reader = ReaderBuilder::new(schema)
.with_projection(vec![1])
.build(Cursor::new(b"1,2\n"))
.unwrap();
let reader_schema = RecordBatchReader::schema(&reader);
let batch = reader.next().unwrap().unwrap();
assert_eq!(reader_schema, batch.schema());
}
```
The assertion fails:
```text
reader schema: [a: Int32, b: Int32]
batch schema: [b: Int32]
```
### Expected behavior
`RecordBatchReader::schema()` should return the projected schema used by the
batches yielded by the reader. In this example, both schemas should contain
only field `b`.
### Additional context
Reproduced on commit
[`cc83f7e`](https://github.com/apache/arrow-rs/commit/cc83f7eb25fbf0f6e8f94656d8fd703470e32c04)(`arrow-csv`
59.1.0).
`Reader::schema()` applies the projection, but the `RecordBatchReader`
implementation returns `self.decoder.schema.clone()` directly:
https://github.com/apache/arrow-rs/blob/cc83f7eb25fbf0f6e8f94656d8fd703470e32c04/arrow-csv/src/reader/mod.rs#L565-L568
The `RecordBatchReader` implementation was originally added in
[#4195](https://github.com/apache/arrow-rs/pull/4195). I could not find an
existing issue or PR reporting this specific mismatch.
I would be happy to work on a fix and add regression tests if the
maintainers agree with the expected behavior described above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
