limenilbuz opened a new issue, #11046:
URL: https://github.com/apache/arrow-rs/issues/11046
### Describe the bug
I recently had a use case where I had to supply both a schema hint and
virtual columns (specifically row numbers) to `ArrowReaderOptions`, which was
then used to construct an `ArrowReaderMetadata`.
The original code only asked for the virtual columns. When I changed the
code to supply both virtual columns AND a schema hint, I observed some
interesting behavior.
When only supplying virtual columns, the result of
`ArrowReaderMetadata::schema` included the virtual column fields as supplied in
`ArrowReaderOptions::with_virtual_columns`.
When supplying BOTH virtual columns and schema hint, the result of
`ArrowReaderMetadata::schema` did NOT include the virtual column fields.
This caused issues for my downstream code that did some inspection on this
schema to fail, since the column no longer exists.
Initially, I tried to work around this issue by including the virtual column
IN the schema hint. This caused issues in
`ArrowReaderMetadata::with_supplied_schema` since the number of fields in the
schema did not match the number of fields in the Parquet file, with the error
"Incompatible supplied Arrow schema: expected X columns received Y".
I eventually landed on a workaround specific to my use case that is not
translatable to a generic solution, but this behavior still left a lot to be
desired.
### To Reproduce
```rust
let row_number_field = Field::new("row_number", DataType::Int64,
false).with_extension_type(RowNumber)
let options =
ArrowReaderOptions::new().with_virtual_fields(vec![row_number_field]);
// uncomment this line:
// let options = options.with_schema(some_schema)
let metadata = ArrowReaderMetadata::load(some_reader, options)?;
assert!(metadata.schema().fields.find("row_number").is_some())
```
### Expected behavior
I'd expect one of two things to happen:
1. Include the virtual column in the resulting schema
2. Don't error when schema hint contains virtual columns
### Additional context
I'd understand if this behavior is not considered a bug. After all, the
library is simply executing the contract of `with_schema`, but it still led to
some head scratching.
I appreciate the work done for this library, thanks all!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
