alamb commented on a change in pull request #1339:
URL: https://github.com/apache/arrow-rs/pull/1339#discussion_r816218983
##########
File path: arrow/src/ipc/reader.rs
##########
@@ -581,14 +603,17 @@ pub struct FileReader<R: Read + Seek> {
/// Metadata version
metadata_version: ipc::MetadataVersion,
+
+ /// Optional projection
Review comment:
```suggestion
/// Optional projection and projected_schema
```
##########
File path: arrow/src/ipc/reader.rs
##########
@@ -845,11 +888,23 @@ impl<R: Read> StreamReader<R> {
// Create an array of optional dictionary value arrays, one per field.
let dictionaries_by_field = vec![None; schema.fields().len()];
+ let projection = projection.map(|projection| {
Review comment:
same here -- `Schema::projection` might make this code easier to read
##########
File path: arrow/src/ipc/reader.rs
##########
@@ -922,7 +977,7 @@ impl<R: Read> StreamReader<R> {
let mut buf = vec![0; message.bodyLength() as usize];
self.reader.read_exact(&mut buf)?;
- read_record_batch(&buf, batch, self.schema(),
&self.dictionaries_by_field).map(Some)
+ read_record_batch(&buf, batch, self.schema(),
&self.dictionaries_by_field, self.projection.as_ref().map(|x|
x.0.as_ref())).map(Some)
Review comment:
maybe it is worth adding something like
```rust
impl<R: Read> StreamReader<R> {
...
/// get projected schema, if any
pub fn projected_schema(&self) -> Option<&Schema> {
...
}
```
##########
File path: arrow/src/ipc/reader.rs
##########
@@ -808,6 +848,9 @@ pub struct StreamReader<R: Read> {
///
/// This value is set to `true` the first time the reader's `next()`
returns `None`.
finished: bool,
+
+ /// Optional projection
Review comment:
```suggestion
/// Optional projection and projected schema
```
##########
File path: arrow/src/ipc/reader.rs
##########
@@ -672,6 +697,18 @@ impl<R: Read + Seek> FileReader<R> {
};
}
+ let projection = projection.map(|projection| {
+ let fields = projection
+ .iter()
+ .map(|x| schema.fields[*x].clone())
Review comment:
Perhaps we could reuse `Schema::project`:
https://docs.rs/arrow/9.1.0/arrow/datatypes/struct.Schema.html#method.project
(which also handles metadata correctly)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
