alamb commented on code in PR #1566:
URL: https://github.com/apache/arrow-rs/pull/1566#discussion_r851256154
##########
arrow/src/datatypes/field.rs:
##########
@@ -116,14 +116,25 @@ impl Field {
/// Returns a (flattened) vector containing all fields contained within
this field (including it self)
pub(crate) fn fields(&self) -> Vec<&Field> {
let mut collected_fields = vec![self];
- match &self.data_type {
+ collected_fields.append(&mut self._fields(&self.data_type));
+
+ collected_fields
+ }
+
+ fn _fields<'a>(&'a self, dt: &'a DataType) -> Vec<&Field> {
+ let mut collected_fields = vec![];
+
+ match dt {
DataType::Struct(fields) | DataType::Union(fields, _) => {
collected_fields.extend(fields.iter().flat_map(|f| f.fields()))
}
DataType::List(field)
| DataType::LargeList(field)
| DataType::FixedSizeList(field, _)
| DataType::Map(field, _) => collected_fields.push(field),
+ DataType::Dictionary(_, value_field) => {
+ collected_fields.append(&mut
self._fields(value_field.as_ref()))
+ }
Review Comment:
I wonder if it is worth adding an error for types that aren't explicitly
supported (I realize this will likely need a bunch of plumbing to return
`Result`)
like
```rust
_ => return Err("Type not supported")
```
##########
arrow/src/datatypes/field.rs:
##########
@@ -116,14 +116,25 @@ impl Field {
/// Returns a (flattened) vector containing all fields contained within
this field (including it self)
pub(crate) fn fields(&self) -> Vec<&Field> {
let mut collected_fields = vec![self];
- match &self.data_type {
+ collected_fields.append(&mut self._fields(&self.data_type));
+
+ collected_fields
+ }
+
+ fn _fields<'a>(&'a self, dt: &'a DataType) -> Vec<&Field> {
+ let mut collected_fields = vec![];
+
+ match dt {
DataType::Struct(fields) | DataType::Union(fields, _) => {
collected_fields.extend(fields.iter().flat_map(|f| f.fields()))
}
DataType::List(field)
| DataType::LargeList(field)
| DataType::FixedSizeList(field, _)
| DataType::Map(field, _) => collected_fields.push(field),
+ DataType::Dictionary(_, value_field) => {
+ collected_fields.append(&mut
self._fields(value_field.as_ref()))
Review Comment:
the key improvement of the PR is that this now recurses into children, right?
##########
arrow/src/ipc/writer.rs:
##########
@@ -175,13 +186,37 @@ impl IpcDataGenerator {
)?;
}
}
+ _ => (),
Review Comment:
maybe this should return an error for unsupported nested types
##########
arrow/src/ipc/reader.rs:
##########
@@ -1439,4 +1440,42 @@ mod tests {
let output_batch = roundtrip_ipc_stream(&input_batch);
assert_eq!(input_batch, output_batch);
}
+
+ #[test]
+ fn test_roundtrip_stream_nested_dict_dict() {
Review Comment:
I tried running this test without the code changes in this PR and it failed
thusly. 👍
```
---- ipc::reader::tests::test_roundtrip_stream_nested_dict_dict stdout ----
thread 'ipc::reader::tests::test_roundtrip_stream_nested_dict_dict' panicked
at 'called `Result::unwrap()` on an `Err` value:
InvalidArgumentError("dictionary id not found in schema")',
arrow/src/ipc/reader.rs:1331:32
```
when I removed the field code
and like this when I removed the writer support
```
---- ipc::reader::tests::test_roundtrip_stream_nested_dict_dict stdout ----
thread 'ipc::reader::tests::test_roundtrip_stream_nested_dict_dict' panicked
at 'called `Option::unwrap()` on a `None` value', arrow/src/ipc/reader.rs:176:64
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
