gabotechs commented on code in PR #25015:
URL: https://github.com/apache/datafusion/pull/25015#discussion_r3948760693
##########
datafusion/substrait/src/logical_plan/consumer/substrait_consumer.rs:
##########
@@ -500,6 +500,16 @@ pub trait SubstraitConsumer: Send + Sync + Sized {
)
}
+ /// Optional Arrow field metadata for a user-defined type, since Arrow has
+ /// no native representation for some UDTs (e.g. JSON maps to plain
+ /// `DataType::Utf8`). `Ok(None)` (the default) attaches nothing.
+ fn consume_user_defined_type_metadata(
+ &self,
+ _user_defined_type: &r#type::UserDefined,
+ ) -> datafusion::common::Result<Option<HashMap<String, String>>> {
+ Ok(None)
Review Comment:
:+1: this makes sense. I think we can afford to handle any type in general,
not just `r#type::UserDefined`.
The problem that the `r#type::UserDefined` you are looking for might be
nested within a list, map or other nested type, so you probably want to change
the signature of this function to something like:
```rust
fn consume_type_metadata(
&self,
_type: Type,
) -> datafusion::common::Result<Option<HashMap<String, String>>> {
Ok(None)
}
```
##########
datafusion/substrait/src/logical_plan/consumer/types.rs:
##########
@@ -349,7 +349,12 @@ fn from_substrait_struct_type(
for (i, f) in s.types.iter().enumerate() {
let name = next_struct_field_name(i, dfs_names, name_idx)?;
let data_type = from_substrait_type(consumer, f, dfs_names, name_idx)?;
- let field = Field::new(name, data_type, type_is_nullable(f)?);
+ let mut field = Field::new(name, data_type, type_is_nullable(f)?);
+ if let Some(r#type::Kind::UserDefined(udt)) = &f.kind
+ && let Some(metadata) =
consumer.consume_user_defined_type_metadata(udt)?
+ {
+ field = field.with_metadata(metadata);
+ }
fields.push(field);
Review Comment:
This is a breaking change that will have a domino effect into some other
functions though.
##########
datafusion/substrait/src/logical_plan/consumer/types.rs:
##########
@@ -349,7 +349,12 @@ fn from_substrait_struct_type(
for (i, f) in s.types.iter().enumerate() {
let name = next_struct_field_name(i, dfs_names, name_idx)?;
let data_type = from_substrait_type(consumer, f, dfs_names, name_idx)?;
- let field = Field::new(name, data_type, type_is_nullable(f)?);
+ let mut field = Field::new(name, data_type, type_is_nullable(f)?);
+ if let Some(r#type::Kind::UserDefined(udt)) = &f.kind
+ && let Some(metadata) =
consumer.consume_user_defined_type_metadata(udt)?
+ {
+ field = field.with_metadata(metadata);
+ }
fields.push(field);
Review Comment:
I see this PR is just implementing this for structs, but there seems to be
several places where we are missing to extract the metadata, and we are
creating `Field`s with empty metadata, where otherwise the
`consume_user_defined_type_metadata` would provide some:
-
https://github.com/apache/datafusion/blob/main/datafusion/substrait/src/logical_plan/consumer/types.rs#L180-L185
-
https://github.com/apache/datafusion/blob/main/datafusion/substrait/src/logical_plan/consumer/types.rs#L60-L68
-
https://github.com/apache/datafusion/blob/main/datafusion/substrait/src/logical_plan/consumer/types.rs#L201-L209
-
https://github.com/apache/datafusion/blob/main/datafusion/substrait/src/logical_plan/consumer/expr/literal.rs#L341-L353
(maybe I have left some others)
This makes me think that also going one by one to every place where a
substrait type is mapped and adding one extra function call for retrieving the
metadata is error-prone, and there's probably a better solution.
The cleanest thing that comes to mind is to do something like this:
```rust
fn consume_user_defined_type(
&self,
user_defined_type: &r#type::UserDefined,
- ) -> datafusion::common::Result<DataType> {
+ ) -> datafusion::common::Result<Field> {
```
That way people would just return the metadata along with the field in one
shot.
##########
datafusion/substrait/src/logical_plan/consumer/types.rs:
##########
@@ -349,7 +349,12 @@ fn from_substrait_struct_type(
for (i, f) in s.types.iter().enumerate() {
let name = next_struct_field_name(i, dfs_names, name_idx)?;
let data_type = from_substrait_type(consumer, f, dfs_names, name_idx)?;
- let field = Field::new(name, data_type, type_is_nullable(f)?);
+ let mut field = Field::new(name, data_type, type_is_nullable(f)?);
+ if let Some(r#type::Kind::UserDefined(udt)) = &f.kind
+ && let Some(metadata) =
consumer.consume_user_defined_type_metadata(udt)?
+ {
+ field = field.with_metadata(metadata);
+ }
fields.push(field);
Review Comment:
Another option that comes to mind is to use the new method of
`SubstraitConsumer` in the existing `field_from_substrait_type`, and in those
places where `from_substrait_type` is used, use `field_from_substrait_type`
instead, that is modified to guarantee that metadata is properly passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
