GitHub user fbx31 added a comment to the discussion: Implementing User Defined
Types and Custom Metadata in DataFusion - Apache DataFusion Blog
Hello, thanks a lot for this article, helped me a lot but unfortunately, I am
still failing to obtain the field metadata. inside a dataframe.
I implemented an async udf which works fine and I added the
return_field_from_args function as you mentioned.
fn return_field_from_args(&self, args: ReturnFieldArgs) ->
datafusion::error::Result<FieldRef> {
let field_name = INDEXED_VALUE.to_string() +
args.arg_fields.get(0).map_or_else(|| "", |f| f.name());
Ok(Arc::new(
Field::new(field_name,
DataType::new_fixed_size_list(DataType::Float32, self.dims as i32,true),
true).with_metadata(
[("OSM:index".to_string(), self.name.to_string())]
.into_iter()
.collect(),
),
))
}
I call df.with_column(alias, my_func_call).create_physical_plan().await then
execute_stream on this plan.
When I print schema with debug (:?), I get all fields but NO metadata attached
to the field I created.
Did I miss something ? Is with_colum erasing my metadata ?
BTW, a similar article explaining how to add medata to a dataframe's schema
could be very helpfull as well. I am struggling with this use case as well.
Thanks for any help
GitHub link:
https://github.com/apache/datafusion-site/discussions/196#discussioncomment-17224814
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
