findepi commented on issue #14247: URL: https://github.com/apache/datafusion/issues/14247#issuecomment-2610778483
> However, since the `ReturnTypeInfo` only provides `DataType` the the `Field` information will not be present and thus UDF writers will not be able to access extension type information i guess we have no option and at some point need to do the Arrow way, with the field metadata. That implies that any type-dealing logic needs to take `(DataType, metadata)` pair as the input. For example, geometry can use Utf8 container type, but it won't use Utf8's functions, casts or coercion rules. Once a field is marked to be "an extension type", it cannot be construed as its container type anymore! We can go about this in two ways - try to evolve incrementally, keep using DataType in _many places_, but add metadata to more and more places incrementally. At the end of this, the code is not beautiful (rather hard to reason about), and in the meantime it's not correct - try to decouple ourselves from arrow type system to query planning. This would be logical types https://github.com/apache/datafusion/issues/12622, explicit extension types https://github.com/apache/datafusion/issues/12644 (thus addressing @paleolimbot 's concern above) - this means more foundational work upfront, maybe even adding stuff like new IR (https://github.com/apache/datafusion/issues/12604) allowing decoupling various DF query processing aspects; but should lead to a design we can be more proud about (I am aware natural gravity always leans towards small increments, irrespective of end-game results) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
