findepi commented on issue #14247: URL: https://github.com/apache/datafusion/issues/14247#issuecomment-2612715603
I am fine with DF not shipping extension types (ie no extension types until we add them explicitly in https://github.com/apache/datafusion/issues/12644). Let's look at the example. I have `Field` information for `a` and `b` columns, both are `DataType::Binary`. What should be the behavior of `a = b` according to DF core's logic? Naive answer would be that given same DataType, the expression is valid and Arrow's comparison function should be used. However, if one (or both) of then happen to be custom-provided extension type (e.g. JSON, VARIANT, ST_Geometry), this logic should not be used at all, not even as a fallback. So the bare minimum is -- given a Field, DF core needs to understand whether this is a type it knows about or a type it doesn't know about.... So we're almost back to explicit extension types. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
