alamb commented on issue #18223: URL: https://github.com/apache/datafusion/issues/18223#issuecomment-3433202429
@tobixdev has a great suggestion on https://github.com/apache/datafusion/issues/12644#issuecomment-3427617206 > > Having something Arc is nice because it limits the points at which a "registry" is necessary. > > That's a valid point. If we further believe that all the metadata handling should be attached to `Field`, maybe the field should not directly have a physical `DataType`. More like something akin to: > > enum FieldDataType { > Native(DataType), > Extension(Arc<dyn ExtensionTypeyThing>) > } > This could i) allow arrow kernels to keep on focusing on `DataType` to interpret the underlying physical layout (which I think was the primary reason for rejecting the `DataType::Extension(...)` solution and ii) prevent accidentally using the wrong implementation of, for example, the `+` operator for an extension type as the type information stems from the `Field`. Again, a huge breaking change but I think this cannot be prevented. We could offer a utility functions that allows ignoring any logical type to ease the migration. > > Any thoughts on that? > > This would be a similar approach to `TypeSignature`(https://github.com/apache/datafusion/blob/main/datafusion/common/src/types/logical.rs) . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
