tustvold commented on issue #8730: URL: https://github.com/apache/arrow-rs/issues/8730#issuecomment-3769372413
_I presume the proposal is to add `DataType::Extension(Box<DataType>, Box<dyn Any>)`, otherwise it isn't clear to me how downcasting would work. Or alternatively the `FieldType` proposal from above, but replacing `DynExtensionType` with `dyn Any`._ If so how do these arrays get created? I presume doing so needs the `Field` information, else how do you know the type? What happens if the array now gets assigned to a field with different metadata? AFAICT this proposal introduces the potential for inconsistency without changing the need for the `Field` information to be plumbed through, alongside potentially some sort of extension registry, whilst also simultaneously increasing number of places this information needs to be propagated? > it's that the people willing to put in the work to implement extension types aren't interested in doing so (purely based on how long some of these tickets have been open). If lack of engagement is the issue, how would moving this into arrow-rs which has even more constrained reviewer bandwidth help? > We are now up to 6 canonical extension types with at least one more being discussed on the mailing list...I think it is reasonable that people (not just DataFusion) want to cast, print, serialize to/from JSON, read and write CSVs, etc. without having to rewrite their APIs. Extension types are no longer metadata that can just be dropped when convenient for many users of arrow-rs. Perhaps this is the key question, is the eventual goal for arrow-rs to support canonical extension types natively. If so, then that would perhaps motivate defining a mechanism for an extension type registry, etc... If not, as is currently the case, it is unclear to me what benefit is derived from such an abstraction being defined in arrow-rs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
