paleolimbot commented on issue #7063: URL: https://github.com/apache/arrow-rs/issues/7063#issuecomment-2794478219
> not based on what is expedient for one particular downstream use-case Agreed! There are also a number of arrow-rs' own crates that make heavy use of the `DataType` and `ArrayRef` (e.g., the Parquet internals, casting). Another approach might be an `arrow-extensions` crate that includes an extension-aware `cast()`, CSV reader/writer (i.e., extension authors can choose how an array is represented in a CSV file), Parquet reader/writer (the issue here...how to propagate an extension type to Parquet and vice versa), extension-aware pretty printer, etc. All of these utilities are made use of by DataFusion but there is nothing strictly related to DataFusion about them and the work we do to (e.g.) support Variant, Geometry/Geography, or UUIDs doesn't have to only benefit DataFusion users. > what do you think about filing a ticket in DataFusion like "Better extension type support" where we can start listing the APIs that would need to be updated to Field instead of DataType? Yes, I'll continue my efforts there, too! In addition to requiring a `Field` a number of them also require a `Context` of some kind to look up extension-specific behaviour (e.g., whether a cast should or should not happen). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
