tustvold commented on issue #8730: URL: https://github.com/apache/arrow-rs/issues/8730#issuecomment-3772825484
> The central idea is that in arrow-rs, once an array has been associated with a Field (where there is potentially metadata), there is a type NativeOrExtensionArray that encodes the possibility of metadata in the type system. As a result, if any code encounters a NativeOrExtensionArray there is a possibility that an array must be interpreted differently. If the code encounters an ArrayRef it must be interpreted as the underlying DataType. From my experience, the RecordBatch will be the central point of use for this new type. This hopefully should make it clear to downstream users (e.g., DataFusion) that metadata needs to be preserved and opting out of it needs an explicit method call. I think the challenge with this approach will be nested types like ListArray and StructArray, which contain Field -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
