tustvold commented on issue #4472:
URL: https://github.com/apache/arrow-rs/issues/4472#issuecomment-1615757110
> If we can think of another place to put this information, I'm open to that.
I personally think Field is the correct location for such metadata, imo
DataType should only contain the information for kernels to interpret the
physical array data. Concerns about nullability, specialized kernels for
extension types, etc... I think should be handled at a higher level. Whilst it
isn't a true logical vs physical separation, I think that is a helpful way to
conceptualize it.
> But it would be nice to find a way to not have to do that.
This feels like a limitation of the way the python conversion is implemented
```
impl ToPyArrow for ArrayData {
fn to_pyarrow(&self, py: Python) -> PyResult<PyObject> {
let array = FFI_ArrowArray::new(self);
let schema =
FFI_ArrowSchema::try_from(self.data_type()).map_err(to_py_err)?;
let module = py.import("pyarrow")?;
let class = module.getattr("Array")?;
let array = class.call_method1(
"_import_from_c",
(
addr_of!(array) as Py_uintptr_t,
addr_of!(schema) as Py_uintptr_t,
),
)?;
Ok(array.to_object(py))
}
}
```
In particular the schema is inferred from the array's data type. If instead
there were a way to provide a `Field` then this would allow propagating not
only extension metadata, but also nullability, dictionary ordering, etc... It
would also potentially allow performing the schema conversion once and using
the result for multiple arrays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
