[
https://issues.apache.org/jira/browse/ARROW-11263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qingyou Meng updated ARROW-11263:
---------------------------------
Priority: Minor (was: Major)
> [Rust] problem of Field nullable
> --------------------------------
>
> Key: ARROW-11263
> URL: https://issues.apache.org/jira/browse/ARROW-11263
> Project: Apache Arrow
> Issue Type: Bug
> Reporter: Qingyou Meng
> Priority: Minor
>
> Quoting from section *Schema message*
>
> [https://github.com/apache/arrow/blob/master/docs/source/format/Columnar.rst#schema-message]
> {noformat}
> Whether the field is semantically nullable. While this has no bearing on the
> array's physical layout, many systems distinguish nullable and non-nullable
> fields and we want to allow them to preserve this metadata to enable faithful
> schema round trips.{noformat}
> This can be read as: for a field with nullable set as true, when encounters
> null array data from the field, data processor CAN continue or refuse to
> process.
> In current rust implementation, apart from read Fields from schema, we also
> construct `Field` with datafusion and`Field::new`in
> arrow::array::*StructArray*.
> * in datafusion, the nullable is determined by DF schema
> * in arrow::array::StructArray::try_from(values: Vec<(&str, ArrayRef)>) ,
> the nullable is determined actual data. This is error-prone if ArrayRef's
> null buffer are all 1s (built by builder).
> Conclusions:
> * It's questionable to set Field's nullable according to data.
> * Perhaps builders should set null buffer back to None when the buffer has
> all bits set.
> * Enhance StructArray::
> try_from(values: Vec<(&str, ArrayRef)>) don't fail when null buffer has all
> bits set.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
