emilk opened a new issue, #10657: URL: https://github.com/apache/arrow-rs/issues/10657
## Propsal The `Display` of `Datatype` currently adds `non-null` markers for fields with `nullability=false`. I want to instead mark the nullable fields with a `nullable` marker: Proposed change: * `List(non-null Uint8)` -> `List(Uint8)` * `List(Uint8)` -> `List(nullable Uint8)` ## Motivation ### `nullable` is a concept in arrow-rs In the source code, the term `nullable` is ubiquitous, but NOT `non-null`. For instance: * https://docs.rs/arrow/latest/arrow/array/trait.Array.html#method.is_nullable * https://docs.rs/arrow/latest/arrow/datatypes/struct.Field.html#method.is_nullable * https://docs.rs/arrow/latest/arrow/ffi/struct.FFI_ArrowSchema.html#method.nullable ### `non-null` has a negation `non-null` describes what something _isn't_. This is completely ~backwards~ non-forwards. We should describe what things _are_, rather than what they _aren't_. ### Nullability as a type Modern languages recognizes that the ability to assign `null` (or `nil` or `None`…) to something is an _addition_ to its type. In Rust, `Option` marks something as "nullable". In haskell it is `Maybe`. The point is: if something can be _absent_ we _add more text to the type_. That is also what a `nullable` prefix would do. ## Migration One big problem is backwards compatibility. In particular: how should `DataType::from_str` handle old strings? When parsing `"List(Uint8)"` from an old Arrow version, we should set `nullable=true`. But when parsing new strings, `"List(Uint8)"` means `nullable=false`. ### Solution A: break compatibility The simple solution: break compatibility of the `Display/parse` of `DataType`. We do major releases anyway, so that's fine. Also: there are so many `DataType`s we still don't properly parse. ### Solution B: add some other marker to distinguish Maybe we change `(…)` into `<…>`, so that going forward we write `List<Uint8>`. That way we can tell old string from new strings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
