Personally, I find the nullability flag to be confusing in general. I
guess a larger question is whether an Arrow schema is intended to just
describe the data itself (this batch) or whether it is intended to describe
something larger / more persistent (e.g. a table, dataset, stream of data,
etc.)
If it's the former (just describes the data) then it is redundant. The
presence or absence of a validity bitmap is usually enough to know if the
data has nulls.
If it's the latter (describing something larger) then it is difficult to
interpret without knowing what exactly we are describing.
> 1) a non-nullable child field in a struct with some nulls in the parent
> (implying that some child elements are logically null)
This one seems the least controversial to me. Let's say we have `outer:
struct<inner: int32>` and `outer` is nullable and `inner` is non-nullable.
Valid: [{"inner": 3}, null, null, {"inner": 5}]
Invalid: [{"inner": 3}, null, {"inner": null}]
For cases 2 & 3 I would consider the field semantically nullable if either
of the two options were nullable.
> What do other implementations do / what do you suggest implementations
> *should* do?
This falls into the same "it's probably a moot point this late in the game"
argument from the thread Raúl linked but I'd argue that implementations
should at least (and probably at most) validate the nullability upon array
creation.
1. If a child array is marked non-nullable but the parent struct has a null
then the child array should not have a validity bitmap and should have a
garbage value in the slot.
2. If a dictionary is non-nullable then the dictionary should not have a
validity bitmap even if the indices contain nulls. If the indices are
non-nullable then the indices should not have a validity bitmap even if the
dictionary has nulls.
3. The parent nullability flag is completely unused during validation.
At this point, however, I have no faith in the nullability flag as proof
that the array has been validated and I completely ignore it and always do
my own validation when receiving outside data (even if it is from a trusted
source).
On Tue, Jul 21, 2026 at 9:00 AM Raúl Cumplido <[email protected]> wrote:
> Just remembered that case (1) started a ML thread some months ago, in
> case someone else recalls, here is the link:
>
> https://lists.apache.org/thread/7gbqjwykh1ob3xbvwph3ljsdl5c7kxpd
>
> El mar, 21 jul 2026 a las 17:48, Antoine Pitrou (<[email protected]>)
> escribió:
> >
> >
> > Hello,
> >
> > https://github.com/apache/arrow/issues/50515 and its associated PR
> > https://github.com/apache/arrow/pull/50546 raise the question of what
> > the `nullable` flag in the IPC format means exactly for non-trivial
> fields.
> >
> > For example:
> >
> > 1) a non-nullable child field in a struct with some nulls in the parent
> > (implying that some child elements are logically null)
> > 2) a dictionary array with nulls in the dictionary (but no nulls in the
> > indices)
> > 3) a union or run-end-encoded array with nulls in the child values
> >
> > Arrow C++ has historically not been very considerate of the `nullable`
> > flag internally, treating it as mere metadata with no semantic
> implications.
> >
> > What do other implementations do / what do you suggest implementations
> > *should* do?
> >
> > Regards
> >
> > Antoine.
> >
>
