There are some valid use cases for not polluting an inner struct with NULLs to match parent array lengths.. Running data validation like "quantity > 0" doesn't work if it has been polluted with Zeros or Nulls to match parent array lengths for nullable slots.
-----Original Message----- From: Lee, David (ITE) <[email protected]> Sent: Wednesday, July 22, 2026 1:56 PM To: [email protected] Subject: RE: [DISCUSS] Semantics of non-nullable fields with non-trivial types Here are my thoughts since I opened the original issue for this current discussion.. I think we an official format change. There is a difference between a nullable array and a struct that a nullable field using that nullable array. The nullable array on its own has a bitmap mask which works fine as is if it is nullable. The only question is whether the item in the nullable slot can be NULL or has to be some non-null garbage value like 0, empty string, 1970 date, etc.. Here's the proposed change for a nullable field in a struct. There should be another index with positions for the non-nullable values. So if the inner struct array which is non-nullable has values of [a, b, c, d] But the outer struct has values of [(1, inner=a), (2, inner=b), (3, Null), (4, Null), (5, inner=c), ), (6, inner=d)] The outer struct should have a positional index with values of [0,1, None, None, 2, 3] to track the nullability of [1,2,3,4] without having to add Nones to [1,2,None, None, 3, 4]. This would also support inner structs which are Nullable with NULL values using a positional index of [1,2, None, 3, 4, 5]. [a, b, c, Null, d] would look different in the outer struct with a Null for not present vs inner=Null as a null value. [(1, inner=a), (2, inner=b), (3, Null), (4, inner=Null), (5, inner=c), ), (6, inner=d)] -----Original Message----- From: Weston Pace <[email protected]> Sent: Tuesday, July 21, 2026 1:07 PM To: [email protected] Subject: Re: [DISCUSS] Semantics of non-nullable fields with non-trivial types External Email: Use caution with links and attachments Personally, I find the nullability flag to be confusing in general. I guess a larger question is whether an Arrow schema is intended to just describe the data itself (this batch) or whether it is intended to describe something larger / more persistent (e.g. a table, dataset, stream of data, etc.) If it's the former (just describes the data) then it is redundant. The presence or absence of a validity bitmap is usually enough to know if the data has nulls. If it's the latter (describing something larger) then it is difficult to interpret without knowing what exactly we are describing. > 1) a non-nullable child field in a struct with some nulls in the > parent (implying that some child elements are logically null) This one seems the least controversial to me. Let's say we have `outer: struct<inner: int32>` and `outer` is nullable and `inner` is non-nullable. Valid: [{"inner": 3}, null, null, {"inner": 5}] Invalid: [{"inner": 3}, null, {"inner": null}] For cases 2 & 3 I would consider the field semantically nullable if either of the two options were nullable. > What do other implementations do / what do you suggest implementations > *should* do? This falls into the same "it's probably a moot point this late in the game" argument from the thread Raúl linked but I'd argue that implementations should at least (and probably at most) validate the nullability upon array creation. 1. If a child array is marked non-nullable but the parent struct has a null then the child array should not have a validity bitmap and should have a garbage value in the slot. 2. If a dictionary is non-nullable then the dictionary should not have a validity bitmap even if the indices contain nulls. If the indices are non-nullable then the indices should not have a validity bitmap even if the dictionary has nulls. 3. The parent nullability flag is completely unused during validation. At this point, however, I have no faith in the nullability flag as proof that the array has been validated and I completely ignore it and always do my own validation when receiving outside data (even if it is from a trusted source). On Tue, Jul 21, 2026 at 9:00 AM Raúl Cumplido <[email protected]> wrote: > Just remembered that case (1) started a ML thread some months ago, in > case someone else recalls, here is the link: > > https://urldefense.com/v3/__https://lists.apache.org/thread/7gbqjwykh1 > ob3xbvwph3ljsdl5c7kxpd__;!!KSjYCgUGsB4!aptxjOXsTsGLVbjrhlMekgqP7EySoww > eSBW9adPvnlFHzenFMJmAeNXZEikm1R3FAMnyVDgV9a7vFtKg9pVhRxA$ > > El mar, 21 jul 2026 a las 17:48, Antoine Pitrou (<[email protected]>) > escribió: > > > > > > Hello, > > > > https://urldefense.com/v3/__https://github.com/apache/arrow/issues/5 > > 0515__;!!KSjYCgUGsB4!aptxjOXsTsGLVbjrhlMekgqP7EySowweSBW9adPvnlFHzen > > FMJmAeNXZEikm1R3FAMnyVDgV9a7vFtKgHPbp0UM$ and its associated PR > > https://urldefense.com/v3/__https://github.com/apache/arrow/pull/505 > > 46__;!!KSjYCgUGsB4!aptxjOXsTsGLVbjrhlMekgqP7EySowweSBW9adPvnlFHzenFM > > JmAeNXZEikm1R3FAMnyVDgV9a7vFtKgSXfNfA0$ raise the question of what > > the `nullable` flag in the IPC format means exactly for non-trivial > fields. > > > > For example: > > > > 1) a non-nullable child field in a struct with some nulls in the > > parent (implying that some child elements are logically null) > > 2) a dictionary array with nulls in the dictionary (but no nulls in > > the > > indices) > > 3) a union or run-end-encoded array with nulls in the child values > > > > Arrow C++ has historically not been very considerate of the > > `nullable` flag internally, treating it as mere metadata with no > > semantic > implications. > > > > What do other implementations do / what do you suggest > > implementations > > *should* do? > > > > Regards > > > > Antoine. > > > This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See https://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to https://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy. For a list of BlackRock's office addresses worldwide, see https://www.blackrock.com/corporate/about-us/contacts-locations . © 2026 BlackRock, Inc. All rights reserved.
