What you are describing sounds like a sparse array representation. I think it's fine to have both. There are definitely use cases where sparse arrays are more efficient than their dense counterparts. However, we also use dense arrays for a variety of reasons. They tend to yield simple and efficient branchless kernels.
> Even though it isn't stated, all child arrays (fields) must have the exact same length > as the parent struct array which is also problematic if you are injecting NULLs into a > non-nullable child array because the parent array's field for that array is nullable. > This could break data validation checks on child arrays if you are adding dummy values to it. What checks break? If the child array is non-nullable you should not be injecting NULLS. You should be injecting dummy values. > The alternative if you want to even out the child struct is would be to inject Nones and > duplicate values to match the parent length, but for this scenario I think you would > automatically copy or change the child struct fields to nullable if the parent field is also > nullable to eliminate the need to check both the parent and child nullability masks. First, I still don't really see why you would need to change the child to nullable. You can just inject dummy values into the child. So it won't contain nulls. Second, you always need to check both parent and child nullability masks. If you have masks at each level then they indicate different things (a null vehicle vs. a vehicle with a null model for example). There are ways to combine the fields (e.g. definition levels) but that is, again, a different representation and it will carry its own pros and cons. On Thu, Jul 23, 2026 at 1:02 PM Lee, David (ITE) <[email protected]> wrote: > This is essentially a take index for nested structs for the struct field. > > https://arrow.apache.org/docs/python/generated/pyarrow.compute.take.html > > import pyarrow as pa > > arr = pa.array(["a", "b", "c", None, "e", "f"]) > indices = pa.array([0, None, 4, 3]) > > arr.take(indices) > <pyarrow.lib.StringArray object at ...> > [ > "a", > null, > "e", > null > ] > This message may contain information that is confidential or privileged. > If you are not the intended recipient, please advise the sender immediately > and delete this message. See > https://www.blackrock.com/corporate/compliance/email-disclaimers for > further information. Please refer to > https://www.blackrock.com/corporate/compliance/privacy-policy for more > information about BlackRock’s Privacy Policy. > For a list of BlackRock's office addresses worldwide, see > https://www.blackrock.com/corporate/about-us/contacts-locations. > > © 2026 BlackRock, Inc. All rights reserved. >
