bkirwi commented on PR #6524: URL: https://github.com/apache/arrow-rs/pull/6524#issuecomment-2407994015
> I think this will solve this issue for PrimitiveArray, but other array types will also run into the same issue. I worked through all the "leaf" types methodically, and it seemed that all of them ended up wanting the exact same implementation as I had for primitive types - dropping the null buffer iff the column type was not optional - so I shifted the logic into the reader impl itself. > The same approach can be applied in most cases, apart from potentially DictionaryArray - where nulls may not actually have a valid value. This _appears_ to work out with the implementation above: the `validate_non_nullable` validation just looks at the dictionary array's null buffer (and not, for example, `logical_nulls` or anything else in the key array) which is precisely the array that we're skipping over. I've added a case to the test that _should_ check this specifically. I've gone ahead and pushed these changes, and I think this addresses all the issues I'm aware of... taking this back out of draft. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
