zeroshade opened a new pull request, #325: URL: https://github.com/apache/arrow-go/pull/325
### Rationale for this change Related to https://github.com/apache/arrow/issues/38503, it was found in https://github.com/apache/iceberg-go/issues/357 that the Parquet file being written by pqarrow was incompatible with pyarrow in some testing. After some digging I determined the cause. So we should fix this to finally put #38503 to bed ### What changes are included in this PR? Fixing the computation of definition levels when writing from Arrow data with lists of non-nullable elements. Previously we were basing the `nullableInParent` on whether it was a list or a map, assuming that the element was *always* nullable in a list. This, of course, is incorrect and we need to actually *check* the nullability of the list element to compute the correct definition level. This way we don't produce incongruous levels that are larger than what the max definition level should be. ### Are these changes tested? Yes, I've updated the corresponding test, which wasn't *actually* testing a non-nullable element until now. ### Are there any user-facing changes? Only fixing writing of files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
