1fanwang opened a new pull request, #55704:
URL: https://github.com/apache/arrow/pull/55704

   ### Rationale for this change
   
   A struct can be null while a fixed-size list field underneath it is declared 
non-nullable in the schema. In that case the list keeps no validity information 
of its own, so nothing records that the struct's null row left no physical 
values behind for it. Reading such a column back raises "Expected all lists to 
be of size=N but index M had size=0" even though the file is well-formed. This 
follows on from GH-35692, which fixed the same symptom for the case where the 
list field itself carries the null.
   
   ### What changes are included in this PR?
   
   The reader decided whether to pad a missing fixed-size list slot based on 
whether the list had its own null count, which is always zero for a 
non-nullable field regardless of what its ancestor did. The fix instead looks 
at the decoded element offsets directly: a null ancestor still leaves a 
zero-length span where a full list was expected, and that gap is now detected 
and padded the same way as a list that is null in its own right.
   
   ### Are these changes tested?
   
   Yes, with a new regression test reproducing the reported schema (a nullable 
struct holding a non-nullable fixed-size list field):
   
   ```
   ninja parquet-arrow-reader-writer-test
   ./debug/parquet-arrow-reader-writer-test 
--gtest_filter=ArrowReadWrite.NonNullableFixedSizeListInsideNullableStruct
   ```
   
   Before the fix, this fails with the exact reported error:
   
   ```
   '_error_or_value34.status()' failed with Invalid: Expected all lists to be 
of size=2 but index 2 had size=0
   ```
   
   After the fix it passes, and the roundtripped table decodes the null struct 
row's fixed-size list as nulls instead of erroring:
   
   ```
   x:
     [
       -- is_valid:
         [true, false, true]
       -- child 0 type: fixed_size_list<element: int32>[2]
         [[1, 2], [null, null], [3, 4]]
     ]
   ```
   
   ### Are there any user-facing changes?
   
   No API changes; this fixes a read failure on previously-unreadable but 
well-formed files.
   
   ### Was AI used for this PR?
   
   In accordance to the [AI generation 
guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code),
 please disclose below whether and how AI was used in this PR.
   
   **PR code and description written by:**
   
   - [ ] Human
   - [X] AI
   
   **Reviewed before submission by:**
   
   - [ ] Human
   - [X] AI
   - [ ] Not reviewed
   
   * GitHub Issue: #55682
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to