fallintoplace opened a new pull request, #1195: URL: https://github.com/apache/arrow-go/pull/1195
## Summary - keep the existing zero-null fixed-size-list path unchanged - validate nullable parent spans once - use the existing slice concatenation for a small number of validity runs - build one child array with a typed Take for highly fragmented validity - avoid one temporary Arrow array per validity run ## Why Parquet does not write child values for null fixed-size-list parents. Arrow still needs `list_size` child slots for every parent, so the reader has to add null child values. The old nullable path creates one temporary array for each validity run and concatenates them. Alternating validity can therefore create one temporary array per parent row. ## Benchmark Apple M1 Pro, 65,536 `FixedSizeList<int32, 4>` parents: | validity | old | new | old allocs | new allocs | | --- | ---: | ---: | ---: | ---: | | 10% nulls | ~4.4 ms | ~1.3 ms | 85,220 | 46 | | alternating nulls | ~23–30 ms | ~1.1 ms | 458,770 | 46 | | one clustered null run | ~0.3 ms | ~0.3 ms | 28 | 27 | The low-run path is kept for the clustered case, where creating a single pair of slices is cheaper than building an index array. ## Tests - `go vet ./parquet/pqarrow` - `go test -race ./parquet/pqarrow -run 'TestBuildFixedSizeListArray|TestParquetArrowIO/TestFixedSizeList'` - `PARQUET_TEST_DATA=parquet-testing/data ARROW_TEST_DATA=arrow-testing/data go test ./...` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
