sdf-jkl opened a new pull request, #12224: URL: https://github.com/apache/arrow-rs/pull/12224
# Which issue does this PR close? - Closes #11050. - Supersedes #11052. # Rationale for this change This carries @emecii's fix from #11052, which was approved but has had no activity since October. The commit is unchanged and keeps their authorship, rebased onto current `main`. An out-of-bounds list lookup and a present Variant null both produce null child columns after `take`. Without tracking index validity separately, shredded input turns missing list positions into valid Variant nulls, unlike unshredded input. # What changes are included in this PR? - Carry the list index validity mask through shredded path traversal (@emecii's commit). - Apply that mask to the output while preserving explicit Variant null elements as valid values. - Regression test distinguishing an empty list, an explicit null element, and a typed element. - When no row has the requested index in bounds, the path step is `Missing` and `take` is skipped. This addresses the all-out-of-bounds slowdown measured in #11052. `value` is not consulted because arrays are never stored there when `typed_value` is a list, which matches Spark (`SparkShreddingUtils` returns null for an out-of-bounds index without reading `value`). - For the same reason, the two existing no-match exits of the list index step (elements are not a struct, or elements hold no values) now also return `Missing` instead of falling back to `value` row by row. For valid data this changes no results, since `value` can only hold non-arrays there, which index to NULL. - A `Missing` path step with no `as_type` now returns an all-null `VariantArray`, as documented on `GetOptions::as_type`, instead of a `NullArray`. # Are these changes tested? Yes. `test_shredded_list_out_of_bounds_is_null` distinguishes an empty list, an explicit null element, a typed element, and a non-list row, for both a mixed and an all-out-of-bounds index. The existing missing-path Variant output test now also runs without `as_type`. `variant_get_list_index_all_oob_int64_64_rows` goes from 1.32 µs to 0.47 µs locally compared to #11052. The other `variant_get_list_index` benchmarks are within noise. # Are there any significant user-facing changes? `variant_get` without `as_type` now returns an all-null `VariantArray` instead of a `NullArray` when the path cannot match a shredded column, for example a field that does not exist on a fully shredded object. # AI usage disclosure The original change was written with OpenAI Codex (see #11052). Claude Code carried the commit over and drafted this description. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Yifan Chen <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
