[
https://issues.apache.org/jira/browse/ARROW-9728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Dipale updated ARROW-9728:
----------------------------------
Summary: [Rust] [Parquet] Compute nested definition and repetition for
structs (was: [Rust] [Parquet] Compute nested spacing)
> [Rust] [Parquet] Compute nested definition and repetition for structs
> ---------------------------------------------------------------------
>
> Key: ARROW-9728
> URL: https://issues.apache.org/jira/browse/ARROW-9728
> Project: Apache Arrow
> Issue Type: Sub-task
> Components: Rust
> Affects Versions: 1.0.0
> Reporter: Neville Dipale
> Assignee: Neville Dipale
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When computing definition levels for deeply nested arrays that include lists,
> the definition levels are correctly calculated, but they are not translated
> into correct indexes for the eventual primitive arrays.
> For example, an int32 array could have no null values, but be a child of a
> list that has null values. If say the first 5 values of the int32 array are
> members of the first list item (i.e. list_array[0] = [1,2,3,4,5], and that
> list is itself a child of a struct whose index is null, the whole 5 values of
> the int32 array *should* be skipped. Further, the list's definition and
> repetition levels will be represented by 1 slot instead of the 5.
> The current logic cannot cater for this, and potentially results in slicing
> the int32 array incorrectly (sometimes including some of those first 5
> values).
> This Jira is for the work necessary to compute the index into the eventual
> leaf arrays correctly.
> I started doing it as part of the initial writer PR, but it's complex and is
> blocking progress.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
