zhuqi-lucas commented on issue #10624: URL: https://github.com/apache/arrow-rs/issues/10624#issuecomment-5290730383
Just a note that this missing capability is also the **root cause of a wrong-results bug family** in DataFusion, not only a performance/ergonomics improvement. Because DataFusion collapses the per-row-group selections into one global `RowSelection` and then rebuilds the push decoder to drop row groups at runtime (`into_builder().with_row_groups(...)`), the global selection is **not sliced to match the dropped row groups** — selectors intended for a dropped RG get applied to the next surviving one, silently returning wrong rows: - apache/datafusion#24355 — runtime dynamic row-group pruning drops RGs without slicing the carried `RowSelection`. - apache/datafusion#24352 — a related drift in the same parallel-state path. The behavior you list — *"Rebuilding a push decoder preserves the remaining local selections"* — is exactly what makes both bugs structurally impossible: if selections stay row-group-local, dropping a RG on rebuild keeps every survivor's selection aligned by construction, and DataFusion can also delete the parallel `rg_plan` bookkeeping it maintains today. DataFusion has shipped minimal stop-the-bleeding fixes for both (apache/datafusion#24354, apache/datafusion#24359), and is tracking the structural fix on its side in apache/datafusion#24358, which depends on this API landing here. Happy to help with the DataFusion-side migration once `with_row_group_selections` is available. Thanks @haohuaijin for writing this up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
