haohuaijin opened a new issue, #10624:
URL: https://github.com/apache/arrow-rs/issues/10624
### Is your feature request related to a problem or challenge?
The Parquet reader currently accepts selected row groups and one global
`RowSelection`:
```rust
builder
.with_row_groups(row_group_indexes)
.with_row_selection(row_selection)
```
### Current DataFusion flow
DataFusion stores the scan decision for each row group separately in
[`ParquetAccessPlan`](https://github.com/apache/datafusion/blob/0ef844e43cfbe5a5576f9d937dfc5850f16aceed/datafusion/datasource-parquet/src/access_plan.rs#L96-L158).
Row-level filters update each entry through
[`scan_selection`](https://github.com/apache/datafusion/blob/0ef844e43cfbe5a5576f9d937dfc5850f16aceed/datafusion/datasource-parquet/src/access_plan.rs#L383-L398):
```text
row group 0: bitmap selection
row group 1: skip
row group 2: selector selection
row group 3: scan all
```
Before constructing the Parquet reader,
[`prepare`](https://github.com/apache/datafusion/blob/0ef844e43cfbe5a5576f9d937dfc5850f16aceed/datafusion/datasource-parquet/src/access_plan.rs#L568-L604)
calls
[`into_overall_row_selection`](https://github.com/apache/datafusion/blob/0ef844e43cfbe5a5576f9d937dfc5850f16aceed/datafusion/datasource-parquet/src/access_plan.rs#L470-L532)
to convert this plan into two values:
```text
row groups: [0, 2, 3]
selection: one global RowSelection covering row groups 0, 2, and 3
```
Full row groups become `select(row_group_rows)`, while partial selections
are appended to the same selection stream. Mixed bitmap and selector inputs
must also be converted to one common representation.
DataFusion then passes both values to arrow-rs in
[`DecoderBuilderConfig::build`](https://github.com/apache/datafusion/blob/0ef844e43cfbe5a5576f9d937dfc5850f16aceed/datafusion/datasource-parquet/src/push_decoder.rs#L81-L105):
```rust
builder
.with_row_groups(row_group_indexes)
.with_row_selection(global_selection)
```
When arrow-rs starts reading,
[`RowGroupFrontier::next_readable_row_group`](https://github.com/apache/arrow-rs/blob/f2476ab9409368afb660b789ea84436abc396eb7/parquet/src/arrow/push_decoder/remaining.rs#L142-L176)
uses the row-group metadata to partition this global selection back into
row-group-local selections. The complete flow is therefore:
```text
per-row-group selections
-> one global RowSelection
-> per-row-group selections during decoding
```
### Describe the solution you'd like
Allow the push decoder to accept row-group-local selections directly:
```rust
pub struct RowGroupSelection {
row_group_index: usize,
selection: Option<RowSelection>,
}
builder.with_row_group_selections(vec![
RowGroupSelection::new(0, Some(bitmap_selection)),
RowGroupSelection::new(2, Some(selector_selection)),
RowGroupSelection::new(3, None),
]);
```
Proposed behavior:
- Entries are decoded in the supplied order.
- Omitted row groups are skipped.
- `None` reads the entire row group.
- Each row group retains its bitmap or selector representation.
- Existing `with_row_groups` and `with_row_selection` APIs remain supported.
- Rebuilding a push decoder preserves the remaining local selections.
### Describe alternatives you've considered
keep current behaviour
### Additional context
find this when work on https://github.com/apache/datafusion/pull/24186
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
