hhhizzz opened a new issue, #10733:
URL: https://github.com/apache/arrow-rs/issues/10733
### Describe the bug
The async Parquet reader can fail with `Invalid offset in sparse column chunk
data: ..., no matching page found` when all of the following are true:
- page indexes are available and page pruning produces a sparse column chunk;
- a row filter is evaluated with the predicate cache enabled;
- the row selection uses `RowSelectionPolicy::Mask`, or `Auto` resolves to
Mask; and
- selected rows are separated by one or more unloaded pages.
The offset index is present in this case, so the error's suggestion to
provide
an offset index does not address the failure. The predicate cache is enabled
by
default for the async reader, which makes this a default-path correctness
bug.
This was originally observed through DataFusion predicate pushdown. TPC-DS
SF10 queries 66, 75, and 81 failed in the same way.
### To Reproduce
On arrow-rs main at `bb1e6cd070`, add this test to
`parquet/tests/arrow_reader/row_filter/async.rs`. It uses the existing
`make_two_column_i64_file` helper, so it creates all data in memory and does
not
need an external Parquet file.
```rust
#[tokio::test]
async fn reproduce_cached_mask_sparse_page_failure() {
let values = (0..60).collect::<Vec<i64>>();
let data = make_two_column_i64_file(&values, 20);
for policy in [
RowSelectionPolicy::Auto { threshold: 32 },
RowSelectionPolicy::Mask,
] {
let builder = ParquetRecordBatchStreamBuilder::new_with_options(
TestReader::new(data.clone()),
ArrowReaderOptions::new().with_page_index_policy(PageIndexPolicy::Required),
)
.await
.unwrap();
let schema = builder.parquet_schema().clone();
let projection = ProjectionMask::leaves(&schema, [0]);
let predicate = ArrowPredicateFn::new(projection.clone(), |batch:
RecordBatch| {
Ok(BooleanArray::from(vec![true; batch.num_rows()]))
});
let stream = builder
.with_projection(projection)
.with_row_filter(RowFilter::new(vec![Box::new(predicate)]))
.with_row_selection(RowSelection::from(vec![
RowSelector::select(1),
RowSelector::skip(39),
RowSelector::select(1),
]))
.with_batch_size(8)
.with_max_predicate_cache_size(1024)
.with_row_selection_policy(policy)
.build()
.unwrap();
let output_schema = stream.schema().clone();
let batches: Vec<RecordBatch> = stream.try_collect().await.unwrap();
let output = concat_batches(&output_schema, &batches).unwrap();
assert_eq!(
output
.column(0)
.as_any()
.downcast_ref::<Int64Array>()
.unwrap()
.values(),
&[0, 40],
"policy={policy:?}"
);
}
}
```
Run:
```shell
cargo test -p parquet --features async --test arrow_reader \
'row_filter::r#async::reproduce_cached_mask_sparse_page_failure' \
-- --exact --nocapture
```
The first iteration, where `Auto` resolves to Mask, fails with:
```text
Parquet error: Invalid offset in sparse column chunk data: 534,
no matching page found.
```
Changing `with_max_predicate_cache_size(1024)` to
`with_max_predicate_cache_size(0)` avoids the failure, which isolates the
predicate-cache interaction.
### Expected behavior
The stream should complete and return values `[0, 40]` for both Auto and
Mask.
Mask execution should skip the unloaded middle page rather than requesting a
read from it.
### Additional context
`MaskCursor::next_chunk` correctly stops scanning at the end of the current
loaded row range, but it currently uses that scan position as both
`self.position` and `chunk_rows`. If the loaded range contains a selected row
followed by skipped rows, those trailing skipped rows are therefore included
in
the read request.
That is unsafe for a fixed-size cached read: the request can extend beyond
the
cached segment into a page that was not loaded. The trailing skips should
instead be left for the next cursor step, so `skip_records` crosses the gap
and
each decoded chunk ends immediately after its last selected row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
