rank-p opened a new issue, #27162:
URL: https://github.com/apache/datafusion/issues/27162
### Describe the bug
Reading a single Parquet file with individually sorted row groups can return
unsorted rows for `ORDER BY`. Adding `LIMIT` can also select the wrong rows
because DataFusion treats the whole file as sorted and removes a required sort.
### To Reproduce
Reproduced with Python 3.11, `datafusion==55.0.0`, and `pyarrow==24.0.0`.
The file below has two row groups: `[1, 4]` and `[2, 3]`. Each group
correctly declares ascending order, but the combined rows are not sorted.
Save as `repro.py` and run:
```sh
uv run --no-project --python 3.11 --with datafusion==55.0.0 --with
pyarrow==24.0.0 repro.py
```
```python
import tempfile
from pathlib import Path
import pyarrow as pa
import pyarrow.parquet as pq
from datafusion import SessionConfig, SessionContext
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "data.parquet"
pq.write_table(
pa.table({"x": pa.array([1, 4, 2, 3], type=pa.int64())}),
path,
row_group_size=2,
sorting_columns=[pq.SortingColumn(0, nulls_first=True)],
)
ctx = SessionContext(SessionConfig().with_target_partitions(1))
ctx.register_parquet("t", str(path))
for suffix in ["", " LIMIT 2"]:
result = ctx.sql("SELECT x FROM t ORDER BY x ASC NULLS FIRST" +
suffix)
print([row["x"] for batch in result.collect() for row in
batch.to_pylist()])
```
Actual results, without and with `LIMIT 2`:
```text
[1, 4, 2, 3]
[1, 4]
```
### Expected behavior
```text
[1, 2, 3, 4]
[1, 2]
```
A row group's sort declaration does not establish ordering across row-group
boundaries.
### Additional context
The current
[`ordering_from_parquet_metadata`](https://github.com/apache/datafusion/blob/90b55e38b5de8e74a17e24bf7ea7259cbcdab022/datafusion/datasource-parquet/src/metadata.rs#L1196-L1221)
uses the first row group's `sorting_columns` to declare an ordering for the
file, without checking the other groups or their boundaries.
Related to lakehq/sail#2730. This is separate from #26278, which covered
NULL bounds between files.
A fix and regression tests are ready locally.
### Willingness to contribute
I plan to contribute a fix for this bug independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
