Huxley Hu created ARROW-13982:
---------------------------------
Summary: dataset scanner stalls when reading parquet with
filtering.
Key: ARROW-13982
URL: https://issues.apache.org/jira/browse/ARROW-13982
Project: Apache Arrow
Issue Type: Bug
Components: C++
Affects Versions: 5.0.0
Environment: ubuntu 18.04 LTS
Reporter: Huxley Hu
Reading parquet files using dataset scanner may stall due to a never-finished
future.
To reproduce this case, one needs two parquet files and sets the filter
expression to something that could filter one file completely. After that,
calling `AsyncScanner::ToRecordBatchReader` and read data continually.
I also have dug this bug a little. It's caused by the
`MakeEmptyGenerator<std::shared_ptr<RecordBatch>>` when filtered row groups is
empty, which's ignored by `FragmentToBatches` and causes SequencingGenerator to
stall.
A quick fix is to return a record batch with 0 rows instead of returning a
nullptr there.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
