adriangb commented on PR #25752:
URL: https://github.com/apache/datafusion/pull/25752#issuecomment-6023807134
`ceaab34cb0` ("keep a filter above a scan that cannot give the target
partitions") also keeps the filter out of a scan that reads several files but
has fewer file groups than `target_partitions`. The scan then emits every row
of every projected column, including columns that only the filter needs, and a
round-robin repartition moves all of it to a `FilterExec`.
MRE with `datafusion-cli`:
```sql
COPY (SELECT v AS id, CASE WHEN v % 100 = 0 THEN 'timeout here' ELSE 'ok'
END AS message, 'svc' || (v % 5) AS service_name, v * 1.0 AS duration FROM
generate_series(1, 200000) t(v)) TO 'few/a.parquet';
COPY (SELECT v AS id, CASE WHEN v % 100 = 0 THEN 'timeout here' ELSE 'ok'
END AS message, 'svc' || (v % 5) AS service_name, v * 1.0 AS duration FROM
generate_series(1, 200000) t(v)) TO 'few/b.parquet';
SET datafusion.execution.target_partitions = 8;
SET datafusion.execution.parquet.pushdown_filters = true;
SET datafusion.optimizer.repartition_file_scans = false;
CREATE EXTERNAL TABLE few STORED AS PARQUET LOCATION 'few/';
EXPLAIN SELECT service_name, count(*), max(duration) FROM few WHERE message
LIKE '%timeout%' GROUP BY 1;
```
- `main` (`6e319bdd`): the predicate is in `DataSourceExec`, and there is no
`FilterExec`.
- This branch: `DataSourceExec` (predicate used for pruning only) ->
`RepartitionExec: RoundRobinBatch(8)` -> `FilterExec`.
- With 16 copies of the file in the directory, this branch pushes the filter
into the scan again.
The heuristic fits the case in the commit message (one small file, where the
filter would otherwise run in one partition). But "fewer partitions than
`target_partitions`" also matches scans that read many files in fewer, larger
partitions on purpose. For a selective filter on wide columns, that costs much
more than the lost parallelism of the filter. Maybe the rule could apply only
when the scan cannot produce enough rows to keep the extra partitions busy (for
example one file, or an estimated row count below some multiple of
`batch_size`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
