matthewmturner commented on issue #26267: URL: https://github.com/apache/datafusion/issues/26267#issuecomment-6646459118
Hi @adriangb - i checked the reproducer on latest main and we get the expected results now. ``` ❯ datafusion-cli DataFusion CLI v57.0.0-dev > COPY (SELECT value AS n FROM generate_series(1, 1000)) TO '/tmp/ev.parquet'; CREATE EXTERNAL TABLE ev STORED AS PARQUET LOCATION '/tmp/ev.parquet'; SET datafusion.explain.format = 'indent'; SET datafusion.execution.parquet.pushdown_filters = true; +-------+ | count | +-------+ | 1000 | +-------+ 1 row(s) fetched. Elapsed 0.051 seconds. 0 row(s) fetched. Elapsed 0.001 seconds. 0 row(s) fetched. Elapsed 0.000 seconds. 0 row(s) fetched. Elapsed 0.000 seconds. > SELECT count(*) FROM (SELECT random() AS x FROM ev) WHERE x < 0.5 OR x >= 0.5; +----------+ | count(*) | +----------+ | 1000 | +----------+ 1 row(s) fetched. Elapsed 0.035 seconds. > SELECT count(*), max(x) FROM (SELECT random() AS x FROM ev) WHERE x < 0.1; +----------+---------------------+ | count(*) | max(x) | +----------+---------------------+ | 95 | 0.09988944740880379 | +----------+---------------------+ 1 row(s) fetched. Elapsed 0.002 seconds. ``` These are the new plans after a [related fix](https://github.com/apache/datafusion/pull/26850) I did on volatile expressions. Youll notice that the volatile expressions are now projected before the filter - so the volatile expression isnt evaluated multiple times anymore. Query 1 ``` > EXPLAIN SELECT count(*) FROM (SELECT random() AS x FROM ev) WHERE x < 0.5 OR x >= 0.5; +---------------+----------------------------------------------------------------------------------------------------------------------+ | plan_type | plan | +---------------+----------------------------------------------------------------------------------------------------------------------+ | logical_plan | Projection: count(Int64(1)) AS count(*) | | | Aggregate: groupBy=[[]], aggr=[[count(Int64(1))]] | | | Projection: | | | Filter: x < Float64(0.5) OR x >= Float64(0.5) | | | Projection: random() AS x | | | TableScan: ev projection=[] | | physical_plan | ProjectionExec: expr=[count(Int64(1))@0 as count(*)] | | | AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))] | | | CoalescePartitionsExec | | | AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))] | | | RepartitionExec: partitioning=RoundRobinBatch(32), input_partitions=1 | | | FilterExec: x@0 < 0.5 OR x@0 >= 0.5, projection=[] | | | DataSourceExec: file_groups={1 group: [[tmp/ev.parquet]]}, projection=[random() as x], file_type=parquet | | | | +---------------+----------------------------------------------------------------------------------------------------------------------+ ``` Query 2 ``` > EXPLAIN SELECT count(*), max(x) FROM (SELECT random() AS x FROM ev) WHERE x < 0.1; +---------------+----------------------------------------------------------------------------------------------------------------------+ | plan_type | plan | +---------------+----------------------------------------------------------------------------------------------------------------------+ | logical_plan | Projection: count(Int64(1)) AS count(*), max(x) | | | Aggregate: groupBy=[[]], aggr=[[count(Int64(1)), max(x)]] | | | Filter: x < Float64(0.1) | | | Projection: random() AS x | | | TableScan: ev projection=[] | | physical_plan | ProjectionExec: expr=[count(Int64(1))@0 as count(*), max(x)@1 as max(x)] | | | AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1)), max(x)] | | | CoalescePartitionsExec | | | AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1)), max(x)] | | | RepartitionExec: partitioning=RoundRobinBatch(32), input_partitions=1 | | | FilterExec: x@0 < 0.1 | | | DataSourceExec: file_groups={1 group: [[tmp/ev.parquet]]}, projection=[random() as x], file_type=parquet | | | | +---------------+----------------------------------------------------------------------------------------------------------------------+ ``` Do you think we can close this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
