adriangb commented on issue #14406: URL: https://github.com/apache/datafusion/issues/14406#issuecomment-2629268321
Here's an assortment of test plans under slight variations that both contain
the bug and not.
```
ataFusion CLI v44.0.0
> explain analyze
with selection as (
select *
from 'parquet_files/*'
limit 1
)
select 1 as foo
from selection
order by duration
limit 1000;
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan
|
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | ProjectionExec: expr=[foo@0 as foo],
metrics=[output_rows=2, elapsed_compute=166ns]
|
| | SortExec: TopK(fetch=1000), expr=[duration@1 ASC
NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=2,
elapsed_compute=1.021375ms, row_replacements=2]
|
| | ProjectionExec: expr=[1 as foo, duration@0 as
duration], metrics=[output_rows=2, elapsed_compute=199.833µs]
|
| | CoalescePartitionsExec, metrics=[output_rows=2,
elapsed_compute=10.875µs]
|
| | ParquetExec: file_groups={2 groups:
[[Users/adriangb/GitHub/platform/parquet_files/day=1970-01-01/file_0.parquet],
[Users/adriangb/GitHub/platform/parquet_files/day=1970-01-02/file_1.parquet]]},
projection=[duration], limit=1, metrics=[output_rows=2, elapsed_compute=2ns,
bytes_scanned=128, file_open_errors=0, file_scan_errors=0,
num_predicate_creation_errors=0, page_index_rows_matched=0,
page_index_rows_pruned=0, predicate_evaluation_errors=0,
pushdown_rows_matched=0, pushdown_rows_pruned=0,
row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0,
row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0,
bloom_filter_eval_time=4ns, metadata_load_time=329.086µs,
page_index_eval_time=4ns, row_pushdown_eval_time=4ns, statistics_eval_time=4ns,
time_elapsed_opening=1.038209ms, time_elapsed_processing=5.046085ms,
time_elapsed_scanning_total=4.058458ms,
time_elapsed_scanning_until_data=4.058417ms] |
| |
|
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.076 seconds.
> explain analyze
with selection as (
select *
from 'parquet_files/*'
limit 1
)
select 1 as foo
from selection
order by duration;
+-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan
|
+-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | ProjectionExec: expr=[foo@0 as foo],
metrics=[output_rows=1, elapsed_compute=375ns]
|
| | SortExec: expr=[duration@1 ASC NULLS LAST],
preserve_partitioning=[false], metrics=[output_rows=1,
elapsed_compute=2.785583ms, spill_count=0, spilled_bytes=0, spilled_rows=0]
|
| | ProjectionExec: expr=[1 as foo, duration@0 as
duration], metrics=[output_rows=1, elapsed_compute=2.25µs]
|
| | GlobalLimitExec: skip=0, fetch=1,
metrics=[output_rows=1, elapsed_compute=1.458µs]
|
| | CoalescePartitionsExec,
metrics=[output_rows=1, elapsed_compute=18.958µs]
|
| | ParquetExec: file_groups={2 groups:
[[Users/adriangb/GitHub/platform/parquet_files/day=1970-01-01/file_0.parquet],
[Users/adriangb/GitHub/platform/parquet_files/day=1970-01-02/file_1.parquet]]},
projection=[duration], limit=1, metrics=[output_rows=1, elapsed_compute=2ns,
bytes_scanned=128, file_open_errors=0, file_scan_errors=0,
num_predicate_creation_errors=0, page_index_rows_matched=0,
page_index_rows_pruned=0, predicate_evaluation_errors=0,
pushdown_rows_matched=0, pushdown_rows_pruned=0,
row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0,
row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0,
bloom_filter_eval_time=4ns, metadata_load_time=157.751µs,
page_index_eval_time=4ns, row_pushdown_eval_time=4ns, statistics_eval_time=4ns,
time_elapsed_opening=175.042µs, time_elapsed_processing=131.083µs,
time_elapsed_scanning_total=83.084µs,
time_elapsed_scanning_until_data=83.085µs] |
| |
|
+-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.023 seconds.
> explain analyze
with selection as (
select *
from 'parquet_files/*'
limit 1
)
select 1 as foo
from selection;
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan
|
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | ProjectionExec: expr=[1 as foo],
metrics=[output_rows=1, elapsed_compute=2.458µs]
|
| | GlobalLimitExec: skip=0, fetch=1,
metrics=[output_rows=1, elapsed_compute=1.417µs]
|
| | CoalescePartitionsExec, metrics=[output_rows=1,
elapsed_compute=4.125µs]
|
| | ParquetExec: file_groups={2 groups:
[[Users/adriangb/GitHub/platform/parquet_files/day=1970-01-01/file_0.parquet],
[Users/adriangb/GitHub/platform/parquet_files/day=1970-01-02/file_1.parquet]]},
limit=1, metrics=[output_rows=1, elapsed_compute=2ns, bytes_scanned=0,
file_open_errors=0, file_scan_errors=0, num_predicate_creation_errors=0,
page_index_rows_matched=0, page_index_rows_pruned=0,
predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0,
row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0,
row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0,
bloom_filter_eval_time=4ns, metadata_load_time=188.544µs,
page_index_eval_time=4ns, row_pushdown_eval_time=4ns, statistics_eval_time=4ns,
time_elapsed_opening=206.584µs, time_elapsed_processing=128µs,
time_elapsed_scanning_total=38.876µs,
time_elapsed_scanning_until_data=38.876µs] |
| |
|
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.026 seconds.
> explain analyze
with selection as (
select *
from 'parquet_files/*'
limit 1
)
select 1 as foo
from selection
limit 1000;
+-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan
|
+-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | ProjectionExec: expr=[1 as foo],
metrics=[output_rows=1, elapsed_compute=2.208µs]
|
| | GlobalLimitExec: skip=0, fetch=1,
metrics=[output_rows=1, elapsed_compute=1.583µs]
|
| | CoalescePartitionsExec, metrics=[output_rows=1,
elapsed_compute=10.333µs]
|
| | ParquetExec: file_groups={2 groups:
[[Users/adriangb/GitHub/platform/parquet_files/day=1970-01-01/file_0.parquet],
[Users/adriangb/GitHub/platform/parquet_files/day=1970-01-02/file_1.parquet]]},
limit=1, metrics=[output_rows=1, elapsed_compute=2ns, bytes_scanned=0,
file_open_errors=0, file_scan_errors=0, num_predicate_creation_errors=0,
page_index_rows_matched=0, page_index_rows_pruned=0,
predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0,
row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0,
row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0,
bloom_filter_eval_time=4ns, metadata_load_time=213.96µs,
page_index_eval_time=4ns, row_pushdown_eval_time=4ns, statistics_eval_time=4ns,
time_elapsed_opening=230.583µs, time_elapsed_processing=132.794µs,
time_elapsed_scanning_total=44.209µs,
time_elapsed_scanning_until_data=44.168µs] |
| |
|
+-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.020 seconds.
> SET datafusion.execution.target_partitions = 1;
0 row(s) fetched.
Elapsed 0.001 seconds.
> explain analyze
with selection as (
select *
from 'parquet_files/*'
limit 1
)
select 1 as foo
from selection
limit 1000;
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan
|
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | ProjectionExec: expr=[1 as foo],
metrics=[output_rows=1, elapsed_compute=2.334µs]
|
| | ParquetExec: file_groups={1 group:
[[Users/adriangb/GitHub/platform/parquet_files/day=1970-01-01/file_0.parquet,
Users/adriangb/GitHub/platform/parquet_files/day=1970-01-02/file_1.parquet]]},
limit=1, metrics=[output_rows=1, elapsed_compute=1ns, bytes_scanned=0,
file_open_errors=0, file_scan_errors=0, num_predicate_creation_errors=0,
page_index_rows_matched=0, page_index_rows_pruned=0,
predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0,
row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0,
row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0,
bloom_filter_eval_time=4ns, metadata_load_time=148.419µs,
page_index_eval_time=4ns, row_pushdown_eval_time=4ns, statistics_eval_time=4ns,
time_elapsed_opening=103.875µs, time_elapsed_processing=84.624µs,
time_elapsed_scanning_total=50.5µs, time_elapsed_scanning_until_data=50.5µs] |
| |
|
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.027 seconds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
