mingmwang commented on PR #6003:
URL:
https://github.com/apache/arrow-datafusion/pull/6003#issuecomment-1508194574
For **Q5,** there is downgrade between the versions. But I think it is
unrelated to my PRs, the aggregation is very lightweight.
```
=== Physical plan with metrics ===
SortExec: expr=[revenue@1 DESC], metrics=[output_rows=5,
elapsed_compute=5.335µs, spill_count=0, spilled_bytes=0]
ProjectionExec: expr=[n_name@0 as n_name, SUM(lineitem.l_extendedprice *
Int64(1) - lineitem.l_discount)@1 as revenue], metrics=[output_rows=5,
elapsed_compute=583ns, spill_count=0, spilled_bytes=0, mem_used=0]
AggregateExec: mode=Single, gby=[n_name@2 as n_name],
aggr=[SUM(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)],
metrics=[output_rows=5, elapsed_compute=3.773667ms, spill_count=0,
spilled_bytes=0, mem_used=0]
ProjectionExec: expr=[l_extendedprice@0 as l_extendedprice,
l_discount@1 as l_discount, n_name@2 as n_name], metrics=[output_rows=72985,
elapsed_compute=541ns, spill_count=0, spilled_bytes=0, mem_used=0]
CoalesceBatchesExec: target_batch_size=8192,
metrics=[output_rows=72985, elapsed_compute=1.764µs, spill_count=0,
spilled_bytes=0, mem_used=0]
HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(Column {
name: "n_regionkey", index: 3 }, Column { name: "r_regionkey", index: 0 })],
metrics=[output_rows=1, input_batches=1, output_batches=1,
build_input_batches=1, input_rows=1, build_input_rows=364380,
build_mem_used=37529364, join_time=1.260917ms, build_time=3.941506121s]
ProjectionExec: expr=[l_extendedprice@0 as l_extendedprice,
l_discount@1 as l_discount, n_name@4 as n_name, n_regionkey@5 as n_regionkey],
metrics=[output_rows=364380, elapsed_compute=1.041µs, spill_count=0,
spilled_bytes=0, mem_used=0]
CoalesceBatchesExec: target_batch_size=8192,
metrics=[output_rows=364380, elapsed_compute=1.767µs, spill_count=0,
spilled_bytes=0, mem_used=0]
HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(Column
{ name: "s_nationkey", index: 2 }, Column { name: "n_nationkey", index: 0 })],
metrics=[output_rows=25, input_batches=1, output_batches=1,
build_input_batches=13, input_rows=25, build_input_rows=364380,
build_mem_used=31885080, join_time=7.462876ms, build_time=3.931845754s]
ProjectionExec: expr=[l_extendedprice@2 as
l_extendedprice, l_discount@3 as l_discount, s_nationkey@5 as s_nationkey],
metrics=[output_rows=364380, elapsed_compute=4.209µs, spill_count=0,
spilled_bytes=0, mem_used=0]
CoalesceBatchesExec: target_batch_size=8192,
metrics=[output_rows=364380, elapsed_compute=8.226µs, spill_count=0,
spilled_bytes=0, mem_used=0]
HashJoinExec: mode=CollectLeft, join_type=Inner,
on=[(Column { name: "l_suppkey", index: 1 }, Column { name: "s_suppkey", index:
0 }), (Column { name: "c_nationkey", index: 0 }, Column { name: "s_nationkey",
index: 1 })], metrics=[output_rows=100000, input_batches=13, output_batches=13,
build_input_batches=1046, input_rows=100000, build_input_rows=9103367,
build_mem_used=1038498128, join_time=74.74725ms, build_time=3.853093588s]
ProjectionExec: expr=[c_nationkey@0 as c_nationkey,
l_suppkey@3 as l_suppkey, l_extendedprice@4 as l_extendedprice, l_discount@5 as
l_discount], metrics=[output_rows=9103367, elapsed_compute=215.36µs,
spill_count=0, spilled_bytes=0, mem_used=0]
CoalesceBatchesExec: target_batch_size=8192,
metrics=[output_rows=9103367, elapsed_compute=31.512778ms, spill_count=0,
spilled_bytes=0, mem_used=0]
HashJoinExec: mode=CollectLeft, join_type=Inner,
on=[(Column { name: "o_orderkey", index: 1 }, Column { name: "l_orderkey",
index: 0 })], metrics=[output_rows=59986052, input_batches=7323,
output_batches=7323, build_input_batches=262, input_rows=59986052,
build_input_rows=2275919, build_mem_used=174938288, join_time=807.006477ms,
build_time=702.906984ms]
ProjectionExec: expr=[c_nationkey@1 as
c_nationkey, o_orderkey@2 as o_orderkey], metrics=[output_rows=2275919,
elapsed_compute=48.584µs, spill_count=0, spilled_bytes=0, mem_used=0]
CoalesceBatchesExec: target_batch_size=8192,
metrics=[output_rows=2275919, elapsed_compute=18.668µs, spill_count=0,
spilled_bytes=0, mem_used=0]
HashJoinExec: mode=CollectLeft,
join_type=Inner, on=[(Column { name: "c_custkey", index: 0 }, Column { name:
"o_custkey", index: 1 })], metrics=[output_rows=2275919, input_batches=262,
output_batches=262, build_input_batches=184, input_rows=2275919,
build_input_rows=1500000, build_mem_used=93390992, join_time=206.31563ms,
build_time=49.711972ms]
ParquetExec: limit=None, partitions={1
group:
[[Users/mingmwang/gitrepo/apache/arrow-datafusion/benchmarks/parquet_data10/customer/part-0.parquet]]},
projection=[c_custkey, c_nationkey], metrics=[output_rows=1500000,
elapsed_compute=1ns, spill_count=0, spilled_bytes=0, mem_used=0,
page_index_rows_filtered=0, num_predicate_creation_errors=0,
bytes_scanned=3006119, row_groups_pruned=0, pushdown_rows_filtered=0,
predicate_evaluation_errors=0, time_elapsed_scanning_until_data=2.157083ms,
time_elapsed_scanning_total=16.696249ms, time_elapsed_processing=15.288754ms,
time_elapsed_opening=1.074375ms, page_index_eval_time=2ns,
pushdown_eval_time=2ns]
ProjectionExec: expr=[o_orderkey@0 as
o_orderkey, o_custkey@1 as o_custkey], metrics=[output_rows=2275919,
elapsed_compute=25.543µs, spill_count=0, spilled_bytes=0, mem_used=0]
CoalesceBatchesExec:
target_batch_size=8192, metrics=[output_rows=2275919,
elapsed_compute=2.349505ms, spill_count=0, spilled_bytes=0, mem_used=0]
FilterExec: o_orderdate@2 >= 8766
AND o_orderdate@2 < 9131, metrics=[output_rows=2275919,
elapsed_compute=19.347372ms, spill_count=0, spilled_bytes=0, mem_used=0]
ParquetExec: limit=None,
partitions={1 group:
[[Users/mingmwang/gitrepo/apache/arrow-datafusion/benchmarks/parquet_data10/orders/part-0.parquet]]},
predicate=o_orderdate@4 >= 8766 AND o_orderdate@4 < 9131,
pruning_predicate=o_orderdate_max@0 >= 8766 AND o_orderdate_min@1 < 9131,
projection=[o_orderkey, o_custkey, o_orderdate], metrics=[output_rows=15000000,
elapsed_compute=1ns, spill_count=0, spilled_bytes=0, mem_used=0,
page_index_rows_filtered=0, num_predicate_creation_errors=0,
bytes_scanned=94686185, row_groups_pruned=0, pushdown_rows_filtered=0,
predicate_evaluation_errors=0, time_elapsed_scanning_until_data=4.577708ms,
time_elapsed_scanning_total=613.686009ms, time_elapsed_processing=365.906294ms,
time_elapsed_opening=1.525625ms, page_index_eval_time=2ns,
pushdown_eval_time=2ns]
ParquetExec: limit=None, partitions={1 group:
[[Users/mingmwang/gitrepo/apache/arrow-datafusion/benchmarks/parquet_data10/lineitem/part-0.parquet]]},
projection=[l_orderkey, l_suppkey, l_extendedprice, l_discount],
metrics=[output_rows=59986052, elapsed_compute=1ns, spill_count=0,
spilled_bytes=0, mem_used=0, page_index_rows_filtered=0,
num_predicate_creation_errors=0, bytes_scanned=453645745, row_groups_pruned=0,
pushdown_rows_filtered=0, predicate_evaluation_errors=0,
time_elapsed_scanning_until_data=6.237208ms,
time_elapsed_scanning_total=2.314007854s, time_elapsed_processing=1.391610509s,
time_elapsed_opening=1.374334ms, page_index_eval_time=2ns,
pushdown_eval_time=2ns]
ParquetExec: limit=None, partitions={1 group:
[[Users/mingmwang/gitrepo/apache/arrow-datafusion/benchmarks/parquet_data10/supplier/part-0.parquet]]},
projection=[s_suppkey, s_nationkey], metrics=[output_rows=100000,
elapsed_compute=1ns, spill_count=0, spilled_bytes=0, mem_used=0,
page_index_rows_filtered=0, num_predicate_creation_errors=0,
bytes_scanned=372287, row_groups_pruned=0, pushdown_rows_filtered=0,
predicate_evaluation_errors=0, time_elapsed_scanning_until_data=1.480458ms,
time_elapsed_scanning_total=76.493501ms, time_elapsed_processing=1.467501ms,
time_elapsed_opening=1.877375ms, page_index_eval_time=2ns,
pushdown_eval_time=2ns]
ParquetExec: limit=None, partitions={1 group:
[[Users/mingmwang/gitrepo/apache/arrow-datafusion/benchmarks/parquet_data10/nation/part-0.parquet]]},
projection=[n_nationkey, n_name, n_regionkey], metrics=[output_rows=25,
elapsed_compute=1ns, spill_count=0, spilled_bytes=0, mem_used=0,
page_index_rows_filtered=0, num_predicate_creation_errors=0, bytes_scanned=544,
row_groups_pruned=0, pushdown_rows_filtered=0, predicate_evaluation_errors=0,
time_elapsed_scanning_until_data=790.959µs,
time_elapsed_scanning_total=8.265166ms, time_elapsed_processing=122.376µs,
time_elapsed_opening=1.493584ms, page_index_eval_time=2ns,
pushdown_eval_time=2ns]
ProjectionExec: expr=[r_regionkey@0 as r_regionkey],
metrics=[output_rows=1, elapsed_compute=125ns, spill_count=0, spilled_bytes=0,
mem_used=0]
CoalesceBatchesExec: target_batch_size=8192,
metrics=[output_rows=1, elapsed_compute=1.336µs, spill_count=0,
spilled_bytes=0, mem_used=0]
FilterExec: r_name@1 = ASIA, metrics=[output_rows=1,
elapsed_compute=6.208µs, spill_count=0, spilled_bytes=0, mem_used=0]
ParquetExec: limit=None, partitions={1 group:
[[Users/mingmwang/gitrepo/apache/arrow-datafusion/benchmarks/parquet_data10/region/part-0.parquet]]},
predicate=r_name@1 = ASIA, pruning_predicate=r_name_min@0 <= ASIA AND ASIA <=
r_name_max@1, projection=[r_regionkey, r_name], metrics=[output_rows=5,
elapsed_compute=1ns, spill_count=0, spilled_bytes=0, mem_used=0,
page_index_rows_filtered=0, num_predicate_creation_errors=0, bytes_scanned=230,
row_groups_pruned=0, pushdown_rows_filtered=0, predicate_evaluation_errors=0,
time_elapsed_scanning_until_data=289.083µs,
time_elapsed_scanning_total=297.375µs, time_elapsed_processing=96.583µs,
time_elapsed_opening=900.083µs, page_index_eval_time=2ns,
pushdown_eval_time=2ns]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
