Dandandan commented on issue #5999:
URL:
https://github.com/apache/arrow-datafusion/issues/5999#issuecomment-1509129584
For q1 tpch-h the metrics are as follows for the complete plan are as
follows (only including ):
```
SortExec: [l_returnflag@0 ASC NULLS LAST,l_linestatus@1 ASC NULLS LAST],
metrics=[output_rows=4, elapsed_compute=25.042µs, spill_count=0,
spilled_bytes=0]
CoalescePartitionsExec, metrics=[output_rows=4, elapsed_compute=3.167µs,
spill_count=0, spilled_bytes=0, mem_used=0]
ProjectionExec: expr=[l_returnflag@0 as l_returnflag, l_linestatus@1 as
l_linestatus, SUM(lineitem.l_quantity)@2 as sum_qty,
SUM(lineitem.l_extendedprice)@3 as sum_base_price, SUM(lineitem.l_extendedprice
* Int64(1) - lineitem.l_discount)@4 as sum_disc_price,
SUM(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount * Int64(1) +
lineitem.l_tax)@5 as sum_charge, AVG(lineitem.l_quantity)@6 as avg_qty,
AVG(lineitem.l_extendedprice)@7 as avg_price, AVG(lineitem.l_discount)@8 as
avg_disc, COUNT(UInt8(1))@9 as count_order], metrics=[output_rows=4,
elapsed_compute=8.376µs, spill_count=0, spilled_bytes=0, mem_used=0]
AggregateExec: mode=FinalPartitioned, gby=[l_returnflag@0 as
l_returnflag, l_linestatus@1 as l_linestatus], aggr=[SUM(lineitem.l_quantity),
SUM(lineitem.l_extendedprice), SUM(lineitem.l_extendedprice * Int64(1) -
lineitem.l_discount), SUM(lineitem.l_extendedprice * Int64(1) -
lineitem.l_discount * Int64(1) + lineitem.l_tax), AVG(lineitem.l_quantity),
AVG(lineitem.l_extendedprice), AVG(lineitem.l_discount), COUNT(UInt8(1))],
metrics=[output_rows=4, elapsed_compute=181.588µs, spill_count=0,
spilled_bytes=0, mem_used=0]
CoalesceBatchesExec: target_batch_size=8192,
metrics=[output_rows=64, elapsed_compute=350.731µs, spill_count=0,
spilled_bytes=0, mem_used=0]
RepartitionExec: partitioning=Hash([Column { name: "l_returnflag",
index: 0 }, Column { name: "l_linestatus", index: 1 }], 16),
input_partitions=16, metrics=[repart_time=296.707µs, fetch_time=1.652179502s,
send_time=110.333µs]
AggregateExec: mode=Partial, gby=[l_returnflag@5 as
l_returnflag, l_linestatus@6 as l_linestatus], aggr=[SUM(lineitem.l_quantity),
SUM(lineitem.l_extendedprice), SUM(lineitem.l_extendedprice * Int64(1) -
lineitem.l_discount), SUM(lineitem.l_extendedprice * Int64(1) -
lineitem.l_discount * Int64(1) + lineitem.l_tax), AVG(lineitem.l_quantity),
AVG(lineitem.l_extendedprice), AVG(lineitem.l_discount), COUNT(UInt8(1))],
metrics=[output_rows=64, elapsed_compute=670.144742ms, spill_count=0,
spilled_bytes=0, mem_used=0]
ProjectionExec: expr=[CAST(l_extendedprice@1 AS Decimal128(38,
4)) * CAST(Some(100),23,2 - CAST(l_discount@2 AS Decimal128(23, 2)) AS
Decimal128(38, 4)) as CAST(lineitem.l_extendedprice AS Decimal128(38, 4)) *
CAST(Decimal128(Some(100),23,2) - CAST(lineitem.l_discount AS Decimal128(23,
2)) AS Decimal128(38, 4))CAST(Decimal128(Some(100),23,2) -
CAST(lineitem.l_discount AS Decimal128(23, 2)) AS Decimal128(38,
4))Decimal128(Some(100),23,2) - CAST(lineitem.l_discount AS Decimal128(23,
2))CAST(lineitem.l_discount AS Decimal128(23,
2))lineitem.l_discountDecimal128(Some(100),23,2)CAST(lineitem.l_extendedprice
AS Decimal128(38, 4))lineitem.l_extendedprice, l_quantity@0 as l_quantity,
l_extendedprice@1 as l_extendedprice, l_discount@2 as l_discount, l_tax@3 as
l_tax, l_returnflag@4 as l_returnflag, l_linestatus@5 as l_linestatus],
metrics=[output_rows=5916591, elapsed_compute=162.828798ms, spill_count=0,
spilled_bytes=0, mem_used=0]
CoalesceBatchesExec: target_batch_size=8192,
metrics=[output_rows=5916591, elapsed_compute=29.656446ms, spill_count=0,
spilled_bytes=0, mem_used=0]
FilterExec: l_shipdate@6 <= 10471,
metrics=[output_rows=5916591, elapsed_compute=54.707373ms, spill_count=0,
spilled_bytes=0, mem_used=0]
ParquetExec: limit=None, partitions={16 groups:
[[...]]}, predicate=l_shipdate <= Date32("10471"),
pruning_predicate=l_shipdate_min@0 <= 10471, projection=[l_quantity,
l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate],
metrics=[output_rows=6001215, elapsed_compute=16ns, spill_count=0,
spilled_bytes=0, mem_used=0, row_groups_pruned=0,
predicate_evaluation_errors=0, bytes_scanned=44132502,
num_predicate_creation_errors=0, pushdown_rows_filtered=0,
page_index_rows_filtered=0, pushdown_eval_time=32ns,
time_elapsed_scanning_until_data=493.033123ms,
time_elapsed_scanning_total=1.643740915s, time_elapsed_processing=290.136543ms,
time_elapsed_opening=6.494334ms, page_index_eval_time=32ns]
```
Relevant metrics:
* ParquetExec time_elapsed_scanning_until_data=493.033123ms
* ParquetExec time_elapsed_scanning_total=1.643740915s
* ParquetExec time_elapsed_processing=290.136543ms
* ParquetExec time_elapsed_opening=6.494334ms
* FilterExec elapsed_compute=54.707373ms
* CoalesceBatchesExec elapsed_compute=29.656446ms
* ProjectionExec elapsed_compute=162.828798ms
* RepartitionExec elapsed_compute=670.144742ms
* AggregateExec elapsed_compute=670.144742ms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
