Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11967 )
Change subject: IMPALA-1048: show sinks in exec summary ...................................................................... IMPALA-1048: show sinks in exec summary The exec summary now includes the total time taken and memory consumed by the data sink at the root of each fragment. Previously the exec summary could hide where time and memory went while executing a query. The high-level changes are: * Generalising logic in the exec summary and runtime profile to handle data sinks, not just plan nodes, including adding richer metadata to runtime profile nodes. * Threading through metadata about the data sinks, like names and estimates, so that it can appear in the exec summary. The major potential downside is that the new timings reported for data stream sender can overlap with the receiver's time and potentially cause confusion. [localhost:21000] default> select count(distinct l_comment) from tpch_parquet.lineitem; summary; Query: select count(distinct l_comment) from tpch_parquet.lineitem Query submitted at: 2018-11-20 16:47:03 (Coordinator: http://tarmstrong-box:25000) Query progress can be monitored at: http://tarmstrong-box:25000/query_plan?query_id=f5464383a3bb6878:54b5252b00000000 +---------------------------+ | count(distinct l_comment) | +---------------------------+ | 4580667 | +---------------------------+ Fetched 1 row(s) in 4.53s +---------------------+--------+----------+----------+-------+------------+-----------+---------------+-----------------------+ | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail | +---------------------+--------+----------+----------+-------+------------+-----------+---------------+-----------------------+ | F02:ROOT | 1 | 50.56us | 50.56us | | | 0 B | 0 B | | | 06:AGGREGATE | 1 | 275.89us | 275.89us | 1 | 1 | 16.00 KB | 10.00 MB | FINALIZE | | 05:EXCHANGE | 1 | 49.08us | 49.08us | 3 | 1 | 32.00 KB | 16.00 KB | UNPARTITIONED | | F01:EXCHANGE SENDER | 3 | 100.06us | 113.49us | | | 16.00 KB | 0 B | | | 02:AGGREGATE | 3 | 19.32ms | 19.57ms | 3 | 1 | 16.00 KB | 10.00 MB | | | 04:AGGREGATE | 3 | 1.29s | 1.43s | 4.58M | 4.65M | 98.02 MB | 62.63 MB | | | 03:EXCHANGE | 3 | 241.64ms | 246.54ms | 5.01M | 4.65M | 9.05 MB | 10.12 MB | HASH(l_comment) | | F00:EXCHANGE SENDER | 3 | 2.43s | 2.58s | | | 337.53 KB | 0 B | | | 01:AGGREGATE | 3 | 1.26s | 1.46s | 5.01M | 4.65M | 97.20 MB | 121.17 MB | STREAMING | | 00:SCAN HDFS | 3 | 39.87ms | 41.36ms | 6.00M | 6.00M | 27.87 MB | 80.00 MB | tpch_parquet.lineitem | +---------------------+--------+----------+----------+-------+------------+-----------+---------------+-----------------------+ Testing: Added a basic observability test. Change-Id: I3fdf7bacae8ff597b255da65af453e174ba53544 Reviewed-on: http://gerrit.cloudera.org:8080/11967 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/hbase-table-sink.cc M be/src/exec/hbase-table-sink.h M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/exec/kudu-table-sink.cc M be/src/exec/kudu-table-sink.h M be/src/exec/nested-loop-join-builder.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/data-stream-test.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M be/src/util/summary-util.cc M common/thrift/DataSinks.thrift M common/thrift/ExecStats.thrift M common/thrift/RuntimeProfile.thrift M common/thrift/Types.thrift M fe/src/main/java/org/apache/impala/planner/DataSink.java M fe/src/main/java/org/apache/impala/planner/DataStreamSink.java M fe/src/main/java/org/apache/impala/planner/HBaseTableSink.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/planner/JoinBuildSink.java M fe/src/main/java/org/apache/impala/planner/KuduTableSink.java M fe/src/main/java/org/apache/impala/planner/PlanRootSink.java M shell/impala_client.py M tests/beeswax/impala_beeswax.py M tests/query_test/test_observability.py 37 files changed, 348 insertions(+), 136 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11967 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3fdf7bacae8ff597b255da65af453e174ba53544 Gerrit-Change-Number: 11967 Gerrit-PatchSet: 16 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
