Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11992 )
Change subject: IMPALA-2343: Add lifecycle timeline to plan nodes ...................................................................... IMPALA-2343: Add lifecycle timeline to plan nodes Track the time when various significant events in the lifecycle of an ExecNode occur the timeline of fragment execution. The events tracked are: 'Open Started', 'Open Finished', 'First Batch Fetched', 'First Batch Returned', 'Last Batch Returned', 'Closed'. This uses the existing EventSequence infrastructure so that time will correspond to the fragment instance lifecycle timelines. It's implemented mostly using scoped objects that add the events when entering or exiting Open() and GetNext(). These times are not set inside subplans because it would be mostly redundant with the counters in the containing subplan node. Also fix MaterializeTupleTime for Kudu scan nodes to match the timing of other scanners, where it measures the time spent in the scanner threads, not the time in the main fragment execution thread. Testing: Added a basic test to verify that the event sequence is present. Manually inspected some profiles. Verified the Kudu timer by running a query with scanner parallelism and checking that MaterializeTupleTime was > the wallclock time. set num_nodes=1; select * from tpch_kudu.lineitem where lower(l_comment) = 'foo'; Perf: Ran TPC-H 10 locally. There was no significant perf change. Ran TPC-H Nested locally. There was no significant perf change. Change-Id: I15341bdb15022bad9814882689ce5cb2939f4653 Reviewed-on: http://gerrit.cloudera.org:8080/11992 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M be/src/exec/aggregation-node.cc M be/src/exec/analytic-eval-node.cc M be/src/exec/cardinality-check-node.cc M be/src/exec/data-source-scan-node.cc M be/src/exec/empty-set-node.cc M be/src/exec/empty-set-node.h M be/src/exec/exchange-node.cc A be/src/exec/exec-node-util.h M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/hbase-scan-node.cc M be/src/exec/hdfs-scan-node.cc M be/src/exec/kudu-scan-node.cc M be/src/exec/kudu-scanner.cc M be/src/exec/nested-loop-join-node.cc M be/src/exec/partial-sort-node.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/select-node.cc M be/src/exec/sort-node.cc M be/src/exec/streaming-aggregation-node.cc M be/src/exec/subplan-node.cc M be/src/exec/topn-node.cc M be/src/exec/union-node.cc M be/src/exec/unnest-node.cc M tests/query_test/test_observability.py 25 files changed, 214 insertions(+), 45 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11992 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I15341bdb15022bad9814882689ce5cb2939f4653 Gerrit-Change-Number: 11992 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Lars Volker <l...@cloudera.com> Gerrit-Reviewer: Philip Zeyliger <phi...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
