Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12229 )
Change subject: IMPALA-7731: Add Read/Exchange counters to profile ...................................................................... Patch Set 4: (10 comments) http://gerrit.cloudera.org:8080/#/c/12229/2/be/src/runtime/coordinator-backend-state.cc File be/src/runtime/coordinator-backend-state.cc: http://gerrit.cloudera.org:8080/#/c/12229/2/be/src/runtime/coordinator-backend-state.cc@251 PS2, Line 251: } > Is this how we determine whether the fragment had a scan in it or not? Poss Done http://gerrit.cloudera.org:8080/#/c/12229/3/be/src/runtime/coordinator-backend-state.cc File be/src/runtime/coordinator-backend-state.cc: http://gerrit.cloudera.org:8080/#/c/12229/3/be/src/runtime/coordinator-backend-state.cc@524 PS3, Line 524: RuntimeProfile::Counter* c = p->GetCounter(ScanNode::SCAN_RANGES_COMPLETE_COUNTER); > ?? Sry, done. http://gerrit.cloudera.org:8080/#/c/12229/3/be/src/runtime/coordinator-backend-state.cc@531 PS3, Line 531: p->GetCounter(KrpcDataStreamSender::TOTAL_BYTES_SENT_COUNTER); > This this be a constant somewhere? Done http://gerrit.cloudera.org:8080/#/c/12229/2/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/12229/2/be/src/runtime/coordinator.cc@779 PS2, Line 779: total_utilization.bytes_read); > Should we add comments here about what these new counters mean? (Especially I added comments to all of them. Once IMPALA-7550 comes around we can mode them to the counter definition. (WIP here: https://gerrit.cloudera.org/#/c/12116/) http://gerrit.cloudera.org:8080/#/c/12229/2/be/src/runtime/coordinator.cc@794 PS2, Line 794: double xchg_scan_ratio = 0; > nit: extra line Done http://gerrit.cloudera.org:8080/#/c/12229/3/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/12229/3/be/src/runtime/coordinator.cc@790 PS3, Line 790: // node. > Is it the case that It follows directly from these three COUNTER_SET calls, the addition for TotalBytesSent happens in L783. I added a test to the python tests. http://gerrit.cloudera.org:8080/#/c/12229/2/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/12229/2/tests/query_test/test_observability.py@391 PS2, Line 391: "TotalInnerBytesSent", "ExchangeScanRatio", > line has trailing whitespace Done http://gerrit.cloudera.org:8080/#/c/12229/2/tests/query_test/test_observability.py@391 PS2, Line 391: > This query takes 3.4 secs on my machine, which is far from extreme, but I w I picked a query that does a partitioned exchange without being able to push predicates down to the scan, so that the TxRatio would be large. This was the simplest one that I could come up with. It takes 2.4 s on my machine. Do you have an idea for a faster one. http://gerrit.cloudera.org:8080/#/c/12229/2/tests/query_test/test_observability.py@391 PS2, Line 391: > flake8: W291 trailing whitespace Done http://gerrit.cloudera.org:8080/#/c/12229/3/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/12229/3/tests/query_test/test_observability.py@390 PS3, Line 390: expected_counters = ["TotalBytesRead", "TotalBytesSent", "TotalScanBytesSent", > Should we add small tests for the other new metrics you have? I added a second test that checks all counters are present, and amended this one to check that the *BytesSent counters add up. -- To view, visit http://gerrit.cloudera.org:8080/12229 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife7ec78fe42558429c1cbe6e5eba79842bffd648 Gerrit-Change-Number: 12229 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Comment-Date: Fri, 18 Jan 2019 23:20:41 +0000 Gerrit-HasComments: Yes
