Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12229 )
Change subject: IMPALA-7731: Add Read/Exchange counters to profile ...................................................................... IMPALA-7731: Add Read/Exchange counters to profile Selective scans (and by extension selective fragment instances) take higher performance hits when reading data remotely. They can be identified by a low ratio between data being transmitted vs data being read from HDFS. This change adds several counters to the profile to make it easier to identify queries based on their scan instance selectivity. * TotalBytesSent - The total number of bytes sent by a query in exchange nodes. Does not include remote reads, data written to disk, or data sent to the client. * TotalScanBytesSent - The total number of bytes sent by fragment instances that had a scan node in their plan. * TotalInnerBytesSent - The total number of bytes sent by fragment instances that did not have a scan node in their plan, i.e. that received their input data from other instances through exchange node. * ExchangeScanRatio - The ratio between TotalScanBytesSent and TotalBytesRead, i.e. the selectivity over all fragment instances that had a scan node in their plan. This counter is also added to each fragment instance. * InnerNodeSelectivityRatio - The ratio between bytes sent by instances with a scan node in their plan and instances without a scan node in their plan. This indicates how well the inner nodes of the execution plan reduced the data volume. Change-Id: Ife7ec78fe42558429c1cbe6e5eba79842bffd648 Reviewed-on: http://gerrit.cloudera.org:8080/12229 Reviewed-by: Lars Volker <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/scan-node.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/util/pretty-printer.h M tests/query_test/test_observability.py 12 files changed, 141 insertions(+), 3 deletions(-) Approvals: Lars Volker: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12229 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ife7ec78fe42558429c1cbe6e5eba79842bffd648 Gerrit-Change-Number: 12229 Gerrit-PatchSet: 6 Gerrit-Owner: Lars Volker <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]>
