Hello Lars Volker, Mostafa Mokhtar, Dan Hecht, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/9690 to look at the new patch set (#3). Change subject: IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and KrpcDataStreamSender ...................................................................... IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and KrpcDataStreamSender This change implements a couple of improvements to the profiles of KrpcDataStreamRecvr and KrpcDataStreamSender: - track pending number of deferred row batches over time in KrpcDataStreamRecvr - track the number of bytes dequeued over time in KrpcDataStreamRecvr - track the min/max/avg of time row batches spent in deferred queue - track the number of bytes sent from KrpcDataStreamSender over time Sample profile of an Exchange node instance: EXCHANGE_NODE (id=22):(Total: 378.797ms, non-child: 123.042ms, % non-child: 32.48%) - ConvertRowBatchTime: 70.316ms - PeakMemoryUsage: 80.93 KB (82875) - RowsReturned: 66.86K (66855) - RowsReturnedRate: 176.49 K/sec Buffer pool: - AllocTime: 271.165us - CumulativeAllocationBytes: 1.54 MB (1613824) - CumulativeAllocations: 132 (132) - PeakReservation: 80.00 KB (81920) - PeakUnpinnedBytes: 0 - PeakUsedReservation: 80.00 KB (81920) - ReadIoBytes: 0 - ReadIoOps: 0 (0) - ReadIoWaitTime: 0.000ns - WriteIoBytes: 0 - WriteIoOps: 0 (0) - WriteIoWaitTime: 0.000ns Dequeue: BytesDequeued(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.34 MB - FirstBatchWaitTime: 0.000ns - TotalBytesDequeued: 1.34 MB (1403955) - TotalEosReceived: 1 (1) - TotalGetBatchTime: 308.074ms - DataWaitTime: 255.763ms Enqueue: BytesReceived(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 1.06 MB DeferredQueueSize(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0 - DeferredQueueTime: (Avg: 324.328ms ; Min: 316.144us ; Max: 13s611ms ; Number of samples: 43) - DeserializeRowBatchTime: 1.764ms - TotalBatchesDeferred: 43 (43) - TotalBatchesEnqueued: 66 (66) - TotalBatchesReceived: 66 (66) - TotalBytesReceived: 1.06 MB (1114795) - TotalEarlySenders: 0 (0) Sample sender's profile: KrpcDataStreamSender (dst_id=22):(Total: 83.234ms, non-child: 83.234ms, % non-child: 100.00%) BytesSent(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB - EosSent: 1 (1) - NetworkThroughput: 21.08 MB/sec - PeakMemoryUsage: 9.84 KB (10080) - RowsReturned: 66.86K (66855) - RpcFailure: 0 (0) - RpcRetry: 0 (0) - SerializeBatchTime: 24.319ms - TotalBytesSent: 1.06 MB (1114795) - TotalNetworkTime: 50.424ms - UncompressedRowBatchSize: 1.34 MB (1403955) Change-Id: I8ba405921b3df920c1e85b940ce9c8d02fc647cd --- M be/src/runtime/data-stream-test.cc M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-mgr.h M be/src/runtime/krpc-data-stream-recvr.cc M be/src/runtime/krpc-data-stream-recvr.h M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/runtime-state.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M common/protobuf/data_stream_service.proto 11 files changed, 227 insertions(+), 129 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/9690/3 -- To view, visit http://gerrit.cloudera.org:8080/9690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8ba405921b3df920c1e85b940ce9c8d02fc647cd Gerrit-Change-Number: 9690 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho <k...@cloudera.com> Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com> Gerrit-Reviewer: Lars Volker <l...@cloudera.com> Gerrit-Reviewer: Michael Ho <k...@cloudera.com> Gerrit-Reviewer: Mostafa Mokhtar <mmokh...@cloudera.com>