Hello Lars Volker, Mostafa Mokhtar, Dan Hecht,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/9690
to look at the new patch set (#4).
Change subject: IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and
KrpcDataStreamSender
......................................................................
IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and KrpcDataStreamSender
This change implements a couple of improvements to the profiles of
KrpcDataStreamRecvr and KrpcDataStreamSender:
- track pending number of deferred row batches over time in KrpcDataStreamRecvr
- track the number of bytes dequeued over time in KrpcDataStreamRecvr
- track the min/max/avg of time row batches spent in deferred queue
- track the number of bytes sent from KrpcDataStreamSender over time
Sample profile of an Exchange node instance:
EXCHANGE_NODE (id=82):(Total: 1s367ms, non-child: 418.734us, %
non-child: 0.03%)
- ConvertRowBatchTime: 229.386us
- PeakMemoryUsage: 54.24 KB (55541)
- RowsReturned: 1.80K (1800)
- RowsReturnedRate: 1.32 K/sec
Buffer pool:
- AllocTime: 8.537us
- CumulativeAllocationBytes: 48.00 KB (49152)
- CumulativeAllocations: 4 (4)
- PeakReservation: 48.00 KB (49152)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 48.00 KB (49152)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Dequeue:
BytesDequeued(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 36.91 KB, 36.91 KB, 36.91 KB
- FirstBatchWaitTime: 0.000ns
- TotalBytesDequeued: 36.91 KB (37800)
- TotalGetBatchTime: 1s366ms
- DataWaitTime: 1s366ms
Enqueue:
BytesReceived(500.000ms): 0, 0, 0, 0, 0, 0, 0, 8.10 KB, 14.24 KB,
14.24 KB, 14.24 KB, 14.24 KB, 14.24 KB, 14.24 KB, 14.24 KB, 14.24 KB, 14.24 KB,
14.24 KB, 14.24 KB, 14.24 KB, 14.24 KB, 14.24 KB, 14.24 KB, 14.24 KB, 14.24 KB,
14.24 KB, 14.24 KB
DeferredQueueSize(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0
- DispatchTime: (Avg: 258.755us ; Min: 193.825us ; Max:
323.688us ; Number of samples: 2)
- DeserializeRowBatchTime: 60.318us
- TotalBatchesEnqueued: 2 (2)
- TotalBatchesReceived: 2 (2)
- TotalBytesReceived: 14.24 KB (14579)
- TotalEarlySenders: 0 (0)
- TotalEosReceived: 1 (1)
- TotalRPCsDeferralTime: 8s171ms
- TotalRPCsDeferred: 1 (1)
Sample sender's profile:
KrpcDataStreamSender (dst_id=82):(Total: 9s605ms, non-child: 125.726ms,
% non-child: 1.31%)
BytesSent(500.000ms): 0, 0, 0, 0, 24.29 KB, 24.29 KB, 24.29 KB, 24.29
KB, 24.29 KB, 24.29 KB, 24.29 KB, 24.29 KB, 24.29 KB, 24.29 KB, 24.29 KB, 24.29
KB, 24.29 KB, 24.29 KB, 24.29 KB, 24.29 KB, 24.29 KB, 30.43 KB, 30.43 KB, 30.43
KB
- EosSent: 3 (3)
- NetworkThroughput: 290.09 KB/sec
- PeakMemoryUsage: 29.53 KB (30240)
- RowsReturned: 1.80K (1800)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 649.087us
- TotalBytesSent: 42.71 KB (43737)
- UncompressedRowBatchSize: 110.74 KB (113400)
Change-Id: I8ba405921b3df920c1e85b940ce9c8d02fc647cd
---
M be/src/kudu/rpc/rpc_context.cc
M be/src/kudu/rpc/rpc_context.h
M be/src/runtime/data-stream-test.cc
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
M be/src/runtime/runtime-state.h
M be/src/service/data-stream-service.cc
M be/src/service/data-stream-service.h
M common/protobuf/data_stream_service.proto
12 files changed, 282 insertions(+), 139 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/9690/4
--
To view, visit http://gerrit.cloudera.org:8080/9690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8ba405921b3df920c1e85b940ce9c8d02fc647cd
Gerrit-Change-Number: 9690
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Ho <k...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com>
Gerrit-Reviewer: Lars Volker <l...@cloudera.com>
Gerrit-Reviewer: Michael Ho <k...@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mmokh...@cloudera.com>
