Michael Ho has uploaded this change for review. (
http://gerrit.cloudera.org:8080/9690
Change subject: IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and
KrpcDataStreamSender
......................................................................
IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and KrpcDataStreamSender
This change implements a couple of improvements to the profiles of
KrpcDataStreamRecvr and KrpcDataStreamSender:
- track the number of deferred row batches over time in KrpcDataStreamRecvr
- track the number of bytes dequeued over time in KrpcDataStreamRecvr
- track the amount of time row batches spent in deferred queue
- track the portion of wait time in KrpcDataStreamRecvr waiting for data to
arrive from sender
- track the number of bytes sent from KrpcDataStreamSender over time
A sample profile of an Exchange node instance:
EXCHANGE_NODE (id=33):(Total: 12s359ms, non-child: 86.379ms, %
non-child: 0.70%)
- ConvertRowBatchTime: 32.374ms
- PeakMemoryUsage: 158.90 KB (162710)
- RowsReturned: 177.37K (177374)
- RowsReturnedRate: 14.35 K/sec
Buffer pool:
- AllocTime: 1.861ms
- CumulativeAllocationBytes: 8.83 MB (9256960)
- CumulativeAllocations: 568 (568)
- PeakReservation: 128.00 KB (131072)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 128.00 KB (131072)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Dequeue:
BytesDequeued(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
1.24 MB, 1.61 MB, 1.89 MB, 2.07 MB, 2.25 MB, 2.50 MB, 2.78 MB, 3.10 MB, 3.38
MB, 3.66 MB, 3.84 MB, 4.26 MB, 4.64 MB, 4.90 MB, 5.12 MB, 5.48 MB, 5.78 MB,
6.01 MB, 6.33 MB, 6.59 MB, 6.64 MB, 6.77 MB, 6.84 MB, 6.95 MB, 7.05 MB
- FirstBatchArrivalWaitTime: 0.000ns
- TotalBytesDequeued: 7.10 MB (7449708)
- TotalEosReceived: 3 (3)
- TotalGetBatchTime: 12s324ms
- DataWaitTime: 12s273ms
- ProducerWaitTime: 12s175ms
Enqueue:
BytesReceived(500.000ms): 0, 0, 0, 0, 0, 0, 0, 30.56 KB, 40.90 KB,
40.90 KB, 40.90 KB, 40.90 KB, 40.90 KB, 40.90 KB, 502.04 KB, 655.77 KB, 768.57
KB, 840.35 KB, 912.10 KB, 1014.75 KB, 1.10 MB, 1.23 MB, 1.34 MB, 1.45 MB, 1.52
MB, 1.69 MB, 1.84 MB, 1.94 MB, 2.03 MB, 2.17 MB, 2.29 MB, 2.38 MB, 2.51 MB,
2.61 MB, 2.63 MB, 2.68 MB, 2.71 MB, 2.75 MB, 2.79 MB
DeferredBatches(500.000ms): 0, 0, 0, 0, 0, 0, 0, 2, 3, 3, 3, 3, 3,
3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
- DeferredQueueTime: 9s775ms
- DeserializeRowBatchTime: 9.351ms
- TotalBatchesDeferred: 35 (35)
- TotalBatchesEnqueued: 284 (284)
- TotalBatchesReceived: 284 (284)
- TotalBytesReceived: 2.82 MB (2954847)
- TotalEarlySenders: 0 (0)
A sample profile a KrpcDataStreamSender's profile:
KrpcDataStreamSender (dst_id=49):(Total: 204.228ms, non-child:
204.228ms, % non-child: 100.00%)
BytesSent(1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 40.40 KB,
46.15 KB, 46.15 KB, 46.15 KB, 46.15 KB, 46.15 KB, 46.15 KB, 775.45 KB, 861.84
KB, 925.29 KB, 994.57 KB, 1017.76 KB, 1.06 MB, 1.06 MB, 1.11 MB, 1.22 MB, 1.23
MB, 1.29 MB, 1.34 MB, 1.39 MB, 1.46 MB, 1.52 MB, 1.64 MB, 1.70 MB, 1.76 MB,
1.82 MB, 1.87 MB, 1.95 MB, 2.11 MB, 2.24 MB, 2.36 MB, 2.49 MB, 2.61 MB, 2.69
MB, 2.81 MB, 2.96 MB, 3.07 MB, 3.26 MB, 3.53 MB, 3.74 MB, 3.96 MB
- EosSent: 3 (3)
- OverallThroughput: 20.54 MB/sec
- PeakMemoryUsage: 106.55 KB (109104)
- RowsReturned: 178.80K (178799)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 136.159ms
- TotalBytesSent: 4.19 MB (4398283)
- UncompressedRowBatchSize: 8.53 MB (8939950)
Change-Id: I8ba405921b3df920c1e85b940ce9c8d02fc647cd
---
M be/src/runtime/krpc-data-stream-mgr.h
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
5 files changed, 144 insertions(+), 74 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/9690/1
--
To view, visit http://gerrit.cloudera.org:8080/9690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8ba405921b3df920c1e85b940ce9c8d02fc647cd
Gerrit-Change-Number: 9690
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Ho <k...@cloudera.com>
