Hello Lars Volker, Mostafa Mokhtar, Dan Hecht,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/9690
to look at the new patch set (#2).
Change subject: IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and
KrpcDataStreamSender
......................................................................
IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and KrpcDataStreamSender
This change implements a couple of improvements to the profiles of
KrpcDataStreamRecvr and KrpcDataStreamSender:
- track pending number of deferred row batches over time in KrpcDataStreamRecvr
- track the number of bytes dequeued over time in KrpcDataStreamRecvr
- track the min/max/avg of time row batches spent in deferred queue
- track the number of bytes sent from KrpcDataStreamSender over time
Sample profile of an Exchange node instance:
EXCHANGE_NODE (id=22):(Total: 378.797ms, non-child: 123.042ms, %
non-child: 32.48%)
- ConvertRowBatchTime: 70.316ms
- PeakMemoryUsage: 80.93 KB (82875)
- RowsReturned: 66.86K (66855)
- RowsReturnedRate: 176.49 K/sec
Buffer pool:
- AllocTime: 271.165us
- CumulativeAllocationBytes: 1.54 MB (1613824)
- CumulativeAllocations: 132 (132)
- PeakReservation: 80.00 KB (81920)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 80.00 KB (81920)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Dequeue:
BytesDequeued(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.34 MB
- FirstBatchWaitTime: 0.000ns
- TotalBytesDequeued: 1.34 MB (1403955)
- TotalEosReceived: 1 (1)
- TotalGetBatchTime: 308.074ms
- DataWaitTime: 255.763ms
Enqueue:
BytesReceived(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 33.17 KB, 33.17
KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17
KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17
KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17 KB, 33.17
KB, 33.17 KB, 1.06 MB
DeferredBatches(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0
- DeferredQueueTime: (Avg: 324.328ms ; Min: 316.144us ; Max:
13s611ms ; Number of samples: 43)
- DeserializeRowBatchTime: 1.764ms
- TotalBatchesDeferred: 43 (43)
- TotalBatchesEnqueued: 66 (66)
- TotalBatchesReceived: 66 (66)
- TotalBytesReceived: 1.06 MB (1114795)
- TotalEarlySenders: 0 (0)
Sample sender's profile:
KrpcDataStreamSender (dst_id=22):(Total: 83.234ms, non-child: 83.234ms,
% non-child: 100.00%)
BytesSent(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 16.60 KB, 16.60 KB,
16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB,
16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB,
16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB, 16.60 KB,
16.60 KB
- EosSent: 1 (1)
- NetworkThroughput: 21.08 MB/sec
- PeakMemoryUsage: 9.84 KB (10080)
- RowsReturned: 66.86K (66855)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 24.319ms
- TotalBytesSent: 1.06 MB (1114795)
- TotalNetworkTime: 50.424ms
- UncompressedRowBatchSize: 1.34 MB (1403955)
Change-Id: I8ba405921b3df920c1e85b940ce9c8d02fc647cd
---
M be/src/runtime/data-stream-test.cc
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
M be/src/runtime/runtime-state.h
M be/src/service/data-stream-service.cc
M be/src/service/data-stream-service.h
M common/protobuf/data_stream_service.proto
11 files changed, 227 insertions(+), 129 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/9690/2
--
To view, visit http://gerrit.cloudera.org:8080/9690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8ba405921b3df920c1e85b940ce9c8d02fc647cd
Gerrit-Change-Number: 9690
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Ho <k...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com>
Gerrit-Reviewer: Lars Volker <l...@cloudera.com>
Gerrit-Reviewer: Michael Ho <k...@cloudera.com>
Gerrit-Reviewer: Mostafa Mokhtar <mmokh...@cloudera.com>
