Hello Lars Volker, Sailesh Mukil, Dan Hecht, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/9202
to look at the new patch set (#6).
Change subject: IMPALA-6396: Exchange node's memory usage should include its
receiver's
......................................................................
IMPALA-6396: Exchange node's memory usage should include its receiver's
A DataStreamRecvr is co-owned by the DataStreamMgr and
an Exchange node. However, the life time of the memory
allocations (e.g. row batches) of a DataStreamRecvr never
exceeds that of its owning Exchange node. Previously, we
used the fragment instance's MemTracker as the parent of
the DataStreamRecvr's MemTracker. This change switches to
using the MemTracker of the owning Exchange node as the
parent tracker of the DataStreamRecvr. This makes it
easier to identify the peak memory usage of the receivers
of different exchange nodes in the runtime profile and
query summary. Most of the exchange node's memory usage
is from its receiver so we don't track the peak memory
usage of the receiver separately.
Sample output from TPCH-Q21:
EXCHANGE_NODE (id=18):(Total: 1s448ms, non-child: 265.818ms, % non-child:
18.35%)
- ConvertRowBatchTime: 223.895ms
- PeakMemoryUsage: 10.04 MB (10524943)
- RowsReturned: 1.27M (1267464)
- RowsReturnedRate: 875.19 K/sec
RecvrSide:
BytesReceived(500.000ms): 0, 1.64 MB, 9.98 MB, 9.98 MB, 10.01 MB, 10.01 MB,
10.01 MB, 31.79 MB, 60.19 MB, 87.84 MB
- FirstBatchArrivalWaitTime: 0.000ns
- TotalBytesReceived: 93.07 MB (97594728)
- TotalGetBatchTime: 1s194ms
- DataArrivalTimer: 1s183ms
SenderSide:
- DeserializeRowBatchTime: 344.343ms
- NumBatchesAccepted: 3.80K (3796)
- NumBatchesDeferred: 5 (5)
- NumEarlySenders: 0 (0)
Testing done: Updated test_observability.py to verify the
peak memory usage of exchange node is not 0.
Change-Id: I8ca3c47d87bfcd221d34565eda1878f3c15d5c45
---
M be/src/exec/exchange-node.cc
M be/src/runtime/data-stream-mgr-base.h
M be/src/runtime/data-stream-mgr.cc
M be/src/runtime/data-stream-mgr.h
M be/src/runtime/data-stream-test.cc
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M tests/query_test/test_observability.py
8 files changed, 42 insertions(+), 40 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/9202/6
--
To view, visit http://gerrit.cloudera.org:8080/9202
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8ca3c47d87bfcd221d34565eda1878f3c15d5c45
Gerrit-Change-Number: 9202
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Ho <k...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker <l...@cloudera.com>
Gerrit-Reviewer: Michael Ho <k...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sail...@cloudera.com>
