Mostafa Mokhtar created IMPALA-6652:
---------------------------------------
Summary: KRPC : Data Stream Manager Deferred RPCs in memz page
isn't correct
Key: IMPALA-6652
URL: https://issues.apache.org/jira/browse/IMPALA-6652
Project: IMPALA
Issue Type: Bug
Components: Distributed Exec
Affects Versions: Impala 2.11.0
Reporter: Mostafa Mokhtar
Assignee: Lars Volker
While loading data into a Kudu table against the latest Impala 2.11.0 I noticed
that "Data Stream Manager Deferred RPCs" in the memz isn't accurate.
>From memz on worker
{code}
Process: Limit=201.73 GB Total=85.41 GB Peak=85.41 GB Buffer Pool: Free
Buffers: Total=43.64 MB Buffer Pool: Clean Pages: Total=0 Buffer Pool: Unused
Reservation: Total=-17.84 MB Data Stream Service Queue: Limit=10.09 GB Total=0
Peak=512.97 MB Data Stream Manager Deferred RPCs: Total=0 Peak=0 TCMalloc
Overhead: Total=124.07 MB Free Disk IO Buffers: Total=984.97 MB Peak=984.97 MB
RequestPool=root.default: Total=83.92 GB Peak=83.92 GB
Query(844a0200d7876345:20bb38b900000000): Reservation=70.44 GB
ReservationLimit=161.39 GB OtherMemory=13.48 GB Total=83.92 GB Peak=83.92 GB
Fragment 844a0200d7876345:20bb38b9000000a3: Reservation=70.44 GB
OtherMemory=38.08 MB Total=70.47 GB Peak=70.47 GB SORT_NODE (id=2):
Reservation=70.44 GB OtherMemory=8.00 KB Total=70.44 GB Peak=70.44 GB
EXCHANGE_NODE (id=1): Reservation=18.06 MB OtherMemory=0 Total=18.06 MB
Peak=19.53 MB KrpcDeferredRpcs: Total=0 Peak=1.47 MB KuduTableSink: Total=20.00
MB Peak=20.00 MB CodeGen: Total=438.00 B Peak=306.00 KB Fragment
844a0200d7876345:20bb38b900000022: Reservation=0 OtherMemory=13.44 GB
Total=13.44 GB Peak=13.97 GB HDFS_SCAN_NODE (id=0): Total=13.44 GB Peak=13.97
GB KrpcDataStreamSender (dst_id=1): Total=2.57 MB Peak=3.61 MB CodeGen:
Total=234.00 B Peak=52.50 KB Untracked Memory: Total=389.18 MB
{code}
And snapshot from query profile
{code}
Instance 844a0200d7876345:20bb38b9000000a3
(host=va1030.halxg.cloudera.com:22000):(Total: 1s172ms, non-child: 200.411ms, %
non-child: 17.09%) Fragment Instance Lifecycle Event Timeline: 1s173ms -
Prepare Finished: 199.691ms (199.691ms) - Open Finished: 1s173ms (973.902ms)
MemoryUsage(1m4s): 4.77 GB, 13.21 GB, 19.60 GB, 23.70 GB, 26.67 GB, 29.21 GB,
31.50 GB, 33.63 GB, 35.40 GB, 37.14 GB, 38.54 GB, 39.79 GB, 41.09 GB, 42.37 GB,
43.60 GB, 44.80 GB, 45.95 GB, 47.01 GB, 48.09 GB, 49.17 GB, 50.22 GB, 51.21 GB,
52.40 GB, 53.46 GB, 54.58 GB, 55.61 GB, 56.58 GB, 57.53 GB, 58.45 GB, 59.39 GB,
60.31 GB, 61.20 GB, 62.12 GB, 63.04 GB, 64.15 GB, 65.11 GB, 66.15 GB, 67.06 GB,
67.87 GB, 68.66 GB, 69.49 GB ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - BloomFilterBytes: 0 -
PeakMemoryUsage: 70.47 GB (75663229366) - PeakReservation: 70.43 GB
(75623301120) - PeakUsedReservation: 0 - PerHostPeakMemUsage: 83.91 GB
(90098285809) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 34m43s -
TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns -
TotalThreadsInvoluntaryContextSwitches: 7 (7) - TotalThreadsTotalWallClockTime:
973.873ms - TotalThreadsSysTime: 2.000ms - TotalThreadsUserTime: 55.991ms -
TotalThreadsVoluntaryContextSwitches: 25 (25) Buffer pool: - AllocTime: 0.000ns
- CumulativeAllocationBytes: 0 - CumulativeAllocations: 0 (0) -
PeakReservation: 0 - PeakUnpinnedBytes: 0 - PeakUsedReservation: 0 -
ReadIoBytes: 0 - ReadIoOps: 0 (0) - ReadIoWaitTime: 0.000ns - ReservationLimit:
0 - WriteIoBytes: 0 - WriteIoOps: 0 (0) - WriteIoWaitTime: 0.000ns Fragment
Instance Lifecycle Timings: - ExecTime: 0.000ns - ExecTreeExecTime: 0.000ns -
OpenTime: 973.876ms - ExecTreeOpenTime: 915.567ms - PrepareTime: 198.988ms -
ExecTreePrepareTime: 155.134us KuduTableSink:(Total: 12.589us, non-child:
12.589us, % non-child: 100.00%) - KuduApplyTimer: 0.000ns - NumRowErrors: 0 (0)
- PeakMemoryUsage: 20.00 MB (20971520) - RowsProcessedRate: 0 - TotalNumRows: 0
(0) SORT_NODE (id=2):(Total: 915.718ms, non-child: 0.000ns, % non-child: 0.00%)
SortType: Partial ExecOption: Codegen Enabled - NumRowsPerRun: 0 (0) (Number of
samples: 0) - InMemorySortTime: 0.000ns - PeakMemoryUsage: 70.43 GB
(75623309312) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - RunsCreated: 1 (1)
- SortDataSize: 0 Buffer pool: - AllocTime: 2m47s - CumulativeAllocationBytes:
70.43 GB (75623301120) - CumulativeAllocations: 36.06K (36060) -
PeakReservation: 70.43 GB (75623301120) - PeakUnpinnedBytes: 0 -
PeakUsedReservation: 70.43 GB (75623301120) - ReadIoBytes: 0 - ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns - WriteIoBytes: 0 - WriteIoOps: 0 (0) -
WriteIoWaitTime: 0.000ns EXCHANGE_NODE (id=1):(Total: 34m53s, non-child:
17s052ms, % non-child: 0.81%) - ConvertRowBatchTime: 7s479ms - PeakMemoryUsage:
19.53 MB (20481319) - RowsReturned: 276.19M (276187128) - RowsReturnedRate:
131.93 K/sec Buffer pool: - AllocTime: 15m30s - CumulativeAllocationBytes:
125.43 GB (134674104320) - CumulativeAllocations: 6.58M (6575884) -
PeakReservation: 18.06 MB (18939904) - PeakUnpinnedBytes: 0 -
PeakUsedReservation: 18.06 MB (18939904) - ReadIoBytes: 0 - ReadIoOps: 0 (0) -
ReadIoWaitTime: 0.000ns - WriteIoBytes: 0 - WriteIoOps: 0 (0) -
WriteIoWaitTime: 0.000ns RecvrSide: BytesReceived(1m4s): 4.87 GB, 13.37 GB,
19.82 GB, 23.95 GB, 26.95 GB, 29.52 GB, 31.82 GB, 33.98 GB, 35.77 GB, 37.52 GB,
38.94 GB, 40.20 GB, 41.51 GB, 42.80 GB, 44.05 GB, 45.26 GB, 46.42 GB, 47.49 GB,
48.58 GB, 49.68 GB, 50.74 GB, 51.74 GB, 52.94 GB, 54.01 GB, 55.15 GB, 56.19 GB,
57.16 GB, 58.12 GB, 59.05 GB, 60.00 GB, 60.93 GB, 61.84 GB, 62.76 GB, 63.70 GB,
64.82 GB, 65.79 GB, 66.84 GB, 67.76 GB, 68.57 GB, 69.38 GB, 70.22 GB -
FirstBatchArrivalWaitTime: 915.459ms - TotalBytesReceived: 71.19 GB
(76444406332) - TotalGetBatchTime: 34m43s - DataArrivalTimer: 34m36s
SenderSide: - DeserializeRowBatchTime: 18m52s - NumBatchesArrived: 3.29M
(3287942) - NumBatchesDeferred: 26.43K (26428) - NumBatchesEnqueued: 3.29M
(3287942) - NumBatchesReceived: 3.29M (3287942) - NumEarlySenders: 0 (0) -
NumEosReceived: 0 (0)
\{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
