[ https://issues.apache.org/jira/browse/IMPALA-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Volker resolved IMPALA-6652. --------------------------------- Resolution: Fixed Fix Version/s: Impala 2.12.0 IMPALA-6652: Rename label of MemTracker for early RPCs This change renames the label of the MemTracker in KrpcDataStreamMgr for tracking payloads of early RPCs to "Data Stream Manager Early RPCs". This is to distinguish these RPCs from the deferred RPCs in a receiver. The early RPCs refer to those RPCs which arrive before a receiver is ready. The responses to these RPCs are deferred until the receiver is created. The receiver may also defer responses to RPCs if the deserialized payloads of RPCs in an inbound queue exceed FLAGS_exchg_node_buffer_size_bytes. In this case, the RPCs won't be responded to until the inbound queue is drained. Change-Id: I5bb72c28e8d660a6b78543dbc8b5b156e0e7c843 Reviewed-on: http://gerrit.cloudera.org:8080/9633 Reviewed-by: Dan Hecht <dhe...@cloudera.com> Reviewed-by: Michael Ho <k...@cloudera.com> Tested-by: Impala Public Jenkins > KRPC : Data Stream Manager Deferred RPCs in memz page should be renamed > ----------------------------------------------------------------------- > > Key: IMPALA-6652 > URL: https://issues.apache.org/jira/browse/IMPALA-6652 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec > Affects Versions: Impala 2.12.0 > Reporter: Mostafa Mokhtar > Assignee: Lars Volker > Priority: Major > Labels: metrics, supportability > Fix For: Impala 2.12.0 > > Attachments: Partial query profile.txt > > > While loading data into a Kudu table against the latest Impala 2.11.0 I > noticed that "Data Stream Manager Deferred RPCs" in the memz isn't accurate. > > From memz on worker > {code} > Process: Limit=201.73 GB Total=85.41 GB Peak=85.41 GB > Buffer Pool: Free Buffers: Total=43.64 MB > Buffer Pool: Clean Pages: Total=0 > Buffer Pool: Unused Reservation: Total=-17.84 MB > Data Stream Service Queue: Limit=10.09 GB Total=0 Peak=512.97 MB > Data Stream Manager Deferred RPCs: Total=0 Peak=0 > TCMalloc Overhead: Total=124.07 MB > Free Disk IO Buffers: Total=984.97 MB Peak=984.97 MB > RequestPool=root.default: Total=83.92 GB Peak=83.92 GB > Query(844a0200d7876345:20bb38b900000000): Reservation=70.44 GB > ReservationLimit=161.39 GB OtherMemory=13.48 GB Total=83.92 GB Peak=83.92 GB > Fragment 844a0200d7876345:20bb38b9000000a3: Reservation=70.44 GB > OtherMemory=38.08 MB Total=70.47 GB Peak=70.47 GB > SORT_NODE (id=2): Reservation=70.44 GB OtherMemory=8.00 KB > Total=70.44 GB Peak=70.44 GB > EXCHANGE_NODE (id=1): Reservation=18.06 MB OtherMemory=0 Total=18.06 > MB Peak=19.53 MB > KrpcDeferredRpcs: Total=0 Peak=1.47 MB > KuduTableSink: Total=20.00 MB Peak=20.00 MB > CodeGen: Total=438.00 B Peak=306.00 KB > Fragment 844a0200d7876345:20bb38b900000022: Reservation=0 > OtherMemory=13.44 GB Total=13.44 GB Peak=13.97 GB > HDFS_SCAN_NODE (id=0): Total=13.44 GB Peak=13.97 GB > KrpcDataStreamSender (dst_id=1): Total=2.57 MB Peak=3.61 MB > CodeGen: Total=234.00 B Peak=52.50 KB > Untracked Memory: Total=389.18 MB > {code} > > And snapshot from query profile > {code} > Instance 844a0200d7876345:20bb38b9000000a3 > (host=va1030.halxg.cloudera.com:22000):(Total: 1s172ms, non-child: 200.411ms, > % non-child: 17.09%) > Fragment Instance Lifecycle Event Timeline: 1s173ms > - Prepare Finished: 199.691ms (199.691ms) > - Open Finished: 1s173ms (973.902ms) > MemoryUsage(1m4s): 4.77 GB, 13.21 GB, 19.60 GB, 23.70 GB, 26.67 GB, > 29.21 GB, 31.50 GB, 33.63 GB, 35.40 GB, 37.14 GB, 38.54 GB, 39.79 GB, 41.09 > GB, 42.37 GB, 43.60 GB, 44.80 GB, 45.95 GB, 47.01 GB, 48.09 GB, 49.17 GB, > 50.22 GB, 51.21 GB, 52.40 GB, 53.46 GB, 54.58 GB, 55.61 GB, 56.58 GB, 57.53 > GB, 58.45 GB, 59.39 GB, 60.31 GB, 61.20 GB, 62.12 GB, 63.04 GB, 64.15 GB, > 65.11 GB, 66.15 GB, 67.06 GB, 67.87 GB, 68.66 GB, 69.49 GB, 70.29 GB, 71.07 > GB, 71.83 GB, 72.61 GB > ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, > 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, > 1, 1 > - AverageThreadTokens: 1.00 > - BloomFilterBytes: 0 > - PeakMemoryUsage: 73.69 GB (79123530166) > - PeakReservation: 73.65 GB (79083601920) > - PeakUsedReservation: 0 > - PerHostPeakMemUsage: 87.02 GB (93435941671) > - RowsProduced: 0 (0) > - TotalNetworkReceiveTime: 38m28s > - TotalNetworkSendTime: 0.000ns > - TotalStorageWaitTime: 0.000ns > - TotalThreadsInvoluntaryContextSwitches: 7 (7) > - TotalThreadsTotalWallClockTime: 973.873ms > - TotalThreadsSysTime: 2.000ms > - TotalThreadsUserTime: 55.991ms > - TotalThreadsVoluntaryContextSwitches: 25 (25) > Buffer pool: > - AllocTime: 0.000ns > - CumulativeAllocationBytes: 0 > - CumulativeAllocations: 0 (0) > - PeakReservation: 0 > - PeakUnpinnedBytes: 0 > - PeakUsedReservation: 0 > - ReadIoBytes: 0 > - ReadIoOps: 0 (0) > - ReadIoWaitTime: 0.000ns > - ReservationLimit: 0 > - WriteIoBytes: 0 > - WriteIoOps: 0 (0) > - WriteIoWaitTime: 0.000ns > Fragment Instance Lifecycle Timings: > - ExecTime: 0.000ns > - ExecTreeExecTime: 0.000ns > - OpenTime: 973.876ms > - ExecTreeOpenTime: 915.567ms > - PrepareTime: 198.988ms > - ExecTreePrepareTime: 155.134us > KuduTableSink:(Total: 12.589us, non-child: 12.589us, % non-child: > 100.00%) > - KuduApplyTimer: 0.000ns > - NumRowErrors: 0 (0) > - PeakMemoryUsage: 20.00 MB (20971520) > - RowsProcessedRate: 0 > - TotalNumRows: 0 (0) > SORT_NODE (id=2):(Total: 915.718ms, non-child: 0.000ns, % non-child: > 0.00%) > SortType: Partial > ExecOption: Codegen Enabled > - NumRowsPerRun: 0 (0) (Number of samples: 0) > - InMemorySortTime: 0.000ns > - PeakMemoryUsage: 73.65 GB (79083610112) > - RowsReturned: 0 (0) > - RowsReturnedRate: 0 > - RunsCreated: 1 (1) > - SortDataSize: 0 > Buffer pool: > - AllocTime: 3m9s > - CumulativeAllocationBytes: 73.65 GB (79083601920) > - CumulativeAllocations: 37.71K (37710) > - PeakReservation: 73.65 GB (79083601920) > - PeakUnpinnedBytes: 0 > - PeakUsedReservation: 73.65 GB (79083601920) > - ReadIoBytes: 0 > - ReadIoOps: 0 (0) > - ReadIoWaitTime: 0.000ns > - WriteIoBytes: 0 > - WriteIoOps: 0 (0) > - WriteIoWaitTime: 0.000ns > EXCHANGE_NODE (id=1):(Total: 38m39s, non-child: 18s181ms, % > non-child: 0.78%) > - ConvertRowBatchTime: 7s926ms > - PeakMemoryUsage: 19.53 MB (20481319) > - RowsReturned: 288.83M (288830892) > - RowsReturnedRate: 124.54 K/sec > Buffer pool: > - AllocTime: 17m12s > - CumulativeAllocationBytes: 131.17 GB (140839444480) > - CumulativeAllocations: 6.88M (6876926) > - PeakReservation: 18.06 MB (18939904) > - PeakUnpinnedBytes: 0 > - PeakUsedReservation: 18.06 MB (18939904) > - ReadIoBytes: 0 > - ReadIoOps: 0 (0) > - ReadIoWaitTime: 0.000ns > - WriteIoBytes: 0 > - WriteIoOps: 0 (0) > - WriteIoWaitTime: 0.000ns > RecvrSide: > BytesReceived(1m4s): 4.87 GB, 13.37 GB, 19.82 GB, 23.95 GB, 26.95 > GB, 29.52 GB, 31.82 GB, 33.98 GB, 35.77 GB, 37.52 GB, 38.94 GB, 40.20 GB, > 41.51 GB, 42.80 GB, 44.05 GB, 45.26 GB, 46.42 GB, 47.49 GB, 48.58 GB, 49.68 > GB, 50.74 GB, 51.74 GB, 52.94 GB, 54.01 GB, 55.15 GB, 56.19 GB, 57.16 GB, > 58.12 GB, 59.05 GB, 60.00 GB, 60.93 GB, 61.84 GB, 62.76 GB, 63.70 GB, 64.82 > GB, 65.79 GB, 66.84 GB, 67.76 GB, 68.57 GB, 69.38 GB, 70.22 GB, 71.02 GB, > 71.81 GB, 72.58 GB, 73.37 GB > - FirstBatchArrivalWaitTime: 915.459ms > - TotalBytesReceived: 74.45 GB (79944017735) > - TotalGetBatchTime: 38m28s > - DataArrivalTimer: 38m21s > SenderSide: > - DeserializeRowBatchTime: 20m44s > - NumBatchesArrived: 3.44M (3438463) > - NumBatchesDeferred: 27.14K (27144) > - NumBatchesEnqueued: 3.44M (3438463) > - NumBatchesReceived: 3.44M (3438463) > - NumEarlySenders: 0 (0) > - NumEosReceived: 0 (0) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)