It is recommended to set a per query memory limit as part of admission and not rely on estimates as they are sometimes inaccurate. Can you please include the full query profile?
On Thu, Feb 22, 2018 at 12:13 PM, Fawze Abujaber <[email protected]> wrote: > Hi Mostafa, > > It's not a specific query, almost all the query has such differene between > the 2 values. > > I can see even queries showing the estimate per node is 8.2 Tib > > > - User: psanalytics > - Database: default > - Query Type: QUERY > - Coordinator: slpr-dhc014.lpdomain.com > <http://slvr-cmc101:7180/cmf/hardware/hosts/22/status> > - Duration: 6.48s > - Rows Produced: 708 > - Estimated per Node Peak Memory: 8.2 TiB > - Per Node Peak Memory Usage: 1.1 GiB > - Pool: root.impanalytics > - Threads: CPU Time: 20.1m > > > > How you can explain this behavior, and for sure i don't have 8.2 Tib > memory per node to give neither you. > > Can you please explain me how i should treat Estimated per Node Peak > Memory and if it used by impala for the resource pool and admission control > and what is the relation of this value to the default memory per node that > i setup for each resource pool? > > Below is part of one of the queries profile which the estimate per node > was ~ @GB and the used was 200MB per node. > > > Instance 744de1b6228736fa:b54bfaa70000000f > (host=slpr-dhc004.lpdomain.com:22000):(Total: > 1s455ms, non-child: 1s292ms, % non-child: 88.82%) > Hdfs split stats (<volume id>:<# splits>/<split lengths>): > 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB 9:1/191.64 > KB 5:2/289.57 KB 3:2/1012.83 KB > MemoryUsage(500.000ms): 6.09 MB, 6.09 MB, 3.00 MB > ThreadUsage(500.000ms): 1, 1, 1 > - AverageThreadTokens: 1.00 > - BloomFilterBytes: 0 > - PeakMemoryUsage: 7.17 MB (7521751) > - PeakReservation: 0 > - PeakUsedReservation: 0 > - PerHostPeakMemUsage: 106.53 MB (111709581) > - RowsProduced: 32.83K (32826) > - TotalNetworkReceiveTime: 0.000ns > - TotalNetworkSendTime: 1s297ms > - TotalStorageWaitTime: 234.356ms > - TotalThreadsInvoluntaryContextSwitches: 66 (66) > - TotalThreadsTotalWallClockTime: 1s715ms > - TotalThreadsSysTime: 5.998ms > - TotalThreadsUserTime: 124.975ms > - TotalThreadsVoluntaryContextSwitches: 303 (303) > Fragment Instance Lifecycle Timings: > - ExecTime: 1s394ms > - ExecTreeExecTime: 67.115ms > - OpenTime: 32.795ms > - ExecTreeOpenTime: 73.243us > - PrepareTime: 27.602ms > - ExecTreePrepareTime: 243.141us > DataStreamSender (dst_id=11):(Total: 38.747ms, non-child: > 38.747ms, % non-child: 100.00%) > - BytesSent: 39.71 MB (41643000) > - NetworkThroughput(*): 1.97 GB/sec > - OverallThroughput: 1.00 GB/sec > - PeakMemoryUsage: 59.38 KB (60800) > - RowsReturned: 32.83K (32826) > - SerializeBatchTime: 16.860ms > - TransmitDataRPCTime: 19.698ms > - UncompressedRowBatchSize: 77.58 MB (81350840) > CodeGen:(Total: 56.573ms, non-child: 56.573ms, % non-child: > 100.00%) > - CodegenTime: 1.299ms > - CompileTime: 10.672ms > - LoadTime: 0.000ns > - ModuleBitcodeSize: 1.96 MB (2050180) > - NumFunctions: 16 (16) > - NumInstructions: 250 (250) > - OptimizationTime: 21.023ms > - PeakMemoryUsage: 125.00 KB (128000) > - PrepareTime: 24.116ms > SUBPLAN_NODE (id=6):(Total: 67.311ms, non-child: 12.013ms, % > non-child: 17.85%) > - PeakMemoryUsage: 627.94 KB (643015) > - RowsReturned: 32.77K (32768) > - RowsReturnedRate: 486.81 K/sec > NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.999ms, non-child: > 25.197ms, % non-child: 74.11%) > - BuildRows: 0 (0) > - BuildTime: 0.000ns > - PeakMemoryUsage: 24.00 KB (24576) > - ProbeRows: 32.83K (32826) > - ProbeTime: 0.000ns > - RowsReturned: 16.80M (16795311) > - RowsReturnedRate: 493.99 M/sec > Nested Loop Join Builder: > - PeakMemoryUsage: 8.00 KB (8192) > SINGULAR_ROW_SRC_NODE (id=7): > - PeakMemoryUsage: 0 > - RowsReturned: 0 (0) > - RowsReturnedRate: 0 > UNNEST_NODE (id=8):(Total: 8.801ms, non-child: 8.801ms, % > non-child: 100.00%) > - AvgCollectionSize: 1.00 > - MaxCollectionSize: 1 (1) > - MinCollectionSize: 1 (1) > - NumCollections: 32.83K (32826) > - PeakMemoryUsage: 0 > - RowsReturned: 1 (1) > - RowsReturnedRate: 113.00 /sec > HDFS_SCAN_NODE (id=5):(Total: 21.299ms, non-child: 21.299ms, % > non-child: 100.00%) > Hdfs split stats (<volume id>:<# splits>/<split lengths>): > 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB 9:1/191.64 > KB 5:2/289.57 KB 3:2/1012.83 KB > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 11 out of > 11 > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% > File Formats: PARQUET/SNAPPY:66 > BytesRead(500.000ms): 2.26 MB, 2.26 MB, 2.26 MB > - FooterProcessingTime: (Avg: 12.082ms ; Min: 9.079ms ; Max: > 29.275ms ; Number of samples: 11) > - AverageHdfsReadThreadConcurrency: 0.00 > - AverageScannerThreadConcurrency: 0.00 > - BytesRead: 2.26 MB (2373738) > - BytesReadDataNodeCache: 0 > - BytesReadLocal: 2.26 MB (2373738) > - BytesReadRemoteUnexpected: 0 > - BytesReadShortCircuit: 2.26 MB (2373738) > - CachedFileHandlesHitCount: 0 (0) > - CachedFileHandlesMissCount: 77 (77) > - DecompressionTime: 1.543ms > - MaxCompressedTextFileLength: 0 > - NumColumns: 6 (6) > - NumDictFilteredRowGroups: 0 (0) > - NumDisksAccessed: 8 (8) > - NumRowGroups: 11 (11) > - NumScannerThreadsStarted: 11 (11) > - NumScannersWithNoReads: 0 (0) > - NumStatsFilteredRowGroups: 0 (0) > - PeakMemoryUsage: 6.51 MB (6821203) > - PerReadThreadRawHdfsThroughput: 764.51 MB/sec > - RemoteScanRanges: 0 (0) > - RowBatchQueueGetWaitTime: 18.825ms > - RowBatchQueuePutWaitTime: 0.000ns > - RowsRead: 73.46K (73464) > - RowsReturned: 32.83K (32826) > - RowsReturnedRate: 1.54 M/sec > - ScanRangesComplete: 11 (11) > - ScannerThreadsInvoluntaryContextSwitches: 1 (1) > - ScannerThreadsTotalWallClockTime: 287.702ms > - MaterializeTupleTime(*): 11.921ms > - ScannerThreadsSysTime: 999.000us > - ScannerThreadsUserTime: 26.990ms > - ScannerThreadsVoluntaryContextSwitches: 232 (232) > - TotalRawHdfsReadTime(*): 2.961ms > - TotalReadThroughput: 1.51 MB/sec > Instance 744de1b6228736fa:b54bfaa70000000d > (host=slpr-dhc002.lpdomain.com:22000):(Total: 1s453ms, non-child: > 1s254ms, % non-child: 86.28%) > Hdfs split stats (<volume id>:<# splits>/<split lengths>): > 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB > MemoryUsage(500.000ms): 5.32 MB, 5.32 MB, 1.27 MB > ThreadUsage(500.000ms): 1, 1, 1 > - AverageThreadTokens: 1.00 > - BloomFilterBytes: 0 > - PeakMemoryUsage: 5.86 MB (6145964) > - PeakReservation: 0 > - PeakUsedReservation: 0 > - PerHostPeakMemUsage: 106.66 MB (111842766) > - RowsProduced: 30.34K (30340) > - TotalNetworkReceiveTime: 0.000ns > - TotalNetworkSendTime: 1s256ms > - TotalStorageWaitTime: 106.530ms > - TotalThreadsInvoluntaryContextSwitches: 177 (177) > - TotalThreadsTotalWallClockTime: 1s557ms > - TotalThreadsSysTime: 6.998ms > - TotalThreadsUserTime: 143.976ms > - TotalThreadsVoluntaryContextSwitches: 136 (136) > Fragment Instance Lifecycle Timings: > - ExecTime: 1s390ms > - ExecTreeExecTime: 91.500ms > - OpenTime: 34.085ms > - ExecTreeOpenTime: 81.436us > - PrepareTime: 28.524ms > - ExecTreePrepareTime: 217.949us > DataStreamSender (dst_id=11):(Total: 48.674ms, non-child: > 48.674ms, % non-child: 100.00%) > - BytesSent: 36.61 MB (38388760) > - NetworkThroughput(*): 1.51 GB/sec > - OverallThroughput: 752.15 MB/sec > - PeakMemoryUsage: 59.38 KB (60800) > - RowsReturned: 30.34K (30340) > - SerializeBatchTime: 23.132ms > - TransmitDataRPCTime: 23.609ms > - UncompressedRowBatchSize: 71.70 MB (75187520) > CodeGen:(Total: 59.024ms, non-child: 59.024ms, % non-child: > 100.00%) > - CodegenTime: 1.382ms > - CompileTime: 10.435ms > - LoadTime: 0.000ns > - ModuleBitcodeSize: 1.96 MB (2050180) > - NumFunctions: 16 (16) > - NumInstructions: 250 (250) > - OptimizationTime: 22.325ms > - PeakMemoryUsage: 125.00 KB (128000) > - PrepareTime: 25.459ms > SUBPLAN_NODE (id=6):(Total: 91.711ms, non-child: 20.997ms, % > non-child: 22.90%) > - PeakMemoryUsage: 699.55 KB (716335) > - RowsReturned: 29.70K (29696) > - RowsReturnedRate: 323.80 K/sec > NESTED_LOOP_JOIN_NODE (id=9):(Total: 53.723ms, non-child: > 40.263ms, % non-child: 74.95%) > - BuildRows: 0 (0) > - BuildTime: 0.000ns > - PeakMemoryUsage: 24.00 KB (24576) > - ProbeRows: 30.34K (30340) > - ProbeTime: 0.000ns > - RowsReturned: 15.43M (15426890) > - RowsReturnedRate: 287.16 M/sec > Nested Loop Join Builder: > - PeakMemoryUsage: 8.00 KB (8192) > SINGULAR_ROW_SRC_NODE (id=7): > - PeakMemoryUsage: 0 > - RowsReturned: 0 (0) > - RowsReturnedRate: 0 > UNNEST_NODE (id=8):(Total: 13.460ms, non-child: 13.460ms, % > non-child: 100.00%) > - AvgCollectionSize: 1.00 > - MaxCollectionSize: 1 (1) > - MinCollectionSize: 1 (1) > - NumCollections: 30.34K (30340) > - PeakMemoryUsage: 0 > - RowsReturned: 1 (1) > - RowsReturnedRate: 74.00 /sec > HDFS_SCAN_NODE (id=5):(Total: 16.990ms, non-child: 16.990ms, % > non-child: 100.00%) > Hdfs split stats (<volume id>:<# splits>/<split lengths>): > 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 out of 5 > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% > File Formats: PARQUET/SNAPPY:30 > BytesRead(500.000ms): 1.56 MB, 1.56 MB, 1.56 MB > - FooterProcessingTime: (Avg: 12.351ms ; Min: 7.816ms ; Max: > 28.687ms ; Number of samples: 5) > - AverageHdfsReadThreadConcurrency: 0.00 > - AverageScannerThreadConcurrency: 0.00 > - BytesRead: 1.56 MB (1630546) > - BytesReadDataNodeCache: 0 > - BytesReadLocal: 1.56 MB (1630546) > - BytesReadRemoteUnexpected: 0 > - BytesReadShortCircuit: 1.56 MB (1630546) > - CachedFileHandlesHitCount: 0 (0) > - CachedFileHandlesMissCount: 35 (35) > - DecompressionTime: 1.267ms > - MaxCompressedTextFileLength: 0 > - NumColumns: 6 (6) > - NumDictFilteredRowGroups: 0 (0) > - NumDisksAccessed: 4 (4) > - NumRowGroups: 5 (5) > - NumScannerThreadsStarted: 5 (5) > - NumScannersWithNoReads: 0 (0) > - NumStatsFilteredRowGroups: 0 (0) > - PeakMemoryUsage: 5.59 MB (5859290) > - PerReadThreadRawHdfsThroughput: 920.32 MB/sec > - RemoteScanRanges: 0 (0) > - RowBatchQueueGetWaitTime: 15.335ms > - RowBatchQueuePutWaitTime: 0.000ns > - RowsRead: 64.69K (64692) > - RowsReturned: 30.34K (30340) > - RowsReturnedRate: 1.79 M/sec > - ScanRangesComplete: 5 (5) > - ScannerThreadsInvoluntaryContextSwitches: 1 (1) > - ScannerThreadsTotalWallClockTime: 132.548ms > - MaterializeTupleTime(*): 10.585ms > - ScannerThreadsSysTime: 999.000us > - ScannerThreadsUserTime: 18.995ms > - ScannerThreadsVoluntaryContextSwitches: 79 (79) > - TotalRawHdfsReadTime(*): 1.689ms > - TotalReadThroughput: 1.04 MB/sec > Instance 744de1b6228736fa:b54bfaa700000002 > (host=slpr-dhc003.lpdomain.com:22000):(Total: 1s452ms, non-child: > 1s255ms, % non-child: 86.41%) > Hdfs split stats (<volume id>:<# splits>/<split lengths>): > 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB > MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 3.94 MB > ThreadUsage(500.000ms): 1, 1, 1 > - AverageThreadTokens: 1.00 > - BloomFilterBytes: 0 > - PeakMemoryUsage: 6.17 MB (6473729) > - PeakReservation: 0 > - PeakUsedReservation: 0 > - PerHostPeakMemUsage: 107.32 MB (112537640) > - RowsProduced: 29.97K (29969) > - TotalNetworkReceiveTime: 0.000ns > - TotalNetworkSendTime: 1s260ms > - TotalStorageWaitTime: 130.815ms > - TotalThreadsInvoluntaryContextSwitches: 15 (15) > - TotalThreadsTotalWallClockTime: 1s586ms > - TotalThreadsSysTime: 11.995ms > - TotalThreadsUserTime: 146.975ms > - TotalThreadsVoluntaryContextSwitches: 197 (197) > Fragment Instance Lifecycle Timings: > - ExecTime: 1s354ms > - ExecTreeExecTime: 67.791ms > - OpenTime: 52.012ms > - ExecTreeOpenTime: 101.232us > - PrepareTime: 45.946ms > - ExecTreePrepareTime: 309.644us > DataStreamSender (dst_id=11):(Total: 38.995ms, non-child: > 38.995ms, % non-child: 100.00%) > - BytesSent: 35.95 MB (37697260) > - NetworkThroughput(*): 2.14 GB/sec > - OverallThroughput: 921.92 MB/sec > - PeakMemoryUsage: 59.38 KB (60800) > - RowsReturned: 29.97K (29969) > - SerializeBatchTime: 18.055ms > - TransmitDataRPCTime: 16.428ms > - UncompressedRowBatchSize: 70.83 MB (74269800) > CodeGen:(Total: 90.385ms, non-child: 90.385ms, % non-child: > 100.00%) > - CodegenTime: 2.224ms > - CompileTime: 15.517ms > - LoadTime: 0.000ns > - ModuleBitcodeSize: 1.96 MB (2050180) > - NumFunctions: 16 (16) > - NumInstructions: 250 (250) > - OptimizationTime: 35.099ms > - PeakMemoryUsage: 125.00 KB (128000) > - PrepareTime: 38.847ms > SUBPLAN_NODE (id=6):(Total: 68.126ms, non-child: 12.625ms, % > non-child: 18.53%) > - PeakMemoryUsage: 636.17 KB (651439) > - RowsReturned: 29.70K (29696) > - RowsReturnedRate: 435.89 K/sec > NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.347ms, non-child: > 27.325ms, % non-child: 77.31%) > - BuildRows: 0 (0) > - BuildTime: 0.000ns > - PeakMemoryUsage: 24.00 KB (24576) > - ProbeRows: 29.97K (29969) > - ProbeTime: 0.000ns > - RowsReturned: 15.26M (15256601) > - RowsReturnedRate: 431.62 M/sec > Nested Loop Join Builder: > - PeakMemoryUsage: 8.00 KB (8192) > SINGULAR_ROW_SRC_NODE (id=7): > - PeakMemoryUsage: 0 > - RowsReturned: 0 (0) > - RowsReturnedRate: 0 > UNNEST_NODE (id=8):(Total: 8.021ms, non-child: 8.021ms, % > non-child: 100.00%) > - AvgCollectionSize: 1.00 > - MaxCollectionSize: 1 (1) > - MinCollectionSize: 1 (1) > - NumCollections: 29.97K (29969) > - PeakMemoryUsage: 0 > - RowsReturned: 1 (1) > - RowsReturnedRate: 124.00 /sec > HDFS_SCAN_NODE (id=5):(Total: 20.153ms, non-child: 20.153ms, % > non-child: 100.00%) > Hdfs split stats (<volume id>:<# splits>/<split lengths>): > 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7 > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% > File Formats: PARQUET/SNAPPY:42 > BytesRead(500.000ms): 1.75 MB, 1.75 MB, 1.75 MB > - FooterProcessingTime: (Avg: 8.683ms ; Min: 3.937ms ; Max: > 10.464ms ; Number of samples: 7) > - AverageHdfsReadThreadConcurrency: 0.00 > - AverageScannerThreadConcurrency: 0.00 > - BytesRead: 1.75 MB (1833560) > - BytesReadDataNodeCache: 0 > - BytesReadLocal: 1.75 MB (1833560) > - BytesReadRemoteUnexpected: 0 > - BytesReadShortCircuit: 1.75 MB (1833560) > - CachedFileHandlesHitCount: 0 (0) > - CachedFileHandlesMissCount: 49 (49) > - DecompressionTime: 1.758ms > - MaxCompressedTextFileLength: 0 > - NumColumns: 6 (6) > - NumDictFilteredRowGroups: 0 (0) > - NumDisksAccessed: 5 (5) > - NumRowGroups: 7 (7) > - NumScannerThreadsStarted: 7 (7) > - NumScannersWithNoReads: 0 (0) > - NumStatsFilteredRowGroups: 0 (0) > - PeakMemoryUsage: 5.82 MB (6098991) > - PerReadThreadRawHdfsThroughput: 621.09 MB/sec > - RemoteScanRanges: 0 (0) > - RowBatchQueueGetWaitTime: 17.938ms > - RowBatchQueuePutWaitTime: 0.000ns > - RowsRead: 65.17K (65166) > - RowsReturned: 29.97K (29969) > - RowsReturnedRate: 1.49 M/sec > - ScanRangesComplete: 7 (7) > - ScannerThreadsInvoluntaryContextSwitches: 1 (1) > - ScannerThreadsTotalWallClockTime: 179.698ms > - MaterializeTupleTime(*): 13.922ms > - ScannerThreadsSysTime: 2.997ms > - ScannerThreadsUserTime: 29.993ms > - ScannerThreadsVoluntaryContextSwitches: 130 (130) > - TotalRawHdfsReadTime(*): 2.815ms > - TotalReadThroughput: 1.17 MB/sec > Instance 744de1b6228736fa:b54bfaa700000008 > (host=slpr-dhc009.lpdomain.com:22000):(Total: 1s450ms, non-child: > 1s305ms, % non-child: 90.01%) > Hdfs split stats (<volume id>:<# splits>/<split lengths>): > 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB > MemoryUsage(500.000ms): 5.97 MB, 5.88 MB, 2.24 MB > ThreadUsage(500.000ms): 1, 1, 1 > - AverageThreadTokens: 1.00 > - BloomFilterBytes: 0 > - PeakMemoryUsage: 6.69 MB (7017157) > - PeakReservation: 0 > - PeakUsedReservation: 0 > - PerHostPeakMemUsage: 106.39 MB (111556990) > - RowsProduced: 42.68K (42681) > - TotalNetworkReceiveTime: 0.000ns > - TotalNetworkSendTime: 1s315ms > - TotalStorageWaitTime: 201.411ms > - TotalThreadsInvoluntaryContextSwitches: 43 (43) > - TotalThreadsTotalWallClockTime: 1s658ms > - TotalThreadsSysTime: 6.997ms > - TotalThreadsUserTime: 108.980ms > - TotalThreadsVoluntaryContextSwitches: 179 (179) > Fragment Instance Lifecycle Timings: > - ExecTime: 1s411ms > - ExecTreeExecTime: 68.017ms > - OpenTime: 19.935ms > - ExecTreeOpenTime: 48.883us > - PrepareTime: 19.160ms > - ExecTreePrepareTime: 134.576us > DataStreamSender (dst_id=11):(Total: 40.671ms, non-child: > 40.671ms, % non-child: 100.00%) > - BytesSent: 51.36 MB (53856680) > - NetworkThroughput(*): 2.44 GB/sec > - OverallThroughput: 1.23 GB/sec > - PeakMemoryUsage: 59.38 KB (60800) > - RowsReturned: 42.68K (42681) > - SerializeBatchTime: 18.299ms > - TransmitDataRPCTime: 20.591ms > - UncompressedRowBatchSize: 100.88 MB (105782860) > CodeGen:(Total: 36.171ms, non-child: 36.171ms, % non-child: > 100.00%) > - CodegenTime: 828.865us > - CompileTime: 6.170ms > - LoadTime: 0.000ns > - ModuleBitcodeSize: 1.96 MB (2050180) > - NumFunctions: 16 (16) > - NumInstructions: 250 (250) > - OptimizationTime: 12.988ms > - PeakMemoryUsage: 125.00 KB (128000) > - PrepareTime: 16.608ms > SUBPLAN_NODE (id=6):(Total: 68.046ms, non-child: 12.434ms, % > non-child: 18.27%) > - PeakMemoryUsage: 689.95 KB (706507) > - RowsReturned: 41.98K (41984) > - RowsReturnedRate: 616.99 K/sec > NESTED_LOOP_JOIN_NODE (id=9):(Total: 39.837ms, non-child: > 29.422ms, % non-child: 73.86%) > - BuildRows: 0 (0) > - BuildTime: 0.000ns > - PeakMemoryUsage: 24.00 KB (24576) > - ProbeRows: 42.68K (42681) > - ProbeTime: 0.000ns > - RowsReturned: 21.76M (21760053) > - RowsReturnedRate: 546.22 M/sec > Nested Loop Join Builder: > - PeakMemoryUsage: 8.00 KB (8192) > SINGULAR_ROW_SRC_NODE (id=7): > - PeakMemoryUsage: 0 > - RowsReturned: 0 (0) > - RowsReturnedRate: 0 > UNNEST_NODE (id=8):(Total: 10.415ms, non-child: 10.415ms, % > non-child: 100.00%) > - AvgCollectionSize: 1.00 > - MaxCollectionSize: 1 (1) > - MinCollectionSize: 1 (1) > - NumCollections: 42.68K (42681) > - PeakMemoryUsage: 0 > - RowsReturned: 1 (1) > - RowsReturnedRate: 96.00 /sec > HDFS_SCAN_NODE (id=5):(Total: 15.774ms, non-child: 15.774ms, % > non-child: 100.00%) > Hdfs split stats (<volume id>:<# splits>/<split lengths>): > 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of 6 > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% > File Formats: PARQUET/SNAPPY:36 > BytesRead(500.000ms): 2.08 MB, 2.08 MB, 2.08 MB > - FooterProcessingTime: (Avg: 22.052ms ; Min: 7.783ms ; Max: > 42.732ms ; Number of samples: 6) > - AverageHdfsReadThreadConcurrency: 0.00 > - AverageScannerThreadConcurrency: 0.00 > - BytesRead: 2.08 MB (2186159) > - BytesReadDataNodeCache: 0 > - BytesReadLocal: 2.08 MB (2186159) > - BytesReadRemoteUnexpected: 0 > - BytesReadShortCircuit: 2.08 MB (2186159) > - CachedFileHandlesHitCount: 0 (0) > - CachedFileHandlesMissCount: 42 (42) > - DecompressionTime: 1.480ms > - MaxCompressedTextFileLength: 0 > - NumColumns: 6 (6) > - NumDictFilteredRowGroups: 0 (0) > - NumDisksAccessed: 4 (4) > - NumRowGroups: 6 (6) > - NumScannerThreadsStarted: 6 (6) > - NumScannersWithNoReads: 0 (0) > - NumStatsFilteredRowGroups: 0 (0) > - PeakMemoryUsage: 6.42 MB (6730467) > - PerReadThreadRawHdfsThroughput: 77.02 MB/sec > - RemoteScanRanges: 0 (0) > - RowBatchQueueGetWaitTime: 13.979ms > - RowBatchQueuePutWaitTime: 0.000ns > - RowsRead: 89.91K (89910) > - RowsReturned: 42.68K (42681) > - RowsReturnedRate: 2.71 M/sec > - ScanRangesComplete: 6 (6) > - ScannerThreadsInvoluntaryContextSwitches: 3 (3) > - ScannerThreadsTotalWallClockTime: 227.825ms > - MaterializeTupleTime(*): 9.931ms > - ScannerThreadsSysTime: 1.998ms > - ScannerThreadsUserTime: 18.994ms > - ScannerThreadsVoluntaryContextSwitches: 85 (85) > - TotalRawHdfsReadTime(*): 27.069ms > - TotalReadThroughput: 1.39 MB/sec > Instance 744de1b6228736fa:b54bfaa700000005 > (host=slpr-dhc001.lpdomain.com:22000):(Total: 1s447ms, non-child: > 1s301ms, % non-child: 89.88%) > Hdfs split stats (<volume id>:<# splits>/<split lengths>): > 6:1/1.20 MB 1:2/2.50 MB 3:1/1.32 MB > MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 1.25 MB > ThreadUsage(500.000ms): 1, 1, 1 > ... > > [Message clipped]
