Thanks jezy for your detailed response. Yes I read the documentation.
Let simplify my question: I have pools set up with memory limit per node and concurrency. If I’m looking on the historical impala queries that I have and the metrics I have per query, on which metrics I can understand that impala will kill the query, for example if I have a query with estimate of 2GB and the used per node is 200mb, what is the default memory values that i need to setup so the query will not fail. The second one is the distribution between pools, if one query is running which metrics o have to look into to know if I submit a query it fail or not. On Fri, 23 Feb 2018 at 10:48 Jeszy <[email protected]> wrote: > Hey Fawze, > > Answers inline. > > On 23 February 2018 at 01:23, Fawze Abujaber <[email protected]> wrote: > > There is no option in the admission control to setup memory limit per > query, > > the memory limit is per pool and there is a default memory per node for > > query. > > per node for query memory limit multiplied by number of nodes gives > you a per query memory limit. I agree its confusing that the > configurations mix and match between per-node and aggregated values. > In this case there's a good reason though, as a single node running > out of memory will lead to query failure, meaning that in addition to > total memory used, distribution of memory usage between hosts also > matters. > > > I have hundreds of impala queries and more add hoc queries, making a pool > > for each query is not a visible solution. > > > > still waiting to understand how the estimate per node related to the > default > > memory per node I set up per pool, is it used in the decision of queuing > and > > killing the query? and if this is true how it was not kill a query that > was > > estimated it needs 8.2TB memory per node. > > > > Understanding on which parameters impala decides to kill a query can help > > understand to define and divide the memory between the pools. > > If you set mem_limit at any level (service level, pool level, or query > level), it will be used for admission control purposes instead of > estimates. So a 8.2TB estimate would not be a problem, if impala can > reserve mem_limit amount on each host, it will start running the > query. > > > Passing memory limit per query manually is also not visible and such > > settings not needs admission control. > > > > I have support pool that runs ad hoc query and I can not ask them to use > > memory limit per query, and I have analytics pool which is fully business > > and I can rely on admission control if it extremely in accurate. > > It's a bit tricky to use memory-based admission control with > non-trivial ad hoc queries. For simple ad-hoc queries, you can try to > come up with a 'good enough' mem_limit, or omit mem_limit and trust > impala's estimations. You can check the estimated vs. actual values > for a representative set of ad hoc queries to see what would work in > your case. I've found that people tend to go with a large enough > mem_limit for the ad hoc pool. > > > Can someone explain me exactly which recommended setting to use per pool > and > > which of them rely on impala memory estimates? > > The documentation of admission control > (https://impala.apache.org/docs/build/html/topics/impala_admission.html) > gives you a good view on how stuff works, but you will have to figure > out how to use these features for your specific use case. That said, > when using memory based admission control, it is best practice to > always use a mem_limit due to potential inaccuracy of estimates as > well as potential variance of estimates between Impala releases. Keep > in mind that you can opt to set a default mem_limit for one pool and > leave it unset for another. > > > So my conclusion right now to avoid using any settings rely on the > estimates > > and to ignore the estimates when I want to evaluate query. > > Sounds good. > > > @mostafa, since my issue with all the query, I think the profile will not > > help me to solve such huge issue. > > > > I’m planning to move a way from Vertica and rely on impala as a sql > engine > > and now fully confused how I can do this if I can’t use the admission > > control. > > > > Last think, is it recommend to use the impala admission control? > > Yes. Admission control can take a while to understand, but if done > right, it works. > > HTH > > > On Fri, 23 Feb 2018 at 1:56 Alexander Behm <[email protected]> > wrote: > >> > >> The planner memory estimates are conservative and sometimes extremely > >> inaccurate. In their current form, they are rarely appropriate for > admission > >> decisions. > >> > >> The recommended practice for memory-based admission control it to set a > >> mem_limit for every query. You can make this easier by setting up > different > >> pools with different mem_limits, e.g. a small/medium/big queries pool or > >> similar. > >> > >> On Thu, Feb 22, 2018 at 3:00 PM, Mostafa Mokhtar <[email protected] > > > >> wrote: > >>> > >>> It is recommended to set a per query memory limit as part of admission > >>> and not rely on estimates as they are sometimes inaccurate. > >>> Can you please include the full query profile? > >>> > >>> > >>> On Thu, Feb 22, 2018 at 12:13 PM, Fawze Abujaber <[email protected]> > >>> wrote: > >>>> > >>>> Hi Mostafa, > >>>> > >>>> It's not a specific query, almost all the query has such differene > >>>> between the 2 values. > >>>> > >>>> I can see even queries showing the estimate per node is 8.2 Tib > >>>> > >>>> User: psanalytics > >>>> > >>>> Database: default > >>>> > >>>> Query Type: QUERY > >>>> Coordinator: slpr-dhc014.lpdomain.com > >>>> > >>>> Duration: 6.48s > >>>> > >>>> Rows Produced: 708 > >>>> Estimated per Node Peak Memory: 8.2 TiB > >>>> > >>>> Per Node Peak Memory Usage: 1.1 GiB > >>>> > >>>> Pool: root.impanalytics > >>>> Threads: CPU Time: 20.1m > >>>> > >>>> > >>>> > >>>> How you can explain this behavior, and for sure i don't have 8.2 Tib > >>>> memory per node to give neither you. > >>>> > >>>> Can you please explain me how i should treat Estimated per Node Peak > >>>> Memory and if it used by impala for the resource pool and admission > control > >>>> and what is the relation of this value to the default memory per node > that i > >>>> setup for each resource pool? > >>>> > >>>> Below is part of one of the queries profile which the estimate per > node > >>>> was ~ @GB and the used was 200MB per node. > >>>> > >>>> > >>>> Instance 744de1b6228736fa:b54bfaa70000000f > >>>> (host=slpr-dhc004.lpdomain.com:22000):(Total: 1s455ms, non-child: > 1s292ms, % > >>>> non-child: 88.82%) > >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>): > >>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB > 9:1/191.64 > >>>> KB 5:2/289.57 KB 3:2/1012.83 KB > >>>> MemoryUsage(500.000ms): 6.09 MB, 6.09 MB, 3.00 MB > >>>> ThreadUsage(500.000ms): 1, 1, 1 > >>>> - AverageThreadTokens: 1.00 > >>>> - BloomFilterBytes: 0 > >>>> - PeakMemoryUsage: 7.17 MB (7521751) > >>>> - PeakReservation: 0 > >>>> - PeakUsedReservation: 0 > >>>> - PerHostPeakMemUsage: 106.53 MB (111709581) > >>>> - RowsProduced: 32.83K (32826) > >>>> - TotalNetworkReceiveTime: 0.000ns > >>>> - TotalNetworkSendTime: 1s297ms > >>>> - TotalStorageWaitTime: 234.356ms > >>>> - TotalThreadsInvoluntaryContextSwitches: 66 (66) > >>>> - TotalThreadsTotalWallClockTime: 1s715ms > >>>> - TotalThreadsSysTime: 5.998ms > >>>> - TotalThreadsUserTime: 124.975ms > >>>> - TotalThreadsVoluntaryContextSwitches: 303 (303) > >>>> Fragment Instance Lifecycle Timings: > >>>> - ExecTime: 1s394ms > >>>> - ExecTreeExecTime: 67.115ms > >>>> - OpenTime: 32.795ms > >>>> - ExecTreeOpenTime: 73.243us > >>>> - PrepareTime: 27.602ms > >>>> - ExecTreePrepareTime: 243.141us > >>>> DataStreamSender (dst_id=11):(Total: 38.747ms, non-child: > >>>> 38.747ms, % non-child: 100.00%) > >>>> - BytesSent: 39.71 MB (41643000) > >>>> - NetworkThroughput(*): 1.97 GB/sec > >>>> - OverallThroughput: 1.00 GB/sec > >>>> - PeakMemoryUsage: 59.38 KB (60800) > >>>> - RowsReturned: 32.83K (32826) > >>>> - SerializeBatchTime: 16.860ms > >>>> - TransmitDataRPCTime: 19.698ms > >>>> - UncompressedRowBatchSize: 77.58 MB (81350840) > >>>> CodeGen:(Total: 56.573ms, non-child: 56.573ms, % non-child: > >>>> 100.00%) > >>>> - CodegenTime: 1.299ms > >>>> - CompileTime: 10.672ms > >>>> - LoadTime: 0.000ns > >>>> - ModuleBitcodeSize: 1.96 MB (2050180) > >>>> - NumFunctions: 16 (16) > >>>> - NumInstructions: 250 (250) > >>>> - OptimizationTime: 21.023ms > >>>> - PeakMemoryUsage: 125.00 KB (128000) > >>>> - PrepareTime: 24.116ms > >>>> SUBPLAN_NODE (id=6):(Total: 67.311ms, non-child: 12.013ms, % > >>>> non-child: 17.85%) > >>>> - PeakMemoryUsage: 627.94 KB (643015) > >>>> - RowsReturned: 32.77K (32768) > >>>> - RowsReturnedRate: 486.81 K/sec > >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.999ms, non-child: > >>>> 25.197ms, % non-child: 74.11%) > >>>> - BuildRows: 0 (0) > >>>> - BuildTime: 0.000ns > >>>> - PeakMemoryUsage: 24.00 KB (24576) > >>>> - ProbeRows: 32.83K (32826) > >>>> - ProbeTime: 0.000ns > >>>> - RowsReturned: 16.80M (16795311) > >>>> - RowsReturnedRate: 493.99 M/sec > >>>> Nested Loop Join Builder: > >>>> - PeakMemoryUsage: 8.00 KB (8192) > >>>> SINGULAR_ROW_SRC_NODE (id=7): > >>>> - PeakMemoryUsage: 0 > >>>> - RowsReturned: 0 (0) > >>>> - RowsReturnedRate: 0 > >>>> UNNEST_NODE (id=8):(Total: 8.801ms, non-child: 8.801ms, % > >>>> non-child: 100.00%) > >>>> - AvgCollectionSize: 1.00 > >>>> - MaxCollectionSize: 1 (1) > >>>> - MinCollectionSize: 1 (1) > >>>> - NumCollections: 32.83K (32826) > >>>> - PeakMemoryUsage: 0 > >>>> - RowsReturned: 1 (1) > >>>> - RowsReturnedRate: 113.00 /sec > >>>> HDFS_SCAN_NODE (id=5):(Total: 21.299ms, non-child: 21.299ms, % > >>>> non-child: 100.00%) > >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>): > >>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB > 9:1/191.64 > >>>> KB 5:2/289.57 KB 3:2/1012.83 KB > >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 11 out > >>>> of 11 > >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% > >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% > 16:0% > >>>> File Formats: PARQUET/SNAPPY:66 > >>>> BytesRead(500.000ms): 2.26 MB, 2.26 MB, 2.26 MB > >>>> - FooterProcessingTime: (Avg: 12.082ms ; Min: 9.079ms ; > Max: > >>>> 29.275ms ; Number of samples: 11) > >>>> - AverageHdfsReadThreadConcurrency: 0.00 > >>>> - AverageScannerThreadConcurrency: 0.00 > >>>> - BytesRead: 2.26 MB (2373738) > >>>> - BytesReadDataNodeCache: 0 > >>>> - BytesReadLocal: 2.26 MB (2373738) > >>>> - BytesReadRemoteUnexpected: 0 > >>>> - BytesReadShortCircuit: 2.26 MB (2373738) > >>>> - CachedFileHandlesHitCount: 0 (0) > >>>> - CachedFileHandlesMissCount: 77 (77) > >>>> - DecompressionTime: 1.543ms > >>>> - MaxCompressedTextFileLength: 0 > >>>> - NumColumns: 6 (6) > >>>> - NumDictFilteredRowGroups: 0 (0) > >>>> - NumDisksAccessed: 8 (8) > >>>> - NumRowGroups: 11 (11) > >>>> - NumScannerThreadsStarted: 11 (11) > >>>> - NumScannersWithNoReads: 0 (0) > >>>> - NumStatsFilteredRowGroups: 0 (0) > >>>> - PeakMemoryUsage: 6.51 MB (6821203) > >>>> - PerReadThreadRawHdfsThroughput: 764.51 MB/sec > >>>> - RemoteScanRanges: 0 (0) > >>>> - RowBatchQueueGetWaitTime: 18.825ms > >>>> - RowBatchQueuePutWaitTime: 0.000ns > >>>> - RowsRead: 73.46K (73464) > >>>> - RowsReturned: 32.83K (32826) > >>>> - RowsReturnedRate: 1.54 M/sec > >>>> - ScanRangesComplete: 11 (11) > >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1) > >>>> - ScannerThreadsTotalWallClockTime: 287.702ms > >>>> - MaterializeTupleTime(*): 11.921ms > >>>> - ScannerThreadsSysTime: 999.000us > >>>> - ScannerThreadsUserTime: 26.990ms > >>>> - ScannerThreadsVoluntaryContextSwitches: 232 (232) > >>>> - TotalRawHdfsReadTime(*): 2.961ms > >>>> - TotalReadThroughput: 1.51 MB/sec > >>>> Instance 744de1b6228736fa:b54bfaa70000000d > >>>> (host=slpr-dhc002.lpdomain.com:22000):(Total: 1s453ms, non-child: > 1s254ms, % > >>>> non-child: 86.28%) > >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>): > >>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB > >>>> MemoryUsage(500.000ms): 5.32 MB, 5.32 MB, 1.27 MB > >>>> ThreadUsage(500.000ms): 1, 1, 1 > >>>> - AverageThreadTokens: 1.00 > >>>> - BloomFilterBytes: 0 > >>>> - PeakMemoryUsage: 5.86 MB (6145964) > >>>> - PeakReservation: 0 > >>>> - PeakUsedReservation: 0 > >>>> - PerHostPeakMemUsage: 106.66 MB (111842766) > >>>> - RowsProduced: 30.34K (30340) > >>>> - TotalNetworkReceiveTime: 0.000ns > >>>> - TotalNetworkSendTime: 1s256ms > >>>> - TotalStorageWaitTime: 106.530ms > >>>> - TotalThreadsInvoluntaryContextSwitches: 177 (177) > >>>> - TotalThreadsTotalWallClockTime: 1s557ms > >>>> - TotalThreadsSysTime: 6.998ms > >>>> - TotalThreadsUserTime: 143.976ms > >>>> - TotalThreadsVoluntaryContextSwitches: 136 (136) > >>>> Fragment Instance Lifecycle Timings: > >>>> - ExecTime: 1s390ms > >>>> - ExecTreeExecTime: 91.500ms > >>>> - OpenTime: 34.085ms > >>>> - ExecTreeOpenTime: 81.436us > >>>> - PrepareTime: 28.524ms > >>>> - ExecTreePrepareTime: 217.949us > >>>> DataStreamSender (dst_id=11):(Total: 48.674ms, non-child: > >>>> 48.674ms, % non-child: 100.00%) > >>>> - BytesSent: 36.61 MB (38388760) > >>>> - NetworkThroughput(*): 1.51 GB/sec > >>>> - OverallThroughput: 752.15 MB/sec > >>>> - PeakMemoryUsage: 59.38 KB (60800) > >>>> - RowsReturned: 30.34K (30340) > >>>> - SerializeBatchTime: 23.132ms > >>>> - TransmitDataRPCTime: 23.609ms > >>>> - UncompressedRowBatchSize: 71.70 MB (75187520) > >>>> CodeGen:(Total: 59.024ms, non-child: 59.024ms, % non-child: > >>>> 100.00%) > >>>> - CodegenTime: 1.382ms > >>>> - CompileTime: 10.435ms > >>>> - LoadTime: 0.000ns > >>>> - ModuleBitcodeSize: 1.96 MB (2050180) > >>>> - NumFunctions: 16 (16) > >>>> - NumInstructions: 250 (250) > >>>> - OptimizationTime: 22.325ms > >>>> - PeakMemoryUsage: 125.00 KB (128000) > >>>> - PrepareTime: 25.459ms > >>>> SUBPLAN_NODE (id=6):(Total: 91.711ms, non-child: 20.997ms, % > >>>> non-child: 22.90%) > >>>> - PeakMemoryUsage: 699.55 KB (716335) > >>>> - RowsReturned: 29.70K (29696) > >>>> - RowsReturnedRate: 323.80 K/sec > >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 53.723ms, non-child: > >>>> 40.263ms, % non-child: 74.95%) > >>>> - BuildRows: 0 (0) > >>>> - BuildTime: 0.000ns > >>>> - PeakMemoryUsage: 24.00 KB (24576) > >>>> - ProbeRows: 30.34K (30340) > >>>> - ProbeTime: 0.000ns > >>>> - RowsReturned: 15.43M (15426890) > >>>> - RowsReturnedRate: 287.16 M/sec > >>>> Nested Loop Join Builder: > >>>> - PeakMemoryUsage: 8.00 KB (8192) > >>>> SINGULAR_ROW_SRC_NODE (id=7): > >>>> - PeakMemoryUsage: 0 > >>>> - RowsReturned: 0 (0) > >>>> - RowsReturnedRate: 0 > >>>> UNNEST_NODE (id=8):(Total: 13.460ms, non-child: 13.460ms, % > >>>> non-child: 100.00%) > >>>> - AvgCollectionSize: 1.00 > >>>> - MaxCollectionSize: 1 (1) > >>>> - MinCollectionSize: 1 (1) > >>>> - NumCollections: 30.34K (30340) > >>>> - PeakMemoryUsage: 0 > >>>> - RowsReturned: 1 (1) > >>>> - RowsReturnedRate: 74.00 /sec > >>>> HDFS_SCAN_NODE (id=5):(Total: 16.990ms, non-child: 16.990ms, % > >>>> non-child: 100.00%) > >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>): > >>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB > >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 out > of > >>>> 5 > >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% > >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% > 16:0% > >>>> File Formats: PARQUET/SNAPPY:30 > >>>> BytesRead(500.000ms): 1.56 MB, 1.56 MB, 1.56 MB > >>>> - FooterProcessingTime: (Avg: 12.351ms ; Min: 7.816ms ; > Max: > >>>> 28.687ms ; Number of samples: 5) > >>>> - AverageHdfsReadThreadConcurrency: 0.00 > >>>> - AverageScannerThreadConcurrency: 0.00 > >>>> - BytesRead: 1.56 MB (1630546) > >>>> - BytesReadDataNodeCache: 0 > >>>> - BytesReadLocal: 1.56 MB (1630546) > >>>> - BytesReadRemoteUnexpected: 0 > >>>> - BytesReadShortCircuit: 1.56 MB (1630546) > >>>> - CachedFileHandlesHitCount: 0 (0) > >>>> - CachedFileHandlesMissCount: 35 (35) > >>>> - DecompressionTime: 1.267ms > >>>> - MaxCompressedTextFileLength: 0 > >>>> - NumColumns: 6 (6) > >>>> - NumDictFilteredRowGroups: 0 (0) > >>>> - NumDisksAccessed: 4 (4) > >>>> - NumRowGroups: 5 (5) > >>>> - NumScannerThreadsStarted: 5 (5) > >>>> - NumScannersWithNoReads: 0 (0) > >>>> - NumStatsFilteredRowGroups: 0 (0) > >>>> - PeakMemoryUsage: 5.59 MB (5859290) > >>>> - PerReadThreadRawHdfsThroughput: 920.32 MB/sec > >>>> - RemoteScanRanges: 0 (0) > >>>> - RowBatchQueueGetWaitTime: 15.335ms > >>>> - RowBatchQueuePutWaitTime: 0.000ns > >>>> - RowsRead: 64.69K (64692) > >>>> - RowsReturned: 30.34K (30340) > >>>> - RowsReturnedRate: 1.79 M/sec > >>>> - ScanRangesComplete: 5 (5) > >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1) > >>>> - ScannerThreadsTotalWallClockTime: 132.548ms > >>>> - MaterializeTupleTime(*): 10.585ms > >>>> - ScannerThreadsSysTime: 999.000us > >>>> - ScannerThreadsUserTime: 18.995ms > >>>> - ScannerThreadsVoluntaryContextSwitches: 79 (79) > >>>> - TotalRawHdfsReadTime(*): 1.689ms > >>>> - TotalReadThroughput: 1.04 MB/sec > >>>> Instance 744de1b6228736fa:b54bfaa700000002 > >>>> (host=slpr-dhc003.lpdomain.com:22000):(Total: 1s452ms, non-child: > 1s255ms, % > >>>> non-child: 86.41%) > >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>): > >>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB > >>>> MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 3.94 MB > >>>> ThreadUsage(500.000ms): 1, 1, 1 > >>>> - AverageThreadTokens: 1.00 > >>>> - BloomFilterBytes: 0 > >>>> - PeakMemoryUsage: 6.17 MB (6473729) > >>>> - PeakReservation: 0 > >>>> - PeakUsedReservation: 0 > >>>> - PerHostPeakMemUsage: 107.32 MB (112537640) > >>>> - RowsProduced: 29.97K (29969) > >>>> - TotalNetworkReceiveTime: 0.000ns > >>>> - TotalNetworkSendTime: 1s260ms > >>>> - TotalStorageWaitTime: 130.815ms > >>>> - TotalThreadsInvoluntaryContextSwitches: 15 (15) > >>>> - TotalThreadsTotalWallClockTime: 1s586ms > >>>> - TotalThreadsSysTime: 11.995ms > >>>> - TotalThreadsUserTime: 146.975ms > >>>> - TotalThreadsVoluntaryContextSwitches: 197 (197) > >>>> Fragment Instance Lifecycle Timings: > >>>> - ExecTime: 1s354ms > >>>> - ExecTreeExecTime: 67.791ms > >>>> - OpenTime: 52.012ms > >>>> - ExecTreeOpenTime: 101.232us > >>>> - PrepareTime: 45.946ms > >>>> - ExecTreePrepareTime: 309.644us > >>>> DataStreamSender (dst_id=11):(Total: 38.995ms, non-child: > >>>> 38.995ms, % non-child: 100.00%) > >>>> - BytesSent: 35.95 MB (37697260) > >>>> - NetworkThroughput(*): 2.14 GB/sec > >>>> - OverallThroughput: 921.92 MB/sec > >>>> - PeakMemoryUsage: 59.38 KB (60800) > >>>> - RowsReturned: 29.97K (29969) > >>>> - SerializeBatchTime: 18.055ms > >>>> - TransmitDataRPCTime: 16.428ms > >>>> - UncompressedRowBatchSize: 70.83 MB (74269800) > >>>> CodeGen:(Total: 90.385ms, non-child: 90.385ms, % non-child: > >>>> 100.00%) > >>>> - CodegenTime: 2.224ms > >>>> - CompileTime: 15.517ms > >>>> - LoadTime: 0.000ns > >>>> - ModuleBitcodeSize: 1.96 MB (2050180) > >>>> - NumFunctions: 16 (16) > >>>> - NumInstructions: 250 (250) > >>>> - OptimizationTime: 35.099ms > >>>> - PeakMemoryUsage: 125.00 KB (128000) > >>>> - PrepareTime: 38.847ms > >>>> SUBPLAN_NODE (id=6):(Total: 68.126ms, non-child: 12.625ms, % > >>>> non-child: 18.53%) > >>>> - PeakMemoryUsage: 636.17 KB (651439) > >>>> - RowsReturned: 29.70K (29696) > >>>> - RowsReturnedRate: 435.89 K/sec > >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.347ms, non-child: > >>>> 27.325ms, % non-child: 77.31%) > >>>> - BuildRows: 0 (0) > >>>> - BuildTime: 0.000ns > >>>> - PeakMemoryUsage: 24.00 KB (24576) > >>>> - ProbeRows: 29.97K (29969) > >>>> - ProbeTime: 0.000ns > >>>> - RowsReturned: 15.26M (15256601) > >>>> - RowsReturnedRate: 431.62 M/sec > >>>> Nested Loop Join Builder: > >>>> - PeakMemoryUsage: 8.00 KB (8192) > >>>> SINGULAR_ROW_SRC_NODE (id=7): > >>>> - PeakMemoryUsage: 0 > >>>> - RowsReturned: 0 (0) > >>>> - RowsReturnedRate: 0 > >>>> UNNEST_NODE (id=8):(Total: 8.021ms, non-child: 8.021ms, % > >>>> non-child: 100.00%) > >>>> - AvgCollectionSize: 1.00 > >>>> - MaxCollectionSize: 1 (1) > >>>> - MinCollectionSize: 1 (1) > >>>> - NumCollections: 29.97K (29969) > >>>> - PeakMemoryUsage: 0 > >>>> - RowsReturned: 1 (1) > >>>> - RowsReturnedRate: 124.00 /sec > >>>> HDFS_SCAN_NODE (id=5):(Total: 20.153ms, non-child: 20.153ms, % > >>>> non-child: 100.00%) > >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>): > >>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB > >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out > of > >>>> 7 > >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% > >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% > 16:0% > >>>> File Formats: PARQUET/SNAPPY:42 > >>>> BytesRead(500.000ms): 1.75 MB, 1.75 MB, 1.75 MB > >>>> - FooterProcessingTime: (Avg: 8.683ms ; Min: 3.937ms ; Max: > >>>> 10.464ms ; Number of samples: 7) > >>>> - AverageHdfsReadThreadConcurrency: 0.00 > >>>> - AverageScannerThreadConcurrency: 0.00 > >>>> - BytesRead: 1.75 MB (1833560) > >>>> - BytesReadDataNodeCache: 0 > >>>> - BytesReadLocal: 1.75 MB (1833560) > >>>> - BytesReadRemoteUnexpected: 0 > >>>> - BytesReadShortCircuit: 1.75 MB (1833560) > >>>> - CachedFileHandlesHitCount: 0 (0) > >>>> - CachedFileHandlesMissCount: 49 (49) > >>>> - DecompressionTime: 1.758ms > >>>> - MaxCompressedTextFileLength: 0 > >>>> - NumColumns: 6 (6) > >>>> - NumDictFilteredRowGroups: 0 (0) > >>>> - NumDisksAccessed: 5 (5) > >>>> - NumRowGroups: 7 (7) > >>>> - NumScannerThreadsStarted: 7 (7) > >>>> - NumScannersWithNoReads: 0 (0) > >>>> - NumStatsFilteredRowGroups: 0 (0) > >>>> - PeakMemoryUsage: 5.82 MB (6098991) > >>>> - PerReadThreadRawHdfsThroughput: 621.09 MB/sec > >>>> - RemoteScanRanges: 0 (0) > >>>> - RowBatchQueueGetWaitTime: 17.938ms > >>>> - RowBatchQueuePutWaitTime: 0.000ns > >>>> - RowsRead: 65.17K (65166) > >>>> - RowsReturned: 29.97K (29969) > >>>> - RowsReturnedRate: 1.49 M/sec > >>>> - ScanRangesComplete: 7 (7) > >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1) > >>>> - ScannerThreadsTotalWallClockTime: 179.698ms > >>>> - MaterializeTupleTime(*): 13.922ms > >>>> - ScannerThreadsSysTime: 2.997ms > >>>> - ScannerThreadsUserTime: 29.993ms > >>>> - ScannerThreadsVoluntaryContextSwitches: 130 (130) > >>>> - TotalRawHdfsReadTime(*): 2.815ms > >>>> - TotalReadThroughput: 1.17 MB/sec > >>>> Instance 744de1b6228736fa:b54bfaa700000008 > >>>> (host=slpr-dhc009.lpdomain.com:22000):(Total: 1s450ms, non-child: > 1s305ms, % > >>>> non-child: 90.01%) > >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>): > >>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB > >>>> MemoryUsage(500.000ms): 5.97 MB, 5.88 MB, 2.24 MB > >>>> ThreadUsage(500.000ms): 1, 1, 1 > >>>> - AverageThreadTokens: 1.00 > >>>> - BloomFilterBytes: 0 > >>>> - PeakMemoryUsage: 6.69 MB (7017157) > >>>> - PeakReservation: 0 > >>>> - PeakUsedReservation: 0 > >>>> - PerHostPeakMemUsage: 106.39 MB (111556990) > >>>> - RowsProduced: 42.68K (42681) > >>>> - TotalNetworkReceiveTime: 0.000ns > >>>> - TotalNetworkSendTime: 1s315ms > >>>> - TotalStorageWaitTime: 201.411ms > >>>> - TotalThreadsInvoluntaryContextSwitches: 43 (43) > >>>> - TotalThreadsTotalWallClockTime: 1s658ms > >>>> - TotalThreadsSysTime: 6.997ms > >>>> - TotalThreadsUserTime: 108.980ms > >>>> - TotalThreadsVoluntaryContextSwitches: 179 (179) > >>>> Fragment Instance Lifecycle Timings: > >>>> - ExecTime: 1s411ms > >>>> - ExecTreeExecTime: 68.017ms > >>>> - OpenTime: 19.935ms > >>>> - ExecTreeOpenTime: 48.883us > >>>> - PrepareTime: 19.160ms > >>>> - ExecTreePrepareTime: 134.576us > >>>> DataStreamSender (dst_id=11):(Total: 40.671ms, non-child: > >>>> 40.671ms, % non-child: 100.00%) > >>>> - BytesSent: 51.36 MB (53856680) > >>>> - NetworkThroughput(*): 2.44 GB/sec > >>>> - OverallThroughput: 1.23 GB/sec > >>>> - PeakMemoryUsage: 59.38 KB (60800) > >>>> - RowsReturned: 42.68K (42681) > >>>> - SerializeBatchTime: 18.299ms > >>>> - TransmitDataRPCTime: 20.591ms > >>>> - UncompressedRowBatchSize: 100.88 MB (105782860) > >>>> CodeGen:(Total: 36.171ms, non-child: 36.171ms, % non-child: > >>>> 100.00%) > >>>> - CodegenTime: 828.865us > >>>> - CompileTime: 6.170ms > >>>> - LoadTime: 0.000ns > >>>> - ModuleBitcodeSize: 1.96 MB (2050180) > >>>> - NumFunctions: 16 (16) > >>>> - NumInstructions: 250 (250) > >>>> - OptimizationTime: 12.988ms > >>>> - PeakMemoryUsage: 125.00 KB (128000) > >>>> - PrepareTime: 16.608ms > >>>> SUBPLAN_NODE (id=6):(Total: 68.046ms, non-child: 12.434ms, % > >>>> non-child: 18.27%) > >>>> - PeakMemoryUsage: 689.95 KB (706507) > >>>> - RowsReturned: 41.98K (41984) > >>>> - RowsReturnedRate: 616.99 K/sec > >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 39.837ms, non-child: > >>>> 29.422ms, % non-child: 73.86%) > >>>> - BuildRows: 0 (0) > >>>> - BuildTime: 0.000ns > >>>> - PeakMemoryUsage: 24.00 KB (24576) > >>>> - ProbeRows: 42.68K (42681) > >>>> - ProbeTime: 0.000ns > >>>> - RowsReturned: 21.76M (21760053) > >>>> - RowsReturnedRate: 546.22 M/sec > >>>> Nested Loop Join Builder: > >>>> - PeakMemoryUsage: 8.00 KB (8192) > >>>> SINGULAR_ROW_SRC_NODE (id=7): > >>>> - PeakMemoryUsage: 0 > >>>> - RowsReturned: 0 (0) > >>>> - RowsReturnedRate: 0 > >>>> UNNEST_NODE (id=8):(Total: 10.415ms, non-child: 10.415ms, % > >>>> non-child: 100.00%) > >>>> - AvgCollectionSize: 1.00 > >>>> - MaxCollectionSize: 1 (1) > >>>> - MinCollectionSize: 1 (1) > >>>> - NumCollections: 42.68K (42681) > >>>> - PeakMemoryUsage: 0 > >>>> - RowsReturned: 1 (1) > >>>> - RowsReturnedRate: 96.00 /sec > >>>> HDFS_SCAN_NODE (id=5):(Total: 15.774ms, non-child: 15.774ms, % > >>>> non-child: 100.00%) > >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>): > >>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB > >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out > of > >>>> 6 > >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% > >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% > 16:0% > >>>> 17:0% > >>>> File Formats: PARQUET/SNAPPY:36 > >>>> BytesRead(500.000ms): 2.08 MB, 2.08 MB, 2.08 MB > >>>> - FooterProcessingTime: (Avg: 22.052ms ; Min: 7.783ms ; > Max: > >>>> 42.732ms ; Number of samples: 6) > >>>> - AverageHdfsReadThreadConcurrency: 0.00 > >>>> - AverageScannerThreadConcurrency: 0.00 > >>>> - BytesRead: 2.08 MB (2186159) > >>>> - BytesReadDataNodeCache: 0 > >>>> - BytesReadLocal: 2.08 MB (2186159) > >>>> - BytesReadRemoteUnexpected: 0 > >>>> - BytesReadShortCircuit: 2.08 MB (2186159) > >>>> - CachedFileHandlesHitCount: 0 (0) > >>>> - CachedFileHandlesMissCount: 42 (42) > >>>> - DecompressionTime: 1.480ms > >>>> - MaxCompressedTextFileLength: 0 > >>>> - NumColumns: 6 (6) > >>>> - NumDictFilteredRowGroups: 0 (0) > >>>> - NumDisksAccessed: 4 (4) > >>>> - NumRowGroups: 6 (6) > >>>> - NumScannerThreadsStarted: 6 (6) > >>>> - NumScannersWithNoReads: 0 (0) > >>>> - NumStatsFilteredRowGroups: 0 (0) > >>>> - PeakMemoryUsage: 6.42 MB (6730467) > >>>> - PerReadThreadRawHdfsThroughput: 77.02 MB/sec > >>>> - RemoteScanRanges: 0 (0) > >>>> - RowBatchQueueGetWaitTime: 13.979ms > >>>> - RowBatchQueuePutWaitTime: 0.000ns > >>>> - RowsRead: 89.91K (89910) > >>>> - RowsReturned: 42.68K (42681) > >>>> - RowsReturnedRate: 2.71 M/sec > >>>> - ScanRangesComplete: 6 (6) > >>>> - ScannerThreadsInvoluntaryContextSwitches: 3 (3) > >>>> - ScannerThreadsTotalWallClockTime: 227.825ms > >>>> >
