Do queuing query or not is based on the prediction which based on the estimate and of course the concurrency that can run in a pool.
If I have memory limit per pool and memory limit per node for a pool, so it can be used to estimate number of queries that can run concurrently, is this also based on the prediction and not the actual use. I believe with the time and trends we can learn a lot from the admission control but trying to minimize the impact the business at this period of learning ... On Fri, 23 Feb 2018 at 11:26 Jeszy <jes...@gmail.com> wrote: > Queries will be killed based on actual usage (peak memory usage across > hosts), so the 200mb is the interesting value in your example. > > Compare the pool's available memory to the query's mem requirement > (based on estimate or mem_limit, as discussed) to predict admission. > > On 23 February 2018 at 10:06, Fawze Abujaber <fawz...@gmail.com> wrote: > > Thanks jezy for your detailed response. > > > > Yes I read the documentation. > > > > Let simplify my question: > > > > I have pools set up with memory limit per node and concurrency. > > > > If I’m looking on the historical impala queries that I have and the > metrics > > I have per query, on which metrics I can understand that impala will kill > > the query, for example if I have a query with estimate of 2GB and the > used > > per node is 200mb, what is the default memory values that i need to > setup so > > the query will not fail. > > > > The second one is the distribution between pools, if one query is running > > which metrics o have to look into to know if I submit a query it fail or > > not. > > > > On Fri, 23 Feb 2018 at 10:48 Jeszy <jes...@gmail.com> wrote: > >> > >> Hey Fawze, > >> > >> Answers inline. > >> > >> On 23 February 2018 at 01:23, Fawze Abujaber <fawz...@gmail.com> wrote: > >> > There is no option in the admission control to setup memory limit per > >> > query, > >> > the memory limit is per pool and there is a default memory per node > for > >> > query. > >> > >> per node for query memory limit multiplied by number of nodes gives > >> you a per query memory limit. I agree its confusing that the > >> configurations mix and match between per-node and aggregated values. > >> In this case there's a good reason though, as a single node running > >> out of memory will lead to query failure, meaning that in addition to > >> total memory used, distribution of memory usage between hosts also > >> matters. > >> > >> > I have hundreds of impala queries and more add hoc queries, making a > >> > pool > >> > for each query is not a visible solution. > >> > > >> > still waiting to understand how the estimate per node related to the > >> > default > >> > memory per node I set up per pool, is it used in the decision of > queuing > >> > and > >> > killing the query? and if this is true how it was not kill a query > that > >> > was > >> > estimated it needs 8.2TB memory per node. > >> > > >> > Understanding on which parameters impala decides to kill a query can > >> > help > >> > understand to define and divide the memory between the pools. > >> > >> If you set mem_limit at any level (service level, pool level, or query > >> level), it will be used for admission control purposes instead of > >> estimates. So a 8.2TB estimate would not be a problem, if impala can > >> reserve mem_limit amount on each host, it will start running the > >> query. > >> > >> > Passing memory limit per query manually is also not visible and such > >> > settings not needs admission control. > >> > > >> > I have support pool that runs ad hoc query and I can not ask them to > use > >> > memory limit per query, and I have analytics pool which is fully > >> > business > >> > and I can rely on admission control if it extremely in accurate. > >> > >> It's a bit tricky to use memory-based admission control with > >> non-trivial ad hoc queries. For simple ad-hoc queries, you can try to > >> come up with a 'good enough' mem_limit, or omit mem_limit and trust > >> impala's estimations. You can check the estimated vs. actual values > >> for a representative set of ad hoc queries to see what would work in > >> your case. I've found that people tend to go with a large enough > >> mem_limit for the ad hoc pool. > >> > >> > Can someone explain me exactly which recommended setting to use per > pool > >> > and > >> > which of them rely on impala memory estimates? > >> > >> The documentation of admission control > >> (https://impala.apache.org/docs/build/html/topics/impala_admission.html > ) > >> gives you a good view on how stuff works, but you will have to figure > >> out how to use these features for your specific use case. That said, > >> when using memory based admission control, it is best practice to > >> always use a mem_limit due to potential inaccuracy of estimates as > >> well as potential variance of estimates between Impala releases. Keep > >> in mind that you can opt to set a default mem_limit for one pool and > >> leave it unset for another. > >> > >> > So my conclusion right now to avoid using any settings rely on the > >> > estimates > >> > and to ignore the estimates when I want to evaluate query. > >> > >> Sounds good. > >> > >> > @mostafa, since my issue with all the query, I think the profile will > >> > not > >> > help me to solve such huge issue. > >> > > >> > I’m planning to move a way from Vertica and rely on impala as a sql > >> > engine > >> > and now fully confused how I can do this if I can’t use the admission > >> > control. > >> > > >> > Last think, is it recommend to use the impala admission control? > >> > >> Yes. Admission control can take a while to understand, but if done > >> right, it works. > >> > >> HTH > >> > >> > On Fri, 23 Feb 2018 at 1:56 Alexander Behm <alex.b...@cloudera.com> > >> > wrote: > >> >> > >> >> The planner memory estimates are conservative and sometimes extremely > >> >> inaccurate. In their current form, they are rarely appropriate for > >> >> admission > >> >> decisions. > >> >> > >> >> The recommended practice for memory-based admission control it to > set a > >> >> mem_limit for every query. You can make this easier by setting up > >> >> different > >> >> pools with different mem_limits, e.g. a small/medium/big queries pool > >> >> or > >> >> similar. > >> >> > >> >> On Thu, Feb 22, 2018 at 3:00 PM, Mostafa Mokhtar > >> >> <mmokh...@cloudera.com> > >> >> wrote: > >> >>> > >> >>> It is recommended to set a per query memory limit as part of > admission > >> >>> and not rely on estimates as they are sometimes inaccurate. > >> >>> Can you please include the full query profile? > >> >>> > >> >>> > >> >>> On Thu, Feb 22, 2018 at 12:13 PM, Fawze Abujaber <fawz...@gmail.com > > > >> >>> wrote: > >> >>>> > >> >>>> Hi Mostafa, > >> >>>> > >> >>>> It's not a specific query, almost all the query has such differene > >> >>>> between the 2 values. > >> >>>> > >> >>>> I can see even queries showing the estimate per node is 8.2 Tib > >> >>>> > >> >>>> User: psanalytics > >> >>>> > >> >>>> Database: default > >> >>>> > >> >>>> Query Type: QUERY > >> >>>> Coordinator: slpr-dhc014.lpdomain.com > >> >>>> > >> >>>> Duration: 6.48s > >> >>>> > >> >>>> Rows Produced: 708 > >> >>>> Estimated per Node Peak Memory: 8.2 TiB > >> >>>> > >> >>>> Per Node Peak Memory Usage: 1.1 GiB > >> >>>> > >> >>>> Pool: root.impanalytics > >> >>>> Threads: CPU Time: 20.1m > >> >>>> > >> >>>> > >> >>>> > >> >>>> How you can explain this behavior, and for sure i don't have 8.2 > Tib > >> >>>> memory per node to give neither you. > >> >>>> > >> >>>> Can you please explain me how i should treat Estimated per Node > Peak > >> >>>> Memory and if it used by impala for the resource pool and admission > >> >>>> control > >> >>>> and what is the relation of this value to the default memory per > node > >> >>>> that i > >> >>>> setup for each resource pool? > >> >>>> > >> >>>> Below is part of one of the queries profile which the estimate per > >> >>>> node > >> >>>> was ~ @GB and the used was 200MB per node. > >> >>>> > >> >>>> > >> >>>> Instance 744de1b6228736fa:b54bfaa70000000f > >> >>>> (host=slpr-dhc004.lpdomain.com:22000):(Total: 1s455ms, non-child: > >> >>>> 1s292ms, % > >> >>>> non-child: 88.82%) > >> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>): > >> >>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB > >> >>>> 9:1/191.64 > >> >>>> KB 5:2/289.57 KB 3:2/1012.83 KB > >> >>>> MemoryUsage(500.000ms): 6.09 MB, 6.09 MB, 3.00 MB > >> >>>> ThreadUsage(500.000ms): 1, 1, 1 > >> >>>> - AverageThreadTokens: 1.00 > >> >>>> - BloomFilterBytes: 0 > >> >>>> - PeakMemoryUsage: 7.17 MB (7521751) > >> >>>> - PeakReservation: 0 > >> >>>> - PeakUsedReservation: 0 > >> >>>> - PerHostPeakMemUsage: 106.53 MB (111709581) > >> >>>> - RowsProduced: 32.83K (32826) > >> >>>> - TotalNetworkReceiveTime: 0.000ns > >> >>>> - TotalNetworkSendTime: 1s297ms > >> >>>> - TotalStorageWaitTime: 234.356ms > >> >>>> - TotalThreadsInvoluntaryContextSwitches: 66 (66) > >> >>>> - TotalThreadsTotalWallClockTime: 1s715ms > >> >>>> - TotalThreadsSysTime: 5.998ms > >> >>>> - TotalThreadsUserTime: 124.975ms > >> >>>> - TotalThreadsVoluntaryContextSwitches: 303 (303) > >> >>>> Fragment Instance Lifecycle Timings: > >> >>>> - ExecTime: 1s394ms > >> >>>> - ExecTreeExecTime: 67.115ms > >> >>>> - OpenTime: 32.795ms > >> >>>> - ExecTreeOpenTime: 73.243us > >> >>>> - PrepareTime: 27.602ms > >> >>>> - ExecTreePrepareTime: 243.141us > >> >>>> DataStreamSender (dst_id=11):(Total: 38.747ms, non-child: > >> >>>> 38.747ms, % non-child: 100.00%) > >> >>>> - BytesSent: 39.71 MB (41643000) > >> >>>> - NetworkThroughput(*): 1.97 GB/sec > >> >>>> - OverallThroughput: 1.00 GB/sec > >> >>>> - PeakMemoryUsage: 59.38 KB (60800) > >> >>>> - RowsReturned: 32.83K (32826) > >> >>>> - SerializeBatchTime: 16.860ms > >> >>>> - TransmitDataRPCTime: 19.698ms > >> >>>> - UncompressedRowBatchSize: 77.58 MB (81350840) > >> >>>> CodeGen:(Total: 56.573ms, non-child: 56.573ms, % non-child: > >> >>>> 100.00%) > >> >>>> - CodegenTime: 1.299ms > >> >>>> - CompileTime: 10.672ms > >> >>>> - LoadTime: 0.000ns > >> >>>> - ModuleBitcodeSize: 1.96 MB (2050180) > >> >>>> - NumFunctions: 16 (16) > >> >>>> - NumInstructions: 250 (250) > >> >>>> - OptimizationTime: 21.023ms > >> >>>> - PeakMemoryUsage: 125.00 KB (128000) > >> >>>> - PrepareTime: 24.116ms > >> >>>> SUBPLAN_NODE (id=6):(Total: 67.311ms, non-child: 12.013ms, > % > >> >>>> non-child: 17.85%) > >> >>>> - PeakMemoryUsage: 627.94 KB (643015) > >> >>>> - RowsReturned: 32.77K (32768) > >> >>>> - RowsReturnedRate: 486.81 K/sec > >> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.999ms, non-child: > >> >>>> 25.197ms, % non-child: 74.11%) > >> >>>> - BuildRows: 0 (0) > >> >>>> - BuildTime: 0.000ns > >> >>>> - PeakMemoryUsage: 24.00 KB (24576) > >> >>>> - ProbeRows: 32.83K (32826) > >> >>>> - ProbeTime: 0.000ns > >> >>>> - RowsReturned: 16.80M (16795311) > >> >>>> - RowsReturnedRate: 493.99 M/sec > >> >>>> Nested Loop Join Builder: > >> >>>> - PeakMemoryUsage: 8.00 KB (8192) > >> >>>> SINGULAR_ROW_SRC_NODE (id=7): > >> >>>> - PeakMemoryUsage: 0 > >> >>>> - RowsReturned: 0 (0) > >> >>>> - RowsReturnedRate: 0 > >> >>>> UNNEST_NODE (id=8):(Total: 8.801ms, non-child: 8.801ms, % > >> >>>> non-child: 100.00%) > >> >>>> - AvgCollectionSize: 1.00 > >> >>>> - MaxCollectionSize: 1 (1) > >> >>>> - MinCollectionSize: 1 (1) > >> >>>> - NumCollections: 32.83K (32826) > >> >>>> - PeakMemoryUsage: 0 > >> >>>> - RowsReturned: 1 (1) > >> >>>> - RowsReturnedRate: 113.00 /sec > >> >>>> HDFS_SCAN_NODE (id=5):(Total: 21.299ms, non-child: > 21.299ms, > >> >>>> % > >> >>>> non-child: 100.00%) > >> >>>> Hdfs split stats (<volume id>:<# splits>/<split > lengths>): > >> >>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB > >> >>>> 9:1/191.64 > >> >>>> KB 5:2/289.57 KB 3:2/1012.83 KB > >> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 11 > >> >>>> out > >> >>>> of 11 > >> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% > 3:0% > >> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% > >> >>>> 16:0% > >> >>>> File Formats: PARQUET/SNAPPY:66 > >> >>>> BytesRead(500.000ms): 2.26 MB, 2.26 MB, 2.26 MB > >> >>>> - FooterProcessingTime: (Avg: 12.082ms ; Min: 9.079ms ; > >> >>>> Max: > >> >>>> 29.275ms ; Number of samples: 11) > >> >>>> - AverageHdfsReadThreadConcurrency: 0.00 > >> >>>> - AverageScannerThreadConcurrency: 0.00 > >> >>>> - BytesRead: 2.26 MB (2373738) > >> >>>> - BytesReadDataNodeCache: 0 > >> >>>> - BytesReadLocal: 2.26 MB (2373738) > >> >>>> - BytesReadRemoteUnexpected: 0 > >> >>>> - BytesReadShortCircuit: 2.26 MB (2373738) > >> >>>> - CachedFileHandlesHitCount: 0 (0) > >> >>>> - CachedFileHandlesMissCount: 77 (77) > >> >>>> - DecompressionTime: 1.543ms > >> >>>> - MaxCompressedTextFileLength: 0 > >> >>>> - NumColumns: 6 (6) > >> >>>> - NumDictFilteredRowGroups: 0 (0) > >> >>>> - NumDisksAccessed: 8 (8) > >> >>>> - NumRowGroups: 11 (11) > >> >>>> - NumScannerThreadsStarted: 11 (11) > >> >>>> - NumScannersWithNoReads: 0 (0) > >> >>>> - NumStatsFilteredRowGroups: 0 (0) > >> >>>> - PeakMemoryUsage: 6.51 MB (6821203) > >> >>>> - PerReadThreadRawHdfsThroughput: 764.51 MB/sec > >> >>>> - RemoteScanRanges: 0 (0) > >> >>>> - RowBatchQueueGetWaitTime: 18.825ms > >> >>>> - RowBatchQueuePutWaitTime: 0.000ns > >> >>>> - RowsRead: 73.46K (73464) > >> >>>> - RowsReturned: 32.83K (32826) > >> >>>> - RowsReturnedRate: 1.54 M/sec > >> >>>> - ScanRangesComplete: 11 (11) > >> >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1) > >> >>>> - ScannerThreadsTotalWallClockTime: 287.702ms > >> >>>> - MaterializeTupleTime(*): 11.921ms > >> >>>> - ScannerThreadsSysTime: 999.000us > >> >>>> - ScannerThreadsUserTime: 26.990ms > >> >>>> - ScannerThreadsVoluntaryContextSwitches: 232 (232) > >> >>>> - TotalRawHdfsReadTime(*): 2.961ms > >> >>>> - TotalReadThroughput: 1.51 MB/sec > >> >>>> Instance 744de1b6228736fa:b54bfaa70000000d > >> >>>> (host=slpr-dhc002.lpdomain.com:22000):(Total: 1s453ms, non-child: > >> >>>> 1s254ms, % > >> >>>> non-child: 86.28%) > >> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>): > >> >>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB > >> >>>> MemoryUsage(500.000ms): 5.32 MB, 5.32 MB, 1.27 MB > >> >>>> ThreadUsage(500.000ms): 1, 1, 1 > >> >>>> - AverageThreadTokens: 1.00 > >> >>>> - BloomFilterBytes: 0 > >> >>>> - PeakMemoryUsage: 5.86 MB (6145964) > >> >>>> - PeakReservation: 0 > >> >>>> - PeakUsedReservation: 0 > >> >>>> - PerHostPeakMemUsage: 106.66 MB (111842766) > >> >>>> - RowsProduced: 30.34K (30340) > >> >>>> - TotalNetworkReceiveTime: 0.000ns > >> >>>> - TotalNetworkSendTime: 1s256ms > >> >>>> - TotalStorageWaitTime: 106.530ms > >> >>>> - TotalThreadsInvoluntaryContextSwitches: 177 (177) > >> >>>> - TotalThreadsTotalWallClockTime: 1s557ms > >> >>>> - TotalThreadsSysTime: 6.998ms > >> >>>> - TotalThreadsUserTime: 143.976ms > >> >>>> - TotalThreadsVoluntaryContextSwitches: 136 (136) > >> >>>> Fragment Instance Lifecycle Timings: > >> >>>> - ExecTime: 1s390ms > >> >>>> - ExecTreeExecTime: 91.500ms > >> >>>> - OpenTime: 34.085ms > >> >>>> - ExecTreeOpenTime: 81.436us > >> >>>> - PrepareTime: 28.524ms > >> >>>> - ExecTreePrepareTime: 217.949us > >> >>>> DataStreamSender (dst_id=11):(Total: 48.674ms, non-child: > >> >>>> 48.674ms, % non-child: 100.00%) > >> >>>> - BytesSent: 36.61 MB (38388760) > >> >>>> - NetworkThroughput(*): 1.51 GB/sec > >> >>>> - OverallThroughput: 752.15 MB/sec > >> >>>> - PeakMemoryUsage: 59.38 KB (60800) > >> >>>> - RowsReturned: 30.34K (30340) > >> >>>> - SerializeBatchTime: 23.132ms > >> >>>> - TransmitDataRPCTime: 23.609ms > >> >>>> - UncompressedRowBatchSize: 71.70 MB (75187520) > >> >>>> CodeGen:(Total: 59.024ms, non-child: 59.024ms, % non-child: > >> >>>> 100.00%) > >> >>>> - CodegenTime: 1.382ms > >> >>>> - CompileTime: 10.435ms > >> >>>> - LoadTime: 0.000ns > >> >>>> - ModuleBitcodeSize: 1.96 MB (2050180) > >> >>>> - NumFunctions: 16 (16) > >> >>>> - NumInstructions: 250 (250) > >> >>>> - OptimizationTime: 22.325ms > >> >>>> - PeakMemoryUsage: 125.00 KB (128000) > >> >>>> - PrepareTime: 25.459ms > >> >>>> SUBPLAN_NODE (id=6):(Total: 91.711ms, non-child: 20.997ms, > % > >> >>>> non-child: 22.90%) > >> >>>> - PeakMemoryUsage: 699.55 KB (716335) > >> >>>> - RowsReturned: 29.70K (29696) > >> >>>> - RowsReturnedRate: 323.80 K/sec > >> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 53.723ms, non-child: > >> >>>> 40.263ms, % non-child: 74.95%) > >> >>>> - BuildRows: 0 (0) > >> >>>> - BuildTime: 0.000ns > >> >>>> - PeakMemoryUsage: 24.00 KB (24576) > >> >>>> - ProbeRows: 30.34K (30340) > >> >>>> - ProbeTime: 0.000ns > >> >>>> - RowsReturned: 15.43M (15426890) > >> >>>> - RowsReturnedRate: 287.16 M/sec > >> >>>> Nested Loop Join Builder: > >> >>>> - PeakMemoryUsage: 8.00 KB (8192) > >> >>>> SINGULAR_ROW_SRC_NODE (id=7): > >> >>>> - PeakMemoryUsage: 0 > >> >>>> - RowsReturned: 0 (0) > >> >>>> - RowsReturnedRate: 0 > >> >>>> UNNEST_NODE (id=8):(Total: 13.460ms, non-child: > 13.460ms, % > >> >>>> non-child: 100.00%) > >> >>>> - AvgCollectionSize: 1.00 > >> >>>> - MaxCollectionSize: 1 (1) > >> >>>> - MinCollectionSize: 1 (1) > >> >>>> - NumCollections: 30.34K (30340) > >> >>>> - PeakMemoryUsage: 0 > >> >>>> - RowsReturned: 1 (1) > >> >>>> - RowsReturnedRate: 74.00 /sec > >> >>>> HDFS_SCAN_NODE (id=5):(Total: 16.990ms, non-child: > 16.990ms, > >> >>>> % > >> >>>> non-child: 100.00%) > >> >>>> Hdfs split stats (<volume id>:<# splits>/<split > lengths>): > >> >>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB > >> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 > out > >> >>>> of > >> >>>> 5 > >> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% > 3:0% > >> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% > >> >>>> 16:0% > >> >>>> File Formats: PARQUET/SNAPPY:30 > >> >>>> BytesRead(500.000ms): 1.56 MB, 1.56 MB, 1.56 MB > >> >>>> - FooterProcessingTime: (Avg: 12.351ms ; Min: 7.816ms ; > >> >>>> Max: > >> >>>> 28.687ms ; Number of samples: 5) > >> >>>> - AverageHdfsReadThreadConcurrency: 0.00 > >> >>>> - AverageScannerThreadConcurrency: 0.00 > >> >>>> - BytesRead: 1.56 MB (1630546) > >> >>>> - BytesReadDataNodeCache: 0 > >> >>>> - BytesReadLocal: 1.56 MB (1630546) > >> >>>> - BytesReadRemoteUnexpected: 0 > >> >>>> - BytesReadShortCircuit: 1.56 MB (1630546) > >> >>>> - CachedFileHandlesHitCount: 0 (0) > >> >>>> - CachedFileHandlesMissCount: 35 (35) > >> >>>> - DecompressionTime: 1.267ms > >> >>>> - MaxCompressedTextFileLength: 0 > >> >>>> - NumColumns: 6 (6) > >> >>>> - NumDictFilteredRowGroups: 0 (0) > >> >>>> - NumDisksAccessed: 4 (4) > >> >>>> - NumRowGroups: 5 (5) > >> >>>> - NumScannerThreadsStarted: 5 (5) > >> >>>> - NumScannersWithNoReads: 0 (0) > >> >>>> - NumStatsFilteredRowGroups: 0 (0) > >> >>>> - PeakMemoryUsage: 5.59 MB (5859290) > >> >>>> - PerReadThreadRawHdfsThroughput: 920.32 MB/sec > >> >>>> - RemoteScanRanges: 0 (0) > >> >>>> - RowBatchQueueGetWaitTime: 15.335ms > >> >>>> - RowBatchQueuePutWaitTime: 0.000ns > >> >>>> - RowsRead: 64.69K (64692) > >> >>>> - RowsReturned: 30.34K (30340) > >> >>>> - RowsReturnedRate: 1.79 M/sec > >> >>>> - ScanRangesComplete: 5 (5) > >> >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1) > >> >>>> - ScannerThreadsTotalWallClockTime: 132.548ms > >> >>>> - MaterializeTupleTime(*): 10.585ms > >> >>>> - ScannerThreadsSysTime: 999.000us > >> >>>> - ScannerThreadsUserTime: 18.995ms > >> >>>> - ScannerThreadsVoluntaryContextSwitches: 79 (79) > >> >>>> - TotalRawHdfsReadTime(*): 1.689ms > >> >>>> - TotalReadThroughput: 1.04 MB/sec > >> >>>> Instance 744de1b6228736fa:b54bfaa700000002 > >> >>>> (host=slpr-dhc003.lpdomain.com:22000):(Total: 1s452ms, non-child: > >> >>>> 1s255ms, % > >> >>>> non-child: 86.41%) > >> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>): > >> >>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB > >> >>>> MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 3.94 MB > >> >>>> ThreadUsage(500.000ms): 1, 1, 1 > >> >>>> - AverageThreadTokens: 1.00 > >> >>>> - BloomFilterBytes: 0 > >> >>>> - PeakMemoryUsage: 6.17 MB (6473729) > >> >>>> - PeakReservation: 0 > >> >>>> - PeakUsedReservation: 0 > >> >>>> - PerHostPeakMemUsage: 107.32 MB (112537640) > >> >>>> - RowsProduced: 29.97K (29969) > >> >>>> - TotalNetworkReceiveTime: 0.000ns > >> >>>> - TotalNetworkSendTime: 1s260ms > >> >>>> - TotalStorageWaitTime: 130.815ms > >> >>>> - TotalThreadsInvoluntaryContextSwitches: 15 (15) > >> >>>> - TotalThreadsTotalWallClockTime: 1s586ms > >> >>>> - TotalThreadsSysTime: 11.995ms > >> >>>> - TotalThreadsUserTime: 146.975ms > >> >>>> - TotalThreadsVoluntaryContextSwitches: 197 (197) > >> >>>> Fragment Instance Lifecycle Timings: > >> >>>> - ExecTime: 1s354ms > >> >>>> - ExecTreeExecTime: 67.791ms > >> >>>> - OpenTime: 52.012ms > >> >>>> - ExecTreeOpenTime: 101.232us > >> >>>> - PrepareTime: 45.946ms > >> >>>> - ExecTreePrepareTime: 309.644us > >> >>>> DataStreamSender (dst_id=11):(Total: 38.995ms, non-child: > >> >>>> 38.995ms, % non-child: 100.00%) > >> >>>> - BytesSent: 35.95 MB (37697260) > >> >>>> - NetworkThroughput(*): 2.14 GB/sec > >> >>>> - OverallThroughput: 921.92 MB/sec > >> >>>> - PeakMemoryUsage: 59.38 KB (60800) > >> >>>> - RowsReturned: 29.97K (29969) > >> >>>> - SerializeBatchTime: 18.055ms > >> >>>> - TransmitDataRPCTime: 16.428ms > >> >>>> - UncompressedRowBatchSize: 70.83 MB (74269800) > >> >>>> CodeGen:(Total: 90.385ms, non-child: 90.385ms, % non-child: > >> >>>> 100.00%) > >> >>>> - CodegenTime: 2.224ms > >> >>>> - CompileTime: 15.517ms > >> >>>> - LoadTime: 0.000ns > >> >>>> - ModuleBitcodeSize: 1.96 MB (2050180) > >> >>>> - NumFunctions: 16 (16) > >> >>>> - NumInstructions: 250 (250) > >> >>>> - OptimizationTime: 35.099ms > >> >>>> - PeakMemoryUsage: 125.00 KB (128000) > >> >>>> - PrepareTime: 38.847ms > >> >>>> SUBPLAN_NODE (id=6):(Total: 68.126ms, non-child: 12.625ms, > % > >> >>>> non-child: 18.53%) > >> >>>> - PeakMemoryUsage: 636.17 KB (651439) > >> >>>> - RowsReturned: 29.70K (29696) > >> >>>> - RowsReturnedRate: 435.89 K/sec > >> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.347ms, non-child: > >> >>>> 27.325ms, % non-child: 77.31%) > >> >>>> - BuildRows: 0 (0) > >> >>>> - BuildTime: 0.000ns > >> >>>> - PeakMemoryUsage: 24.00 KB (24576) > >> >>>> - ProbeRows: 29.97K (29969) > >> >>>> - ProbeTime: 0.000ns > >> >>>> - RowsReturned: 15.26M (15256601) > >> >>>> - RowsReturnedRate: 431.62 M/sec > >> >>>> Nested Loop Join Builder: > >> >>>> - PeakMemoryUsage: 8.00 KB (8192) > >> >>>> SINGULAR_ROW_SRC_NODE (id=7): > >> >>>> - PeakMemoryUsage: 0 > >> >>>> - RowsReturned: 0 (0) > >> >>>> - RowsReturnedRate: 0 > >> >>>> UNNEST_NODE (id=8):(Total: 8.021ms, non-child: 8.021ms, % > >> >>>> non-child: 100.00%) > >> >>>> - AvgCollectionSize: 1.00 > >> >>>> - MaxCollectionSize: 1 (1) > >> >>>> - MinCollectionSize: 1 (1) > >> >>>> - NumCollections: 29.97K (29969) > >> >>>> - PeakMemoryUsage: 0 > >> >>>> - RowsReturned: 1 (1) > >> >>>> - RowsReturnedRate: 124.00 /sec > >> >>>> HDFS_SCAN_NODE (id=5):(Total: 20.153ms, non-child: > 20.153ms, > >> >>>> % > >> >>>> non-child: 100.00%) > >> >>>> Hdfs split stats (<volume id>:<# splits>/<split > lengths>): > >> >>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB > >> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 > out > >> >>>> of > >> >>>> 7 > >> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% > 3:0% > >> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% > >> >>>> 16:0% > >> >>>> File Formats: PARQUET/SNAPPY:42 > >> >>>> BytesRead(500.000ms): 1.75 MB, 1.75 MB, 1.75 MB > >> >>>> - FooterProcessingTime: (Avg: 8.683ms ; Min: 3.937ms ; > >> >>>> Max: > >> >>>> 10.464ms ; Number of samples: 7) > >> >>>> - AverageHdfsReadThreadConcurrency: 0.00 > >> >>>> - AverageScannerThreadConcurrency: 0.00 > >> >>>> - BytesRead: 1.75 MB (1833560) > >> >>>> - BytesReadDataNodeCache: 0 > >> >>>> - BytesReadLocal: 1.75 MB (1833560) > >> >>>> - BytesReadRemoteUnexpected: 0 > >> >>>> - BytesReadShortCircuit: 1.75 MB (1833560) > >> >>>> - CachedFileHandlesHitCount: 0 (0) > >> >>>> - CachedFileHandlesMissCount: 49 (49) > >> >>>> - DecompressionTime: 1.758ms > >> >>>> - MaxCompressedTextFileLength: 0 > >> >>>> - NumColumns: 6 (6) > >> >>>> - NumDictFilteredRowGroups: 0 (0) > >> >>>> - NumDisksAccessed: 5 (5) > >> >>>> - NumRowGroups: 7 (7) > >> >>>> - NumScannerThreadsStarted: 7 (7) > >> >>>> - NumScannersWithNoReads: 0 (0) > >> >>>> - NumStatsFilteredRowGroups: 0 (0) > >> >>>> - PeakMemoryUsage: 5.82 MB (6098991) > >> >>>> - PerReadThreadRawHdfsThroughput: 621.09 MB/sec > >> >>>> - RemoteScanRanges: 0 (0) > >> >>>> - RowBatchQueueGetWaitTime: 17.938ms > >> >>>> - RowBatchQueuePutWaitTime: 0.000ns > >> >>>> - RowsRead: 65.17K (65166) > >> >>>> - RowsReturned: 29.97K (29969) > >> >>>> - RowsReturnedRate: 1.49 M/sec > >> >>>> - ScanRangesComplete: 7 (7) > >> >>>> - ScannerThreadsInvoluntaryContextSwitches: 1 (1) > >> >>>> - ScannerThreadsTotalWallClockTime: 179.698ms > >> >>>> - MaterializeTupleTime(*): 13.922ms > >> >>>> - ScannerThreadsSysTime: 2.997ms > >> >>>> - ScannerThreadsUserTime: 29.993ms > >> >>>> - ScannerThreadsVoluntaryContextSwitches: 130 (130) > >> >>>> - TotalRawHdfsReadTime(*): 2.815ms > >> >>>> - TotalReadThroughput: 1.17 MB/sec > >> >>>> Instance 744de1b6228736fa:b54bfaa700000008 > >> >>>> (host=slpr-dhc009.lpdomain.com:22000):(Total: 1s450ms, non-child: > >> >>>> 1s305ms, % > >> >>>> non-child: 90.01%) > >> >>>> Hdfs split stats (<volume id>:<# splits>/<split lengths>): > >> >>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB > >> >>>> MemoryUsage(500.000ms): 5.97 MB, 5.88 MB, 2.24 MB > >> >>>> ThreadUsage(500.000ms): 1, 1, 1 > >> >>>> - AverageThreadTokens: 1.00 > >> >>>> - BloomFilterBytes: 0 > >> >>>> - PeakMemoryUsage: 6.69 MB (7017157) > >> >>>> - PeakReservation: 0 > >> >>>> - PeakUsedReservation: 0 > >> >>>> - PerHostPeakMemUsage: 106.39 MB (111556990) > >> >>>> - RowsProduced: 42.68K (42681) > >> >>>> - TotalNetworkReceiveTime: 0.000ns > >> >>>> - TotalNetworkSendTime: 1s315ms > >> >>>> - TotalStorageWaitTime: 201.411ms > >> >>>> - TotalThreadsInvoluntaryContextSwitches: 43 (43) > >> >>>> - TotalThreadsTotalWallClockTime: 1s658ms > >> >>>> - TotalThreadsSysTime: 6.997ms > >> >>>> - TotalThreadsUserTime: 108.980ms > >> >>>> - TotalThreadsVoluntaryContextSwitches: 179 (179) > >> >>>> Fragment Instance Lifecycle Timings: > >> >>>> - ExecTime: 1s411ms > >> >>>> - ExecTreeExecTime: 68.017ms > >> >>>> - OpenTime: 19.935ms > >> >>>> - ExecTreeOpenTime: 48.883us > >> >>>> - PrepareTime: 19.160ms > >> >>>> - ExecTreePrepareTime: 134.576us > >> >>>> DataStreamSender (dst_id=11):(Total: 40.671ms, non-child: > >> >>>> 40.671ms, % non-child: 100.00%) > >> >>>> - BytesSent: 51.36 MB (53856680) > >> >>>> - NetworkThroughput(*): 2.44 GB/sec > >> >>>> - OverallThroughput: 1.23 GB/sec > >> >>>> - PeakMemoryUsage: 59.38 KB (60800) > >> >>>> - RowsReturned: 42.68K (42681) > >> >>>> - SerializeBatchTime: 18.299ms > >> >>>> - TransmitDataRPCTime: 20.591ms > >> >>>> - UncompressedRowBatchSize: 100.88 MB (105782860) > >> >>>> CodeGen:(Total: 36.171ms, non-child: 36.171ms, % non-child: > >> >>>> 100.00%) > >> >>>> - CodegenTime: 828.865us > >> >>>> - CompileTime: 6.170ms > >> >>>> - LoadTime: 0.000ns > >> >>>> - ModuleBitcodeSize: 1.96 MB (2050180) > >> >>>> - NumFunctions: 16 (16) > >> >>>> - NumInstructions: 250 (250) > >> >>>> - OptimizationTime: 12.988ms > >> >>>> - PeakMemoryUsage: 125.00 KB (128000) > >> >>>> - PrepareTime: 16.608ms > >> >>>> SUBPLAN_NODE (id=6):(Total: 68.046ms, non-child: 12.434ms, > % > >> >>>> non-child: 18.27%) > >> >>>> - PeakMemoryUsage: 689.95 KB (706507) > >> >>>> - RowsReturned: 41.98K (41984) > >> >>>> - RowsReturnedRate: 616.99 K/sec > >> >>>> NESTED_LOOP_JOIN_NODE (id=9):(Total: 39.837ms, non-child: > >> >>>> 29.422ms, % non-child: 73.86%) > >> >>>> - BuildRows: 0 (0) > >> >>>> - BuildTime: 0.000ns > >> >>>> - PeakMemoryUsage: 24.00 KB (24576) > >> >>>> - ProbeRows: 42.68K (42681) > >> >>>> - ProbeTime: 0.000ns > >> >>>> - RowsReturned: 21.76M (21760053) > >> >>>> - RowsReturnedRate: 546.22 M/sec > >> >>>> Nested Loop Join Builder: > >> >>>> - PeakMemoryUsage: 8.00 KB (8192) > >> >>>> SINGULAR_ROW_SRC_NODE (id=7): > >> >>>> - PeakMemoryUsage: 0 > >> >>>> - RowsReturned: 0 (0) > >> >>>> - RowsReturnedRate: 0 > >> >>>> UNNEST_NODE (id=8):(Total: 10.415ms, non-child: > 10.415ms, % > >> >>>> non-child: 100.00%) > >> >>>> - AvgCollectionSize: 1.00 > >> >>>> - MaxCollectionSize: 1 (1) > >> >>>> - MinCollectionSize: 1 (1) > >> >>>> - NumCollections: 42.68K (42681) > >> >>>> - PeakMemoryUsage: 0 > >> >>>> - RowsReturned: 1 (1) > >> >>>> - RowsReturnedRate: 96.00 /sec > >> >>>> HDFS_SCAN_NODE (id=5):(Total: 15.774ms, non-child: > 15.774ms, > >> >>>> % > >> >>>> non-child: 100.00%) > >> >>>> Hdfs split stats (<volume id>:<# splits>/<split > lengths>): > >> >>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB > >> >>>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 > out > >> >>>> of > >> >>>> 6 > >> >>>> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% > 3:0% > >> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% > >> >>>> 16:0% > >> >>>> 17:0% > >> >>>> File Formats: PARQUET/SNAPPY:36 > >> >>>> BytesRead(500.000ms): 2.08 MB, 2.08 MB, 2.08 MB > >> >>>> - FooterProcessingTime: (Avg: 22.052ms ; Min: 7.783ms ; > >> >>>> Max: > >> >>>> 42.732ms ; Number of samples: 6) > >> >>>> - AverageHdfsReadThreadConcurrency: 0.00 > >> >>>> - AverageScannerThreadConcurrency: 0.00 > >> >>>> - BytesRead: 2.08 MB (2186159) > >> >>>> - BytesReadDataNodeCache: 0 > >> >>>> - BytesReadLocal: 2.08 MB (2186159) > >> >>>> - BytesReadRemoteUnexpected: 0 > >> >>>> - BytesReadShortCircuit: 2.08 MB (2186159) > >> >>>> - CachedFileHandlesHitCount: 0 (0) > >> >>>> - CachedFileHandlesMissCount: 42 (42) > >> >>>> - DecompressionTime: 1.480ms > >> >>>> - MaxCompressedTextFileLength: 0 > >> >>>> - NumColumns: 6 (6) > >> >>>> - NumDictFilteredRowGroups: 0 (0) > >> >>>> - NumDisksAccessed: 4 (4) > >> >>>> - NumRowGroups: 6 (6) > >> >>>> - NumScannerThreadsStarted: 6 (6) > >> >>>> - NumScannersWithNoReads: 0 (0) > >> >>>> - NumStatsFilteredRowGroups: 0 (0) > >> >>>> - PeakMemoryUsage: 6.42 MB (6730467) > >> >>>> - PerReadThreadRawHdfsThroughput: 77.02 MB/sec > >> >>>> - RemoteScanRanges: 0 (0) > >> >>>> - RowBatchQueueGetWaitTime: 13.979ms > >> >>>> - RowBatchQueuePutWaitTime: 0.000ns > >> >>>> - RowsRead: 89.91K (89910) > >> >>>> - RowsReturned: 42.68K (42681) > >> >>>> - RowsReturnedRate: 2.71 M/sec > >> >>>> - ScanRangesComplete: 6 (6) > >> >>>> - ScannerThreadsInvoluntaryContextSwitches: 3 (3) > >> >>>> - ScannerThreadsTotalWallClockTime: 227.825ms > >> >>>> >
