Thanks jezy for your detailed response.

Yes I read the documentation.

Let simplify my question:

I have pools set up with memory limit per node and concurrency.

If I’m looking on the historical impala queries that I have and the metrics
I have per query, on which metrics I can understand that impala will kill
the query, for example if I have a query with estimate of 2GB and the used
per node is 200mb, what is the default memory values that i need to setup
so the query will not fail.

The second one is the distribution between pools, if one query is running
which metrics o have to look into to know if I submit a query it fail or
not.

On Fri, 23 Feb 2018 at 10:48 Jeszy <jes...@gmail.com> wrote:

> Hey Fawze,
>
> Answers inline.
>
> On 23 February 2018 at 01:23, Fawze Abujaber <fawz...@gmail.com> wrote:
> > There is no option in the admission control to setup memory limit per
> query,
> > the memory limit is per pool and there is a default memory per node for
> > query.
>
> per node for query memory limit multiplied by number of nodes gives
> you a per query memory limit. I agree its confusing that the
> configurations mix and match between per-node and aggregated values.
> In this case there's a good reason though, as a single node running
> out of memory will lead to query failure, meaning that in addition to
> total memory used, distribution of memory usage between hosts also
> matters.
>
> > I have hundreds of impala queries and more add hoc queries, making a pool
> > for each query is not a visible solution.
> >
> > still waiting to understand how the estimate per node related to the
> default
> > memory per node I set up per pool, is it used in the decision of queuing
> and
> > killing the query? and if this is true how it was not kill a query that
> was
> > estimated it needs 8.2TB memory per node.
> >
> > Understanding on which parameters impala decides to kill a query can help
> > understand to define and divide the memory between the pools.
>
> If you set mem_limit at any level (service level, pool level, or query
> level), it will be used for admission control purposes instead of
> estimates. So a 8.2TB estimate would not be a problem, if impala can
> reserve mem_limit amount on each host, it will start running the
> query.
>
> > Passing memory limit per query manually is also not visible and such
> > settings not needs admission control.
> >
> > I have support pool that runs ad hoc query and I can not ask them to use
> > memory limit per query, and I have analytics pool which is fully business
> > and I can rely on admission control if it extremely in accurate.
>
> It's a bit tricky to use memory-based admission control with
> non-trivial ad hoc queries. For simple ad-hoc queries, you can try to
> come up with a 'good enough' mem_limit, or omit mem_limit and trust
> impala's estimations. You can check the estimated vs. actual values
> for a representative set of ad hoc queries to see what would work in
> your case. I've found that people tend to go with a large enough
> mem_limit for the ad hoc pool.
>
> > Can someone explain me exactly which recommended setting to use per pool
> and
> > which of them rely on impala memory estimates?
>
> The documentation of admission control
> (https://impala.apache.org/docs/build/html/topics/impala_admission.html)
> gives you a good view on how stuff works, but you will have to figure
> out how to use these features for your specific use case. That said,
> when using memory based admission control, it is best practice to
> always use a mem_limit due to potential inaccuracy of estimates as
> well as potential variance of estimates between Impala releases. Keep
> in mind that you can opt to set a default mem_limit for one pool and
> leave it unset for another.
>
> > So my conclusion right now to avoid using any settings rely on the
> estimates
> > and to ignore the estimates when I want to evaluate query.
>
> Sounds good.
>
> > @mostafa, since my issue with all the query, I think the profile will not
> > help me to solve such huge issue.
> >
> > I’m planning to move a way from Vertica and rely on impala as a sql
> engine
> > and now fully confused how I can do this if I can’t use the admission
> > control.
> >
> > Last think, is it recommend to use the impala admission control?
>
> Yes. Admission control can take a while to understand, but if done
> right, it works.
>
> HTH
>
> > On Fri, 23 Feb 2018 at 1:56 Alexander Behm <alex.b...@cloudera.com>
> wrote:
> >>
> >> The planner memory estimates are conservative and sometimes extremely
> >> inaccurate. In their current form, they are rarely appropriate for
> admission
> >> decisions.
> >>
> >> The recommended practice for memory-based admission control it to set a
> >> mem_limit for every query. You can make this easier by setting up
> different
> >> pools with different mem_limits, e.g. a small/medium/big queries pool or
> >> similar.
> >>
> >> On Thu, Feb 22, 2018 at 3:00 PM, Mostafa Mokhtar <mmokh...@cloudera.com
> >
> >> wrote:
> >>>
> >>> It is recommended to set a per query memory limit as part of admission
> >>> and not rely on estimates as they are sometimes inaccurate.
> >>> Can you please include the full query profile?
> >>>
> >>>
> >>> On Thu, Feb 22, 2018 at 12:13 PM, Fawze Abujaber <fawz...@gmail.com>
> >>> wrote:
> >>>>
> >>>> Hi Mostafa,
> >>>>
> >>>> It's not a specific query, almost all the query has such differene
> >>>> between the 2 values.
> >>>>
> >>>> I can see even queries showing the estimate per node is 8.2 Tib
> >>>>
> >>>> User: psanalytics
> >>>>
> >>>> Database: default
> >>>>
> >>>> Query Type: QUERY
> >>>> Coordinator: slpr-dhc014.lpdomain.com
> >>>>
> >>>> Duration: 6.48s
> >>>>
> >>>> Rows Produced: 708
> >>>> Estimated per Node Peak Memory: 8.2 TiB
> >>>>
> >>>> Per Node Peak Memory Usage: 1.1 GiB
> >>>>
> >>>> Pool: root.impanalytics
> >>>> Threads: CPU Time: 20.1m
> >>>>
> >>>>
> >>>>
> >>>> How you can explain this behavior, and for sure i don't have 8.2 Tib
> >>>> memory per node to give neither you.
> >>>>
> >>>> Can you please explain me how i should treat Estimated per Node Peak
> >>>> Memory and if it used by impala for the resource pool and admission
> control
> >>>> and what is the relation of this value to the default memory per node
> that i
> >>>> setup for each resource pool?
> >>>>
> >>>> Below is part of one of the queries profile which the estimate per
> node
> >>>> was ~ @GB and the used was 200MB per node.
> >>>>
> >>>>
> >>>>  Instance 744de1b6228736fa:b54bfaa70000000f
> >>>> (host=slpr-dhc004.lpdomain.com:22000):(Total: 1s455ms, non-child:
> 1s292ms, %
> >>>> non-child: 88.82%)
> >>>>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB
> 9:1/191.64
> >>>> KB 5:2/289.57 KB 3:2/1012.83 KB
> >>>>         MemoryUsage(500.000ms): 6.09 MB, 6.09 MB, 3.00 MB
> >>>>         ThreadUsage(500.000ms): 1, 1, 1
> >>>>          - AverageThreadTokens: 1.00
> >>>>          - BloomFilterBytes: 0
> >>>>          - PeakMemoryUsage: 7.17 MB (7521751)
> >>>>          - PeakReservation: 0
> >>>>          - PeakUsedReservation: 0
> >>>>          - PerHostPeakMemUsage: 106.53 MB (111709581)
> >>>>          - RowsProduced: 32.83K (32826)
> >>>>          - TotalNetworkReceiveTime: 0.000ns
> >>>>          - TotalNetworkSendTime: 1s297ms
> >>>>          - TotalStorageWaitTime: 234.356ms
> >>>>          - TotalThreadsInvoluntaryContextSwitches: 66 (66)
> >>>>          - TotalThreadsTotalWallClockTime: 1s715ms
> >>>>            - TotalThreadsSysTime: 5.998ms
> >>>>            - TotalThreadsUserTime: 124.975ms
> >>>>          - TotalThreadsVoluntaryContextSwitches: 303 (303)
> >>>>         Fragment Instance Lifecycle Timings:
> >>>>            - ExecTime: 1s394ms
> >>>>              - ExecTreeExecTime: 67.115ms
> >>>>            - OpenTime: 32.795ms
> >>>>              - ExecTreeOpenTime: 73.243us
> >>>>            - PrepareTime: 27.602ms
> >>>>              - ExecTreePrepareTime: 243.141us
> >>>>         DataStreamSender (dst_id=11):(Total: 38.747ms, non-child:
> >>>> 38.747ms, % non-child: 100.00%)
> >>>>            - BytesSent: 39.71 MB (41643000)
> >>>>            - NetworkThroughput(*): 1.97 GB/sec
> >>>>            - OverallThroughput: 1.00 GB/sec
> >>>>            - PeakMemoryUsage: 59.38 KB (60800)
> >>>>            - RowsReturned: 32.83K (32826)
> >>>>            - SerializeBatchTime: 16.860ms
> >>>>            - TransmitDataRPCTime: 19.698ms
> >>>>            - UncompressedRowBatchSize: 77.58 MB (81350840)
> >>>>         CodeGen:(Total: 56.573ms, non-child: 56.573ms, % non-child:
> >>>> 100.00%)
> >>>>            - CodegenTime: 1.299ms
> >>>>            - CompileTime: 10.672ms
> >>>>            - LoadTime: 0.000ns
> >>>>            - ModuleBitcodeSize: 1.96 MB (2050180)
> >>>>            - NumFunctions: 16 (16)
> >>>>            - NumInstructions: 250 (250)
> >>>>            - OptimizationTime: 21.023ms
> >>>>            - PeakMemoryUsage: 125.00 KB (128000)
> >>>>            - PrepareTime: 24.116ms
> >>>>         SUBPLAN_NODE (id=6):(Total: 67.311ms, non-child: 12.013ms, %
> >>>> non-child: 17.85%)
> >>>>            - PeakMemoryUsage: 627.94 KB (643015)
> >>>>            - RowsReturned: 32.77K (32768)
> >>>>            - RowsReturnedRate: 486.81 K/sec
> >>>>           NESTED_LOOP_JOIN_NODE (id=9):(Total: 33.999ms, non-child:
> >>>> 25.197ms, % non-child: 74.11%)
> >>>>              - BuildRows: 0 (0)
> >>>>              - BuildTime: 0.000ns
> >>>>              - PeakMemoryUsage: 24.00 KB (24576)
> >>>>              - ProbeRows: 32.83K (32826)
> >>>>              - ProbeTime: 0.000ns
> >>>>              - RowsReturned: 16.80M (16795311)
> >>>>              - RowsReturnedRate: 493.99 M/sec
> >>>>             Nested Loop Join Builder:
> >>>>                - PeakMemoryUsage: 8.00 KB (8192)
> >>>>             SINGULAR_ROW_SRC_NODE (id=7):
> >>>>                - PeakMemoryUsage: 0
> >>>>                - RowsReturned: 0 (0)
> >>>>                - RowsReturnedRate: 0
> >>>>           UNNEST_NODE (id=8):(Total: 8.801ms, non-child: 8.801ms, %
> >>>> non-child: 100.00%)
> >>>>              - AvgCollectionSize: 1.00
> >>>>              - MaxCollectionSize: 1 (1)
> >>>>              - MinCollectionSize: 1 (1)
> >>>>              - NumCollections: 32.83K (32826)
> >>>>              - PeakMemoryUsage: 0
> >>>>              - RowsReturned: 1 (1)
> >>>>              - RowsReturnedRate: 113.00 /sec
> >>>>         HDFS_SCAN_NODE (id=5):(Total: 21.299ms, non-child: 21.299ms, %
> >>>> non-child: 100.00%)
> >>>>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 6:1/1.20 MB 2:1/199.83 KB 1:1/1.20 MB 10:2/1.42 MB 8:1/225.57 KB
> 9:1/191.64
> >>>> KB 5:2/289.57 KB 3:2/1012.83 KB
> >>>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 11 out
> >>>> of 11
> >>>>           Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
> 16:0%
> >>>>           File Formats: PARQUET/SNAPPY:66
> >>>>           BytesRead(500.000ms): 2.26 MB, 2.26 MB, 2.26 MB
> >>>>            - FooterProcessingTime: (Avg: 12.082ms ; Min: 9.079ms ;
> Max:
> >>>> 29.275ms ; Number of samples: 11)
> >>>>            - AverageHdfsReadThreadConcurrency: 0.00
> >>>>            - AverageScannerThreadConcurrency: 0.00
> >>>>            - BytesRead: 2.26 MB (2373738)
> >>>>            - BytesReadDataNodeCache: 0
> >>>>            - BytesReadLocal: 2.26 MB (2373738)
> >>>>            - BytesReadRemoteUnexpected: 0
> >>>>            - BytesReadShortCircuit: 2.26 MB (2373738)
> >>>>            - CachedFileHandlesHitCount: 0 (0)
> >>>>            - CachedFileHandlesMissCount: 77 (77)
> >>>>            - DecompressionTime: 1.543ms
> >>>>            - MaxCompressedTextFileLength: 0
> >>>>            - NumColumns: 6 (6)
> >>>>            - NumDictFilteredRowGroups: 0 (0)
> >>>>            - NumDisksAccessed: 8 (8)
> >>>>            - NumRowGroups: 11 (11)
> >>>>            - NumScannerThreadsStarted: 11 (11)
> >>>>            - NumScannersWithNoReads: 0 (0)
> >>>>            - NumStatsFilteredRowGroups: 0 (0)
> >>>>            - PeakMemoryUsage: 6.51 MB (6821203)
> >>>>            - PerReadThreadRawHdfsThroughput: 764.51 MB/sec
> >>>>            - RemoteScanRanges: 0 (0)
> >>>>            - RowBatchQueueGetWaitTime: 18.825ms
> >>>>            - RowBatchQueuePutWaitTime: 0.000ns
> >>>>            - RowsRead: 73.46K (73464)
> >>>>            - RowsReturned: 32.83K (32826)
> >>>>            - RowsReturnedRate: 1.54 M/sec
> >>>>            - ScanRangesComplete: 11 (11)
> >>>>            - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
> >>>>            - ScannerThreadsTotalWallClockTime: 287.702ms
> >>>>              - MaterializeTupleTime(*): 11.921ms
> >>>>              - ScannerThreadsSysTime: 999.000us
> >>>>              - ScannerThreadsUserTime: 26.990ms
> >>>>            - ScannerThreadsVoluntaryContextSwitches: 232 (232)
> >>>>            - TotalRawHdfsReadTime(*): 2.961ms
> >>>>            - TotalReadThroughput: 1.51 MB/sec
> >>>>       Instance 744de1b6228736fa:b54bfaa70000000d
> >>>> (host=slpr-dhc002.lpdomain.com:22000):(Total: 1s453ms, non-child:
> 1s254ms, %
> >>>> non-child: 86.28%)
> >>>>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
> >>>>         MemoryUsage(500.000ms): 5.32 MB, 5.32 MB, 1.27 MB
> >>>>         ThreadUsage(500.000ms): 1, 1, 1
> >>>>          - AverageThreadTokens: 1.00
> >>>>          - BloomFilterBytes: 0
> >>>>          - PeakMemoryUsage: 5.86 MB (6145964)
> >>>>          - PeakReservation: 0
> >>>>          - PeakUsedReservation: 0
> >>>>          - PerHostPeakMemUsage: 106.66 MB (111842766)
> >>>>          - RowsProduced: 30.34K (30340)
> >>>>          - TotalNetworkReceiveTime: 0.000ns
> >>>>          - TotalNetworkSendTime: 1s256ms
> >>>>          - TotalStorageWaitTime: 106.530ms
> >>>>          - TotalThreadsInvoluntaryContextSwitches: 177 (177)
> >>>>          - TotalThreadsTotalWallClockTime: 1s557ms
> >>>>            - TotalThreadsSysTime: 6.998ms
> >>>>            - TotalThreadsUserTime: 143.976ms
> >>>>          - TotalThreadsVoluntaryContextSwitches: 136 (136)
> >>>>         Fragment Instance Lifecycle Timings:
> >>>>            - ExecTime: 1s390ms
> >>>>              - ExecTreeExecTime: 91.500ms
> >>>>            - OpenTime: 34.085ms
> >>>>              - ExecTreeOpenTime: 81.436us
> >>>>            - PrepareTime: 28.524ms
> >>>>              - ExecTreePrepareTime: 217.949us
> >>>>         DataStreamSender (dst_id=11):(Total: 48.674ms, non-child:
> >>>> 48.674ms, % non-child: 100.00%)
> >>>>            - BytesSent: 36.61 MB (38388760)
> >>>>            - NetworkThroughput(*): 1.51 GB/sec
> >>>>            - OverallThroughput: 752.15 MB/sec
> >>>>            - PeakMemoryUsage: 59.38 KB (60800)
> >>>>            - RowsReturned: 30.34K (30340)
> >>>>            - SerializeBatchTime: 23.132ms
> >>>>            - TransmitDataRPCTime: 23.609ms
> >>>>            - UncompressedRowBatchSize: 71.70 MB (75187520)
> >>>>         CodeGen:(Total: 59.024ms, non-child: 59.024ms, % non-child:
> >>>> 100.00%)
> >>>>            - CodegenTime: 1.382ms
> >>>>            - CompileTime: 10.435ms
> >>>>            - LoadTime: 0.000ns
> >>>>            - ModuleBitcodeSize: 1.96 MB (2050180)
> >>>>            - NumFunctions: 16 (16)
> >>>>            - NumInstructions: 250 (250)
> >>>>            - OptimizationTime: 22.325ms
> >>>>            - PeakMemoryUsage: 125.00 KB (128000)
> >>>>            - PrepareTime: 25.459ms
> >>>>         SUBPLAN_NODE (id=6):(Total: 91.711ms, non-child: 20.997ms, %
> >>>> non-child: 22.90%)
> >>>>            - PeakMemoryUsage: 699.55 KB (716335)
> >>>>            - RowsReturned: 29.70K (29696)
> >>>>            - RowsReturnedRate: 323.80 K/sec
> >>>>           NESTED_LOOP_JOIN_NODE (id=9):(Total: 53.723ms, non-child:
> >>>> 40.263ms, % non-child: 74.95%)
> >>>>              - BuildRows: 0 (0)
> >>>>              - BuildTime: 0.000ns
> >>>>              - PeakMemoryUsage: 24.00 KB (24576)
> >>>>              - ProbeRows: 30.34K (30340)
> >>>>              - ProbeTime: 0.000ns
> >>>>              - RowsReturned: 15.43M (15426890)
> >>>>              - RowsReturnedRate: 287.16 M/sec
> >>>>             Nested Loop Join Builder:
> >>>>                - PeakMemoryUsage: 8.00 KB (8192)
> >>>>             SINGULAR_ROW_SRC_NODE (id=7):
> >>>>                - PeakMemoryUsage: 0
> >>>>                - RowsReturned: 0 (0)
> >>>>                - RowsReturnedRate: 0
> >>>>           UNNEST_NODE (id=8):(Total: 13.460ms, non-child: 13.460ms, %
> >>>> non-child: 100.00%)
> >>>>              - AvgCollectionSize: 1.00
> >>>>              - MaxCollectionSize: 1 (1)
> >>>>              - MinCollectionSize: 1 (1)
> >>>>              - NumCollections: 30.34K (30340)
> >>>>              - PeakMemoryUsage: 0
> >>>>              - RowsReturned: 1 (1)
> >>>>              - RowsReturnedRate: 74.00 /sec
> >>>>         HDFS_SCAN_NODE (id=5):(Total: 16.990ms, non-child: 16.990ms, %
> >>>> non-child: 100.00%)
> >>>>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 0:1/1.32 MB 4:1/1.18 MB 5:1/1.19 MB 10:2/1.06 MB
> >>>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 out
> of
> >>>> 5
> >>>>           Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
> 16:0%
> >>>>           File Formats: PARQUET/SNAPPY:30
> >>>>           BytesRead(500.000ms): 1.56 MB, 1.56 MB, 1.56 MB
> >>>>            - FooterProcessingTime: (Avg: 12.351ms ; Min: 7.816ms ;
> Max:
> >>>> 28.687ms ; Number of samples: 5)
> >>>>            - AverageHdfsReadThreadConcurrency: 0.00
> >>>>            - AverageScannerThreadConcurrency: 0.00
> >>>>            - BytesRead: 1.56 MB (1630546)
> >>>>            - BytesReadDataNodeCache: 0
> >>>>            - BytesReadLocal: 1.56 MB (1630546)
> >>>>            - BytesReadRemoteUnexpected: 0
> >>>>            - BytesReadShortCircuit: 1.56 MB (1630546)
> >>>>            - CachedFileHandlesHitCount: 0 (0)
> >>>>            - CachedFileHandlesMissCount: 35 (35)
> >>>>            - DecompressionTime: 1.267ms
> >>>>            - MaxCompressedTextFileLength: 0
> >>>>            - NumColumns: 6 (6)
> >>>>            - NumDictFilteredRowGroups: 0 (0)
> >>>>            - NumDisksAccessed: 4 (4)
> >>>>            - NumRowGroups: 5 (5)
> >>>>            - NumScannerThreadsStarted: 5 (5)
> >>>>            - NumScannersWithNoReads: 0 (0)
> >>>>            - NumStatsFilteredRowGroups: 0 (0)
> >>>>            - PeakMemoryUsage: 5.59 MB (5859290)
> >>>>            - PerReadThreadRawHdfsThroughput: 920.32 MB/sec
> >>>>            - RemoteScanRanges: 0 (0)
> >>>>            - RowBatchQueueGetWaitTime: 15.335ms
> >>>>            - RowBatchQueuePutWaitTime: 0.000ns
> >>>>            - RowsRead: 64.69K (64692)
> >>>>            - RowsReturned: 30.34K (30340)
> >>>>            - RowsReturnedRate: 1.79 M/sec
> >>>>            - ScanRangesComplete: 5 (5)
> >>>>            - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
> >>>>            - ScannerThreadsTotalWallClockTime: 132.548ms
> >>>>              - MaterializeTupleTime(*): 10.585ms
> >>>>              - ScannerThreadsSysTime: 999.000us
> >>>>              - ScannerThreadsUserTime: 18.995ms
> >>>>            - ScannerThreadsVoluntaryContextSwitches: 79 (79)
> >>>>            - TotalRawHdfsReadTime(*): 1.689ms
> >>>>            - TotalReadThroughput: 1.04 MB/sec
> >>>>       Instance 744de1b6228736fa:b54bfaa700000002
> >>>> (host=slpr-dhc003.lpdomain.com:22000):(Total: 1s452ms, non-child:
> 1s255ms, %
> >>>> non-child: 86.41%)
> >>>>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
> >>>>         MemoryUsage(500.000ms): 5.58 MB, 5.58 MB, 3.94 MB
> >>>>         ThreadUsage(500.000ms): 1, 1, 1
> >>>>          - AverageThreadTokens: 1.00
> >>>>          - BloomFilterBytes: 0
> >>>>          - PeakMemoryUsage: 6.17 MB (6473729)
> >>>>          - PeakReservation: 0
> >>>>          - PeakUsedReservation: 0
> >>>>          - PerHostPeakMemUsage: 107.32 MB (112537640)
> >>>>          - RowsProduced: 29.97K (29969)
> >>>>          - TotalNetworkReceiveTime: 0.000ns
> >>>>          - TotalNetworkSendTime: 1s260ms
> >>>>          - TotalStorageWaitTime: 130.815ms
> >>>>          - TotalThreadsInvoluntaryContextSwitches: 15 (15)
> >>>>          - TotalThreadsTotalWallClockTime: 1s586ms
> >>>>            - TotalThreadsSysTime: 11.995ms
> >>>>            - TotalThreadsUserTime: 146.975ms
> >>>>          - TotalThreadsVoluntaryContextSwitches: 197 (197)
> >>>>         Fragment Instance Lifecycle Timings:
> >>>>            - ExecTime: 1s354ms
> >>>>              - ExecTreeExecTime: 67.791ms
> >>>>            - OpenTime: 52.012ms
> >>>>              - ExecTreeOpenTime: 101.232us
> >>>>            - PrepareTime: 45.946ms
> >>>>              - ExecTreePrepareTime: 309.644us
> >>>>         DataStreamSender (dst_id=11):(Total: 38.995ms, non-child:
> >>>> 38.995ms, % non-child: 100.00%)
> >>>>            - BytesSent: 35.95 MB (37697260)
> >>>>            - NetworkThroughput(*): 2.14 GB/sec
> >>>>            - OverallThroughput: 921.92 MB/sec
> >>>>            - PeakMemoryUsage: 59.38 KB (60800)
> >>>>            - RowsReturned: 29.97K (29969)
> >>>>            - SerializeBatchTime: 18.055ms
> >>>>            - TransmitDataRPCTime: 16.428ms
> >>>>            - UncompressedRowBatchSize: 70.83 MB (74269800)
> >>>>         CodeGen:(Total: 90.385ms, non-child: 90.385ms, % non-child:
> >>>> 100.00%)
> >>>>            - CodegenTime: 2.224ms
> >>>>            - CompileTime: 15.517ms
> >>>>            - LoadTime: 0.000ns
> >>>>            - ModuleBitcodeSize: 1.96 MB (2050180)
> >>>>            - NumFunctions: 16 (16)
> >>>>            - NumInstructions: 250 (250)
> >>>>            - OptimizationTime: 35.099ms
> >>>>            - PeakMemoryUsage: 125.00 KB (128000)
> >>>>            - PrepareTime: 38.847ms
> >>>>         SUBPLAN_NODE (id=6):(Total: 68.126ms, non-child: 12.625ms, %
> >>>> non-child: 18.53%)
> >>>>            - PeakMemoryUsage: 636.17 KB (651439)
> >>>>            - RowsReturned: 29.70K (29696)
> >>>>            - RowsReturnedRate: 435.89 K/sec
> >>>>           NESTED_LOOP_JOIN_NODE (id=9):(Total: 35.347ms, non-child:
> >>>> 27.325ms, % non-child: 77.31%)
> >>>>              - BuildRows: 0 (0)
> >>>>              - BuildTime: 0.000ns
> >>>>              - PeakMemoryUsage: 24.00 KB (24576)
> >>>>              - ProbeRows: 29.97K (29969)
> >>>>              - ProbeTime: 0.000ns
> >>>>              - RowsReturned: 15.26M (15256601)
> >>>>              - RowsReturnedRate: 431.62 M/sec
> >>>>             Nested Loop Join Builder:
> >>>>                - PeakMemoryUsage: 8.00 KB (8192)
> >>>>             SINGULAR_ROW_SRC_NODE (id=7):
> >>>>                - PeakMemoryUsage: 0
> >>>>                - RowsReturned: 0 (0)
> >>>>                - RowsReturnedRate: 0
> >>>>           UNNEST_NODE (id=8):(Total: 8.021ms, non-child: 8.021ms, %
> >>>> non-child: 100.00%)
> >>>>              - AvgCollectionSize: 1.00
> >>>>              - MaxCollectionSize: 1 (1)
> >>>>              - MinCollectionSize: 1 (1)
> >>>>              - NumCollections: 29.97K (29969)
> >>>>              - PeakMemoryUsage: 0
> >>>>              - RowsReturned: 1 (1)
> >>>>              - RowsReturnedRate: 124.00 /sec
> >>>>         HDFS_SCAN_NODE (id=5):(Total: 20.153ms, non-child: 20.153ms, %
> >>>> non-child: 100.00%)
> >>>>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 0:1/875.15 KB 10:1/927.27 KB 6:1/1.07 MB 9:1/215.54 KB 8:3/1.84 MB
> >>>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out
> of
> >>>> 7
> >>>>           Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
> 16:0%
> >>>>           File Formats: PARQUET/SNAPPY:42
> >>>>           BytesRead(500.000ms): 1.75 MB, 1.75 MB, 1.75 MB
> >>>>            - FooterProcessingTime: (Avg: 8.683ms ; Min: 3.937ms ; Max:
> >>>> 10.464ms ; Number of samples: 7)
> >>>>            - AverageHdfsReadThreadConcurrency: 0.00
> >>>>            - AverageScannerThreadConcurrency: 0.00
> >>>>            - BytesRead: 1.75 MB (1833560)
> >>>>            - BytesReadDataNodeCache: 0
> >>>>            - BytesReadLocal: 1.75 MB (1833560)
> >>>>            - BytesReadRemoteUnexpected: 0
> >>>>            - BytesReadShortCircuit: 1.75 MB (1833560)
> >>>>            - CachedFileHandlesHitCount: 0 (0)
> >>>>            - CachedFileHandlesMissCount: 49 (49)
> >>>>            - DecompressionTime: 1.758ms
> >>>>            - MaxCompressedTextFileLength: 0
> >>>>            - NumColumns: 6 (6)
> >>>>            - NumDictFilteredRowGroups: 0 (0)
> >>>>            - NumDisksAccessed: 5 (5)
> >>>>            - NumRowGroups: 7 (7)
> >>>>            - NumScannerThreadsStarted: 7 (7)
> >>>>            - NumScannersWithNoReads: 0 (0)
> >>>>            - NumStatsFilteredRowGroups: 0 (0)
> >>>>            - PeakMemoryUsage: 5.82 MB (6098991)
> >>>>            - PerReadThreadRawHdfsThroughput: 621.09 MB/sec
> >>>>            - RemoteScanRanges: 0 (0)
> >>>>            - RowBatchQueueGetWaitTime: 17.938ms
> >>>>            - RowBatchQueuePutWaitTime: 0.000ns
> >>>>            - RowsRead: 65.17K (65166)
> >>>>            - RowsReturned: 29.97K (29969)
> >>>>            - RowsReturnedRate: 1.49 M/sec
> >>>>            - ScanRangesComplete: 7 (7)
> >>>>            - ScannerThreadsInvoluntaryContextSwitches: 1 (1)
> >>>>            - ScannerThreadsTotalWallClockTime: 179.698ms
> >>>>              - MaterializeTupleTime(*): 13.922ms
> >>>>              - ScannerThreadsSysTime: 2.997ms
> >>>>              - ScannerThreadsUserTime: 29.993ms
> >>>>            - ScannerThreadsVoluntaryContextSwitches: 130 (130)
> >>>>            - TotalRawHdfsReadTime(*): 2.815ms
> >>>>            - TotalReadThroughput: 1.17 MB/sec
> >>>>       Instance 744de1b6228736fa:b54bfaa700000008
> >>>> (host=slpr-dhc009.lpdomain.com:22000):(Total: 1s450ms, non-child:
> 1s305ms, %
> >>>> non-child: 90.01%)
> >>>>         Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
> >>>>         MemoryUsage(500.000ms): 5.97 MB, 5.88 MB, 2.24 MB
> >>>>         ThreadUsage(500.000ms): 1, 1, 1
> >>>>          - AverageThreadTokens: 1.00
> >>>>          - BloomFilterBytes: 0
> >>>>          - PeakMemoryUsage: 6.69 MB (7017157)
> >>>>          - PeakReservation: 0
> >>>>          - PeakUsedReservation: 0
> >>>>          - PerHostPeakMemUsage: 106.39 MB (111556990)
> >>>>          - RowsProduced: 42.68K (42681)
> >>>>          - TotalNetworkReceiveTime: 0.000ns
> >>>>          - TotalNetworkSendTime: 1s315ms
> >>>>          - TotalStorageWaitTime: 201.411ms
> >>>>          - TotalThreadsInvoluntaryContextSwitches: 43 (43)
> >>>>          - TotalThreadsTotalWallClockTime: 1s658ms
> >>>>            - TotalThreadsSysTime: 6.997ms
> >>>>            - TotalThreadsUserTime: 108.980ms
> >>>>          - TotalThreadsVoluntaryContextSwitches: 179 (179)
> >>>>         Fragment Instance Lifecycle Timings:
> >>>>            - ExecTime: 1s411ms
> >>>>              - ExecTreeExecTime: 68.017ms
> >>>>            - OpenTime: 19.935ms
> >>>>              - ExecTreeOpenTime: 48.883us
> >>>>            - PrepareTime: 19.160ms
> >>>>              - ExecTreePrepareTime: 134.576us
> >>>>         DataStreamSender (dst_id=11):(Total: 40.671ms, non-child:
> >>>> 40.671ms, % non-child: 100.00%)
> >>>>            - BytesSent: 51.36 MB (53856680)
> >>>>            - NetworkThroughput(*): 2.44 GB/sec
> >>>>            - OverallThroughput: 1.23 GB/sec
> >>>>            - PeakMemoryUsage: 59.38 KB (60800)
> >>>>            - RowsReturned: 42.68K (42681)
> >>>>            - SerializeBatchTime: 18.299ms
> >>>>            - TransmitDataRPCTime: 20.591ms
> >>>>            - UncompressedRowBatchSize: 100.88 MB (105782860)
> >>>>         CodeGen:(Total: 36.171ms, non-child: 36.171ms, % non-child:
> >>>> 100.00%)
> >>>>            - CodegenTime: 828.865us
> >>>>            - CompileTime: 6.170ms
> >>>>            - LoadTime: 0.000ns
> >>>>            - ModuleBitcodeSize: 1.96 MB (2050180)
> >>>>            - NumFunctions: 16 (16)
> >>>>            - NumInstructions: 250 (250)
> >>>>            - OptimizationTime: 12.988ms
> >>>>            - PeakMemoryUsage: 125.00 KB (128000)
> >>>>            - PrepareTime: 16.608ms
> >>>>         SUBPLAN_NODE (id=6):(Total: 68.046ms, non-child: 12.434ms, %
> >>>> non-child: 18.27%)
> >>>>            - PeakMemoryUsage: 689.95 KB (706507)
> >>>>            - RowsReturned: 41.98K (41984)
> >>>>            - RowsReturnedRate: 616.99 K/sec
> >>>>           NESTED_LOOP_JOIN_NODE (id=9):(Total: 39.837ms, non-child:
> >>>> 29.422ms, % non-child: 73.86%)
> >>>>              - BuildRows: 0 (0)
> >>>>              - BuildTime: 0.000ns
> >>>>              - PeakMemoryUsage: 24.00 KB (24576)
> >>>>              - ProbeRows: 42.68K (42681)
> >>>>              - ProbeTime: 0.000ns
> >>>>              - RowsReturned: 21.76M (21760053)
> >>>>              - RowsReturnedRate: 546.22 M/sec
> >>>>             Nested Loop Join Builder:
> >>>>                - PeakMemoryUsage: 8.00 KB (8192)
> >>>>             SINGULAR_ROW_SRC_NODE (id=7):
> >>>>                - PeakMemoryUsage: 0
> >>>>                - RowsReturned: 0 (0)
> >>>>                - RowsReturnedRate: 0
> >>>>           UNNEST_NODE (id=8):(Total: 10.415ms, non-child: 10.415ms, %
> >>>> non-child: 100.00%)
> >>>>              - AvgCollectionSize: 1.00
> >>>>              - MaxCollectionSize: 1 (1)
> >>>>              - MinCollectionSize: 1 (1)
> >>>>              - NumCollections: 42.68K (42681)
> >>>>              - PeakMemoryUsage: 0
> >>>>              - RowsReturned: 1 (1)
> >>>>              - RowsReturnedRate: 96.00 /sec
> >>>>         HDFS_SCAN_NODE (id=5):(Total: 15.774ms, non-child: 15.774ms, %
> >>>> non-child: 100.00%)
> >>>>           Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >>>> 7:1/1.08 MB 6:2/2.52 MB 3:2/2.19 MB 8:1/810.47 KB
> >>>>           ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out
> of
> >>>> 6
> >>>>           Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
> >>>> 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
> 16:0%
> >>>> 17:0%
> >>>>           File Formats: PARQUET/SNAPPY:36
> >>>>           BytesRead(500.000ms): 2.08 MB, 2.08 MB, 2.08 MB
> >>>>            - FooterProcessingTime: (Avg: 22.052ms ; Min: 7.783ms ;
> Max:
> >>>> 42.732ms ; Number of samples: 6)
> >>>>            - AverageHdfsReadThreadConcurrency: 0.00
> >>>>            - AverageScannerThreadConcurrency: 0.00
> >>>>            - BytesRead: 2.08 MB (2186159)
> >>>>            - BytesReadDataNodeCache: 0
> >>>>            - BytesReadLocal: 2.08 MB (2186159)
> >>>>            - BytesReadRemoteUnexpected: 0
> >>>>            - BytesReadShortCircuit: 2.08 MB (2186159)
> >>>>            - CachedFileHandlesHitCount: 0 (0)
> >>>>            - CachedFileHandlesMissCount: 42 (42)
> >>>>            - DecompressionTime: 1.480ms
> >>>>            - MaxCompressedTextFileLength: 0
> >>>>            - NumColumns: 6 (6)
> >>>>            - NumDictFilteredRowGroups: 0 (0)
> >>>>            - NumDisksAccessed: 4 (4)
> >>>>            - NumRowGroups: 6 (6)
> >>>>            - NumScannerThreadsStarted: 6 (6)
> >>>>            - NumScannersWithNoReads: 0 (0)
> >>>>            - NumStatsFilteredRowGroups: 0 (0)
> >>>>            - PeakMemoryUsage: 6.42 MB (6730467)
> >>>>            - PerReadThreadRawHdfsThroughput: 77.02 MB/sec
> >>>>            - RemoteScanRanges: 0 (0)
> >>>>            - RowBatchQueueGetWaitTime: 13.979ms
> >>>>            - RowBatchQueuePutWaitTime: 0.000ns
> >>>>            - RowsRead: 89.91K (89910)
> >>>>            - RowsReturned: 42.68K (42681)
> >>>>            - RowsReturnedRate: 2.71 M/sec
> >>>>            - ScanRangesComplete: 6 (6)
> >>>>            - ScannerThreadsInvoluntaryContextSwitches: 3 (3)
> >>>>            - ScannerThreadsTotalWallClockTime: 227.825ms
> >>>>
>

Reply via email to