Hi Community,
I'm looking on issue with CM and Impala where i see in impala UI the query
still executing and in the query state it shown as FINISHED
Looking at the query profile ( attached here), i see the query has no
finished time and from the other side i see the first rows fecthed and all
rows available.
What i'm intersting in here to know if such query still hanging some impala
resources or not, and why my configuration didn't help here and the query
wasn't killed.
BTW:
I have the configuration of:
Impala Command Line Argument Advanced Configuration Snippet (Safety Valve)
Impala (Service-Wide)
idle_session_timeout=600
idle_query_timeout=600
[image: image.png]
Part of the query profile:
Query Timeline
Query submitted: 119.16us (119160)
Planning finished: 482ms (482689616)
Ready to start on 7 backends: 561ms (561747089)
All 7 execution backends (8 fragment instances) started: 581ms
(581185065)
* Rows available: 34.73s (34732649614)*
* First row fetched: 36.81s (36806528086)*
--
Take Care
Fawze Abujaber
Query (id=434ed55d986c7a61:10605b1b00000000)
Summary
Session ID: 5f46fa06711e153c:f4f15670b074638b
Session Type: HIVESERVER2
HiveServer2 Protocol Version: V6
Start Time: 2018-11-20 19:41:56.291766000
End Time:
Query Type: QUERY
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 2.10.0-cdh5.13.0 RELEASE (build
2511805f1eaa991df1460276c7e9f19d819cd4e4)
User: archanav
Connected User: archanav
Delegated User:
Default Db: default
Coordinator: xxxxxxxxxxxxxxxxxxxx:22000
Query Options (set by configuration): SYNC_DDL=1,QUERY_TIMEOUT_S=600
Query Options (set by configuration and planner):
SYNC_DDL=1,QUERY_TIMEOUT_S=600,MT_DOP=0
Query Timeline
Query submitted: 119.16us (119160)
Planning finished: 482ms (482689616)
Ready to start on 7 backends: 561ms (561747089)
All 7 execution backends (8 fragment instances) started: 581ms (581185065)
Rows available: 34.73s (34732649614)
First row fetched: 36.81s (36806528086)
Plan:
----------------
Max Per-Host Resource Reservation: Memory=12.00MB
Per-Host Resource Estimates: Memory=92.00MB
F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
| Per-Host Resources: mem-estimate=0B mem-reservation=0B
PLAN-ROOT SINK
| mem-estimate=0B mem-reservation=0B
|
02:MERGING-EXCHANGE [UNPARTITIONED]
| order by:
| mem-estimate=0B mem-reservation=0B
| tuple-ids=1 row-size=68B cardinality=unavailable
|
F00:PLAN FRAGMENT [RANDOM] hosts=7 instances=7
Per-Host Resources: mem-estimate=92.00MB mem-reservation=12.00MB
01:SORT
| order by:
| mem-estimate=12.00MB mem-reservation=12.00MB spill-buffer=2.00MB
| tuple-ids=1 row-size=68B cardinality=unavailable
|
00:SCAN HDFS [default.dv_messagepublishevent_prq_local msg, RANDOM]
partitions=501/622 files=27038 size=6.56GB
predicates:
stats-rows=unavailable extrapolated-rows=disabled
table stats: rows=unavailable size=unavailable
column stats: all
parquet dictionary predicates:
mem-estimate=80.00MB mem-reservation=0B
tuple-ids=0 row-size=68B cardinality=unavailable
----------------
Estimated Per-Host Mem: 96468992
Tables Missing Stats: default.dv_messagepublishevent_prq_local
Per Host Min Reservation: xxxxxxxxxxxxxx:22000(12.00 MB)
bbbbbbbbbbbbbbbbbbbb:22000(12.00 MB) ccccccccccccccccccccc:22000(12.00 MB)
dddddddddddddddd:22000(12.00 MB) eeeeeeeeeeeeeeeeee:22000(12.00 MB)
ffffffffffffffffff:22000(12.00 MB) ggggggggggggggg:22000(12.00 MB)
Request Pool: default-pool
- ComputeScanRangeAssignmentTimer: 42ms (42482994)
- InactiveTotalTime: 0ns (0)
- TotalTime: 0ns (0)
Planner Timeline
Analysis finished: 17ms (17384392)
Equivalence classes computed: 17ms (17606164)
Single node plan created: 367ms (367124182)
Runtime filters computed: 367ms (367185164)
Distributed plan created: 367ms (367366629)
Lineage info computed: 367ms (367547180)
Planning finished: 394ms (394287091)
ImpalaServer
- ClientFetchWaitTimer: 2.07s (2073991745)
- InactiveTotalTime: 0ns (0)
- RowMaterializationTimer: 1ms (1793160)
- TotalTime: 0ns (0)
Execution Profile 434ed55d986c7a61:10605b1b00000000
Number of filters: 0
Filter routing table:
ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected)
First arrived Completed Enabled
-------------------------------------------------------------------------------------------------------------------
Backend startup latencies: Count: 7, min / max: 9ms / 12ms, 25th %-ile:
10ms, 50th %-ile: 10ms, 75th %-ile: 11ms, 90th %-ile: 12ms, 95th %-ile: 12ms,
99.9th %-ile: 12ms
- FiltersReceived: 0 (0)
- FinalizationTimer: 0ns (0)
- InactiveTotalTime: 0ns (0)
- TotalTime: 34.17s (34171231035)
Averaged Fragment F01
split sizes: min: 0, max: 0, avg: 0, stddev: 0
completion times: min:36s231ms max:36s231ms mean: 36s231ms
stddev:0.000ns
execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 0.00 (0.0)
- BloomFilterBytes: 0 B (0)
- InactiveTotalTime: 0ns (0)
- PeakMemoryUsage: 241.5 KiB (247296)
- PeakReservation: 0 B (0)
- PeakUsedReservation: 0 B (0)
- PerHostPeakMemUsage: 14.8 MiB (15544253)
- RowsProduced: 60 (60)
- TotalNetworkReceiveTime: 0ns (0)
- TotalNetworkSendTime: 0ns (0)
- TotalStorageWaitTime: 0ns (0)
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 36.21s (36206328622)
- TotalThreadsSysTime: 0ns (0)
- TotalThreadsUserTime: 10ms (10998000)
- TotalThreadsVoluntaryContextSwitches: 4 (4)
- TotalTime: 36.23s (36227654799)
Fragment Instance Lifecycle Timings
- ExecTime: 2.07s (2074245151)
- ExecTreeExecTime: 27.41us (27415)
- InactiveTotalTime: 0ns (0)
- OpenTime: 34.13s (34132092177)
- ExecTreeOpenTime: 34.12s (34120830332)
- PrepareTime: 21ms (21237305)
- ExecTreePrepareTime: 57.85us (57855)
- TotalTime: 0ns (0)
PLAN_ROOT_SINK
- InactiveTotalTime: 0ns (0)
- PeakMemoryUsage: 0 B (0)
- TotalTime: 0ns (0)
CodeGen
- CodegenTime: 964.47us (964469)
- CompileTime: 3ms (3113371)
- InactiveTotalTime: 0ns (0)
- LoadTime: 0ns (0)
- ModuleBitcodeSize: 2.0 MiB (2050180)
- NumFunctions: 25 (25)
- NumInstructions: 467 (467)
- OptimizationTime: 7ms (7542323)
- PeakMemoryUsage: 233.5 KiB (239104)
- PrepareTime: 20ms (20821669)
- TotalTime: 32ms (32036786)
EXCHANGE_NODE (id=2)
- ConvertRowBatchTime: 0ns (0)
- InactiveTotalTime: 0ns (0)
- MergeGetNext: 21.83us (21833)
- MergeGetNextBatch: 7.24us (7239)
- PeakMemoryUsage: 0 B (0)
- RowsReturned: 60 (60)
- RowsReturnedRate: 1 per second (1)
- TotalTime: 34.12s (34120854130)
DataStreamReceiver
- BytesReceived: 9.6 KiB (9845)
- DeserializeRowBatchTimer: 82.66us (82660)
- FirstBatchArrivalWaitTime: 34.12s (34120728155)
- InactiveTotalTime: 0ns (0)
- PeakMemoryUsage: 26.3 KiB (26931)
- SendersBlockedTimer: 0ns (0)
- SendersBlockedTotalTimer(*): 0ns (0)
- TotalTime: 0ns (0)
Coordinator Fragment F01
Instance 434ed55d986c7a61:10605b1b00000000
(host=sspr-dhc001.lpdomain.com:22000)
- AverageThreadTokens: 0.00 (0.0)
- BloomFilterBytes: 0 B (0)
- InactiveTotalTime: 0ns (0)
- PeakMemoryUsage: 241.5 KiB (247296)
- PeakReservation: 0 B (0)
- PeakUsedReservation: 0 B (0)
- PerHostPeakMemUsage: 14.8 MiB (15544253)
- RowsProduced: 60 (60)
- TotalNetworkReceiveTime: 0ns (0)
- TotalNetworkSendTime: 0ns (0)
- TotalStorageWaitTime: 0ns (0)
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 36.21s (36206328622)
- TotalThreadsSysTime: 0ns (0)
- TotalThreadsUserTime: 10ms (10998000)
- TotalThreadsVoluntaryContextSwitches: 4 (4)
- TotalTime: 36.23s (36227654799)
Fragment Instance Lifecycle Timings
- ExecTime: 2.07s (2074245151)
- ExecTreeExecTime: 27.41us (27415)
- InactiveTotalTime: 0ns (0)
- OpenTime: 34.13s (34132092177)
- ExecTreeOpenTime: 34.12s (34120830332)
- PrepareTime: 21ms (21237305)
- ExecTreePrepareTime: 57.85us (57855)
- TotalTime: 0ns (0)
PLAN_ROOT_SINK
- InactiveTotalTime: 0ns (0)
- PeakMemoryUsage: 0 B (0)
- TotalTime: 0ns (0)
CodeGen
- CodegenTime: 964.47us (964469)
- CompileTime: 3ms (3113371)
- InactiveTotalTime: 0ns (0)
- LoadTime: 0ns (0)
- ModuleBitcodeSize: 2.0 MiB (2050180)
- NumFunctions: 25 (25)
- NumInstructions: 467 (467)
- OptimizationTime: 7ms (7542323)
- PeakMemoryUsage: 233.5 KiB (239104)
- PrepareTime: 20ms (20821669)
- TotalTime: 32ms (32036786)
EXCHANGE_NODE (id=2)
ExecOption: Codegen Enabled
- ConvertRowBatchTime: 0ns (0)
- InactiveTotalTime: 0ns (0)
- MergeGetNext: 21.83us (21833)
- MergeGetNextBatch: 7.24us (7239)
- PeakMemoryUsage: 0 B (0)
- RowsReturned: 60 (60)
- RowsReturnedRate: 1 per second (1)
- TotalTime: 34.12s (34120854130)
DataStreamReceiver
- BytesReceived: 9.6 KiB (9845)
- DeserializeRowBatchTimer: 82.66us (82660)
- FirstBatchArrivalWaitTime: 34.12s (34120728155)
- InactiveTotalTime: 34.12s (34120732570)
- PeakMemoryUsage: 26.3 KiB (26931)
- SendersBlockedTimer: 0ns (0)
- SendersBlockedTotalTimer(*): 0ns (0)
- TotalTime: 0ns (0)
Averaged Fragment F00
split sizes: min: 958.39 MB, max: 960.62 MB, avg: 959.50 MB, stddev:
822.24 KB
completion times: min:23s448ms max:34s156ms mean: 29s373ms
stddev:3s105ms
execution rates: min:28.11 MB/sec max:40.97 MB/sec mean:33.06 MB/sec
stddev:3.76 MB/sec
num instances: 7
- AverageThreadTokens: 24.93 (24.934463440548633)
- BloomFilterBytes: 0 B (0)
- InactiveTotalTime: 0ns (0)
- PeakMemoryUsage: 14.9 MiB (15580019)
- PeakReservation: 12.0 MiB (12582912)
- PeakUsedReservation: 0 B (0)
- PerHostPeakMemUsage: 14.9 MiB (15581202)
- RowsProduced: 8 (8)
- TotalNetworkReceiveTime: 0ns (0)
- TotalNetworkSendTime: 279.85us (279852)
- TotalStorageWaitTime: 10.1m (603258428717)
- TotalThreadsInvoluntaryContextSwitches: 116 (116)
- TotalThreadsTotalWallClockTime: 12.2m (730142241932)
- TotalThreadsSysTime: 323ms (323226571)
- TotalThreadsUserTime: 5.45s (5451016571)
- TotalThreadsVoluntaryContextSwitches: 32,024 (32024)
- TotalTime: 29.37s (29370631116)
Fragment Instance Lifecycle Timings
- ExecTime: 458.56us (458560)
- ExecTreeExecTime: 6.29us (6288)
- InactiveTotalTime: 0ns (0)
- OpenTime: 29.34s (29342861986)
- ExecTreeOpenTime: 29.32s (29316302918)
- PrepareTime: 27ms (27277796)
- ExecTreePrepareTime: 9ms (9851189)
- TotalTime: 0ns (0)
DataStreamSender (dst_id=2)
- BytesSent: 1.4 KiB (1406)
- InactiveTotalTime: 0ns (0)
- NetworkThroughput(*): 66.4 MiB/s (69592906)
- OverallThroughput: 9.5 MiB/s (9944432)
- PeakMemoryUsage: 1.9 KiB (1896)
- RowsReturned: 8 (8)
- SerializeBatchTime: 16.23us (16228)
- TotalTime: 145.79us (145792)
- TransmitDataRPCTime: 18.23us (18233)
- UncompressedRowBatchSize: 2.6 KiB (2694)
CodeGen
- CodegenTime: 1ms (1487994)
- CompileTime: 7ms (7574439)
- InactiveTotalTime: 0ns (0)
- LoadTime: 0ns (0)
- ModuleBitcodeSize: 2.0 MiB (2050180)
- NumFunctions: 42 (42)
- NumInstructions: 815 (815)
- OptimizationTime: 18ms (18423670)
- PeakMemoryUsage: 407.5 KiB (417280)
- PrepareTime: 16ms (16346011)
- TotalTime: 42ms (42791987)
SORT_NODE (id=1)
- InMemorySortTime: 2.52us (2520)
- InactiveTotalTime: 0ns (0)
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 12.3 MiB (12948114)
- RowsReturned: 8 (8)
- RowsReturnedRate: 0 per second (0)
- SortDataSize: 2.6 KiB (2656)
- SpilledRuns: 0 (0)
- TotalMergesPerformed: 0 (0)
- TotalTime: 29.33s (29326156801)
Buffer pool
- AllocTime: 37.04us (37038)
- CumulativeAllocationBytes: 6.0 MiB (6291456)
- CumulativeAllocations: 3 (3)
- InactiveTotalTime: 0ns (0)
- PeakReservation: 12.0 MiB (12582912)
- PeakUnpinnedBytes: 0 B (0)
- PeakUsedReservation: 6.0 MiB (6291456)
- ReadIoBytes: 0 B (0)
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0ns (0)
- TotalTime: 0ns (0)
- WriteIoBytes: 0 B (0)
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0ns (0)
HDFS_SCAN_NODE (id=0)
- AverageHdfsReadThreadConcurrency: 0.03 (0.03416677290872328)
- AverageScannerThreadConcurrency: 23.93 (23.934463440548633)
- BytesRead: 392.2 MiB (411244226)
- BytesReadDataNodeCache: 0 B (0)
- BytesReadLocal: 392.2 MiB (411244226)
- BytesReadRemoteUnexpected: 0 B (0)
- BytesReadShortCircuit: 392.2 MiB (411244226)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 9,783 (9783)
- DecompressionTime: 150.76us (150760)
- InactiveTotalTime: 0ns (0)
- MaxCompressedTextFileLength: 0 B (0)
- NumColumns: 2 (2)
- NumDictFilteredRowGroups: 3,861 (3861)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 3,862 (3862)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.8 MiB (2978002)
- PerReadThreadRawHdfsThroughput: 391.1 MiB/s (410066552)
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 29.26s (29256476209)
- RowBatchQueuePutWaitTime: 0ns (0)
- RowsRead: 1,990 (1990)
- RowsReturned: 8 (8)
- RowsReturnedRate: 0 per second (0)
- ScanRangesComplete: 3,862 (3862)
- ScannerThreadsInvoluntaryContextSwitches: 116 (116)
- ScannerThreadsTotalWallClockTime: 11.7m (700798986954)
- MaterializeTupleTime(*): 683.84us (683843)
- ScannerThreadsSysTime: 298ms (298658714)
- ScannerThreadsUserTime: 5.34s (5344889857)
- ScannerThreadsVoluntaryContextSwitches: 28,209 (28209)
- TotalRawHdfsReadTime(*): 1.03s (1033840590)
- TotalReadThroughput: 13.4 MiB/s (14029199)
- TotalTime: 29.32s (29316875770)
Fragment F00
Instance 434ed55d986c7a61:10605b1b00000004
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:244/70.06
MB 9:329/79.58 MB 3:383/84.46 MB 8:329/77.32 MB 5:348/80.98 MB 10:334/87.74 MB
4:325/70.81 MB 2:326/74.79 MB 1:325/83.41 MB 6:361/73.97 MB 7:390/98.61 MB
11:315/78.33 MB
- AverageThreadTokens: 24.79 (24.794117647058822)
- BloomFilterBytes: 0 B (0)
- InactiveTotalTime: 0ns (0)
- PeakMemoryUsage: 14.7 MiB (15449220)
- PeakReservation: 12.0 MiB (12582912)
- PeakUsedReservation: 0 B (0)
- PerHostPeakMemUsage: 14.7 MiB (15449220)
- RowsProduced: 6 (6)
- TotalNetworkReceiveTime: 0ns (0)
- TotalNetworkSendTime: 327.62us (327615)
- TotalStorageWaitTime: 11.8m (709721877112)
- TotalThreadsInvoluntaryContextSwitches: 109 (109)
- TotalThreadsTotalWallClockTime: 14.1m (845388554572)
- TotalThreadsSysTime: 338ms (338938000)
- TotalThreadsUserTime: 5.75s (5749111000)
- TotalThreadsVoluntaryContextSwitches: 32,298 (32298)
- TotalTime: 34.15s (34153773375)
Fragment Instance Lifecycle Timings
- ExecTime: 480.64us (480639)
- ExecTreeExecTime: 8.29us (8288)
- InactiveTotalTime: 0ns (0)
- OpenTime: 34.13s (34126126184)
- ExecTreeOpenTime: 34.10s (34099850647)
- PrepareTime: 27ms (27138592)
- ExecTreePrepareTime: 9ms (9801193)
- TotalTime: 0ns (0)
DataStreamSender (dst_id=2)
- BytesSent: 663 B (663)
- InactiveTotalTime: 0ns (0)
- NetworkThroughput(*): 19.7 MiB/s (20645201)
- OverallThroughput: 4.3 MiB/s (4493361)
- PeakMemoryUsage: 1.9 KiB (1896)
- RowsReturned: 6 (6)
- SerializeBatchTime: 16.40us (16404)
- TotalTime: 147.55us (147551)
- TransmitDataRPCTime: 32.11us (32114)
- UncompressedRowBatchSize: 1.3 KiB (1374)
CodeGen
- CodegenTime: 1ms (1601836)
- CompileTime: 7ms (7605539)
- InactiveTotalTime: 0ns (0)
- LoadTime: 0ns (0)
- ModuleBitcodeSize: 2.0 MiB (2050180)
- NumFunctions: 42 (42)
- NumInstructions: 815 (815)
- OptimizationTime: 17ms (17931014)
- PeakMemoryUsage: 407.5 KiB (417280)
- PrepareTime: 16ms (16313047)
- TotalTime: 42ms (42317621)
SORT_NODE (id=1)
ExecOption: Codegen Enabled
- InMemorySortTime: 2.97us (2974)
- InactiveTotalTime: 0ns (0)
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 12.4 MiB (12961734)
- RowsReturned: 6 (6)
- RowsReturnedRate: 0 per second (0)
- SortDataSize: 1.3 KiB (1346)
- SpilledRuns: 0 (0)
- TotalMergesPerformed: 0 (0)
- TotalTime: 34.11s (34109655674)
Buffer pool
- AllocTime: 37.99us (37989)
- CumulativeAllocationBytes: 6.0 MiB (6291456)
- CumulativeAllocations: 3 (3)
- InactiveTotalTime: 0ns (0)
- PeakReservation: 12.0 MiB (12582912)
- PeakUnpinnedBytes: 0 B (0)
- PeakUsedReservation: 6.0 MiB (6291456)
- ReadIoBytes: 0 B (0)
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0ns (0)
- TotalTime: 0ns (0)
- WriteIoBytes: 0 B (0)
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0ns (0)
HDFS_SCAN_NODE (id=0)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:244/70.06 MB 9:329/79.58 MB 3:383/84.46 MB 8:329/77.32 MB 5:348/80.98 MB
10:334/87.74 MB 4:325/70.81 MB 2:326/74.79 MB 1:325/83.41 MB 6:361/73.97 MB
7:390/98.61 MB 11:315/78.33 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 4009 out of
4009
Hdfs Read Thread Concurrency Bucket: 0:97.06% 1:1.471% 2:1.471%
3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
17:0%
File Formats: PARQUET/NONE:5937 PARQUET/SNAPPY:5992
- AverageHdfsReadThreadConcurrency: 0.04 (0.04411764705882353)
- AverageScannerThreadConcurrency: 23.79 (23.794117647058822)
- BytesRead: 404.9 MiB (424567068)
- BytesReadDataNodeCache: 0 B (0)
- BytesReadLocal: 404.9 MiB (424567068)
- BytesReadRemoteUnexpected: 0 B (0)
- BytesReadShortCircuit: 404.9 MiB (424567068)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 10,000 (10000)
- DecompressionTime: 160.16us (160163)
- InactiveTotalTime: 0ns (0)
- MaxCompressedTextFileLength: 0 B (0)
- NumColumns: 5 (5)
- NumDictFilteredRowGroups: 4,008 (4008)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 4,009 (4009)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.7 MiB (2847203)
- PerReadThreadRawHdfsThroughput: 339.9 MiB/s (356381258)
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 34.03s (34035165850)
- RowBatchQueuePutWaitTime: 0ns (0)
- RowsRead: 1,987 (1987)
- RowsReturned: 6 (6)
- RowsReturnedRate: 0 per second (0)
- ScanRangesComplete: 4,009 (4009)
- ScannerThreadsInvoluntaryContextSwitches: 109 (109)
- ScannerThreadsTotalWallClockTime: 13.5m (811262022134)
- MaterializeTupleTime(*): 572.00us (572002)
- ScannerThreadsSysTime: 310ms (310942000)
- ScannerThreadsUserTime: 5.64s (5639128000)
- ScannerThreadsVoluntaryContextSwitches: 28,319 (28319)
- TotalRawHdfsReadTime(*): 1.19s (1191328268)
- TotalReadThroughput: 11.9 MiB/s (12482486)
- TotalTime: 34.10s (34098799756)
Instance 434ed55d986c7a61:10605b1b00000003 ()
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 5:350/80.36
MB 1:376/93.26 MB 3:333/79.58 MB 9:266/68.96 MB 4:334/79.80 MB 11:248/59.56 MB
8:340/81.01 MB 6:343/79.52 MB 7:351/86.40 MB 10:316/86.61 MB 0:331/80.14 MB
2:355/85.24 MB
- AverageThreadTokens: 25.00 (25.0)
- BloomFilterBytes: 0 B (0)
- InactiveTotalTime: 0ns (0)
- PeakMemoryUsage: 14.9 MiB (15629030)
- PeakReservation: 12.0 MiB (12582912)
- PeakUsedReservation: 0 B (0)
- PerHostPeakMemUsage: 14.9 MiB (15629030)
- RowsProduced: 11 (11)
- TotalNetworkReceiveTime: 0ns (0)
- TotalNetworkSendTime: 460.48us (460477)
- TotalStorageWaitTime: 11.0m (659048680825)
- TotalThreadsInvoluntaryContextSwitches: 106 (106)
- TotalThreadsTotalWallClockTime: 13.3m (798079407027)
- TotalThreadsSysTime: 335ms (335939000)
- TotalThreadsUserTime: 5.45s (5446160000)
- TotalThreadsVoluntaryContextSwitches: 32,777 (32777)
- TotalTime: 32.04s (32042218379)
Fragment Instance Lifecycle Timings
- ExecTime: 688.99us (688990)
- ExecTreeExecTime: 6.50us (6495)
- InactiveTotalTime: 0ns (0)
- OpenTime: 32.02s (32015989659)
- ExecTreeOpenTime: 31.99s (31992043486)
- PrepareTime: 25ms (25505064)
- ExecTreePrepareTime: 9ms (9271084)
- TotalTime: 0ns (0)
DataStreamSender (dst_id=2)
- BytesSent: 2.8 KiB (2889)
- InactiveTotalTime: 0ns (0)
- NetworkThroughput(*): 149.1 MiB/s (156339628)
- OverallThroughput: 15.9 MiB/s (16647842)
- PeakMemoryUsage: 1.9 KiB (1896)
- RowsReturned: 11 (11)
- SerializeBatchTime: 24.61us (24606)
- TotalTime: 173.54us (173536)
- TransmitDataRPCTime: 18.48us (18479)
- UncompressedRowBatchSize: 5.0 KiB (5078)
CodeGen
- CodegenTime: 1ms (1368905)
- CompileTime: 7ms (7012106)
- InactiveTotalTime: 0ns (0)
- LoadTime: 0ns (0)
- ModuleBitcodeSize: 2.0 MiB (2050180)
- NumFunctions: 42 (42)
- NumInstructions: 815 (815)
- OptimizationTime: 16ms (16441784)
- PeakMemoryUsage: 407.5 KiB (417280)
- PrepareTime: 15ms (15138293)
- TotalTime: 39ms (39017360)
SORT_NODE (id=1)
ExecOption: Codegen Enabled
- InMemorySortTime: 2.91us (2913)
- InactiveTotalTime: 0ns (0)
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 12.4 MiB (13007836)
- RowsReturned: 11 (11)
- RowsReturnedRate: 0 per second (0)
- SortDataSize: 4.9 KiB (5030)
- SpilledRuns: 0 (0)
- TotalMergesPerformed: 0 (0)
- TotalTime: 32.00s (32001318106)
Buffer pool
- AllocTime: 41.10us (41105)
- CumulativeAllocationBytes: 6.0 MiB (6291456)
- CumulativeAllocations: 3 (3)
- InactiveTotalTime: 0ns (0)
- PeakReservation: 12.0 MiB (12582912)
- PeakUnpinnedBytes: 0 B (0)
- PeakUsedReservation: 6.0 MiB (6291456)
- ReadIoBytes: 0 B (0)
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0ns (0)
- TotalTime: 0ns (0)
- WriteIoBytes: 0 B (0)
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0ns (0)
HDFS_SCAN_NODE (id=0)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
5:350/80.36 MB 1:376/93.26 MB 3:333/79.58 MB 9:266/68.96 MB 4:334/79.80 MB
11:248/59.56 MB 8:340/81.01 MB 6:343/79.52 MB 7:351/86.40 MB 10:316/86.61 MB
0:331/80.14 MB 2:355/85.24 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 3943 out of
3943
Hdfs Read Thread Concurrency Bucket: 0:98.44% 1:1.562% 2:0% 3:0%
4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/NONE:5979 PARQUET/SNAPPY:5940
- AverageHdfsReadThreadConcurrency: 0.02 (0.015625)
- AverageScannerThreadConcurrency: 24.00 (24.0)
- BytesRead: 399.0 MiB (418338366)
- BytesReadDataNodeCache: 0 B (0)
- BytesReadLocal: 399.0 MiB (418338366)
- BytesReadRemoteUnexpected: 0 B (0)
- BytesReadShortCircuit: 399.0 MiB (418338366)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 9,880 (9880)
- DecompressionTime: 90.40us (90398)
- InactiveTotalTime: 0ns (0)
- MaxCompressedTextFileLength: 0 B (0)
- NumColumns: 1 (1)
- NumDictFilteredRowGroups: 3,942 (3942)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 3,943 (3943)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.9 MiB (3027013)
- PerReadThreadRawHdfsThroughput: 291.0 MiB/s (305115443)
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 31.93s (31932674157)
- RowBatchQueuePutWaitTime: 0ns (0)
- RowsRead: 1,653 (1653)
- RowsReturned: 11 (11)
- RowsReturnedRate: 0 per second (0)
- ScanRangesComplete: 3,943 (3943)
- ScannerThreadsInvoluntaryContextSwitches: 106 (106)
- ScannerThreadsTotalWallClockTime: 12.8m (766062773671)
- MaterializeTupleTime(*): 718.22us (718224)
- ScannerThreadsSysTime: 310ms (310943000)
- ScannerThreadsUserTime: 5.34s (5342176000)
- ScannerThreadsVoluntaryContextSwitches: 28,872 (28872)
- TotalRawHdfsReadTime(*): 1.37s (1371082243)
- TotalReadThroughput: 12.4 MiB/s (13037956)
- TotalTime: 31.99s (31991810654)
Instance 434ed55d986c7a61:10605b1b00000001 (:22000)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 3:323/71.21
MB 4:341/86.24 MB 11:314/75.06 MB 0:307/71.77 MB 2:360/81.55 MB 1:366/94.04 MB
9:274/71.62 MB 5:347/71.35 MB 8:346/73.61 MB 7:330/93.35 MB 10:343/84.51 MB
6:363/84.92 MB
- AverageThreadTokens: 24.83 (24.833333333333332)
- BloomFilterBytes: 0 B (0)
- InactiveTotalTime: 0ns (0)
- PeakMemoryUsage: 14.7 MiB (15403957)
- PeakReservation: 12.0 MiB (12582912)
- PeakUsedReservation: 0 B (0)
- PerHostPeakMemUsage: 14.7 MiB (15403957)
- RowsProduced: 10 (10)
- TotalNetworkReceiveTime: 0ns (0)
- TotalNetworkSendTime: 314.35us (314347)
- TotalStorageWaitTime: 10.4m (623537031522)
- TotalThreadsInvoluntaryContextSwitches: 146 (146)
- TotalThreadsTotalWallClockTime: 12.4m (746025392784)
- TotalThreadsSysTime: 306ms (306943000)
- TotalThreadsUserTime: 5.47s (5471159000)
- TotalThreadsVoluntaryContextSwitches: 31,751 (31751)
- TotalTime: 29.99s (29990622988)
Fragment Instance Lifecycle Timings
- ExecTime: 518.72us (518723)
- ExecTreeExecTime: 6.15us (6146)
- InactiveTotalTime: 0ns (0)
- OpenTime: 29.96s (29964242178)
- ExecTreeOpenTime: 29.94s (29938991308)
- PrepareTime: 25ms (25836070)
- ExecTreePrepareTime: 9ms (9131399)
- TotalTime: 0ns (0)
DataStreamSender (dst_id=2)
- BytesSent: 1013 B (1013)
- InactiveTotalTime: 0ns (0)
- NetworkThroughput(*): 57.4 MiB/s (60190136)
- OverallThroughput: 6.5 MiB/s (6841035)
- PeakMemoryUsage: 1.9 KiB (1896)
- RowsReturned: 10 (10)
- SerializeBatchTime: 17.43us (17431)
- TotalTime: 148.08us (148077)
- TransmitDataRPCTime: 16.83us (16830)
- UncompressedRowBatchSize: 2.2 KiB (2301)
CodeGen
- CodegenTime: 1ms (1347485)
- CompileTime: 7ms (7084038)
- InactiveTotalTime: 0ns (0)
- LoadTime: 0ns (0)
- ModuleBitcodeSize: 2.0 MiB (2050180)
- NumFunctions: 42 (42)
- NumInstructions: 815 (815)
- OptimizationTime: 17ms (17698509)
- PeakMemoryUsage: 407.5 KiB (417280)
- PrepareTime: 15ms (15683691)
- TotalTime: 40ms (40866461)
SORT_NODE (id=1)
ExecOption: Codegen Enabled
- InMemorySortTime: 2.29us (2286)
- InactiveTotalTime: 0ns (0)
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 12.4 MiB (12955017)
- RowsReturned: 10 (10)
- RowsReturnedRate: 0 per second (0)
- SortDataSize: 2.2 KiB (2257)
- SpilledRuns: 0 (0)
- TotalMergesPerformed: 0 (0)
- TotalTime: 29.95s (29948125327)
Buffer pool
- AllocTime: 25.98us (25985)
- CumulativeAllocationBytes: 6.0 MiB (6291456)
- CumulativeAllocations: 3 (3)
- InactiveTotalTime: 0ns (0)
- PeakReservation: 12.0 MiB (12582912)
- PeakUnpinnedBytes: 0 B (0)
- PeakUsedReservation: 6.0 MiB (6291456)
- ReadIoBytes: 0 B (0)
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0ns (0)
- TotalTime: 0ns (0)
- WriteIoBytes: 0 B (0)
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0ns (0)
HDFS_SCAN_NODE (id=0)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
3:323/71.21 MB 4:341/86.24 MB 11:314/75.06 MB 0:307/71.77 MB 2:360/81.55 MB
1:366/94.04 MB 9:274/71.62 MB 5:347/71.35 MB 8:346/73.61 MB 7:330/93.35 MB
10:343/84.51 MB 6:363/84.92 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 4014 out of
4014
Hdfs Read Thread Concurrency Bucket: 0:98.33% 1:1.667% 2:0% 3:0%
4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/NONE:5826 PARQUET/SNAPPY:5960
- AverageHdfsReadThreadConcurrency: 0.02 (0.016666666666666666)
- AverageScannerThreadConcurrency: 23.83 (23.833333333333332)
- BytesRead: 404.3 MiB (423979093)
- BytesReadDataNodeCache: 0 B (0)
- BytesReadLocal: 404.3 MiB (423979093)
- BytesReadRemoteUnexpected: 0 B (0)
- BytesReadShortCircuit: 404.3 MiB (423979093)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 9,973 (9973)
- DecompressionTime: 157.75us (157751)
- InactiveTotalTime: 0ns (0)
- MaxCompressedTextFileLength: 0 B (0)
- NumColumns: 1 (1)
- NumDictFilteredRowGroups: 4,013 (4013)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 4,014 (4014)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.7 MiB (2801940)
- PerReadThreadRawHdfsThroughput: 378.1 MiB/s (396428449)
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 29.88s (29878810342)
- RowBatchQueuePutWaitTime: 0ns (0)
- RowsRead: 1,726 (1726)
- RowsReturned: 10 (10)
- RowsReturnedRate: 0 per second (0)
- ScanRangesComplete: 4,014 (4014)
- ScannerThreadsInvoluntaryContextSwitches: 146 (146)
- ScannerThreadsTotalWallClockTime: 11.9m (716060697079)
- MaterializeTupleTime(*): 681.80us (681800)
- ScannerThreadsSysTime: 281ms (281947000)
- ScannerThreadsUserTime: 5.36s (5364175000)
- ScannerThreadsVoluntaryContextSwitches: 27,851 (27851)
- TotalRawHdfsReadTime(*): 1.07s (1069497141)
- TotalReadThroughput: 13.5 MiB/s (14109605)
- TotalTime: 29.94s (29938335126)
Instance 434ed55d986c7a61:10605b1b00000007 (:22000)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 2:334/80.96
MB 0:331/85.09 MB 7:337/76.16 MB 1:347/87.65 MB 6:328/82.30 MB 9:344/85.78 MB
10:326/66.35 MB 11:315/88.83 MB 8:337/85.52 MB 3:353/67.12 MB 5:352/81.18 MB
4:313/71.93 MB
- AverageThreadTokens: 25.00 (25.0)
- BloomFilterBytes: 0 B (0)
- InactiveTotalTime: 0ns (0)
- PeakMemoryUsage: 14.9 MiB (15661936)
- PeakReservation: 12.0 MiB (12582912)
- PeakUsedReservation: 0 B (0)
- PerHostPeakMemUsage: 14.9 MiB (15661936)
- RowsProduced: 4 (4)
- TotalNetworkReceiveTime: 0ns (0)
- TotalNetworkSendTime: 288.31us (288315)
- TotalStorageWaitTime: 9.7m (584492319539)
- TotalThreadsInvoluntaryContextSwitches: 122 (122)
- TotalThreadsTotalWallClockTime: 12.0m (721638782673)
- TotalThreadsSysTime: 378ms (378933000)
- TotalThreadsUserTime: 6.07s (6070063000)
- TotalThreadsVoluntaryContextSwitches: 33,604 (33604)
- TotalTime: 29.00s (28996225150)
Fragment Instance Lifecycle Timings
- ExecTime: 449.54us (449542)
- ExecTreeExecTime: 7.52us (7519)
- InactiveTotalTime: 0ns (0)
- OpenTime: 28.97s (28969520704)
- ExecTreeOpenTime: 28.95s (28945775792)
- PrepareTime: 26ms (26224924)
- ExecTreePrepareTime: 10ms (10359145)
- TotalTime: 0ns (0)
DataStreamSender (dst_id=2)
- BytesSent: 827 B (827)
- InactiveTotalTime: 0ns (0)
- NetworkThroughput(*): 50.3 MiB/s (52752439)
- OverallThroughput: 6.6 MiB/s (6875166)
- PeakMemoryUsage: 1.9 KiB (1896)
- RowsReturned: 4 (4)
- SerializeBatchTime: 14.58us (14577)
- TotalTime: 120.29us (120288)
- TransmitDataRPCTime: 15.68us (15677)
- UncompressedRowBatchSize: 1.3 KiB (1323)
CodeGen
- CodegenTime: 1ms (1320997)
- CompileTime: 6ms (6820822)
- InactiveTotalTime: 0ns (0)
- LoadTime: 0ns (0)
- ModuleBitcodeSize: 2.0 MiB (2050180)
- NumFunctions: 42 (42)
- NumInstructions: 815 (815)
- OptimizationTime: 16ms (16424337)
- PeakMemoryUsage: 407.5 KiB (417280)
- PrepareTime: 14ms (14826851)
- TotalTime: 38ms (38520929)
SORT_NODE (id=1)
ExecOption: Codegen Enabled
- InMemorySortTime: 2.15us (2145)
- InactiveTotalTime: 0ns (0)
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 12.4 MiB (13034298)
- RowsReturned: 4 (4)
- RowsReturnedRate: 0 per second (0)
- SortDataSize: 1.3 KiB (1303)
- SpilledRuns: 0 (0)
- TotalMergesPerformed: 0 (0)
- TotalTime: 28.96s (28956139285)
Buffer pool
- AllocTime: 41.70us (41698)
- CumulativeAllocationBytes: 6.0 MiB (6291456)
- CumulativeAllocations: 3 (3)
- InactiveTotalTime: 0ns (0)
- PeakReservation: 12.0 MiB (12582912)
- PeakUnpinnedBytes: 0 B (0)
- PeakUsedReservation: 6.0 MiB (6291456)
- ReadIoBytes: 0 B (0)
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0ns (0)
- TotalTime: 0ns (0)
- WriteIoBytes: 0 B (0)
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0ns (0)
HDFS_SCAN_NODE (id=0)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
2:334/80.96 MB 0:331/85.09 MB 7:337/76.16 MB 1:347/87.65 MB 6:328/82.30 MB
9:344/85.78 MB 10:326/66.35 MB 11:315/88.83 MB 8:337/85.52 MB 3:353/67.12 MB
5:352/81.18 MB 4:313/71.93 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 4017 out of
4017
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/NONE:6003 PARQUET/SNAPPY:6026
- AverageHdfsReadThreadConcurrency: 0.02 (0.017241379310344827)
- AverageScannerThreadConcurrency: 24.00 (24.0)
- BytesRead: 406.0 MiB (425765040)
- BytesReadDataNodeCache: 0 B (0)
- BytesReadLocal: 406.0 MiB (425765040)
- BytesReadRemoteUnexpected: 0 B (0)
- BytesReadShortCircuit: 406.0 MiB (425765040)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 10,040 (10040)
- DecompressionTime: 250.19us (250193)
- InactiveTotalTime: 0ns (0)
- MaxCompressedTextFileLength: 0 B (0)
- NumColumns: 1 (1)
- NumDictFilteredRowGroups: 4,015 (4015)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 4,017 (4017)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.9 MiB (3059919)
- PerReadThreadRawHdfsThroughput: 482.5 MiB/s (505940117)
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 28.88s (28876761794)
- RowBatchQueuePutWaitTime: 0ns (0)
- RowsRead: 3,414 (3414)
- RowsReturned: 4 (4)
- RowsReturnedRate: 0 per second (0)
- ScanRangesComplete: 4,017 (4017)
- ScannerThreadsInvoluntaryContextSwitches: 121 (121)
- ScannerThreadsTotalWallClockTime: 11.5m (692668878420)
- MaterializeTupleTime(*): 818.25us (818248)
- ScannerThreadsSysTime: 357ms (357936000)
- ScannerThreadsUserTime: 5.95s (5949082000)
- ScannerThreadsVoluntaryContextSwitches: 29,527 (29527)
- TotalRawHdfsReadTime(*): 841ms (841532476)
- TotalReadThroughput: 14.0 MiB/s (14633257)
- TotalTime: 28.94s (28943830524)
Instance 434ed55d986c7a61:10605b1b00000006 (:22000)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:315/65.80
MB 0:303/61.47 MB 10:349/83.34 MB 1:308/69.80 MB 2:337/100.04 MB 9:353/86.50 MB
4:342/75.93 MB 7:339/86.05 MB 3:369/76.05 MB 8:375/87.12 MB 5:294/76.79 MB
6:356/89.98 MB
- AverageThreadTokens: 24.91 (24.913793103448278)
- BloomFilterBytes: 0 B (0)
- InactiveTotalTime: 0ns (0)
- PeakMemoryUsage: 15.0 MiB (15740378)
- PeakReservation: 12.0 MiB (12582912)
- PeakUsedReservation: 0 B (0)
- PerHostPeakMemUsage: 15.0 MiB (15740378)
- RowsProduced: 14 (14)
- TotalNetworkReceiveTime: 0ns (0)
- TotalNetworkSendTime: 313.05us (313054)
- TotalStorageWaitTime: 9.8m (587692243836)
- TotalThreadsInvoluntaryContextSwitches: 108 (108)
- TotalThreadsTotalWallClockTime: 12.0m (718225467748)
- TotalThreadsSysTime: 337ms (337940000)
- TotalThreadsUserTime: 5.89s (5887095000)
- TotalThreadsVoluntaryContextSwitches: 34,394 (34394)
- TotalTime: 28.85s (28852640546)
Fragment Instance Lifecycle Timings
- ExecTime: 512.28us (512279)
- ExecTreeExecTime: 5.58us (5578)
- InactiveTotalTime: 0ns (0)
- OpenTime: 28.82s (28823634426)
- ExecTreeOpenTime: 28.80s (28797449745)
- PrepareTime: 28ms (28467715)
- ExecTreePrepareTime: 10ms (10228441)
- TotalTime: 0ns (0)
DataStreamSender (dst_id=2)
- BytesSent: 2.7 KiB (2755)
- InactiveTotalTime: 0ns (0)
- NetworkThroughput(*): 107.4 MiB/s (112619057)
- OverallThroughput: 19.1 MiB/s (20037675)
- PeakMemoryUsage: 1.9 KiB (1896)
- RowsReturned: 14 (14)
- SerializeBatchTime: 22.25us (22250)
- TotalTime: 137.49us (137491)
- TransmitDataRPCTime: 24.46us (24463)
- UncompressedRowBatchSize: 5.0 KiB (5097)
CodeGen
- CodegenTime: 1ms (1516906)
- CompileTime: 7ms (7542926)
- InactiveTotalTime: 0ns (0)
- LoadTime: 0ns (0)
- ModuleBitcodeSize: 2.0 MiB (2050180)
- NumFunctions: 42 (42)
- NumInstructions: 815 (815)
- OptimizationTime: 18ms (18086561)
- PeakMemoryUsage: 407.5 KiB (417280)
- PrepareTime: 17ms (17208458)
- TotalTime: 43ms (43302039)
SORT_NODE (id=1)
ExecOption: Codegen Enabled
- InMemorySortTime: 2.99us (2988)
- InactiveTotalTime: 0ns (0)
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 12.4 MiB (13016261)
- RowsReturned: 14 (14)
- RowsReturnedRate: 0 per second (0)
- SortDataSize: 4.9 KiB (5037)
- SpilledRuns: 0 (0)
- TotalMergesPerformed: 0 (0)
- TotalTime: 28.81s (28807680233)
Buffer pool
- AllocTime: 36.77us (36771)
- CumulativeAllocationBytes: 6.0 MiB (6291456)
- CumulativeAllocations: 3 (3)
- InactiveTotalTime: 0ns (0)
- PeakReservation: 12.0 MiB (12582912)
- PeakUnpinnedBytes: 0 B (0)
- PeakUsedReservation: 6.0 MiB (6291456)
- ReadIoBytes: 0 B (0)
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0ns (0)
- TotalTime: 0ns (0)
- WriteIoBytes: 0 B (0)
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0ns (0)
HDFS_SCAN_NODE (id=0)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
11:315/65.80 MB 0:303/61.47 MB 10:349/83.34 MB 1:308/69.80 MB 2:337/100.04 MB
9:353/86.50 MB 4:342/75.93 MB 7:339/86.05 MB 3:369/76.05 MB 8:375/87.12 MB
5:294/76.79 MB 6:356/89.98 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 4040 out of
4040
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/NONE:6024 PARQUET/SNAPPY:6052
- AverageHdfsReadThreadConcurrency: 0.00 (0.0)
- AverageScannerThreadConcurrency: 23.91 (23.913793103448278)
- BytesRead: 407.2 MiB (427015309)
- BytesReadDataNodeCache: 0 B (0)
- BytesReadLocal: 407.2 MiB (427015309)
- BytesReadRemoteUnexpected: 0 B (0)
- BytesReadShortCircuit: 407.2 MiB (427015309)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 10,086 (10086)
- DecompressionTime: 132.62us (132622)
- InactiveTotalTime: 0ns (0)
- MaxCompressedTextFileLength: 0 B (0)
- NumColumns: 1 (1)
- NumDictFilteredRowGroups: 4,039 (4039)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 4,040 (4040)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 3.0 MiB (3138361)
- PerReadThreadRawHdfsThroughput: 483.0 MiB/s (506467231)
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 28.73s (28734088248)
- RowBatchQueuePutWaitTime: 0ns (0)
- RowsRead: 1,908 (1908)
- RowsReturned: 14 (14)
- RowsReturnedRate: 0 per second (0)
- ScanRangesComplete: 4,040 (4040)
- ScannerThreadsInvoluntaryContextSwitches: 107 (107)
- ScannerThreadsTotalWallClockTime: 11.5m (689401393408)
- MaterializeTupleTime(*): 814.28us (814278)
- ScannerThreadsSysTime: 313ms (313943000)
- ScannerThreadsUserTime: 5.78s (5775112000)
- ScannerThreadsVoluntaryContextSwitches: 30,419 (30419)
- TotalRawHdfsReadTime(*): 843ms (843125246)
- TotalReadThroughput: 14.0 MiB/s (14701080)
- TotalTime: 28.80s (28797892014)
Instance 434ed55d986c7a61:10605b1b00000005 (:22000)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 5:313/79.06
MB 0:315/72.24 MB 4:341/82.17 MB 1:354/79.95 MB 11:334/90.25 MB 6:320/71.48 MB
7:326/82.05 MB 8:343/74.24 MB 10:337/80.35 MB 3:376/86.30 MB 2:329/82.61 MB
9:297/77.69 MB
- AverageThreadTokens: 25.00 (25.0)
- BloomFilterBytes: 0 B (0)
- InactiveTotalTime: 0ns (0)
- PeakMemoryUsage: 14.9 MiB (15639646)
- PeakReservation: 12.0 MiB (12582912)
- PeakUsedReservation: 0 B (0)
- PerHostPeakMemUsage: 14.9 MiB (15639646)
- RowsProduced: 15 (15)
- TotalNetworkReceiveTime: 0ns (0)
- TotalNetworkSendTime: 254.98us (254976)
- TotalStorageWaitTime: 9.6m (573696030689)
- TotalThreadsInvoluntaryContextSwitches: 99 (99)
- TotalThreadsTotalWallClockTime: 11.7m (699286707304)
- TotalThreadsSysTime: 313ms (313941000)
- TotalThreadsUserTime: 5.56s (5560144000)
- TotalThreadsVoluntaryContextSwitches: 32,877 (32877)
- TotalTime: 28.11s (28114962044)
Fragment Instance Lifecycle Timings
- ExecTime: 449.58us (449577)
- ExecTreeExecTime: 5.96us (5955)
- InactiveTotalTime: 0ns (0)
- OpenTime: 28.09s (28087460227)
- ExecTreeOpenTime: 28.06s (28062994084)
- PrepareTime: 27ms (27020296)
- ExecTreePrepareTime: 10ms (10464658)
- TotalTime: 0ns (0)
DataStreamSender (dst_id=2)
- BytesSent: 1.7 KiB (1698)
- InactiveTotalTime: 0ns (0)
- NetworkThroughput(*): 80.7 MiB/s (84603886)
- OverallThroughput: 14.0 MiB/s (14715950)
- PeakMemoryUsage: 1.9 KiB (1896)
- RowsReturned: 15 (15)
- SerializeBatchTime: 18.33us (18328)
- TotalTime: 115.39us (115385)
- TransmitDataRPCTime: 20.07us (20070)
- UncompressedRowBatchSize: 3.6 KiB (3688)
CodeGen
- CodegenTime: 1ms (1378206)
- CompileTime: 7ms (7167805)
- InactiveTotalTime: 0ns (0)
- LoadTime: 0ns (0)
- ModuleBitcodeSize: 2.0 MiB (2050180)
- NumFunctions: 42 (42)
- NumInstructions: 815 (815)
- OptimizationTime: 16ms (16780487)
- PeakMemoryUsage: 407.5 KiB (417280)
- PrepareTime: 15ms (15579504)
- TotalTime: 39ms (39925571)
SORT_NODE (id=1)
ExecOption: Codegen Enabled
- InMemorySortTime: 3.85us (3854)
- InactiveTotalTime: 0ns (0)
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 12.4 MiB (12950454)
- RowsReturned: 15 (15)
- RowsReturnedRate: 0 per second (0)
- SortDataSize: 3.5 KiB (3624)
- SpilledRuns: 0 (0)
- TotalMergesPerformed: 0 (0)
- TotalTime: 28.07s (28073461477)
Buffer pool
- AllocTime: 38.24us (38245)
- CumulativeAllocationBytes: 6.0 MiB (6291456)
- CumulativeAllocations: 3 (3)
- InactiveTotalTime: 0ns (0)
- PeakReservation: 12.0 MiB (12582912)
- PeakUnpinnedBytes: 0 B (0)
- PeakUsedReservation: 6.0 MiB (6291456)
- ReadIoBytes: 0 B (0)
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0ns (0)
- TotalTime: 0ns (0)
- WriteIoBytes: 0 B (0)
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0ns (0)
HDFS_SCAN_NODE (id=0)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
5:313/79.06 MB 0:315/72.24 MB 4:341/82.17 MB 1:354/79.95 MB 11:334/90.25 MB
6:320/71.48 MB 7:326/82.05 MB 8:343/74.24 MB 10:337/80.35 MB 3:376/86.30 MB
2:329/82.61 MB 9:297/77.69 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 3985 out of
3985
Hdfs Read Thread Concurrency Bucket: 0:98.21% 1:1.786% 2:0% 3:0%
4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0%
File Formats: PARQUET/NONE:5883 PARQUET/SNAPPY:5954
- AverageHdfsReadThreadConcurrency: 0.02 (0.017857142857142856)
- AverageScannerThreadConcurrency: 24.00 (24.0)
- BytesRead: 402.9 MiB (422464470)
- BytesReadDataNodeCache: 0 B (0)
- BytesReadLocal: 402.9 MiB (422464470)
- BytesReadRemoteUnexpected: 0 B (0)
- BytesReadShortCircuit: 402.9 MiB (422464470)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 9,937 (9937)
- DecompressionTime: 264.20us (264199)
- InactiveTotalTime: 0ns (0)
- MaxCompressedTextFileLength: 0 B (0)
- NumColumns: 5 (5)
- NumDictFilteredRowGroups: 3,983 (3983)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 3,985 (3985)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.9 MiB (3037629)
- PerReadThreadRawHdfsThroughput: 449.8 MiB/s (471611847)
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 28.00s (28003658191)
- RowBatchQueuePutWaitTime: 0ns (0)
- RowsRead: 3,248 (3248)
- RowsReturned: 15 (15)
- RowsReturnedRate: 0 per second (0)
- ScanRangesComplete: 3,985 (3985)
- ScannerThreadsInvoluntaryContextSwitches: 97 (97)
- ScannerThreadsTotalWallClockTime: 11.2m (671198861506)
- MaterializeTupleTime(*): 843.12us (843125)
- ScannerThreadsSysTime: 286ms (286945000)
- ScannerThreadsUserTime: 5.46s (5460159000)
- ScannerThreadsVoluntaryContextSwitches: 29,115 (29115)
- TotalRawHdfsReadTime(*): 895ms (895788502)
- TotalReadThroughput: 14.3 MiB/s (15008479)
- TotalTime: 28.06s (28064585236)
Instance 434ed55d986c7a61:10605b1b00000002 (:22000)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 9:246/78.25
MB 3:229/90.57 MB 10:281/87.28 MB 5:209/68.46 MB 7:262/77.81 MB 8:261/66.92 MB
4:277/85.43 MB 6:273/76.19 MB 0:259/94.79 MB 2:262/72.81 MB 11:217/68.91 MB
1:254/93.19 MB
- AverageThreadTokens: 25.00 (25.0)
- BloomFilterBytes: 0 B (0)
- InactiveTotalTime: 0ns (0)
- PeakMemoryUsage: 14.8 MiB (15535972)
- PeakReservation: 12.0 MiB (12582912)
- PeakUsedReservation: 0 B (0)
- PerHostPeakMemUsage: 14.8 MiB (15544253)
- RowsProduced: 0 (0)
- TotalNetworkReceiveTime: 0ns (0)
- TotalNetworkSendTime: 181ns (181)
- TotalStorageWaitTime: 8.1m (484620817497)
- TotalThreadsInvoluntaryContextSwitches: 128 (128)
- TotalThreadsTotalWallClockTime: 9.7m (582351381416)
- TotalThreadsSysTime: 249ms (249952000)
- TotalThreadsUserTime: 3.97s (3973384000)
- TotalThreadsVoluntaryContextSwitches: 26,470 (26470)
- TotalTime: 23.44s (23443975336)
Fragment Instance Lifecycle Timings
- ExecTime: 110.17us (110175)
- ExecTreeExecTime: 4.04us (4039)
- InactiveTotalTime: 0ns (0)
- OpenTime: 23.41s (23413060526)
- ExecTreeOpenTime: 23.38s (23377015370)
- PrepareTime: 30ms (30751912)
- ExecTreePrepareTime: 9ms (9702404)
- TotalTime: 0ns (0)
DataStreamSender (dst_id=2)
- BytesSent: 0 B (0)
- InactiveTotalTime: 0ns (0)
- NetworkThroughput(*): 0 B/s (0)
- OverallThroughput: 0 B/s (0)
- PeakMemoryUsage: 1.9 KiB (1896)
- RowsReturned: 0 (0)
- SerializeBatchTime: 0ns (0)
- TotalTime: 178.22us (178217)
- TransmitDataRPCTime: 0ns (0)
- UncompressedRowBatchSize: 0 B (0)
CodeGen
- CodegenTime: 1ms (1881629)
- CompileTime: 9ms (9787839)
- InactiveTotalTime: 0ns (0)
- LoadTime: 0ns (0)
- ModuleBitcodeSize: 2.0 MiB (2050180)
- NumFunctions: 42 (42)
- NumInstructions: 815 (815)
- OptimizationTime: 25ms (25603003)
- PeakMemoryUsage: 407.5 KiB (417280)
- PrepareTime: 19ms (19672239)
- TotalTime: 55ms (55593930)
SORT_NODE (id=1)
ExecOption: Codegen Enabled
- InMemorySortTime: 484ns (484)
- InactiveTotalTime: 0ns (0)
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 12.1 MiB (12711200)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0 per second (0)
- SortDataSize: 0 B (0)
- SpilledRuns: 0 (0)
- TotalMergesPerformed: 0 (0)
- TotalTime: 23.39s (23386717505)
Buffer pool
- AllocTime: 37.47us (37473)
- CumulativeAllocationBytes: 6.0 MiB (6291456)
- CumulativeAllocations: 3 (3)
- InactiveTotalTime: 0ns (0)
- PeakReservation: 12.0 MiB (12582912)
- PeakUnpinnedBytes: 0 B (0)
- PeakUsedReservation: 6.0 MiB (6291456)
- ReadIoBytes: 0 B (0)
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0ns (0)
- TotalTime: 0ns (0)
- WriteIoBytes: 0 B (0)
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0ns (0)
HDFS_SCAN_NODE (id=0)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
9:246/78.25 MB 3:229/90.57 MB 10:281/87.28 MB 5:209/68.46 MB 7:262/77.81 MB
8:261/66.92 MB 4:277/85.43 MB 6:273/76.19 MB 0:259/94.79 MB 2:262/72.81 MB
11:217/68.91 MB 1:254/93.19 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 3030 out of
3030
Hdfs Read Thread Concurrency Bucket: 0:87.23% 1:12.77% 2:0% 3:0%
4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/NONE:7533 PARQUET/SNAPPY:5541
- AverageHdfsReadThreadConcurrency: 0.13 (0.1276595744680851)
- AverageScannerThreadConcurrency: 24.00 (24.0)
- BytesRead: 321.0 MiB (336580237)
- BytesReadDataNodeCache: 0 B (0)
- BytesReadLocal: 321.0 MiB (336580237)
- BytesReadRemoteUnexpected: 0 B (0)
- BytesReadShortCircuit: 321.0 MiB (336580237)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 8,565 (8565)
- DecompressionTime: 0ns (0)
- InactiveTotalTime: 0ns (0)
- MaxCompressedTextFileLength: 0 B (0)
- NumColumns: 5 (5)
- NumDictFilteredRowGroups: 3,030 (3030)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 3,030 (3030)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.8 MiB (2933955)
- PerReadThreadRawHdfsThroughput: 313.3 MiB/s (328521519)
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 23.33s (23334174884)
- RowBatchQueuePutWaitTime: 0ns (0)
- RowsRead: 0 (0)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0 per second (0)
- ScanRangesComplete: 3,030 (3030)
- ScannerThreadsInvoluntaryContextSwitches: 128 (128)
- ScannerThreadsTotalWallClockTime: 9.3m (558938282460)
- MaterializeTupleTime(*): 339.23us (339227)
- ScannerThreadsSysTime: 227ms (227955000)
- ScannerThreadsUserTime: 3.88s (3884397000)
- ScannerThreadsVoluntaryContextSwitches: 23,363 (23363)
- TotalRawHdfsReadTime(*): 1.02s (1024530258)
- TotalReadThroughput: 13.6 MiB/s (14231534)
- TotalTime: 23.38s (23382877083)
