Hi Team,
Our team is working on a huge data load in Kudu and we are currently facing
a performance issue . We are hoping you can guide us on a solution to the
below mentioned concerns.
We have 212 million data loads in Kudu. Currently for such a data load,
when loading through impala, 47 seconds are spent for query processing and
loading overall. We have used default configurations in Kudu and Impala
with 6 node clusters to get these numbers.We haven’t reached the
performance we expected.
I have attached the impala profile and DDL of the table creation. We have
used impala-3.4.0 and kudu-1.15.0 versions.
*What can we do to reduce the time spent for loading 212 million data loads
from 47 seconds to 10 seconds through impala?*
We would be grateful if you can provide us with some solutions at
your earliest possible.
Thank You! Regards,
Sandaruwan Kumarasingha.
CREATE TABLE `table_abc` (
`col_2` STRING NOT NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_1` BIGINT NOT NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_3` BIGINT NOT NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_4` BIGINT NOT NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_5` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_6` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_7` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_8` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_9` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_10` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_11` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_12` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_13` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_14` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_15` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_16` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_17` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_18` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_19` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_20` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_21` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_22` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_23` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_24` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_25` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_26` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_27` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_28` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_29` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_30` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_31` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_32` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_33` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_34` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_35` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_36` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_37` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_38` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_39` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_40` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_41` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_42` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_43` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_44` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_45` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_46` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_47` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_48` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_49` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_50` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_51` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_52` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_53` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_54` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_55` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_56` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_57` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_58` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_59` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_60` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_61` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_62` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_63` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_64` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_65` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_66` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_67` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_68` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_69` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_70` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_71` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_72` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_73` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_74` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_75` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_76` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_77` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_78` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_79` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_80` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_81` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_82` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_83` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_84` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_85` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_86` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_87` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_88` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_89` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_90` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_91` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_92` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_93` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_94` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_95` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_96` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_97` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_98` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_99` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_100` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_101` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_102` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_103` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_104` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_105` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_106` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_107` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_108` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_109` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_110` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_111` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_112` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_113` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_114` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_115` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_116` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_117` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
PRIMARY KEY (col_2, col_1, col_3, col_4)
)
PARTITION BY HASH (col_2) PARTITIONS 4, HASH (col_3) PARTITIONS 64, RANGE
(col_1) (PARTITION 1622505600000000000 <= VALUES < 1625097599999999999)
STORED AS KUDU
TBLPROPERTIES
('kudu.master_addresses'='node-1:8051,node-2:8051,node-3:8051,node-4:8051,node-5:8051',
'kudu.num_tablet_replicas'='3', 'numFiles'='0', 'totalSize'='0')Query (id=test_id):
Summary:
Session ID: test_id_1:test_id_2
Session Type: HIVESERVER2
HiveServer2 Protocol Version: V6
Start Time: 2022-05-02 09:45:06.613721000
End Time: 2022-05-02 09:45:56.470552000
Query Type: QUERY
Query State: FINISHED
Impala Query State: FINISHED
Query Status: OK
Impala Version: impalad version 3.4.0-RELEASE RELEASE (build Could not
obtain git hash)
User:
Connected User:
Delegated User:
Network Address: ***.***.***.***:******
Default Db: default
Sql Statement: SELECT * from table_abc where (col_1>=num_1 AND col_1<num_2)
AND col_2 ='value_1' ORDER BY col_1 ASC, col_3 ASC LIMIT 250000
Coordinator: impala-coordinator-node-1:22000
Query Options (set by configuration): TIMEZONE=timezone
Query Options (set by configuration and planner): MT_DOP=0,TIMEZONE=timezone
Plan:
----------------
Max Per-Host Resource Reservation: Memory=0B Threads=3
Per-Host Resource Estimates: Memory=422MB
Dedicated Coordinator Resource Estimate: Memory=106MB
WARNING: The following tables are missing relevant table and/or column
statistics.
default.table_abc
Analyzed query: SELECT * FROM `default`.table_abc WHERE
col_1 >= CAST(num_1 AS BIGINT) AND col_1 <
CAST(num_2 AS BIGINT) AND col_2 = 'value_1' ORDER BY
col_1 ASC, col_3 ASC LIMIT CAST(250000 AS INT)
F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
| Per-Host Resources: mem-estimate=5.51MB mem-reservation=0B
thread-reservation=1
PLAN-ROOT SINK
| output exprs: col_2, col_1, col_3, col_4, col_5, col_6, col_7, col_8, col_9,
col_10, col_11, col_12, col_13, col_14, col_15, col_16, col_17, col_18, col_19,
col_20, col_21, col_22, col_23, col_24, col_25, col_26, col_27, col_28, col_29,
col_30, col_31, col_32, col_33, col_34, col_35, col_36, col_37, col_38, col_39,
col_40, col_41, col_42, col_43, col_44, col_45, col_46, col_47, col_48, col_49,
col_50, col_51, col_52, col_53, col_54, col_55, col_56, col_57, col_58, col_59,
col_60, col_61, col_62, col_63, col_64, col_65, col_66, col_67, col_68, col_69,
col_70, col_71, col_72, col_73, col_74, col_75, col_76, col_77, col_78, col_79,
col_80, col_81, col_82, col_83, col_84, col_85, col_86, col_87, col_88, col_89,
col_90, col_91, col_92, col_93, col_94, col_95, col_96, col_97, col_98, col_99,
col_100, col_101, col_102, col_103, col_104, col_105, col_106, col_107,
col_108, col_109, col_110, col_111, col_112, col_113, col_114, col_115,
col_116, col_117
| mem-estimate=0B mem-reservation=0B thread-reservation=0
|
02:MERGING-EXCHANGE [UNPARTITIONED]
| order by: col_1 ASC, col_3 ASC
| limit: 250000
| mem-estimate=5.51MB mem-reservation=0B thread-reservation=0
| tuple-ids=1 row-size=936B cardinality=0
| in pipelines: 01(GETNEXT)
|
F00:PLAN FRAGMENT [RANDOM] hosts=6 instances=6
Per-Host Resources: mem-estimate=416.00MB mem-reservation=0B
thread-reservation=2
01:TOP-N [LIMIT=250000]
| order by: col_1 ASC, col_3 ASC
| mem-estimate=0B mem-reservation=0B thread-reservation=0
| tuple-ids=1 row-size=936B cardinality=0
| in pipelines: 01(GETNEXT), 00(OPEN)
|
00:SCAN KUDU [default.table_abc]
kudu predicates: col_1 < CAST(num_2 AS BIGINT), col_1 >= CAST(num_1 AS
BIGINT), col_2 = 'value_1'
mem-estimate=416.00MB mem-reservation=0B thread-reservation=1
tuple-ids=0 row-size=1.09KB cardinality=0
in pipelines: 00(GETNEXT)
----------------
Estimated Per-Host Mem: 441982976
Tables Missing Stats: default.table_abc
Request Pool: default-pool
Per Host Min Memory Reservation: node-1:22000(0) node-5:22000(0)
node-6:22000(0) node-4:22000(0) node-3:22000(0) node-2:22000(0)
impala-coordinator-node-1:22000(0)
Per Host Number of Fragment Instances: node-1:22000(1) node-5:22000(1)
node-6:22000(1) node-4:22000(1) node-3:22000(1) node-2:22000(1)
impala-coordinator-node-1:22000(1)
Admission result: Admitted immediately
Cluster Memory Admitted: 2.57 GB
Executor Group: default
ExecSummary:
Operator #Hosts #Inst Avg Time Max Time #Rows Est. #Rows
Peak Mem Est. Peak Mem Detail
--------------------------------------------------------------------------------------------------------------------------------------------
F01:ROOT 1 1 625.002ms 625.002ms
0 0
02:MERGING-EXCHANGE 1 1 10.000ms 10.000ms 250.00K 0
44.60 MB 5.51 MB UNPARTITIONED
F00:EXCHANGE SENDER 6 6 872.503ms 884.003ms
136.00 B 0
01:TOP-N 6 6 15s490ms 16s627ms 1.50M 0
772.60 MB 0
00:SCAN KUDU 6 6 19s313ms 20s704ms 212.00M 0
33.28 MB 416.00 MB default.table_abc
Errors:
Query Compilation: 7s284ms
- Metadata load started: 86.966ms (86.966ms)
- Metadata load finished. loaded-tables=1/1 load-requests=1
catalog-updates=5 storage-load-time=996ms: 6s062ms (5s975ms)
- Analysis finished: 6s133ms (71.252ms)
- Authorization finished (noop): 6s134ms (1.170ms)
- Value transfer graph computed: 6s173ms (39.176ms)
- Single node plan created: 7s206ms (1s032ms)
- Runtime filters computed: 7s211ms (4.579ms)
- Distributed plan created: 7s212ms (1.348ms)
- Planning finished: 7s284ms (71.641ms)
Query Timeline: 49s896ms
- Query submitted: 1.000ms (1.000ms)
- Planning finished: 7s356ms (7s355ms)
- Submit for admission: 7s356ms (0.000ns)
- Completed admission: 7s356ms (0.000ns)
- Ready to start on 7 backends: 7s357ms (1.000ms)
- All 7 execution backends (7 fragment instances) started: 7s374ms
(17.000ms)
- Rows available: 45s049ms (37s675ms)
- First row fetched: 45s095ms (46.000ms)
- Last row fetched: 49s126ms (4s031ms)
- Released admission control resources: 49s858ms (732.002ms)
- Unregister query: 49s896ms (38.000ms)
- AdmissionControlTimeSinceLastUpdate: 50.000ms
- ComputeScanRangeAssignmentTimer: 0.000ns
Frontend:
ImpalaServer:
- ClientFetchWaitTimer: 3s486ms
- NumRowsFetched: 250.00K (250000)
- NumRowsFetchedFromCache: 0 (0)
- RowMaterializationRate: 183.69 K/sec
- RowMaterializationTimer: 1s361ms
Execution Profile test_id:(Total: 39s053ms, non-child: 0.000ns, % non-child:
0.00%)
Number of filters: 0
Filter routing table:
ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected)
First arrived Completed Enabled
-------------------------------------------------------------------------------------------------------------------
Backend startup latencies: Count: 7, min / max: 4ms / 15ms, 25th %-ile:
8ms, 50th %-ile: 10ms, 75th %-ile: 15ms, 90th %-ile: 15ms, 95th %-ile: 15ms,
99.9th %-ile: 15ms
Slowest backend to start up: node-6:22000
Per Node Peak Memory Usage: node-1:22000(803.20 MB) node-5:22000(803.20 MB)
node-6:22000(804.80 MB) node-4:22000(804.80 MB) node-3:22000(804.80 MB)
node-2:22000(804.30 MB) impala-coordinator-node-1:22000(44.62 MB)
Per Node Bytes Read: node-1:22000(0) node-5:22000(0) node-6:22000(0)
node-4:22000(0) node-3:22000(0) node-2:22000(0)
impala-coordinator-node-1:22000(0)
Per Node User Time: node-1:22000(37s983ms) node-5:22000(38s197ms)
node-6:22000(42s559ms) node-4:22000(42s054ms) node-3:22000(42s405ms)
node-2:22000(41s349ms) impala-coordinator-node-1:22000(816.651ms)
Per Node System Time: node-1:22000(23s847ms) node-5:22000(24s348ms)
node-6:22000(26s663ms) node-4:22000(26s685ms) node-3:22000(27s136ms)
node-2:22000(26s544ms) impala-coordinator-node-1:22000(15.825ms)
- ExchangeScanRatio: 0.00
- FiltersReceived: 0 (0)
- FinalizationTimer: 0.000ns
- InnerNodeSelectivityRatio: 0.00
- NumBackends: 7 (7)
- NumCompletedBackends: 7 (7)
- NumFragmentInstances: 7 (7)
- NumFragments: 2 (2)
- TotalBytesRead: 0
- TotalBytesSent: 105.79 MB (110929234)
- TotalCpuTime: 6m40s
- TotalInnerBytesSent: 105.79 MB (110929234)
- TotalScanBytesSent: 0
Per Node Profiles:
node-1:22000:
- AdmissionSlots: 1 (1)
- BloomFilterBytes: 0
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
node-5:22000:
- AdmissionSlots: 1 (1)
- BloomFilterBytes: 0
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
node-6:22000:
- AdmissionSlots: 1 (1)
- BloomFilterBytes: 0
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
node-4:22000:
- AdmissionSlots: 1 (1)
- BloomFilterBytes: 0
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
node-3:22000:
- AdmissionSlots: 1 (1)
- BloomFilterBytes: 0
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
node-2:22000:
- AdmissionSlots: 1 (1)
- BloomFilterBytes: 0
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
impala-coordinator-node-1:22000:
- AdmissionSlots: 1 (1)
- BloomFilterBytes: 0
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Averaged Fragment F01:(Total: 41s751ms, non-child: 0.000ns, % non-child:
0.00%)
split sizes: min: 0, max: 0, avg: 0, stddev: 0
completion times: min:41s752ms max:41s752ms mean: 41s752ms
stddev:0.000ns
execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 0.00
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 44.62 MB (46784837)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 44.62 MB (46784837)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 41s748ms
- TotalThreadsSysTime: 15.825ms
- TotalThreadsUserTime: 816.651ms
- TotalThreadsVoluntaryContextSwitches: 584 (584)
Fragment Instance Lifecycle Timings:
- ExecTime: 4s065ms
- ExecTreeExecTime: 10.000ms
- OpenTime: 37s683ms
- ExecTreeOpenTime: 37s466ms
- PrepareTime: 2.000ms
- ExecTreePrepareTime: 0.000ns
PLAN_ROOT_SINK:(Total: 4s055ms, non-child: 0.000ns, % non-child: 0.00%)
- PeakMemoryUsage: 0
- RowsSent: 250.00K (250000)
- RowsSentRate: 61.65 K/sec
EXCHANGE_NODE (id=2):(Total: 37s476ms, non-child: 0.000ns, % non-child:
0.00%)
- ConvertRowBatchTime: 0.000ns
- MergeGetNext: 10.000ms
- MergeGetNextBatch: 1.000ms
- PeakMemoryUsage: 44.60 MB (46771278)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 6.67 K/sec
Buffer pool:
- AllocTime: 4.000ms
- CumulativeAllocationBytes: 511.99 MB (536862720)
- CumulativeAllocations: 510 (510)
- PeakReservation: 42.14 MB (44187648)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 42.14 MB (44187648)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- SystemAllocTime: 4.000ms
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Dequeue:
- FirstBatchWaitTime: 37s466ms
- TotalBytesDequeued: 293.49 MB (307742393)
- TotalGetBatchTime: 37s467ms
- DataWaitTime: 37s466ms
Enqueue:
- DeserializeRowBatchTime: 242.000ms
- TotalBatchesEnqueued: 255 (255)
- TotalBatchesReceived: 261 (261)
- TotalBytesReceived: 105.79 MB (110929234)
- TotalEarlySenders: 0 (0)
- TotalEosReceived: 0 (0)
- TotalHasDeferredRPCsTime: 37s114ms
- TotalRPCsDeferred: 246 (246)
CodeGen:(Total: 217.000ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 217.000ms
- CodegenSysTime: 5.794ms
- CodegenUserTime: 136.811ms
- CodegenVoluntaryContextSwitches: 93 (93)
- CompileTime: 96.000ms
- IrGenerationTime: 10.000ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 134 (134)
- NumInstructions: 3.50K (3499)
- OptimizationTime: 97.000ms
- PeakMemoryUsage: 1.71 MB (1791488)
- PrepareTime: 13.000ms
Coordinator Fragment F01:
Instance test_id (host=impala-coordinator-node-1:22000):(Total: 41s751ms,
non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2022-05-02 09:45:55.728
Fragment Instance Lifecycle Event Timeline: 41s751ms
- Prepare Finished: 3.000ms (3.000ms)
- Open Finished: 37s686ms (37s683ms)
- First Batch Produced: 37s686ms (0.000ns)
- First Batch Sent: 37s812ms (126.000ms)
- ExecInternal Finished: 41s751ms (3s939ms)
- MemoryUsage (1s000ms): 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24
KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24
KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24
KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24
KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 21.07 MB, 24.47 MB, 28.06
MB, 33.28 MB, 44.39 MB, 44.59 MB, 44.59 MB
- AverageThreadTokens: 0.00
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 44.62 MB (46784837)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 44.62 MB (46784837)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 41s748ms
- TotalThreadsSysTime: 15.825ms
- TotalThreadsUserTime: 816.651ms
- TotalThreadsVoluntaryContextSwitches: 584 (584)
Fragment Instance Lifecycle Timings:
- ExecTime: 4s065ms
- ExecTreeExecTime: 10.000ms
- OpenTime: 37s683ms
- ExecTreeOpenTime: 37s466ms
- PrepareTime: 2.000ms
- ExecTreePrepareTime: 0.000ns
PLAN_ROOT_SINK:(Total: 4s055ms, non-child: 0.000ns, % non-child: 0.00%)
- PeakMemoryUsage: 0
- RowsSent: 250.00K (250000)
- RowsSentRate: 61.65 K/sec
EXCHANGE_NODE (id=2):(Total: 37s476ms, non-child: 0.000ns, % non-child:
0.00%)
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 41s751ms
- Open Started: 220.000ms (220.000ms)
- Open Finished: 37s686ms (37s466ms)
- First Batch Requested: 37s686ms (0.000ns)
- First Batch Returned: 37s686ms (0.000ns)
- Last Batch Returned: 41s748ms (4s062ms)
- Closed: 41s751ms (3.000ms)
- ConvertRowBatchTime: 0.000ns
- MergeGetNext: 10.000ms
- MergeGetNextBatch: 1.000ms
- PeakMemoryUsage: 44.60 MB (46771278)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 6.67 K/sec
Buffer pool:
- AllocTime: 4.000ms
- CumulativeAllocationBytes: 511.99 MB (536862720)
- CumulativeAllocations: 510 (510)
- PeakReservation: 42.14 MB (44187648)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 42.14 MB (44187648)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- SystemAllocTime: 4.000ms
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Dequeue:
- BytesDequeued (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 601.98 KB, 1.18
MB, 1.18 MB, 4.12 MB, 58.01 MB, 132.81 MB, 205.43 MB
- FirstBatchWaitTime: 37s466ms
- TotalBytesDequeued: 293.49 MB (307742393)
- TotalGetBatchTime: 37s467ms
- DataWaitTime: 37s466ms
Enqueue:
- BytesReceived (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4.91 MB, 5.88
MB, 7.01 MB, 8.17 MB, 25.02 MB, 50.82 MB, 75.68 MB
- DeferredQueueSize (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 5, 5,
5, 6, 6
- DispatchTime: (Avg: -22246.000ns ; Min: -71104.000ns ; Max:
-13189.000ns ; Number of samples: 261)
- DeserializeRowBatchTime: 242.000ms
- TotalBatchesEnqueued: 255 (255)
- TotalBatchesReceived: 261 (261)
- TotalBytesReceived: 105.79 MB (110929234)
- TotalEarlySenders: 0 (0)
- TotalEosReceived: 0 (0)
- TotalHasDeferredRPCsTime: 37s114ms
- TotalRPCsDeferred: 246 (246)
CodeGen:(Total: 217.000ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 217.000ms
- CodegenSysTime: 5.794ms
- CodegenUserTime: 136.811ms
- CodegenVoluntaryContextSwitches: 93 (93)
- CompileTime: 96.000ms
- IrGenerationTime: 10.000ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 134 (134)
- NumInstructions: 3.50K (3499)
- OptimizationTime: 97.000ms
- PeakMemoryUsage: 1.71 MB (1791488)
- PrepareTime: 13.000ms
Averaged Fragment F00:(Total: 42s478ms, non-child: 0.000ns, % non-child:
0.00%)
split sizes: min: 0, max: 0, avg: 0, stddev: 0
completion times: min:42s473ms max:42s490ms mean: 42s480ms
stddev:5.335ms
execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
stddev:0.00 /sec
num instances: 6
- AverageThreadTokens: 9.55
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 804.18 MB (843248438)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 804.18 MB (843248438)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 6s151ms
- TotalStorageWaitTime: 4m50s
- TotalThreadsInvoluntaryContextSwitches: 24 (24)
- TotalThreadsTotalWallClockTime: 6m45s
- TotalThreadsSysTime: 25s871ms
- TotalThreadsUserTime: 40s758ms
- TotalThreadsVoluntaryContextSwitches: 88.47K (88472)
Fragment Instance Lifecycle Timings:
- ExecTime: 7s063ms
- ExecTreeExecTime: 1.833ms
- OpenTime: 35s413ms
- ExecTreeOpenTime: 34s801ms
- PrepareTime: 1.333ms
- ExecTreePrepareTime: 166.666us
KrpcDataStreamSender (dst_id=2):(Total: 7s023ms, non-child: 0.000ns, %
non-child: 0.00%)
- EosSent: 0 (0)
- PeakMemoryUsage: 136.00 B (136)
- RowsSent: 250.00K (250000)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 872.003ms
- TotalBytesSent: 17.63 MB (18488205)
- UncompressedRowBatchSize: 290.62 MB (304736180)
SORT_NODE (id=1):(Total: 34s803ms, non-child: 0.000ns, % non-child: 0.00%)
- InsertBatchTime: 10s491ms
- PeakMemoryUsage: 772.52 MB (810043050)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 7.19 K/sec
- TuplePoolReclamations: 2 (2)
KUDU_SCAN_NODE (id=0):(Total: 19s313ms, non-child: 0.000ns, % non-child:
0.00%)
- AverageScannerThreadConcurrency: 10.45
- KuduClientTime: 4m50s
- KuduRemoteScanTokens: 0 (0)
- MaterializeTupleTime: 5m28s
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 10 (10)
- PeakMemoryUsage: 32.56 MB (34138282)
- PeakScannerThreadConcurrency: 10 (10)
- RowBatchBytesEnqueued: 53.94 GB (57921613141)
- RowBatchQueueGetWaitTime: 19s171ms
- RowBatchQueuePeakMemoryUsage: 31.63 MB (33168384)
- RowBatchQueuePutWaitTime: 31s818ms
- RowBatchesEnqueued: 34.51K (34511)
- RowsRead: 35.33M (35333550)
- RowsReturned: 35.33M (35333550)
- RowsReturnedRate: 1.83 M/sec
- ScanRangesComplete: 10 (10)
- ScannerThreadsInvoluntaryContextSwitches: 11 (11)
- ScannerThreadsTotalWallClockTime: 6m2s
- ScannerThreadsSysTime: 20s596ms
- ScannerThreadsUserTime: 28s914ms
- ScannerThreadsVoluntaryContextSwitches: 58.55K (58545)
- TotalKuduScanRoundTrips: 46.03K (46029)
CodeGen:(Total: 611.501ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 611.334ms
- CodegenSysTime: 15.765ms
- CodegenUserTime: 517.740ms
- CodegenVoluntaryContextSwitches: 92 (92)
- CompileTime: 198.667ms
- IrGenerationTime: 15.666ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 168 (168)
- NumInstructions: 8.93K (8927)
- OptimizationTime: 384.167ms
- PeakMemoryUsage: 4.36 MB (4570624)
- PrepareTime: 12.500ms
Fragment F00:
Instance 6044b4a39c7f7962:eed7043600000006 (host=node-1:22000):(Total:
42s487ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2022-05-02 09:45:56.470
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
Fragment Instance Lifecycle Event Timeline: 42s487ms
- Prepare Finished: 999.998us (999.998us)
- Open Finished: 34s009ms (34s008ms)
- First Batch Produced: 34s009ms (0.000ns)
- First Batch Sent: 34s015ms (6.000ms)
- ExecInternal Finished: 42s487ms (8s472ms)
- MemoryUsage (1s000ms): 2.21 MB, 382.34 MB, 453.66 MB, 389.32 MB,
545.66 MB, 398.46 MB, 450.41 MB, 341.66 MB, 364.66 MB, 374.42 MB, 376.68 MB,
376.12 MB, 376.92 MB, 376.12 MB, 376.12 MB, 376.12 MB, 376.92 MB, 376.12 MB,
376.12 MB, 378.42 MB, 379.92 MB, 379.12 MB, 379.92 MB, 379.12 MB, 379.92 MB,
379.92 MB, 379.12 MB, 379.92 MB, 379.12 MB, 379.92 MB, 379.12 MB, 379.12 MB,
378.87 MB, 371.08 MB, 363.03 MB, 363.03 MB, 363.03 MB, 363.03 MB, 363.03 MB,
363.03 MB, 363.03 MB, 363.03 MB
- ThreadUsage (1s000ms): 6, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 5, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 8.75
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 803.20 MB (842211468)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 803.20 MB (842211468)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 7s563ms
- TotalStorageWaitTime: 4m22s
- TotalThreadsInvoluntaryContextSwitches: 20 (20)
- TotalThreadsTotalWallClockTime: 6m10s
- TotalThreadsSysTime: 23s847ms
- TotalThreadsUserTime: 37s983ms
- TotalThreadsVoluntaryContextSwitches: 83.55K (83551)
Fragment Instance Lifecycle Timings:
- ExecTime: 8s478ms
- ExecTreeExecTime: 1.000ms
- OpenTime: 34s008ms
- ExecTreeOpenTime: 33s379ms
- PrepareTime: 999.998us
- ExecTreePrepareTime: 0.000ns
KrpcDataStreamSender (dst_id=2):(Total: 8s433ms, non-child: 0.000ns, %
non-child: 0.00%)
ExecOption: Unpartitioned Sender Codegen Disabled: not needed
- BytesSent (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3.18 MB, 3.18 MB, 3.18
MB, 3.18 MB, 4.19 MB, 7.82 MB, 11.87 MB, 15.92 MB
- NetworkThroughput: (Avg: 22.40 MB/sec ; Min: 47.87 KB/sec ; Max:
121.23 MB/sec ; Number of samples: 43)
- RpcNetworkTime: (Avg: 357.597ms ; Min: 3.356ms ; Max: 8s565ms ;
Number of samples: 43)
- RpcRecvrTime: (Avg: -178206067.000ns ; Min: -4282486732.000ns ;
Max: -1261033.000ns ; Number of samples: 43)
- EosSent: 0 (0)
- PeakMemoryUsage: 136.00 B (136)
- RowsSent: 250.00K (250000)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 870.010ms
- TotalBytesSent: 17.33 MB (18170174)
- UncompressedRowBatchSize: 290.61 MB (304728781)
SORT_NODE (id=1):(Total: 33s380ms, non-child: 0.000ns, % non-child:
0.00%)
SortType: TopN
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 42s487ms
- Open Started: 629.999ms (629.999ms)
- Open Finished: 34s009ms (33s379ms)
- First Batch Requested: 34s009ms (0.000ns)
- First Batch Returned: 34s009ms (0.000ns)
- Last Batch Returned: 42s442ms (8s433ms)
- Closed: 42s487ms (45.000ms)
- InsertBatchTime: 9s756ms
- PeakMemoryUsage: 772.60 MB (810130432)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 7.49 K/sec
- TuplePoolReclamations: 3 (3)
KUDU_SCAN_NODE (id=0):(Total: 18s935ms, non-child: 0.000ns, %
non-child: 0.00%)
Table Name: default.table_abc
Node Lifecycle Event Timeline: 34s009ms
- Open Started: 629.999ms (629.999ms)
- Open Finished: 649.999ms (19.999ms)
- First Batch Requested: 649.999ms (0.000ns)
- First Batch Returned: 840.998ms (190.999ms)
- Last Batch Returned: 33s810ms (32s969ms)
- Closed: 34s009ms (199.000ms)
- AverageScannerThreadConcurrency: 9.84
- KuduClientTime: 4m22s
- KuduRemoteScanTokens: 0 (0)
- MaterializeTupleTime: 4m57s
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 10 (10)
- PeakMemoryUsage: 30.57 MB (32052224)
- PeakScannerThreadConcurrency: 10 (10)
- RowBatchBytesEnqueued: 50.56 GB (54285777920)
- RowBatchQueueGetWaitTime: 18s802ms
- RowBatchQueuePeakMemoryUsage: 30.56 MB (32044032)
- RowBatchQueuePutWaitTime: 28s692ms
- RowBatchesEnqueued: 32.34K (32345)
- RowsRead: 33.12M (33115154)
- RowsReturned: 33.12M (33115154)
- RowsReturnedRate: 1.75 M/sec
- ScanRangesComplete: 10 (10)
- ScannerThreadsInvoluntaryContextSwitches: 5 (5)
- ScannerThreadsTotalWallClockTime: 5m28s
- ScannerThreadsSysTime: 18s948ms
- ScannerThreadsUserTime: 26s852ms
- ScannerThreadsVoluntaryContextSwitches: 55.05K (55045)
- TotalKuduScanRoundTrips: 43.14K (43142)
CodeGen:(Total: 628.999ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 628.999ms
- CodegenSysTime: 10.024ms
- CodegenUserTime: 546.104ms
- CodegenVoluntaryContextSwitches: 92 (92)
- CompileTime: 195.999ms
- IrGenerationTime: 14.999ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 168 (168)
- NumInstructions: 8.93K (8927)
- OptimizationTime: 404.999ms
- PeakMemoryUsage: 4.36 MB (4570624)
- PrepareTime: 11.999ms
Instance 6044b4a39c7f7962:eed7043600000004 (host=node-6:22000):(Total:
42s480ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2022-05-02 09:45:56.468
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
Fragment Instance Lifecycle Event Timeline: 42s480ms
- Prepare Finished: 2.000ms (2.000ms)
- Open Finished: 36s213ms (36s211ms)
- First Batch Produced: 36s213ms (0.000ns)
- First Batch Sent: 36s220ms (7.000ms)
- ExecInternal Finished: 42s480ms (6s260ms)
- MemoryUsage (1s000ms): 2.21 MB, 365.37 MB, 438.32 MB, 594.37 MB,
403.02 MB, 423.31 MB, 370.23 MB, 458.56 MB, 580.07 MB, 389.77 MB, 417.73 MB,
423.64 MB, 421.23 MB, 422.03 MB, 421.23 MB, 421.23 MB, 422.03 MB, 421.23 MB,
422.84 MB, 422.03 MB, 448.22 MB, 453.73 MB, 461.53 MB, 460.73 MB, 460.73 MB,
461.53 MB, 460.73 MB, 460.73 MB, 460.73 MB, 461.53 MB, 461.53 MB, 462.33 MB,
460.73 MB, 460.73 MB, 459.93 MB, 451.08 MB, 443.03 MB, 443.03 MB, 443.03 MB,
443.03 MB, 443.03 MB, 443.03 MB
- ThreadUsage (1s000ms): 6, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 11, 6, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 10.04
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 804.80 MB (843897996)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 804.80 MB (843897996)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 5s350ms
- TotalStorageWaitTime: 4m58s
- TotalThreadsInvoluntaryContextSwitches: 25 (25)
- TotalThreadsTotalWallClockTime: 7m5s
- TotalThreadsSysTime: 26s663ms
- TotalThreadsUserTime: 42s559ms
- TotalThreadsVoluntaryContextSwitches: 91.66K (91656)
Fragment Instance Lifecycle Timings:
- ExecTime: 6s267ms
- ExecTreeExecTime: 0.000ns
- OpenTime: 36s211ms
- ExecTreeOpenTime: 35s576ms
- PrepareTime: 1.000ms
- ExecTreePrepareTime: 0.000ns
KrpcDataStreamSender (dst_id=2):(Total: 6s223ms, non-child: 0.000ns, %
non-child: 0.00%)
ExecOption: Unpartitioned Sender Codegen Disabled: not needed
- BytesSent (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 379.67 KB,
574.66 KB, 3.36 MB, 7.80 MB, 12.25 MB, 16.50 MB
- NetworkThroughput: (Avg: 2.78 MB/sec ; Min: 134.99 KB/sec ; Max:
30.34 MB/sec ; Number of samples: 44)
- RpcNetworkTime: (Avg: 248.116ms ; Min: 12.222ms ; Max: 2s889ms ;
Number of samples: 44)
- RpcRecvrTime: (Avg: -122756291.000ns ; Min: -1443300748.000ns ;
Max: -1258097.000ns ; Number of samples: 44)
- EosSent: 0 (0)
- PeakMemoryUsage: 136.00 B (136)
- RowsSent: 250.00K (250000)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 873.010ms
- TotalBytesSent: 17.71 MB (18573770)
- UncompressedRowBatchSize: 290.63 MB (304745705)
SORT_NODE (id=1):(Total: 35s576ms, non-child: 0.000ns, % non-child:
0.00%)
SortType: TopN
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 42s480ms
- Open Started: 637.007ms (637.007ms)
- Open Finished: 36s213ms (35s576ms)
- First Batch Requested: 36s213ms (0.000ns)
- First Batch Returned: 36s213ms (0.000ns)
- Last Batch Returned: 42s436ms (6s223ms)
- Closed: 42s480ms (44.000ms)
- InsertBatchTime: 11s497ms
- PeakMemoryUsage: 772.60 MB (810130432)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 7.03 K/sec
- TuplePoolReclamations: 3 (3)
KUDU_SCAN_NODE (id=0):(Total: 18s949ms, non-child: 0.000ns, %
non-child: 0.00%)
Table Name: default.table_abc
Node Lifecycle Event Timeline: 36s213ms
- Open Started: 637.007ms (637.007ms)
- Open Finished: 664.008ms (27.000ms)
- First Batch Requested: 664.008ms (0.000ns)
- First Batch Returned: 888.011ms (224.002ms)
- Last Batch Returned: 36s004ms (35s116ms)
- Closed: 36s213ms (209.002ms)
- AverageScannerThreadConcurrency: 10.82
- KuduClientTime: 4m58s
- KuduRemoteScanTokens: 0 (0)
- MaterializeTupleTime: 5m37s
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 11 (11)
- PeakMemoryUsage: 33.28 MB (34892800)
- PeakScannerThreadConcurrency: 11 (11)
- RowBatchBytesEnqueued: 55.62 GB (59725264896)
- RowBatchQueueGetWaitTime: 18s796ms
- RowBatchQueuePeakMemoryUsage: 32.17 MB (33730560)
- RowBatchQueuePutWaitTime: 42s614ms
- RowBatchesEnqueued: 35.59K (35586)
- RowsRead: 36.43M (36433200)
- RowsReturned: 36.43M (36433200)
- RowsReturnedRate: 1.92 M/sec
- ScanRangesComplete: 11 (11)
- ScannerThreadsInvoluntaryContextSwitches: 13 (13)
- ScannerThreadsTotalWallClockTime: 6m22s
- ScannerThreadsSysTime: 21s219ms
- ScannerThreadsUserTime: 29s716ms
- ScannerThreadsVoluntaryContextSwitches: 60.79K (60793)
- TotalKuduScanRoundTrips: 47.46K (47457)
CodeGen:(Total: 635.007ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 1 (1)
- CodegenTotalWallClockTime: 635.007ms
- CodegenSysTime: 15.991ms
- CodegenUserTime: 528.504ms
- CodegenVoluntaryContextSwitches: 92 (92)
- CompileTime: 207.002ms
- IrGenerationTime: 16.000ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 168 (168)
- NumInstructions: 8.93K (8927)
- OptimizationTime: 399.004ms
- PeakMemoryUsage: 4.36 MB (4570624)
- PrepareTime: 12.000ms
Instance 6044b4a39c7f7962:eed7043600000002 (host=node-3:22000):(Total:
42s479ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2022-05-02 09:45:56.461
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
Fragment Instance Lifecycle Event Timeline: 42s480ms
- Prepare Finished: 1.999ms (1.999ms)
- Open Finished: 37s670ms (37s668ms)
- First Batch Produced: 37s670ms (0.000ns)
- First Batch Sent: 37s676ms (5.999ms)
- ExecInternal Finished: 42s480ms (4s803ms)
- MemoryUsage (1s000ms): 64.34 MB, 401.77 MB, 518.52 MB, 405.93 MB,
564.48 MB, 418.06 MB, 470.89 MB, 600.56 MB, 396.48 MB, 413.73 MB, 413.73 MB,
413.73 MB, 414.53 MB, 413.73 MB, 415.33 MB, 414.53 MB, 414.53 MB, 414.53 MB,
413.73 MB, 413.73 MB, 413.73 MB, 421.53 MB, 420.73 MB, 420.73 MB, 420.73 MB,
421.28 MB, 420.48 MB, 421.53 MB, 420.73 MB, 422.33 MB, 421.53 MB, 420.73 MB,
420.73 MB, 422.34 MB, 420.73 MB, 421.53 MB, 416.71 MB, 404.64 MB, 403.03 MB,
403.03 MB, 403.03 MB, 403.03 MB
- ThreadUsage (1s000ms): 6, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 9, 2, 1, 1, 1, 1
- AverageThreadTokens: 10.41
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 804.80 MB (843897996)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 804.80 MB (843897996)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 3s903ms
- TotalStorageWaitTime: 5m23s
- TotalThreadsInvoluntaryContextSwitches: 34 (34)
- TotalThreadsTotalWallClockTime: 7m22s
- TotalThreadsSysTime: 27s136ms
- TotalThreadsUserTime: 42s405ms
- TotalThreadsVoluntaryContextSwitches: 90.68K (90680)
Fragment Instance Lifecycle Timings:
- ExecTime: 4s808ms
- ExecTreeExecTime: 5.999ms
- OpenTime: 37s668ms
- ExecTreeOpenTime: 37s097ms
- PrepareTime: 1.999ms
- ExecTreePrepareTime: 999.999us
KrpcDataStreamSender (dst_id=2):(Total: 4s769ms, non-child: 0.000ns, %
non-child: 0.00%)
ExecOption: Unpartitioned Sender Codegen Disabled: not needed
- BytesSent (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 585.66 KB,
4.18 MB, 8.62 MB, 12.68 MB, 16.92 MB
- NetworkThroughput: (Avg: 5.50 MB/sec ; Min: 902.23 KB/sec ; Max:
93.65 MB/sec ; Number of samples: 44)
- RpcNetworkTime: (Avg: 183.510ms ; Min: 4.097ms ; Max: 438.466ms ;
Number of samples: 44)
- RpcRecvrTime: (Avg: -91124357.000ns ; Min: -218711905.000ns ; Max:
-1292804.000ns ; Number of samples: 44)
- EosSent: 0 (0)
- PeakMemoryUsage: 136.00 B (136)
- RowsSent: 250.00K (250000)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 865.999ms
- TotalBytesSent: 17.73 MB (18590995)
- UncompressedRowBatchSize: 290.63 MB (304743618)
SORT_NODE (id=1):(Total: 37s103ms, non-child: 0.000ns, % non-child:
0.00%)
SortType: TopN
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 42s479ms
- Open Started: 572.999ms (572.999ms)
- Open Finished: 37s670ms (37s097ms)
- First Batch Requested: 37s670ms (0.000ns)
- First Batch Returned: 37s670ms (0.000ns)
- Last Batch Returned: 42s446ms (4s775ms)
- Closed: 42s479ms (32.999ms)
- InsertBatchTime: 11s157ms
- PeakMemoryUsage: 772.60 MB (810130432)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 6.74 K/sec
- TuplePoolReclamations: 3 (3)
KUDU_SCAN_NODE (id=0):(Total: 20s704ms, non-child: 0.000ns, %
non-child: 0.00%)
Table Name: default.table_abc
Node Lifecycle Event Timeline: 37s670ms
- Open Started: 572.999ms (572.999ms)
- Open Finished: 589.999ms (16.999ms)
- First Batch Requested: 589.999ms (0.000ns)
- First Batch Returned: 788.999ms (198.999ms)
- Last Batch Returned: 37s468ms (36s679ms)
- Closed: 37s670ms (201.999ms)
- AverageScannerThreadConcurrency: 10.81
- KuduClientTime: 5m23s
- KuduRemoteScanTokens: 0 (0)
- MaterializeTupleTime: 6m2s
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 11 (11)
- PeakMemoryUsage: 33.28 MB (34892800)
- PeakScannerThreadConcurrency: 11 (11)
- RowBatchBytesEnqueued: 55.63 GB (59735334912)
- RowBatchQueueGetWaitTime: 20s562ms
- RowBatchQueuePeakMemoryUsage: 32.17 MB (33730560)
- RowBatchQueuePutWaitTime: 35s451ms
- RowBatchesEnqueued: 35.59K (35592)
- RowsRead: 36.44M (36442638)
- RowsReturned: 36.44M (36442638)
- RowsReturnedRate: 1.76 M/sec
- ScanRangesComplete: 11 (11)
- ScannerThreadsInvoluntaryContextSwitches: 18 (18)
- ScannerThreadsTotalWallClockTime: 6m40s
- ScannerThreadsSysTime: 21s588ms
- ScannerThreadsUserTime: 29s950ms
- ScannerThreadsVoluntaryContextSwitches: 60.14K (60141)
- TotalKuduScanRoundTrips: 47.49K (47488)
CodeGen:(Total: 570.999ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 1 (1)
- CodegenTotalWallClockTime: 570.999ms
- CodegenSysTime: 23.250ms
- CodegenUserTime: 490.702ms
- CodegenVoluntaryContextSwitches: 92 (92)
- CompileTime: 185.999ms
- IrGenerationTime: 11.999ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 168 (168)
- NumInstructions: 8.93K (8927)
- OptimizationTime: 359.999ms
- PeakMemoryUsage: 4.36 MB (4570624)
- PrepareTime: 11.999ms
Instance 6044b4a39c7f7962:eed7043600000003 (host=node-4:22000):(Total:
42s476ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2022-05-02 09:45:56.464
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
Fragment Instance Lifecycle Event Timeline: 42s476ms
- Prepare Finished: 999.994us (999.994us)
- Open Finished: 35s745ms (35s744ms)
- First Batch Produced: 35s745ms (0.000ns)
- First Batch Sent: 35s752ms (6.999ms)
- ExecInternal Finished: 42s476ms (6s723ms)
- MemoryUsage (1s000ms): 8.13 KB, 260.07 MB, 429.23 MB, 608.06 MB,
461.59 MB, 591.06 MB, 454.66 MB, 427.68 MB, 392.59 MB, 418.77 MB, 419.73 MB,
420.53 MB, 421.08 MB, 420.03 MB, 420.28 MB, 419.73 MB, 419.73 MB, 420.53 MB,
425.03 MB, 429.34 MB, 445.16 MB, 450.23 MB, 450.23 MB, 450.23 MB, 450.23 MB,
450.23 MB, 451.03 MB, 450.23 MB, 450.23 MB, 450.23 MB, 451.03 MB, 450.24 MB,
451.03 MB, 451.03 MB, 450.23 MB, 436.56 MB, 432.53 MB, 432.53 MB, 432.53 MB,
432.53 MB, 432.53 MB, 432.53 MB
- ThreadUsage (1s000ms): 1, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 11, 3, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 9.84
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 804.80 MB (843897996)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 804.80 MB (843897996)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 5s805ms
- TotalStorageWaitTime: 5m4s
- TotalThreadsInvoluntaryContextSwitches: 30 (30)
- TotalThreadsTotalWallClockTime: 7m
- TotalThreadsSysTime: 26s685ms
- TotalThreadsUserTime: 42s054ms
- TotalThreadsVoluntaryContextSwitches: 90.92K (90924)
Fragment Instance Lifecycle Timings:
- ExecTime: 6s730ms
- ExecTreeExecTime: 0.000ns
- OpenTime: 35s744ms
- ExecTreeOpenTime: 35s123ms
- PrepareTime: 999.994us
- ExecTreePrepareTime: 0.000ns
KrpcDataStreamSender (dst_id=2):(Total: 6s687ms, non-child: 0.000ns, %
non-child: 0.00%)
ExecOption: Unpartitioned Sender Codegen Disabled: not needed
- BytesSent (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 388.12 KB,
388.12 KB, 2.61 MB, 7.15 MB, 11.49 MB, 16.04 MB
- NetworkThroughput: (Avg: 3.05 MB/sec ; Min: 104.92 KB/sec ; Max:
29.33 MB/sec ; Number of samples: 45)
- RpcNetworkTime: (Avg: 263.329ms ; Min: 12.921ms ; Max: 3s823ms ;
Number of samples: 45)
- RpcRecvrTime: (Avg: -130347565.000ns ; Min: -1910204152.000ns ;
Max: -1429922.000ns ; Number of samples: 45)
- EosSent: 0 (0)
- PeakMemoryUsage: 136.00 B (136)
- RowsSent: 250.00K (250000)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 880.994ms
- TotalBytesSent: 18.52 MB (19416128)
- UncompressedRowBatchSize: 290.60 MB (304721211)
SORT_NODE (id=1):(Total: 35s123ms, non-child: 0.000ns, % non-child:
0.00%)
SortType: TopN
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 42s476ms
- Open Started: 621.995ms (621.995ms)
- Open Finished: 35s745ms (35s123ms)
- First Batch Requested: 35s745ms (0.000ns)
- First Batch Returned: 35s745ms (0.000ns)
- Last Batch Returned: 42s432ms (6s686ms)
- Closed: 42s476ms (43.999ms)
- InsertBatchTime: 10s954ms
- PeakMemoryUsage: 772.60 MB (810130432)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 7.12 K/sec
- TuplePoolReclamations: 3 (3)
KUDU_SCAN_NODE (id=0):(Total: 18s931ms, non-child: 0.000ns, %
non-child: 0.00%)
Table Name: default.table_abc
Node Lifecycle Event Timeline: 35s745ms
- Open Started: 621.995ms (621.995ms)
- Open Finished: 645.995ms (23.999ms)
- First Batch Requested: 645.995ms (0.000ns)
- First Batch Returned: 791.994ms (145.999ms)
- Last Batch Returned: 35s544ms (34s752ms)
- Closed: 35s745ms (200.998ms)
- AverageScannerThreadConcurrency: 10.73
- KuduClientTime: 5m4s
- KuduRemoteScanTokens: 0 (0)
- MaterializeTupleTime: 5m42s
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 11 (11)
- PeakMemoryUsage: 33.28 MB (34892800)
- PeakScannerThreadConcurrency: 11 (11)
- RowBatchBytesEnqueued: 55.63 GB (59728621568)
- RowBatchQueueGetWaitTime: 18s785ms
- RowBatchQueuePeakMemoryUsage: 32.17 MB (33730560)
- RowBatchQueuePutWaitTime: 31s936ms
- RowBatchesEnqueued: 35.59K (35588)
- RowsRead: 36.44M (36435994)
- RowsReturned: 36.44M (36435994)
- RowsReturnedRate: 1.92 M/sec
- ScanRangesComplete: 11 (11)
- ScannerThreadsInvoluntaryContextSwitches: 15 (15)
- ScannerThreadsTotalWallClockTime: 6m17s
- ScannerThreadsSysTime: 21s145ms
- ScannerThreadsUserTime: 29s800ms
- ScannerThreadsVoluntaryContextSwitches: 60.77K (60774)
- TotalKuduScanRoundTrips: 47.47K (47465)
CodeGen:(Total: 620.995ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 2 (2)
- CodegenTotalWallClockTime: 620.995ms
- CodegenSysTime: 14.067ms
- CodegenUserTime: 522.296ms
- CodegenVoluntaryContextSwitches: 92 (92)
- CompileTime: 202.998ms
- IrGenerationTime: 18.999ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 168 (168)
- NumInstructions: 8.93K (8927)
- OptimizationTime: 383.997ms
- PeakMemoryUsage: 4.36 MB (4570624)
- PrepareTime: 14.999ms
Instance 6044b4a39c7f7962:eed7043600000005 (host=node-5:22000):(Total:
42s474ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2022-05-02 09:45:56.457
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
Fragment Instance Lifecycle Event Timeline: 42s474ms
- Prepare Finished: 1.000ms (1.000ms)
- Open Finished: 34s258ms (34s257ms)
- First Batch Produced: 34s258ms (0.000ns)
- First Batch Sent: 34s264ms (6.000ms)
- ExecInternal Finished: 42s474ms (8s210ms)
- MemoryUsage (1s000ms): 3.01 MB, 379.71 MB, 464.66 MB, 383.02 MB,
466.51 MB, 373.96 MB, 411.66 MB, 464.23 MB, 474.72 MB, 534.46 MB, 325.42 MB,
325.42 MB, 324.62 MB, 324.62 MB, 324.62 MB, 324.62 MB, 324.62 MB, 325.42 MB,
346.16 MB, 340.62 MB, 340.62 MB, 340.62 MB, 341.43 MB, 340.62 MB, 340.62 MB,
340.62 MB, 340.62 MB, 341.42 MB, 341.42 MB, 341.42 MB, 340.62 MB, 340.62 MB,
339.82 MB, 326.95 MB, 324.53 MB, 324.53 MB, 324.53 MB, 324.53 MB, 324.53 MB,
324.53 MB, 324.53 MB, 324.53 MB
- ThreadUsage (1s000ms): 6, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
11, 10, 2, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 8.67
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 803.20 MB (842211468)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 803.20 MB (842211468)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 7s322ms
- TotalStorageWaitTime: 4m19s
- TotalThreadsInvoluntaryContextSwitches: 20 (20)
- TotalThreadsTotalWallClockTime: 6m7s
- TotalThreadsSysTime: 24s348ms
- TotalThreadsUserTime: 38s197ms
- TotalThreadsVoluntaryContextSwitches: 83.07K (83068)
Fragment Instance Lifecycle Timings:
- ExecTime: 8s216ms
- ExecTreeExecTime: 1.000ms
- OpenTime: 34s257ms
- ExecTreeOpenTime: 33s651ms
- PrepareTime: 1.000ms
- ExecTreePrepareTime: 0.000ns
KrpcDataStreamSender (dst_id=2):(Total: 8s182ms, non-child: 0.000ns, %
non-child: 0.00%)
ExecOption: Unpartitioned Sender Codegen Disabled: not needed
- BytesSent (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 384.21 KB, 384.21 KB,
384.21 KB, 579.52 KB, 3.38 MB, 7.44 MB, 11.30 MB, 15.36 MB
- NetworkThroughput: (Avg: 3.51 MB/sec ; Min: 57.24 KB/sec ; Max:
55.82 MB/sec ; Number of samples: 41)
- RpcNetworkTime: (Avg: 363.379ms ; Min: 6.721ms ; Max: 6s824ms ;
Number of samples: 41)
- RpcRecvrTime: (Avg: -181019758.000ns ; Min: -3411512967.000ns ;
Max: -1296478.000ns ; Number of samples: 41)
- EosSent: 0 (0)
- PeakMemoryUsage: 136.00 B (136)
- RowsSent: 250.00K (250000)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 859.005ms
- TotalBytesSent: 16.58 MB (17383444)
- UncompressedRowBatchSize: 290.62 MB (304741907)
SORT_NODE (id=1):(Total: 33s652ms, non-child: 0.000ns, % non-child:
0.00%)
SortType: TopN
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 42s474ms
- Open Started: 607.003ms (607.003ms)
- Open Finished: 34s258ms (33s651ms)
- First Batch Requested: 34s258ms (0.000ns)
- First Batch Returned: 34s258ms (0.000ns)
- Last Batch Returned: 42s441ms (8s183ms)
- Closed: 42s474ms (33.000ms)
- InsertBatchTime: 9s819ms
- PeakMemoryUsage: 772.60 MB (810130432)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 7.43 K/sec
- TuplePoolReclamations: 3 (3)
KUDU_SCAN_NODE (id=0):(Total: 19s128ms, non-child: 0.000ns, %
non-child: 0.00%)
Table Name: default.table_abc
Node Lifecycle Event Timeline: 34s258ms
- Open Started: 607.003ms (607.003ms)
- Open Finished: 628.003ms (21.000ms)
- First Batch Requested: 628.003ms (0.000ns)
- First Batch Returned: 849.004ms (221.001ms)
- Last Batch Returned: 34s065ms (33s216ms)
- Closed: 34s258ms (193.001ms)
- AverageScannerThreadConcurrency: 9.73
- KuduClientTime: 4m19s
- KuduRemoteScanTokens: 0 (0)
- MaterializeTupleTime: 4m54s
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 10 (10)
- PeakMemoryUsage: 31.67 MB (33206272)
- PeakScannerThreadConcurrency: 10 (10)
- RowBatchBytesEnqueued: 50.58 GB (54312631296)
- RowBatchQueueGetWaitTime: 18s999ms
- RowBatchQueuePeakMemoryUsage: 30.56 MB (32044032)
- RowBatchQueuePutWaitTime: 27s649ms
- RowBatchesEnqueued: 32.36K (32361)
- RowsRead: 33.13M (33131362)
- RowsReturned: 33.13M (33131362)
- RowsReturnedRate: 1.73 M/sec
- ScanRangesComplete: 10 (10)
- ScannerThreadsInvoluntaryContextSwitches: 9 (9)
- ScannerThreadsTotalWallClockTime: 5m24s
- ScannerThreadsSysTime: 19s336ms
- ScannerThreadsUserTime: 27s112ms
- ScannerThreadsVoluntaryContextSwitches: 54.64K (54636)
- TotalKuduScanRoundTrips: 43.15K (43154)
CodeGen:(Total: 606.003ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 606.003ms
- CodegenSysTime: 16.129ms
- CodegenUserTime: 516.952ms
- CodegenVoluntaryContextSwitches: 92 (92)
- CompileTime: 199.001ms
- IrGenerationTime: 15.000ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 168 (168)
- NumInstructions: 8.93K (8927)
- OptimizationTime: 380.002ms
- PeakMemoryUsage: 4.36 MB (4570624)
- PrepareTime: 12.000ms
Instance 6044b4a39c7f7962:eed7043600000001 (host=node-2:22000):(Total:
42s472ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2022-05-02 09:45:56.460
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
Fragment Instance Lifecycle Event Timeline: 42s472ms
- Prepare Finished: 2.000ms (2.000ms)
- Open Finished: 34s592ms (34s590ms)
- First Batch Produced: 34s592ms (0.000ns)
- First Batch Sent: 34s598ms (6.000ms)
- ExecInternal Finished: 42s472ms (7s874ms)
- MemoryUsage (1s000ms): 2.21 MB, 372.32 MB, 468.82 MB, 394.56 MB,
459.07 MB, 598.06 MB, 394.52 MB, 444.49 MB, 474.27 MB, 472.73 MB, 474.09 MB,
472.73 MB, 473.28 MB, 473.53 MB, 472.73 MB, 472.73 MB, 472.73 MB, 472.73 MB,
473.53 MB, 475.78 MB, 478.03 MB, 477.23 MB, 477.23 MB, 477.23 MB, 478.03 MB,
478.83 MB, 477.23 MB, 477.23 MB, 477.23 MB, 477.23 MB, 478.03 MB, 478.59 MB,
477.23 MB, 471.60 MB, 461.14 MB, 459.53 MB, 459.53 MB, 459.53 MB, 459.53 MB,
459.53 MB, 459.53 MB, 459.53 MB
- ThreadUsage (1s000ms): 6, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 8, 2, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 9.61
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 804.30 MB (843373708)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 804.30 MB (843373708)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 6s962ms
- TotalStorageWaitTime: 4m57s
- TotalThreadsInvoluntaryContextSwitches: 19 (19)
- TotalThreadsTotalWallClockTime: 6m46s
- TotalThreadsSysTime: 26s544ms
- TotalThreadsUserTime: 41s349ms
- TotalThreadsVoluntaryContextSwitches: 90.95K (90954)
Fragment Instance Lifecycle Timings:
- ExecTime: 7s880ms
- ExecTreeExecTime: 3.000ms
- OpenTime: 34s590ms
- ExecTreeOpenTime: 33s983ms
- PrepareTime: 2.000ms
- ExecTreePrepareTime: 0.000ns
KrpcDataStreamSender (dst_id=2):(Total: 7s846ms, non-child: 0.000ns, %
non-child: 0.00%)
ExecOption: Unpartitioned Sender Codegen Disabled: not needed
- BytesSent (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 390.60 KB, 781.21 KB,
781.21 KB, 781.21 KB, 3.42 MB, 7.91 MB, 12.00 MB, 16.69 MB
- NetworkThroughput: (Avg: 4.51 MB/sec ; Min: 61.24 KB/sec ; Max:
79.02 MB/sec ; Number of samples: 44)
- RpcNetworkTime: (Avg: 321.779ms ; Min: 4.883ms ; Max: 6s562ms ;
Number of samples: 44)
- RpcRecvrTime: (Avg: -159590226.000ns ; Min: -3280054040.000ns ;
Max: -1265135.000ns ; Number of samples: 44)
- EosSent: 0 (0)
- PeakMemoryUsage: 136.00 B (136)
- RowsSent: 250.00K (250000)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 883.003ms
- TotalBytesSent: 17.92 MB (18794723)
- UncompressedRowBatchSize: 290.62 MB (304735863)
SORT_NODE (id=1):(Total: 33s985ms, non-child: 0.000ns, % non-child:
0.00%)
SortType: TopN
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 42s472ms
- Open Started: 609.002ms (609.002ms)
- Open Finished: 34s592ms (33s983ms)
- First Batch Requested: 34s592ms (0.000ns)
- First Batch Returned: 34s592ms (0.000ns)
- Last Batch Returned: 42s441ms (7s849ms)
- Closed: 42s472ms (31.000ms)
- InsertBatchTime: 9s765ms
- PeakMemoryUsage: 772.10 MB (809606144)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 7.36 K/sec
- TuplePoolReclamations: 2 (2)
KUDU_SCAN_NODE (id=0):(Total: 19s229ms, non-child: 0.000ns, %
non-child: 0.00%)
Table Name: default.table_abc
Node Lifecycle Event Timeline: 34s592ms
- Open Started: 609.002ms (609.002ms)
- Open Finished: 636.002ms (27.000ms)
- First Batch Requested: 636.002ms (0.000ns)
- First Batch Returned: 769.002ms (133.000ms)
- Last Batch Returned: 34s393ms (33s624ms)
- Closed: 34s592ms (199.000ms)
- AverageScannerThreadConcurrency: 10.76
- KuduClientTime: 4m57s
- KuduRemoteScanTokens: 0 (0)
- MaterializeTupleTime: 5m36s
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 11 (11)
- PeakMemoryUsage: 33.28 MB (34892800)
- PeakScannerThreadConcurrency: 11 (11)
- RowBatchBytesEnqueued: 55.64 GB (59742048256)
- RowBatchQueueGetWaitTime: 19s081ms
- RowBatchQueuePeakMemoryUsage: 32.17 MB (33730560)
- RowBatchQueuePutWaitTime: 24s568ms
- RowBatchesEnqueued: 35.60K (35596)
- RowsRead: 36.44M (36442952)
- RowsReturned: 36.44M (36442952)
- RowsReturnedRate: 1.90 M/sec
- ScanRangesComplete: 11 (11)
- ScannerThreadsInvoluntaryContextSwitches: 8 (8)
- ScannerThreadsTotalWallClockTime: 6m3s
- ScannerThreadsSysTime: 21s343ms
- ScannerThreadsUserTime: 30s051ms
- ScannerThreadsVoluntaryContextSwitches: 59.88K (59885)
- TotalKuduScanRoundTrips: 47.47K (47471)
CodeGen:(Total: 607.002ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 1 (1)
- CodegenTotalWallClockTime: 606.002ms
- CodegenSysTime: 15.132ms
- CodegenUserTime: 501.887ms
- CodegenVoluntaryContextSwitches: 92 (92)
- CompileTime: 201.000ms
- IrGenerationTime: 17.000ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 168 (168)
- NumInstructions: 8.93K (8927)
- OptimizationTime: 377.001ms
- PeakMemoryUsage: 4.36 MB (4570624)
- PrepareTime: 12.000ms
