anujphadke has posted comments on this change.
Change subject: IMPALA-3766: Applying LZ4 compression on buffers before
spilling
......................................................................
Patch Set 2:
Before -
[localhost:21000] > profile;
Query Runtime Profile:
Query (id=a348adf5718dd795:e38675e8e7ea1a94):
Summary:
Session ID: 624487d98d041692:7ce44c89f323fc8b
Session Type: BEESWAX
Start Time: 2016-07-07 15:44:59.063728000
End Time: 2016-07-07 16:50:19.658784000
Query Type: QUERY
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 2.7.0-cdh5-INTERNAL DEBUG (build
d43a6771b7a892df2bab8d62c25ec8154ac9fab4)
User: anuj
Connected User: anuj
Delegated User:
Network Address: ::ffff:127.0.0.1:59531
Default Db: tpch
Sql Statement: select * from lineitem order by l_quantity
Coordinator: anuj-OptiPlex-9020:22000
Query Options (non default): MEM_LIMIT=1073741824
Plan:
----------------
Estimated Per-Host Requirements: Memory=864.00MB VCores=1
02:MERGING-EXCHANGE [UNPARTITIONED]
| order by: l_quantity ASC
| hosts=1 per-host-mem=unavailable
| tuple-ids=1 row-size=263B cardinality=6001215
|
01:SORT
| order by: l_quantity ASC
| hosts=1 per-host-mem=160.00MB
| tuple-ids=1 row-size=263B cardinality=6001215
|
00:SCAN HDFS [tpch.lineitem, RANDOM]
partitions=1/1 files=1 size=718.94MB
table stats: 6001215 rows total
column stats: all
hosts=1 per-host-mem=704.00MB
tuple-ids=0 row-size=263B cardinality=6001215
----------------
Estimated Per-Host Mem: 905969664
Estimated Per-Host VCores: 1
Request Pool: default-pool
Admission result: Admitted immediately
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak
Mem Est. Peak Mem Detail
---------------------------------------------------------------------------------------------------------------
02:MERGING-EXCHANGE 1 1s020ms 1s020ms 6.00M 6.00M
0 -1.00 B UNPARTITIONED
01:SORT 1 37s063ms 37s063ms 6.00M 6.00M 816.20
MB 160.00 MB
00:SCAN HDFS 1 344.851ms 344.851ms 6.00M 6.00M 201.39
MB 704.00 MB tpch.lineitem
Planner Timeline: 105.297ms
- Analysis finished: 11.531ms (11.531ms)
- Equivalence classes computed: 31.690ms (20.159ms)
- Single node plan created: 45.135ms (13.444ms)
- Runtime filters computed: 46.123ms (988.831us)
- Distributed plan created: 51.654ms (5.530ms)
- Planning finished: 105.297ms (53.643ms)
Query Timeline: 1h5m
- Start execution: 40.528us (40.528us)
- Planning finished: 5s618ms (5s618ms)
- Submit for admission: 5s620ms (2.164ms)
- Completed admission: 5s620ms (146.718us)
- Ready to start 1 remote fragments: 5s621ms (714.555us)
- All 1 remote fragments started: 5s622ms (1.010ms)
- Rows available: 31s511ms (25s888ms)
- First row fetched: 31s680ms (169.455ms)
- Unregister query: 1h5m (1h4m)
ImpalaServer:
- ClientFetchWaitTimer: 1h4m
- RowMaterializationTimer: 34s415ms
Execution Profile a348adf5718dd795:e38675e8e7ea1a94:(Total: 27s454ms,
non-child: 0.000ns, % non-child: 0.00%)
Number of filters: 0
Filter routing table:
ID Src. Node Tgt. Node(s) Targets Target type Partition filter Pending
(Expected) First arrived Completed
-------------------------------------------------------------------------------------------------------------------
Fragment start latencies: Count: 1, 25th %-ile: 0, 50th %-ile: 0, 75th
%-ile: 0, 90th %-ile: 0, 95th %-ile: 0, 99.9th %-ile: 0
Per Node Peak Memory Usage: anuj-OptiPlex-9020:22000(978.74 MB)
- FiltersReceived: 0 (0)
- FinalizationTimer: 0.000ns
Coordinator Fragment F01:(Total: 26s923ms, non-child: 14.099ms, %
non-child: 0.05%)
MemoryUsage(1m4s): 12.26 MB, 20.12 MB, 22.13 MB, 19.77 MB, 21.58 MB,
20.77 MB, 20.43 MB, 21.78 MB, 19.75 MB, 22.03 MB, 20.41 MB, 20.87 MB, 21.66 MB,
19.83 MB, 22.03 MB, 20.11 MB, 21.33 MB, 21.15 MB, 20.08 MB, 22.14 MB, 19.90 MB,
21.69 MB, 20.84 MB, 20.45 MB, 22.01 MB, 19.73 MB, 22.11 MB, 20.45 MB, 20.92 MB,
21.70 MB, 19.84 MB, 22.17 MB, 20.05 MB, 21.29 MB, 21.27 MB, 20.08 MB, 21.99 MB,
19.89 MB, 21.69 MB, 20.88 MB, 20.44 MB, 22.03 MB, 19.77 MB, 22.00 MB, 20.53 MB,
20.71 MB, 21.70 MB, 19.89 MB, 22.15 MB, 20.20 MB, 21.22 MB, 21.39 MB, 19.93 MB,
22.16 MB, 19.93 MB, 21.56 MB, 20.96 MB, 20.34 MB, 22.05 MB, 19.71 MB, 15.62 MB
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 22.55 MB (23643536)
- PerHostPeakMemUsage: 0
- PrepareTime: 43.673us
- RowsProduced: 6.00M (6001215)
- TotalCpuTime: 1h5m
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 219 (219)
- BlocksRecycled: 32 (32)
- BufferedPins: 74 (74)
- BytesWritten: 944.31 MB (990183996)
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 819.20 MB (858993472)
- PeakMemoryUsage: 816.00 MB (855638016)
- TotalBufferWaitTime: 0.000ns
- TotalEncryptionTime: 0.000ns
- TotalIntegrityCheckTime: 0.000ns
- TotalReadBlockTime: 485.948ms
EXCHANGE_NODE (id=2):(Total: 26s909ms, non-child: 1s020ms, % non-child:
3.79%)
BytesReceived(1m4s): 8.15 MB, 23.92 MB, 36.90 MB, 48.93 MB, 62.21 MB,
74.37 MB, 87.28 MB, 100.01 MB, 112.26 MB, 125.53 MB, 137.54 MB, 150.62 MB,
163.13 MB, 175.50 MB, 188.63 MB, 200.69 MB, 213.86 MB, 226.13 MB, 238.83 MB,
251.80 MB, 263.83 MB, 277.07 MB, 289.19 MB, 302.03 MB, 314.76 MB, 326.91 MB,
340.17 MB, 352.13 MB, 365.15 MB, 377.61 MB, 389.96 MB, 403.11 MB, 415.06 MB,
428.29 MB, 440.57 MB, 453.15 MB, 466.12 MB, 478.23 MB, 491.48 MB, 503.60 MB,
516.43 MB, 529.18 MB, 541.29 MB, 554.54 MB, 566.55 MB, 579.55 MB, 592.15 MB,
604.48 MB, 617.63 MB, 629.62 MB, 642.76 MB, 655.07 MB, 667.58 MB, 680.65 MB,
692.70 MB, 705.94 MB, 718.11 MB, 730.91 MB, 743.72 MB, 755.81 MB, 766.16 MB
- BytesReceived: 766.85 MB (804096787)
- ConvertRowBatchTime: 0.000ns
- DeserializeRowBatchTimer: 3s974ms
- FirstBatchArrivalWaitTime: 25s888ms
- MergeGetNext: 997.745ms
- MergeGetNextBatch: 64.131ms
- PeakMemoryUsage: 0
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 223.02 K/sec
- SendersBlockedTimer: 1h3m
- SendersBlockedTotalTimer(*): 1h3m
Averaged Fragment F00:(Total: 1h4m, non-child: 1h3m, % non-child: 98.77%)
split sizes: min: 718.94 MB, max: 718.94 MB, avg: 718.94 MB, stddev: 0
completion times: min:1h4m max:1h4m mean: 1h4m stddev:0.000ns
execution rates: min:190.64 KB/sec max:190.64 KB/sec mean:190.64 KB/sec
stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 1.03
- BloomFilterBytes: 0
- PeakMemoryUsage: 969.24 MB (1016316584)
- PerHostPeakMemUsage: 978.74 MB (1026286248)
- PrepareTime: 143.403ms
- RowsProduced: 6.00M (6001215)
- TotalCpuTime: 2m42s
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1h3m
- TotalStorageWaitTime: 1s124ms
CodeGen:(Total: 212.028ms, non-child: 212.028ms, % non-child: 100.00%)
- CodegenTime: 1.478ms
- CompileTime: 6.577ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.12 MB (2224192)
- NumFunctions: 29 (29)
- NumInstructions: 529 (529)
- OptimizationTime: 74.084ms
- PrepareTime: 126.830ms
DataStreamSender (dst_id=2):(Total: 9s711ms, non-child: 9s711ms, %
non-child: 100.00%)
- BytesSent: 766.85 MB (804096787)
- NetworkThroughput(*): 1.04 GB/sec
- OverallThroughput: 78.97 MB/sec
- PeakMemoryUsage: 680.00 B (680)
- RowsReturned: 6.00M (6001215)
- SerializeBatchTime: 8s992ms
- TransmitDataRPCTime: 717.249ms
- UncompressedRowBatchSize: 1.51 GB (1625021116)
SORT_NODE (id=1):(Total: 37s408ms, non-child: 37s063ms, % non-child:
99.08%)
- InMemorySortTime: 6s488ms
- InitialRunsCreated: 2 (2)
- MergeGetNext: 11s580ms
- MergeGetNextBatch: 4s490ms
- PeakMemoryUsage: 816.20 MB (855842816)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 160.42 K/sec
- SortDataSize: 1.49 GB (1600992812)
- SpilledRuns: 3 (3)
- TotalMergesPerformed: 1 (1)
HDFS_SCAN_NODE (id=0):(Total: 344.851ms, non-child: 344.851ms, %
non-child: 100.00%)
- AverageHdfsReadThreadConcurrency: 0.10
- AverageScannerThreadConcurrency: 5.76
- BytesRead: 718.94 MB (753867192)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 718.94 MB (753867192)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 718.94 MB (753867192)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumDisksAccessed: 2 (2)
- NumScannerThreadsStarted: 6 (6)
- PeakMemoryUsage: 201.39 MB (211169280)
- PerReadThreadRawHdfsThroughput: 192.58 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 6.00M (6001215)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 17.40 M/sec
- ScanRangesComplete: 6 (6)
- ScannerThreadsInvoluntaryContextSwitches: 371 (371)
- ScannerThreadsTotalWallClockTime: 1m57s
- DelimiterParseTime: 3s697ms
- MaterializeTupleTime(*): 6s048ms
- ScannerThreadsSysTime: 35.892ms
- ScannerThreadsUserTime: 9s863ms
- ScannerThreadsVoluntaryContextSwitches: 5.90K (5903)
- TotalRawHdfsReadTime(*): 3s733ms
- TotalReadThroughput: 35.07 MB/sec
Fragment F00:
Instance a348adf5718dd795:e38675e8e7ea1a96
(host=anuj-OptiPlex-9020:22000):(Total: 1h4m, non-child: 1h3m, % non-child:
98.77%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/718.94
MB
MemoryUsage(1m4s): 270.90 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB,
32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB,
32.52 MB, 32.52 MB, 32.52 MB, 32.58 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB,
32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB,
32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB,
32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB,
32.51 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB,
32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 28.77 MB
ThreadUsage(1m4s): 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.03
- BloomFilterBytes: 0
- PeakMemoryUsage: 969.24 MB (1016316584)
- PerHostPeakMemUsage: 978.74 MB (1026286248)
- PrepareTime: 143.403ms
- RowsProduced: 6.00M (6001215)
- TotalCpuTime: 2m42s
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1h3m
- TotalStorageWaitTime: 1s124ms
CodeGen:(Total: 212.028ms, non-child: 212.028ms, % non-child: 100.00%)
- CodegenTime: 1.478ms
- CompileTime: 6.577ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.12 MB (2224192)
- NumFunctions: 29 (29)
- NumInstructions: 529 (529)
- OptimizationTime: 74.084ms
- PrepareTime: 126.830ms
DataStreamSender (dst_id=2):(Total: 9s711ms, non-child: 9s711ms, %
non-child: 100.00%)
- BytesSent: 766.85 MB (804096787)
- NetworkThroughput(*): 1.04 GB/sec
- OverallThroughput: 78.97 MB/sec
- PeakMemoryUsage: 680.00 B (680)
- RowsReturned: 6.00M (6001215)
- SerializeBatchTime: 8s992ms
- TransmitDataRPCTime: 717.249ms
- UncompressedRowBatchSize: 1.51 GB (1625021116)
SORT_NODE (id=1):(Total: 37s408ms, non-child: 37s063ms, % non-child:
99.08%)
ExecOption: Codegen Enabled
- InMemorySortTime: 6s488ms
- InitialRunsCreated: 2 (2)
- MergeGetNext: 11s580ms
- MergeGetNextBatch: 4s490ms
- PeakMemoryUsage: 816.20 MB (855842816)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 160.42 K/sec
- SortDataSize: 1.49 GB (1600992812)
- SpilledRuns: 3 (3)
- TotalMergesPerformed: 1 (1)
HDFS_SCAN_NODE (id=0):(Total: 344.851ms, non-child: 344.851ms, %
non-child: 100.00%)
ExecOption: Expr Evaluation Codegen Disabled, Codegen enabled: 0 out
of 6
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/718.94
MB
Hdfs Read Thread Concurrency Bucket: 0:92.68% 1:4.878% 2:2.439% 3:0%
4:0%
File Formats: TEXT/NONE:6
BytesRead(500.000ms): 0, 192.00 MB, 192.00 MB, 192.00 MB, 200.00 MB,
240.00 MB, 248.00 MB, 288.00 MB, 296.00 MB, 336.00 MB, 344.00 MB, 384.00 MB,
392.00 MB, 432.00 MB, 448.00 MB, 478.94 MB, 486.94 MB, 486.94 MB, 486.94 MB,
486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB, 486.94 MB,
486.94 MB, 486.94 MB, 486.94 MB, 494.94 MB, 542.94 MB, 550.94 MB, 590.94 MB,
630.94 MB, 638.94 MB, 670.94 MB, 702.94 MB, 710.94 MB, 718.94 MB, 718.94 MB,
718.94 MB
- AverageHdfsReadThreadConcurrency: 0.10
- AverageScannerThreadConcurrency: 5.76
- BytesRead: 718.94 MB (753867192)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 718.94 MB (753867192)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 718.94 MB (753867192)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumDisksAccessed: 2 (2)
- NumScannerThreadsStarted: 6 (6)
- PeakMemoryUsage: 201.39 MB (211169280)
- PerReadThreadRawHdfsThroughput: 192.58 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 6.00M (6001215)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 17.40 M/sec
- ScanRangesComplete: 6 (6)
- ScannerThreadsInvoluntaryContextSwitches: 371 (371)
- ScannerThreadsTotalWallClockTime: 1m57s
- DelimiterParseTime: 3s697ms
- MaterializeTupleTime(*): 6s048ms
- ScannerThreadsSysTime: 35.892ms
- ScannerThreadsUserTime: 9s863ms
- ScannerThreadsVoluntaryContextSwitches: 5.90K (5903)
- TotalRawHdfsReadTime(*): 3s733ms
- TotalReadThroughput: 35.07 MB/sec
After -
[localhost:21000] > profile;
Query Runtime Profile:
Query (id=394ee7185b41be23:949a047c1e09ec82):
Summary:
Session ID: 84ece9f0c0c9e06:f6c333f0e5737cb5
Session Type: BEESWAX
Start Time: 2016-07-07 12:43:01.225510000
End Time: 2016-07-07 13:48:28.978336000
Query Type: QUERY
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 2.7.0-cdh5-INTERNAL DEBUG (build
34e49668e40386c7e50f87f01e4c5ad926983717)
User: anuj
Connected User: anuj
Delegated User:
Network Address: ::ffff:127.0.0.1:57170
Default Db: tpch
Sql Statement: select * from lineitem order by l_quantity
Coordinator: anuj-OptiPlex-9020:22000
Query Options (non default): MEM_LIMIT=1073741824
Plan:
----------------
Estimated Per-Host Requirements: Memory=864.00MB VCores=1
02:MERGING-EXCHANGE [UNPARTITIONED]
| order by: l_quantity ASC
| hosts=1 per-host-mem=unavailable
| tuple-ids=1 row-size=263B cardinality=6001215
|
01:SORT
| order by: l_quantity ASC
| hosts=1 per-host-mem=160.00MB
| tuple-ids=1 row-size=263B cardinality=6001215
|
00:SCAN HDFS [tpch.lineitem, RANDOM]
partitions=1/1 files=1 size=718.94MB
table stats: 6001215 rows total
column stats: all
hosts=1 per-host-mem=704.00MB
tuple-ids=0 row-size=263B cardinality=6001215
----------------
Estimated Per-Host Mem: 905969664
Estimated Per-Host VCores: 1
Request Pool: default-pool
Admission result: Admitted immediately
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak
Mem Est. Peak Mem Detail
---------------------------------------------------------------------------------------------------------------
02:MERGING-EXCHANGE 1 1s021ms 1s021ms 6.00M 6.00M
0 -1.00 B UNPARTITIONED
01:SORT 1 41s010ms 41s010ms 6.00M 6.00M 816.20
MB 160.00 MB
00:SCAN HDFS 1 412.149ms 412.149ms 6.00M 6.00M 201.76
MB 704.00 MB tpch.lineitem
Planner Timeline: 116.792ms
- Analysis finished: 11.939ms (11.939ms)
- Equivalence classes computed: 32.413ms (20.473ms)
- Single node plan created: 46.386ms (13.973ms)
- Runtime filters computed: 47.696ms (1.310ms)
- Distributed plan created: 54.638ms (6.942ms)
- Planning finished: 116.792ms (62.153ms)
Query Timeline: 1h5m
- Start execution: 43.421us (43.421us)
- Planning finished: 4s826ms (4s826ms)
- Submit for admission: 4s826ms (261.600us)
- Completed admission: 4s826ms (149.430us)
- Ready to start 1 remote fragments: 4s828ms (1.831ms)
- All 1 remote fragments started: 4s829ms (1.125ms)
- Rows available: 33s993ms (29s163ms)
- First row fetched: 34s394ms (401.447ms)
- Unregister query: 1h5m (1h4m)
ImpalaServer:
- ClientFetchWaitTimer: 1h4m
- RowMaterializationTimer: 34s291ms
Execution Profile 394ee7185b41be23:949a047c1e09ec82:(Total: 30s738ms,
non-child: 0.000ns, % non-child: 0.00%)
Number of filters: 0
Filter routing table:
ID Src. Node Tgt. Node(s) Targets Target type Partition filter Pending
(Expected) First arrived Completed
-------------------------------------------------------------------------------------------------------------------
Fragment start latencies: Count: 1, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th
%-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms
Per Node Peak Memory Usage: anuj-OptiPlex-9020:22000(970.91 MB)
- FiltersReceived: 0 (0)
- FinalizationTimer: 0.000ns
Coordinator Fragment F01:(Total: 30s200ms, non-child: 15.938ms, %
non-child: 0.05%)
MemoryUsage(1m4s): 10.99 MB, 19.34 MB, 22.10 MB, 19.26 MB, 21.25 MB,
20.63 MB, 19.67 MB, 21.83 MB, 19.01 MB, 21.66 MB, 20.05 MB, 20.10 MB, 21.68 MB,
19.01 MB, 22.03 MB, 19.70 MB, 20.70 MB, 21.08 MB, 19.28 MB, 22.10 MB, 19.27 MB,
21.21 MB, 20.80 MB, 19.59 MB, 21.98 MB, 19.02 MB, 21.66 MB, 20.18 MB, 20.09 MB,
21.72 MB, 18.95 MB, 21.94 MB, 19.71 MB, 20.50 MB, 21.30 MB, 19.15 MB, 22.05 MB,
19.39 MB, 21.13 MB, 20.86 MB, 19.51 MB, 21.99 MB, 19.07 MB, 21.40 MB, 20.29 MB,
19.82 MB, 21.82 MB, 19.00 MB, 21.91 MB, 19.93 MB, 20.42 MB, 21.44 MB, 19.04 MB,
22.03 MB, 19.51 MB, 20.87 MB, 21.00 MB, 19.45 MB, 22.05 MB, 19.22 MB, 17.13 MB
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 22.55 MB (23644896)
- PerHostPeakMemUsage: 0
- PrepareTime: 224.969us
- RowsProduced: 6.00M (6001215)
- TotalCpuTime: 1h5m
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 219 (219)
- BlocksRecycled: 32 (32)
- BufferedPins: 74 (74)
- BytesWritten: 944.32 MB (990186348)
- CompressedBytesWritten: 437.30 MB (458537208)
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 819.20 MB (858993472)
- PeakMemoryUsage: 816.00 MB (855638016)
- TotalBufferWaitTime: 0.000ns
- TotalEncryptionTime: 0.000ns
- TotalIntegrityCheckTime: 0.000ns
- TotalReadBlockTime: 226.005ms
EXCHANGE_NODE (id=2):(Total: 30s184ms, non-child: 1s021ms, % non-child:
3.39%)
BytesReceived(1m4s): 7.20 MB, 23.35 MB, 36.77 MB, 48.83 MB, 62.53 MB,
74.82 MB, 87.88 MB, 101.03 MB, 113.27 MB, 126.98 MB, 139.11 MB, 152.48 MB,
165.37 MB, 177.76 MB, 191.43 MB, 203.47 MB, 217.02 MB, 229.56 MB, 242.35 MB,
255.81 MB, 267.86 MB, 281.57 MB, 293.87 MB, 306.84 MB, 320.09 MB, 332.25 MB,
346.00 MB, 358.15 MB, 371.47 MB, 384.39 MB, 396.76 MB, 410.51 MB, 422.55 MB,
436.12 MB, 448.79 MB, 461.47 MB, 475.04 MB, 487.12 MB, 500.78 MB, 513.14 MB,
526.06 MB, 539.35 MB, 551.49 MB, 565.21 MB, 577.44 MB, 590.70 MB, 603.79 MB,
616.08 MB, 629.79 MB, 641.85 MB, 655.32 MB, 668.05 MB, 680.60 MB, 694.24 MB,
706.31 MB, 719.94 MB, 732.40 MB, 745.26 MB, 758.61 MB, 770.73 MB, 782.37 MB
- BytesReceived: 783.74 MB (821814808)
- ConvertRowBatchTime: 0.000ns
- DeserializeRowBatchTimer: 3s948ms
- FirstBatchArrivalWaitTime: 29s163ms
- MergeGetNext: 997.881ms
- MergeGetNextBatch: 65.561ms
- PeakMemoryUsage: 0
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 198.81 K/sec
- SendersBlockedTimer: 1h3m
- SendersBlockedTotalTimer(*): 1h3m
Averaged Fragment F00:(Total: 1h4m, non-child: 1h3m, % non-child: 98.68%)
split sizes: min: 718.94 MB, max: 718.94 MB, avg: 718.94 MB, stddev: 0
completion times: min:1h4m max:1h4m mean: 1h4m stddev:0.000ns
execution rates: min:190.29 KB/sec max:190.29 KB/sec mean:190.29 KB/sec
stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 1.03
- BloomFilterBytes: 0
- PeakMemoryUsage: 969.61 MB (1016709800)
- PerHostPeakMemUsage: 970.91 MB (1018077864)
- PrepareTime: 167.610ms
- RowsProduced: 6.00M (6001215)
- TotalCpuTime: 3m1s
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1h3m
- TotalStorageWaitTime: 1s303ms
CodeGen:(Total: 202.085ms, non-child: 202.085ms, % non-child: 100.00%)
- CodegenTime: 274.215us
- CompileTime: 3.591ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.12 MB (2224196)
- NumFunctions: 29 (29)
- NumInstructions: 529 (529)
- OptimizationTime: 54.022ms
- PrepareTime: 143.628ms
DataStreamSender (dst_id=2):(Total: 9s619ms, non-child: 9s619ms, %
non-child: 100.00%)
- BytesSent: 783.74 MB (821814808)
- NetworkThroughput(*): 1.03 GB/sec
- OverallThroughput: 81.48 MB/sec
- PeakMemoryUsage: 680.00 B (680)
- RowsReturned: 6.00M (6001215)
- SerializeBatchTime: 8s876ms
- TransmitDataRPCTime: 741.034ms
- UncompressedRowBatchSize: 1.51 GB (1625021116)
SORT_NODE (id=1):(Total: 41s422ms, non-child: 41s010ms, % non-child:
99.01%)
- InMemorySortTime: 6s580ms
- InitialRunsCreated: 2 (2)
- MergeGetNext: 12s295ms
- MergeGetNextBatch: 5s318ms
- PeakMemoryUsage: 816.20 MB (855842816)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 144.88 K/sec
- SortDataSize: 1.49 GB (1600992812)
- SpilledRuns: 3 (3)
- TotalMergesPerformed: 1 (1)
HDFS_SCAN_NODE (id=0):(Total: 412.149ms, non-child: 412.149ms, %
non-child: 100.00%)
- AverageHdfsReadThreadConcurrency: 0.24
- AverageScannerThreadConcurrency: 5.74
- BytesRead: 718.94 MB (753867192)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 718.94 MB (753867192)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 718.94 MB (753867192)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumDisksAccessed: 2 (2)
- NumScannerThreadsStarted: 6 (6)
- PeakMemoryUsage: 201.76 MB (211562496)
- PerReadThreadRawHdfsThroughput: 195.65 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 6.00M (6001215)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 14.56 M/sec
- ScanRangesComplete: 6 (6)
- ScannerThreadsInvoluntaryContextSwitches: 320 (320)
- ScannerThreadsTotalWallClockTime: 2m12s
- DelimiterParseTime: 3s826ms
- MaterializeTupleTime(*): 6s619ms
- ScannerThreadsSysTime: 20.468ms
- ScannerThreadsUserTime: 10s538ms
- ScannerThreadsVoluntaryContextSwitches: 6.01K (6006)
- TotalRawHdfsReadTime(*): 3s674ms
- TotalReadThroughput: 30.59 MB/sec
Fragment F00:
Instance 394ee7185b41be23:949a047c1e09ec84
(host=anuj-OptiPlex-9020:22000):(Total: 1h4m, non-child: 1h3m, % non-child:
98.68%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/718.94
MB
MemoryUsage(1m4s): 297.48 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB,
32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB,
32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB,
32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB,
32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB,
32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB,
32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB,
32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 32.52 MB, 30.89 MB
ThreadUsage(1m4s): 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.03
- BloomFilterBytes: 0
- PeakMemoryUsage: 969.61 MB (1016709800)
- PerHostPeakMemUsage: 970.91 MB (1018077864)
- PrepareTime: 167.610ms
- RowsProduced: 6.00M (6001215)
- TotalCpuTime: 3m1s
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1h3m
- TotalStorageWaitTime: 1s303ms
CodeGen:(Total: 202.085ms, non-child: 202.085ms, % non-child: 100.00%)
- CodegenTime: 274.215us
- CompileTime: 3.591ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.12 MB (2224196)
- NumFunctions: 29 (29)
- NumInstructions: 529 (529)
- OptimizationTime: 54.022ms
- PrepareTime: 143.628ms
DataStreamSender (dst_id=2):(Total: 9s619ms, non-child: 9s619ms, %
non-child: 100.00%)
- BytesSent: 783.74 MB (821814808)
- NetworkThroughput(*): 1.03 GB/sec
- OverallThroughput: 81.48 MB/sec
- PeakMemoryUsage: 680.00 B (680)
- RowsReturned: 6.00M (6001215)
- SerializeBatchTime: 8s876ms
- TransmitDataRPCTime: 741.034ms
- UncompressedRowBatchSize: 1.51 GB (1625021116)
SORT_NODE (id=1):(Total: 41s422ms, non-child: 41s010ms, % non-child:
99.01%)
ExecOption: Codegen Enabled
- InMemorySortTime: 6s580ms
- InitialRunsCreated: 2 (2)
- MergeGetNext: 12s295ms
- MergeGetNextBatch: 5s318ms
- PeakMemoryUsage: 816.20 MB (855842816)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 144.88 K/sec
- SortDataSize: 1.49 GB (1600992812)
- SpilledRuns: 3 (3)
- TotalMergesPerformed: 1 (1)
HDFS_SCAN_NODE (id=0):(Total: 412.149ms, non-child: 412.149ms, %
non-child: 100.00%)
ExecOption: Expr Evaluation Codegen Disabled, Codegen enabled: 0 out
of 6
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/718.94
MB
Hdfs Read Thread Concurrency Bucket: 0:91.3% 1:4.348% 2:0% 3:2.174%
4:2.174%
File Formats: TEXT/NONE:6
BytesRead(500.000ms): 0, 96.00 MB, 192.00 MB, 192.00 MB, 192.00 MB,
200.00 MB, 240.00 MB, 240.00 MB, 288.00 MB, 288.00 MB, 304.00 MB, 336.00 MB,
336.00 MB, 384.00 MB, 384.00 MB, 432.00 MB, 432.00 MB, 478.94 MB, 478.94 MB,
478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB,
478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB, 478.94 MB,
486.94 MB, 534.94 MB, 550.94 MB, 582.94 MB, 622.94 MB, 622.94 MB, 662.94 MB,
662.94 MB, 702.94 MB, 702.94 MB, 718.94 MB, 718.94 MB, 718.94 MB, 718.94 MB
- AverageHdfsReadThreadConcurrency: 0.24
- AverageScannerThreadConcurrency: 5.74
- BytesRead: 718.94 MB (753867192)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 718.94 MB (753867192)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 718.94 MB (753867192)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumDisksAccessed: 2 (2)
- NumScannerThreadsStarted: 6 (6)
- PeakMemoryUsage: 201.76 MB (211562496)
- PerReadThreadRawHdfsThroughput: 195.65 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 6.00M (6001215)
- RowsReturned: 6.00M (6001215)
- RowsReturnedRate: 14.56 M/sec
- ScanRangesComplete: 6 (6)
- ScannerThreadsInvoluntaryContextSwitches: 320 (320)
- ScannerThreadsTotalWallClockTime: 2m12s
- DelimiterParseTime: 3s826ms
- MaterializeTupleTime(*): 6s619ms
- ScannerThreadsSysTime: 20.468ms
- ScannerThreadsUserTime: 10s538ms
- ScannerThreadsVoluntaryContextSwitches: 6.01K (6006)
- TotalRawHdfsReadTime(*): 3s674ms
- TotalReadThroughput: 30.59 MB/sec
[localhost:21000] >
--
To view, visit http://gerrit.cloudera.org:8080/3478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d49bd8d6d7643c84cefd1274c18b52907ca1488
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <[email protected]>
Gerrit-Reviewer: Mostafa Mokhtar <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
Gerrit-Reviewer: anujphadke <[email protected]>
Gerrit-HasComments: No
