Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/10550 )
Change subject: PREVIEW: IMPALA-7078: improve memory consumption of wide Avro
scans
......................................................................
Patch Set 2:
Here's an example of the profile with the new additions:
HDFS_SCAN_NODE (id=0):(Total: 60.443ms, non-child: 60.443ms, % non-child:
100.00%)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 9.36 MB (9816312)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 9.36 MB (9816312)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 9.36 MB (9816312)
- BytesSkipped: 0
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 2 (2)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 21.409ms
- MaxCompressedTextFileLength: 0
- NumDisksAccessed: 1 (1)
- NumScannerThreadReservationsDenied: 0 (0)
- NumScannerThreadsStarted: 1 (1)
- PeakMemoryUsage: 24.99 MB (26205298)
- PerReadThreadRawHdfsThroughput: 3.80 GB/sec
- RemoteScanRanges: 0 (0)
- RowBatchBytesEnqueued: 20.89 MB (21903380)
- RowBatchQueueCapacity: 5 (5)
- RowBatchQueueGetWaitTime: 59.187ms
- RowBatchQueuePeakMemoryUsage: 8.85 MB (9279347)
- RowBatchQueuePutWaitTime: 0.000ns
- RowBatchesEnqueued: 6 (6)
- RowsRead: 200.00K (200000)
- RowsReturned: 747 (747)
- RowsReturnedRate: 12.36 K/sec
- ScanRangesComplete: 1 (1)
- ScannerThreadsInvoluntaryContextSwitches: 1 (1)
- ScannerThreadsTotalWallClockTime: 63.681ms
- MaterializeTupleTime(*): 38.682ms
- ScannerThreadsSysTime: 0.000ns
- ScannerThreadsUserTime: 60.000ms
- ScannerThreadsVoluntaryContextSwitches: 4 (4)
- TotalRawHdfsOpenFileTime(*): 1.449ms
- TotalRawHdfsReadTime(*): 2.408ms
- TotalReadThroughput: 0.00 /sec
--
To view, visit http://gerrit.cloudera.org:8080/10550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iebd2600b4784fd19696c9b92eefb7d7e9db0c80b
Gerrit-Change-Number: 10550
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <[email protected]>
Gerrit-Reviewer: Dan Hecht <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
Gerrit-Comment-Date: Wed, 30 May 2018 23:14:54 +0000
Gerrit-HasComments: No
