[
https://issues.apache.org/jira/browse/HIVE-21391?focusedWorklogId=246410&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246410
]
ASF GitHub Bot logged work on HIVE-21391:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 21/May/19 22:20
Start Date: 21/May/19 22:20
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #641: HIVE-21391: Fix
VRBsQueue size estimate
URL: https://github.com/apache/hive/pull/641
Limit the queue using absolute min and max in addition to that make the
queue length estimate related to actual size.
As minor switched to ArrayBlockingQ to avoid extra GC.
Add fn to drain the pool when done to help GC.
Fixed some NPE issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 246410)
Time Spent: 10m
Remaining Estimate: 0h
> LLAP: Pool of column vector buffers can cause memory pressure
> -------------------------------------------------------------
>
> Key: HIVE-21391
> URL: https://issues.apache.org/jira/browse/HIVE-21391
> Project: Hive
> Issue Type: Bug
> Components: llap
> Affects Versions: 4.0.0, 3.2.0
> Reporter: Prasanth Jayachandran
> Assignee: Prasanth Jayachandran
> Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21391.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Where there are too many columns (in the order of 100s), with decimal, string
> types the column vector pool of buffers created here
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java#L59]
> can cause memory pressure.
> Example:
> 128 (poolSize) * 300 (numCols) * 1024 (batchSize) * 80 (decimalSize) ~= 3GB
> The pool size keeps increasing when there is slow consumer but fast llap io
> (SSDs) leading to GC pressure when all LLAP io threads read splits from same
> table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
