[
https://issues.apache.org/jira/browse/HIVE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973569#comment-16973569
]
Slim Bouguerra commented on HIVE-22492:
---------------------------------------
{code}
IO-Elevator-Thread-9 Blocked CPU usage on sample: 609ms
org.apache.hadoop.hive.llap.cache.LowLevelLrfuCachePolicy.notifyUnlock(LlapCacheableBuffer)
LowLevelLrfuCachePolicy.java:125
org.apache.hadoop.hive.llap.cache.CacheContentsTracker.notifyUnlock(LlapCacheableBuffer)
CacheContentsTracker.java:173
org.apache.hadoop.hive.llap.cache.LowLevelCacheImpl.unlockBuffer(LlapDataBuffer,
boolean) LowLevelCacheImpl.java:391
org.apache.hadoop.hive.llap.cache.LowLevelCacheImpl.decRefBuffers(List)
LowLevelCacheImpl.java:379
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.returnData(Reader$OrcEncodedColumnBatch)
OrcEncodedDataReader.java:759
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.returnData(Object)
OrcEncodedDataReader.java:110
org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.returnSourceData(EncodedColumnBatch)
EncodedDataConsumer.java:100
org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.consumeData(EncodedColumnBatch)
EncodedDataConsumer.java:92
org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.consumeData(Object)
EncodedDataConsumer.java:34
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(int,
StripeInformation, OrcProto$RowIndex[], List, List, boolean[], boolean[],
Consumer) EncodedReaderImpl.java:532
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead()
OrcEncodedDataReader.java:407
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run()
OrcEncodedDataReader.java:266
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run()
OrcEncodedDataReader.java:263
java.security.AccessController.doPrivileged(PrivilegedExceptionAction,
AccessControlContext) AccessController.java (native)
javax.security.auth.Subject.doAs(Subject, PrivilegedExceptionAction)
Subject.java:422
org.apache.hadoop.security.UserGroupInformation.doAs(PrivilegedExceptionAction)
UserGroupInformation.java:1688
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal()
OrcEncodedDataReader.java:263
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal()
OrcEncodedDataReader.java:110
org.apache.tez.common.CallableWithNdc.call() CallableWithNdc.java:36
org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call()
StatsRecordingThreadPool.java:110
java.util.concurrent.FutureTask.run() FutureTask.java:266
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
ThreadPoolExecutor.java:1142
java.util.concurrent.ThreadPoolExecutor$Worker.run()
ThreadPoolExecutor.java:617
java.lang.Thread.run() Thread.java:745
{code}
> Amortize lock contention due to LRFU accounting
> -----------------------------------------------
>
> Key: HIVE-22492
> URL: https://issues.apache.org/jira/browse/HIVE-22492
> Project: Hive
> Issue Type: Improvement
> Reporter: Slim Bouguerra
> Assignee: Slim Bouguerra
> Priority: Major
>
> LRFU eviction policy can be a major source of contention under high load.
> This can be see on the following profiles.
> To fix this the idea is to use a batching wrapper to amortize the locking
> contention.
> The trick i a common way to amortize locking as explained here
> http://www.ece.eng.wayne.edu/~sjiang/pubs/papers/ding-09-BP-Wrapper.pdf
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
