[
https://issues.apache.org/jira/browse/HIVE-24297?focusedWorklogId=505215&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-505215
]
ASF GitHub Bot logged work on HIVE-24297:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Oct/20 13:30
Start Date: 27/Oct/20 13:30
Worklog Time Spent: 10m
Work Description: szlta opened a new pull request #1614:
URL: https://github.com/apache/hive/pull/1614
HIVE-23741 introduced an optimization so that CacheTags are not stored on
buffer level, but rather on file level, as one cache tag can only relate to one
file. With this change a buffer->filecache reference was introduced so that the
buffer's tag can be calculated with an extra indirection i.e.
buffer.filecache.tag.
However during buffer collision in putFileData method, we don't set the
filecache reference of the collided (new) buffer:
https://github.com/apache/hive/commit/2e18a7408a8dd49beecad8d66bfe054b7dc474da#diff-d2ccd7cf3042845a0812a5e118f82db49253d82fc86449ffa408903bf434fb6dR309-R311
Later this cases NPE when the new (instantly decRef'ed) buffer is evicted.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 505215)
Remaining Estimate: 0h
Time Spent: 10m
> LLAP buffer collision causes NPE
> --------------------------------
>
> Key: HIVE-24297
> URL: https://issues.apache.org/jira/browse/HIVE-24297
> Project: Hive
> Issue Type: Bug
> Reporter: Ádám Szita
> Assignee: Ádám Szita
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HIVE-23741 introduced an optimization so that CacheTags are not stored on
> buffer level, but rather on file level, as one cache tag can only relate to
> one file. With this change a buffer->filecache reference was introduced so
> that the buffer's tag can be calculated with an extra indirection i.e.
> buffer.filecache.tag.
> However during buffer collision in putFileData method, we don't set the
> filecache reference of the collided (new) buffer:
> [https://github.com/apache/hive/commit/2e18a7408a8dd49beecad8d66bfe054b7dc474da#diff-d2ccd7cf3042845a0812a5e118f82db49253d82fc86449ffa408903bf434fb6dR309-R311]
> Later this cases NPE when the new (instantly decRef'ed) buffer is evicted:
> {code:java}
> Caused by: java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:778)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546)
> at
> org.apache.hadoop.hive.llap.cache.CacheContentsTracker.getTagState(CacheContentsTracker.java:129)
> at
> org.apache.hadoop.hive.llap.cache.CacheContentsTracker.getTagState(CacheContentsTracker.java:125)
> at
> org.apache.hadoop.hive.llap.cache.CacheContentsTracker.reportRemoved(CacheContentsTracker.java:109)
> at
> org.apache.hadoop.hive.llap.cache.CacheContentsTracker.notifyEvicted(CacheContentsTracker.java:238)
> at
> org.apache.hadoop.hive.llap.cache.LowLevelLrfuCachePolicy.evictSomeBlocks(LowLevelLrfuCachePolicy.java:276)
> at
> org.apache.hadoop.hive.llap.cache.CacheContentsTracker.evictSomeBlocks(CacheContentsTracker.java:177)
> at
> org.apache.hadoop.hive.llap.cache.LowLevelCacheMemoryManager.reserveMemory(LowLevelCacheMemoryManager.java:98)
> at
> org.apache.hadoop.hive.llap.cache.LowLevelCacheMemoryManager.reserveMemory(LowLevelCacheMemoryManager.java:65)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:323)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.allocateMultiple(EncodedReaderImpl.java:1302)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedStream(EncodedReaderImpl.java:930)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:506)
> ... 16 more {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
