[
https://issues.apache.org/jira/browse/HIVE-17411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-17411:
------------------------------------
Resolution: Fixed
Fix Version/s: 2.4.0
3.0.0
2.3.0
Status: Resolved (was: Patch Available)
Pushed to master, branch-2 and branch-2.3. Thanks for the reviews!
> LLAP IO may incorrectly release a refcount in some rare cases
> -------------------------------------------------------------
>
> Key: HIVE-17411
> URL: https://issues.apache.org/jira/browse/HIVE-17411
> Project: Hive
> Issue Type: Bug
> Reporter: Sergey Shelukhin
> Assignee: Sergey Shelukhin
> Fix For: 2.3.0, 3.0.0, 2.4.0
>
> Attachments: HIVE-17411.patch
>
>
> In a large stream whose buffers are not reused, and that is separated into
> many CB (e.g. due to a small ORC compression buffer size), it may happen that
> some, but not all, buffers that are read together as a unit are evicted from
> cache.
> If CacheBuffer follows BufferChunk in the buffer list when a stream like this
> is read, the latter will be converted to ProcCacheChunk; it is possible for
> early refcount release logic from the former to release the refcount (for a
> dictionary stream, the initial refCount is always released early), and then
> backtrack to the latter to see if we can unlock more buffers. It would then
> to decref an uninitialized MemoryBuffer in ProcCacheChunk because
> ProcCacheChunk looks like a CacheChunk. PCC initial refcounts are released
> separately after the data is uncompressed.
> I'm assuming this would almost never happen with non-stripe-level streams
> because one would need a large RG to span 2+ CBs, no overlap with
> next/previous RGs in 2+ buffers for the early release to kick in, and an
> unfortunate eviction order. However it's possible with large-ish dictionaries.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
