[
https://issues.apache.org/jira/browse/HIVE-18269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18269:
-----------------------------------------
Description:
pendingData linked list in Llap IO elevator (LlapRecordReader.java) may grow
indefinitely when Llap IO is faster than processing pipeline. Since we don't
have backpressure to slow down the IO, this can lead to indefinite growth of
pending data leading to severe GC pressure and eventually lead to OOM.
This specific instance of LLAP was running on HDFS on top of EBS volume backed
by SSD. The query that triggered this is issue was ANALYZE STATISTICS .. FOR
COLUMNS which also gather bitvectors. Fast IO and Slow processing case.
was:pendingData linked list in Llap IO elevator (LlapRecordReader.java) may
have grow indefinitely when Llap IO is faster than processing pipeline. Since
we don't have backpressure to slow down the IO, this can lead to indefinite
growth of pending data leading to severe GC pressure and eventually lead to OOM.
> LLAP: Fast llap io with slow processing pipeline can lead to OOM
> ----------------------------------------------------------------
>
> Key: HIVE-18269
> URL: https://issues.apache.org/jira/browse/HIVE-18269
> Project: Hive
> Issue Type: Bug
> Affects Versions: 3.0.0
> Reporter: Prasanth Jayachandran
> Assignee: Prasanth Jayachandran
> Attachments: Screen Shot 2017-12-13 at 1.15.16 AM.png
>
>
> pendingData linked list in Llap IO elevator (LlapRecordReader.java) may grow
> indefinitely when Llap IO is faster than processing pipeline. Since we don't
> have backpressure to slow down the IO, this can lead to indefinite growth of
> pending data leading to severe GC pressure and eventually lead to OOM.
> This specific instance of LLAP was running on HDFS on top of EBS volume
> backed by SSD. The query that triggered this is issue was ANALYZE STATISTICS
> .. FOR COLUMNS which also gather bitvectors. Fast IO and Slow processing case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
