Tathagata Das created SPARK-1592:
------------------------------------
Summary: Streaming input blocks were never removed from the
BlockManagers
Key: SPARK-1592
URL: https://issues.apache.org/jira/browse/SPARK-1592
Project: Spark
Issue Type: Bug
Reporter: Tathagata Das
Assignee: Tathagata Das
Priority: Blocker
The raw input data is stored as blocks in BlockManagers. Earlier they were
cleared by cleaner ttl. Now since streaming does not require cleaner TTL to be
set, the block would not get cleared. This increases up the Spark's memory
usage, which is not even accounted and shown in the Spark storage UI. It may
cause the data blocks to spill over to disk, which eventually slows down the
receiving of data (persisting to memory become bottlenecked by writing to disk).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
