[ https://issues.apache.org/jira/browse/SPARK-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tathagata Das updated SPARK-1592: --------------------------------- Fix Version/s: 1.0.0 > Old streaming input blocks not removed automatically from the BlockManagers > --------------------------------------------------------------------------- > > Key: SPARK-1592 > URL: https://issues.apache.org/jira/browse/SPARK-1592 > Project: Spark > Issue Type: Bug > Reporter: Tathagata Das > Assignee: Tathagata Das > Priority: Blocker > Fix For: 1.0.0 > > > The raw input data is stored as blocks in BlockManagers. Earlier they were > cleared by cleaner ttl. Now since streaming does not require cleaner TTL to > be set, the block would not get cleared. This increases up the Spark's memory > usage, which is not even accounted and shown in the Spark storage UI. It may > cause the data blocks to spill over to disk, which eventually slows down the > receiving of data (persisting to memory become bottlenecked by writing to > disk). -- This message was sent by Atlassian JIRA (v6.2#6252)