GitHub user jeanlyn opened a pull request:
https://github.com/apache/spark/pull/12028
[SPARK-13845][CORE][Backport-1.6]Using onBlockUpdated to replace onTaskEnd
avioding driver OOM
## What changes were proposed in this pull request?
We have a streaming job using `FlumePollInputStream` always driver OOM
after few days, here is some driver heap dump before OOM
```
num #instances #bytes class name
----------------------------------------------
1: 13845916 553836640 org.apache.spark.storage.BlockStatus
2: 14020324 336487776 org.apache.spark.storage.StreamBlockId
3: 13883881 333213144 scala.collection.mutable.DefaultEntry
4: 8907 89043952 [Lscala.collection.mutable.HashEntry;
5: 62360 65107352 [B
6: 163368 24453904 [Ljava.lang.Object;
7: 293651 20342664 [C
...
```
`BlockStatus` and `StreamBlockId` keep on growing, and the driver OOM in
the end.
After investigated, i found the `executorIdToStorageStatus` in
`StorageStatusListener` seems never remove the blocks from `StorageStatus`.
In order to fix the issue, i try to use `onBlockUpdated` replace `onTaskEnd
` , so we can update the block informations(add blocks, drop the block from
memory to disk and delete the blocks) in time.
## How was this patch tested?
Existing unit tests and manual tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jeanlyn/spark fixoom1.6
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12028.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12028
----
commit f37f3594b71df0083880aa5395a77808fd6c72b3
Author: jeanlyn <[email protected]>
Date: 2016-03-29T05:06:03Z
Using onBlockUpdated to replace onTaskEnd avioding driver OOM
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
