GitHub user yuanboliu opened a pull request:
https://github.com/apache/spark/pull/21690
[SPARK-24713]AppMatser of spark streaming kafka OOM if there are hundâ¦
We have hundreds of kafka topics need to be consumed in one application.
The application master will throw OOM exception after hanging for nearly half
of an hour.
OOM happens in the env with a lot of topics, and it's not convenient to set
up such kind of env in the unit test. So I didn't change/add test case.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yuanboliu/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21690.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21690
----
commit 15d03e63453e72675390c65ddf72492caefdadee
Author: yuanbo <yuanbo@...>
Date: 2018-07-02T03:58:53Z
[SPARK-24713]AppMatser of spark streaming kafka OOM if there are hundreds
of topics consumed
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
