GitHub user junhaoMg opened a pull request:
https://github.com/apache/spark/pull/9593
[SPARK-11627] Add initial input rate limit for Spark Streaming backpressure
mechanism.
https://issues.apache.org/jira/browse/SPARK-11627
Spark Streaming backpressure mechanism has no initial input rate limit, it
might cause OOM exception.
In the firest batch task ,receivers receive data at the maximum speed they
can reach,it might exhaust executors memory resources. Add a initial input rate
limit value can make sure the Streaming job execute success in the first
batch,then the backpressure mechanism can adjust receiving rate adaptively.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/junhaoMg/spark junhao-dev
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9593.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9593
----
commit 2de3ce7e8133ec9a8bbb74e980dc3921a4b6d505
Author: junhao <[email protected]>
Date: 2015-11-04T14:14:58Z
fix spark streaming backpressure bugs
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
