GitHub user junhaoMg opened a pull request:
https://github.com/apache/spark/pull/9593
[SPARK-11627] Add initial input rate limit for Spark Streaming backpressure
mechanism.
https://issues.apache.org/jira/browse/SPARK-11627
Spark Streaming backpressure mechanism has
Github user junhaoMg commented on the pull request:
https://github.com/apache/spark/pull/9593#issuecomment-155629739
Setting the rate limit using SparkConf spark.streaming.receiver.maxRate
will cap the receiver rateï¼but select an appropriate value is difficult, if
the value is too
Github user junhaoMg commented on the pull request:
https://github.com/apache/spark/pull/9593#issuecomment-155631922
Given an appropriate initial value can make sure the job success in the
first batch, then the backpressure mechanism can work and adjust data receive
rate according
Github user junhaoMg commented on the pull request:
https://github.com/apache/spark/pull/9593#issuecomment-161936369
I hava add the configuration to the docs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user junhaoMg commented on a diff in the pull request:
https://github.com/apache/spark/pull/9593#discussion_r47876028
--- Diff: docs/configuration.md ---
@@ -1523,6 +1523,15 @@ Apart from these, the following properties are also
available, and may be useful
Github user junhaoMg commented on a diff in the pull request:
https://github.com/apache/spark/pull/9593#discussion_r47870756
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala
---
@@ -18,6 +18,7 @@
package
Github user junhaoMg commented on a diff in the pull request:
https://github.com/apache/spark/pull/9593#discussion_r48112803
--- Diff: docs/configuration.md ---
@@ -1523,6 +1523,15 @@ Apart from these, the following properties are also
available, and may be useful
Github user junhaoMg commented on the pull request:
https://github.com/apache/spark/pull/9593#issuecomment-159794454
I have changed 'spark.streaming.receiver.initialRate' to
'spark.streaming.backpressure.initialRate'.
---
If your project is set up for it, you can reply to this email
