[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/9593


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-185003282
  
@junhaoMg Thanks a lot. Merging to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-184962897
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51396/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-184962894
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-184962774
  
**[Test build #51396 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51396/consoleFull)**
 for PR 9593 at commit 
[`2d750c4`](https://github.com/apache/spark/commit/2d750c4c1cedaff9849137710b58242bcd15bef9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-184939531
  
**[Test build #51396 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51396/consoleFull)**
 for PR 9593 at commit 
[`2d750c4`](https://github.com/apache/spark/commit/2d750c4c1cedaff9849137710b58242bcd15bef9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-184934114
  
Will pull in once tests pass


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-184933638
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-166381528
  
LGTM. Ping @tdas to take a final look.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-166178142
  
**[Test build #48091 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48091/consoleFull)**
 for PR 9593 at commit 
[`2d750c4`](https://github.com/apache/spark/commit/2d750c4c1cedaff9849137710b58242bcd15bef9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-166185197
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[junhaoMg]

Github user junhaoMg commented on a diff in the pull request:

https://github.com/apache/spark/pull/9593#discussion_r48112803
  
--- Diff: docs/configuration.md ---
@@ -1523,6 +1523,15 @@ Apart from these, the following properties are also 
available, and may be useful
   
 
 
+  spark.streaming.backpressure.initialRate
+  not set
+  
+Initial rate for backpressure mechanism (since 1.5). This provides 
maximum receiving rate of
+receivers in the first batch when enables the backpressure mechanism, 
then the maximum receiving
--- End diff --

thank you, I have modified it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-166185198
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48091/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-166185146
  
**[Test build #48091 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48091/consoleFull)**
 for PR 9593 at commit 
[`2d750c4`](https://github.com/apache/spark/commit/2d750c4c1cedaff9849137710b58242bcd15bef9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[BenFradet]

Github user BenFradet commented on a diff in the pull request:

https://github.com/apache/spark/pull/9593#discussion_r47878665
  
--- Diff: docs/configuration.md ---
@@ -1523,6 +1523,15 @@ Apart from these, the following properties are also 
available, and may be useful
   
 
 
+  spark.streaming.backpressure.initialRate
+  not set
+  
+Initial rate for backpressure mechanism (since 1.5). This provides 
maximum receiving rate of
+receivers in the first batch when enables the backpressure mechanism, 
then the maximum receiving
+rate will compute dynamically based on the current batch scheduling 
delays and processing times.
--- End diff --

"will **be** compute**d** dynamically"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[BenFradet]

Github user BenFradet commented on a diff in the pull request:

https://github.com/apache/spark/pull/9593#discussion_r47878960
  
--- Diff: docs/configuration.md ---
@@ -1523,6 +1523,15 @@ Apart from these, the following properties are also 
available, and may be useful
   
 
 
+  spark.streaming.backpressure.initialRate
+  not set
+  
+Initial rate for backpressure mechanism (since 1.5). This provides 
maximum receiving rate of
--- End diff --

"for **the** backpressure mechanism"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[BenFradet]

Github user BenFradet commented on a diff in the pull request:

https://github.com/apache/spark/pull/9593#discussion_r47879107
  
--- Diff: docs/configuration.md ---
@@ -1523,6 +1523,15 @@ Apart from these, the following properties are also 
available, and may be useful
   
 
 
+  spark.streaming.backpressure.initialRate
+  not set
+  
+Initial rate for backpressure mechanism (since 1.5). This provides 
maximum receiving rate of
+receivers in the first batch when enables the backpressure mechanism, 
then the maximum receiving
--- End diff --

I'd say: "This is the initial maximum receiving rate at which each receiver 
will receive data for the first batch when the backpressure mechanism is 
enabled."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[BenFradet]

Github user BenFradet commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-165380452
  
A few remarks regarding the doc.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-165381302
  
**[Test build #47907 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47907/consoleFull)**
 for PR 9593 at commit 
[`5a1dd98`](https://github.com/apache/spark/commit/5a1dd982cc59b360a8bb14b73e778540b1d8f8ea).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-165381420
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-165381421
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47907/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-165399619
  
**[Test build #47900 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47900/consoleFull)**
 for PR 9593 at commit 
[`4f3392f`](https://github.com/apache/spark/commit/4f3392ffc0fadfc31761e8b84a53ec47f82c1245).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-165399693
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-165399695
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47900/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/9593#discussion_r47820904
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala 
---
@@ -18,6 +18,7 @@
 package org.apache.spark.streaming.receiver
 
 import com.google.common.util.concurrent.{RateLimiter => GuavaRateLimiter}
+import org.apache.spark.streaming.scheduler.RateController
--- End diff --

nit: import order. See 
https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide#SparkCodeStyleGuide-Imports


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/9593#discussion_r47820788
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala 
---
@@ -61,4 +62,21 @@ private[receiver] abstract class RateLimiter(conf: 
SparkConf) extends Logging {
 rateLimiter.setRate(newRate)
   }
 }
+
+  /**
+   * Get the initial rateLimit to initial rateLimiter
+   * @return
+   */
+  def getInitialRateLimit() : Long = {
--- End diff --

nit: add `private`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/9593#discussion_r47820758
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala 
---
@@ -61,4 +62,21 @@ private[receiver] abstract class RateLimiter(conf: 
SparkConf) extends Logging {
 rateLimiter.setRate(newRate)
   }
 }
+
+  /**
+   * Get the initial rateLimit to initial rateLimiter
+   * @return
--- End diff --

nit: remove the empty `@return`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/9593#discussion_r47820721
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala 
---
@@ -61,4 +62,21 @@ private[receiver] abstract class RateLimiter(conf: 
SparkConf) extends Logging {
 rateLimiter.setRate(newRate)
   }
 }
+
+  /**
+   * Get the initial rateLimit to initial rateLimiter
+   * @return
+   */
+  def getInitialRateLimit() : Long = {
+if (RateController.isBackPressureEnabled(conf)) {
+  val initialRate = 
conf.getLong("spark.streaming.backpressure.initialRate", 1000)
--- End diff --

The default value should be `maxRateLimit`. And here you can use one line 
code, such as 
```math.max(conf.getLong("spark.streaming.backpressure.initialRate", 
maxRateLimit), maxRateLimit)```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[junhaoMg]

Github user junhaoMg commented on a diff in the pull request:

https://github.com/apache/spark/pull/9593#discussion_r47876028
  
--- Diff: docs/configuration.md ---
@@ -1523,6 +1523,15 @@ Apart from these, the following properties are also 
available, and may be useful
   
 
 
+  spark.streaming.backpressure.initialRate
+  1000
--- End diff --

Thanks for reminding, I have updated it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/9593#discussion_r47870905
  
--- Diff: docs/configuration.md ---
@@ -1523,6 +1523,15 @@ Apart from these, the following properties are also 
available, and may be useful
   
 
 
+  spark.streaming.backpressure.initialRate
+  1000
--- End diff --

Here should be updated as well


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-165341987
  
**[Test build #47900 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47900/consoleFull)**
 for PR 9593 at commit 
[`4f3392f`](https://github.com/apache/spark/commit/4f3392ffc0fadfc31761e8b84a53ec47f82c1245).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[junhaoMg]

Github user junhaoMg commented on a diff in the pull request:

https://github.com/apache/spark/pull/9593#discussion_r47870756
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala 
---
@@ -18,6 +18,7 @@
 package org.apache.spark.streaming.receiver
 
 import com.google.common.util.concurrent.{RateLimiter => GuavaRateLimiter}
+import org.apache.spark.streaming.scheduler.RateController
--- End diff --

Thank you for your suggestions，I've made changes。


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-165368252
  
**[Test build #47907 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47907/consoleFull)**
 for PR 9593 at commit 
[`5a1dd98`](https://github.com/apache/spark/commit/5a1dd982cc59b360a8bb14b73e778540b1d8f8ea).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-161950061
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-161950062
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47195/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-161938126
  
**[Test build #47195 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47195/consoleFull)**
 for PR 9593 at commit 
[`14c3bf4`](https://github.com/apache/spark/commit/14c3bf4bb535a0d5cc026a626b74c575cec150fb).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[junhaoMg]

Github user junhaoMg commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-161936369
  
I hava add the configuration to the docs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[BenFradet]

Github user BenFradet commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159921616
  
If you introduce a new configuration property, don't forget to add it to 
the docs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159795582
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46738/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159795580
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159793849
  
**[Test build #46740 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46740/consoleFull)**
 for PR 9593 at commit 
[`c0478f9`](https://github.com/apache/spark/commit/c0478f948e7769db38a2ff04085325d2740eb431).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159791092
  
**[Test build #46733 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46733/consoleFull)**
 for PR 9593 at commit 
[`8a324ff`](https://github.com/apache/spark/commit/8a324ffe54b0221d854a07ee0ef6e90f137b8534).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159791889
  
**[Test build #46736 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46736/consoleFull)**
 for PR 9593 at commit 
[`995fe9c`](https://github.com/apache/spark/commit/995fe9c03cfb4a5de6e5d41752c6b70e1bca2dd7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[junhaoMg]

Github user junhaoMg commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159794454
  
I have changed 'spark.streaming.receiver.initialRate' to 
'spark.streaming.backpressure.initialRate'.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159799136
  
**[Test build #46736 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46736/consoleFull)**
 for PR 9593 at commit 
[`995fe9c`](https://github.com/apache/spark/commit/995fe9c03cfb4a5de6e5d41752c6b70e1bca2dd7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159799410
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159799412
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46736/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159801849
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46740/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159801848
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159833997
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46733/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159833994
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159833901
  
**[Test build #46733 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46733/consoleFull)**
 for PR 9593 at commit 
[`8a324ff`](https://github.com/apache/spark/commit/8a324ffe54b0221d854a07ee0ef6e90f137b8534).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`class SparkPlanInfo(`\n  * `class SQLMetricInfo(`\n  * `case class 
SparkListenerSQLExecutionStart(`\n  * `case class 
SparkListenerSQLExecutionEnd(executionId: Long, time: Long)`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-159801774
  
**[Test build #46740 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46740/consoleFull)**
 for PR 9593 at commit 
[`c0478f9`](https://github.com/apache/spark/commit/c0478f948e7769db38a2ff04085325d2740eb431).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`public final class OneWayMessage implements RequestMessage `\n  * `case class 
Cast(child: Expression, dataType: DataType) extends UnaryExpression `\n  * 
`class SparkPlanInfo(`\n  * `class SQLMetricInfo(`\n  * `case class 
SparkListenerSQLExecutionStart(`\n  * `case class 
SparkListenerSQLExecutionEnd(executionId: Long, time: Long)`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-158229319
  
**[Test build #46357 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46357/consoleFull)**
 for PR 9593 at commit 
[`cd6e324`](https://github.com/apache/spark/commit/cd6e3248aede20f3aae45d89ccb0a5f3724b5b29).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-158210110
  
Jenkins, this is ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-158211547
  
**[Test build #46357 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46357/consoleFull)**
 for PR 9593 at commit 
[`cd6e324`](https://github.com/apache/spark/commit/cd6e3248aede20f3aae45d89ccb0a5f3724b5b29).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-158229452
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-158229456
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46357/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/9593#discussion_r45007951
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/receiver/RateLimiter.scala 
---
@@ -61,4 +62,21 @@ private[receiver] abstract class RateLimiter(conf: 
SparkConf) extends Logging {
 rateLimiter.setRate(newRate)
   }
 }
+
+  /**
+   * Get the initial rateLimit to initial rateLimiter
+   * @return
+   */
+  def getInitialRateLimit() : Long = {
+if (RateController.isBackPressureEnabled(conf)) {
+  val initialRate = 
conf.getLong("spark.streaming.receiver.initialRate", 1000)
--- End diff --

This should be `spark.streaming.backpressure.initialRate` to keep it in the 
same name space as "spark.streaming.backpressure.enabled"




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-155374156
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[junhaoMg]

GitHub user junhaoMg opened a pull request:

https://github.com/apache/spark/pull/9593

[SPARK-11627] Add initial input rate limit for Spark Streaming backpressure 
mechanism.

https://issues.apache.org/jira/browse/SPARK-11627

Spark Streaming backpressure mechanism has no initial input rate limit, it 
might cause OOM exception.
In the firest batch task ,receivers receive data at the maximum speed they 
can reach,it might exhaust executors memory resources. Add a initial input rate 
limit value can make sure the Streaming job execute  success in the first 
batch,then the backpressure mechanism can adjust receiving rate adaptively.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/junhaoMg/spark junhao-dev

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9593.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9593


commit 2de3ce7e8133ec9a8bbb74e980dc3921a4b6d505
Author: junhao 
Date:   2015-11-04T14:14:58Z

fix spark streaming backpressure bugs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[junhaoMg]

Github user junhaoMg commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-155629739
  
Setting the rate limit using SparkConf spark.streaming.receiver.maxRate 
will cap the receiver rate，but select an appropriate value is difficult, if 
the value is too small, the system can not achieve the maximum processing 
capacity, because the  receiver rate which backpressure mechanism  predicted  
can not exceed the spark.streaming.receiver.maxRate. If the 
spark.streaming.receiver.maxRate is too big, the  the job might failed in the 
first batch.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[junhaoMg]

Github user junhaoMg commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-155631922
  
Given an appropriate initial value can make sure the job success in the 
first batch, then the backpressure mechanism can work and adjust  data receive 
rate according to the processing capacity of the system, the receive rate 
backpressure mechanism predicted can either greater or smaller than the 
initial value, so  the spark.streaming.receiver.maxRate is not reasonable as an 
initial value.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11627] Add initial input rate limit for...

[tdas]

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/9593#issuecomment-155655693
  
test this again


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org