[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[koeninger]

Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/17774
  
merged to master
Thanks @arzt !


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17774
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88226/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17774
  
**[Test build #88226 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88226/testReport)**
 for PR 17774 at commit 
[`1acbe4c`](https://github.com/apache/spark/commit/1acbe4c2a98e4d4c407fd557005d815dfd94851a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17774
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17774
  
**[Test build #88226 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88226/testReport)**
 for PR 17774 at commit 
[`1acbe4c`](https://github.com/apache/spark/commit/1acbe4c2a98e4d4c407fd557005d815dfd94851a).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/17774
  
Jenkins, ok to test


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[koeninger]

Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/17774
  
LGTM
@tdas @zsxwing absent any objections from you in the next couple of days, 
I'll merge this


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[pptaszynski]

Github user pptaszynski commented on the issue:

https://github.com/apache/spark/pull/17774
  
I am looking forward this one to be merged. we are suffering from the issue 
it resolved quite badly. It effectively makes the back-pressure not working for 
us at all.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17774
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17774
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17774
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17774
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/17774
  
I think @tdas @zsxwing should comment if this is the right direction...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17774
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17774
  
**[Test build #78618 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78618/testReport)**
 for PR 17774 at commit 
[`16b9aaf`](https://github.com/apache/spark/commit/16b9aafd8c668a6db2e4bafbeb4b4b0768677b2e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17774
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78618/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17774
  
**[Test build #78618 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78618/testReport)**
 for PR 17774 at commit 
[`16b9aaf`](https://github.com/apache/spark/commit/16b9aafd8c668a6db2e4bafbeb4b4b0768677b2e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[arzt]

Github user arzt commented on the issue:

https://github.com/apache/spark/pull/17774
  
It's been a while. What can I do to draw some attention to this request? Is 
this issue not relevant enough? Thanks for reconsideration @felixcheung @brkyvz 
@zsxwing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/17774
  
@brkyvz @zsxwing 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[arzt]

Github user arzt commented on the issue:

https://github.com/apache/spark/pull/17774
  
@felixcheung will this be merged?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[JasonMWhite]

Github user JasonMWhite commented on the issue:

https://github.com/apache/spark/pull/17774
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[arzt]

Github user arzt commented on the issue:

https://github.com/apache/spark/pull/17774
  
Sorry for being inactive. All good with this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17774
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76263/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17774
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17774
  
**[Test build #76263 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76263/testReport)**
 for PR 17774 at commit 
[`c98b9a4`](https://github.com/apache/spark/commit/c98b9a48516e600cca59bdd14463b5559ebcf5e5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17774
  
**[Test build #76263 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76263/testReport)**
 for PR 17774 at commit 
[`c98b9a4`](https://github.com/apache/spark/commit/c98b9a48516e600cca59bdd14463b5559ebcf5e5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17774
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76256/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17774
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17774
  
**[Test build #76256 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76256/testReport)**
 for PR 17774 at commit 
[`d4a7867`](https://github.com/apache/spark/commit/d4a7867d96aa7c4bbed9cbd03b0753adcf79db9d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17774
  
**[Test build #76256 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76256/testReport)**
 for PR 17774 at commit 
[`d4a7867`](https://github.com/apache/spark/commit/d4a7867d96aa7c4bbed9cbd03b0753adcf79db9d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/17774
  
Jenkins, ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/17774
  
Jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[koeninger]

Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/17774
  
LGTM pending jason's comments on tests


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[JasonMWhite]

Github user JasonMWhite commented on the issue:

https://github.com/apache/spark/pull/17774
  
Tests have some fairly repetitive code, but not sure if that's a problem or 
not. Looks good to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[arzt]

Github user arzt commented on the issue:

https://github.com/apache/spark/pull/17774
  
I changed the max messages per partition to be at least 1. Agreed?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[JasonMWhite]

Github user JasonMWhite commented on the issue:

https://github.com/apache/spark/pull/17774
  
I think @koeninger's suggestion is valid. `effectiveRateLimitPerPartition` 
is the upper bound on the number of messages per partition per second, and 
`maxMessagesPerPartition` sets an upper bound on the number of messages to be 
retrieved per partition per batch window.

Making `effectiveRateLimitPerPartition` a float will allow it to handle 
properly rates of < 1/partition/s, so this is definitely a good idea. 
`maxMessagesPerPartition` must still be an integer, as you can't retrieve 
partial messages. All agreed there.

Setting `maxMessagesPerPartition` to have a minimum of 1 message per window 
is a good safe value to allow progress in all cases. If there isn't 1 message 
to retrieve, `clamp` will prevent it from attempting to retrieve an invalid 
message.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[koeninger]

Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/17774
  
Have you read the function def clamp?

Rate limit of 1 should not imply an attempt to grab 1 message even if it
doesn't exist.

On Apr 27, 2017 11:01, "Sebastian Arzt"  wrote:

> @koeninger  I agree that assuming a long
> batch size is wrong, not sure whether it even matters.
> But what if for one partition there is no lack in the current batch? Then
> fetching 1 message for this partition from kafka, is you suggest, would
> fail. So here zero makes sense in my eyes. This is also the old behaviour
> if rate > 1 and lag == 0 here
> 

> .
> Further, I think that truncating 0.99 to 0 messages per partition is also
> the right thing to do, as one cannot be sure that there is one message
> available if (secsPerBatch * limit) < 1.0. And as you say, in a future
> batch it is very like to become greater than 1.0.
> Do you agree?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[arzt]

Github user arzt commented on the issue:

https://github.com/apache/spark/pull/17774
  
@koeninger I agree that assuming a long batch size is wrong, not sure 
whether it even matters. 
But what if for one partition there is no lack in the current batch? Then 
fetching 1 message for this partition from kafka, is you suggest, would fail. 
So here zero makes sense in my eyes. This is also the old behaviour if `rate > 
1` and `lag == 0` 
[here](https://github.com/apache/spark/blob/master/external/kafka-0-8/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala#L107).
 
Further, I think that truncating 0.99 to 0 messages per partition is also 
the right thing to do, as one cannot be sure that there is one message 
available if `(secsPerBatch * limit) < 1.0`. And as you say, in a future batch 
it is very like to become greater than 1.0.
Do you agree?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[koeninger]

Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/17774
  
@arzt It's entirely possible to have batch times less than a second, and 
I'm not sure I agree that the absolute number of messages allowable for a 
partition should ever be zero.

So to put this another way, right now effectiveRateLimitPerPartition is a 
Map[TopicPartition, Long], which matches the return value of the function 
maxMessagesPerPartition.

You're wanting to change effectiveRateLimitPerPartition to a 
Map[TopicPartition, Double], which is probably a good idea, and should fix the 
bug around treating a very small rate limit as no limit.

But it still needs to be converted to Map[TopicPartition, Long] before 
returning.  Calling .toLong is probably not the right thing to do there, 
because 0.99 will get truncated to 0.  

I think one message per partition per batch is the minimum reasonable rate 
limit, otherwise particular partitions may not make progress.  The relative lag 
calculation might take care of that in future batches, but it still seems 
questionable, even if it's a corner case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[arzt]

Github user arzt commented on the issue:

https://github.com/apache/spark/pull/17774
  
To run tests or debug using IntelliJ:
`mvn test -DforkMode=never -pl external/kafka-0-8 
"-Dsuites=org.apache.spark.streaming.kafka.DirectKafkaStreamSuite 
maxMessagesPerPartition"`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[arzt]

Github user arzt commented on the issue:

https://github.com/apache/spark/pull/17774
  
Thanks for your valuable feedback. I added tests as suggested by 
@JasonMWhite . @koeninger the estimated rate is per second summed over all 
partitions, is it? The batch time usually is longer. So even values less than 1 
but greater than 0 for `backpressureRate` can make sense for one partition. The 
casting to long is only needed when the absolute number of messages is 
computed, but even this number can be zero for some partitions, e.g. when there 
is no lag. I hope I am not confused here. There is also a test covering this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[koeninger]

Github user koeninger commented on the issue:

https://github.com/apache/spark/pull/17774
  
How do you read 0.1 of a kafka message for a given partition of a given 
batch?

Ultimately the floor for a rate limit, assuming one is set, needs to be 1 
message per partition per batch, not a fraction, which is why it's a long.

If you want to delay that conversion by keeping it as a double as long as 
possible, that makes sense, but the lines like

(secsPerBatch * limit).toLong

probably need attention too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[JasonMWhite]

Github user JasonMWhite commented on the issue:

https://github.com/apache/spark/pull/17774
  
Code looks sound. Could you add or modify a test to illustrate/verify?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17774
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org