[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10089 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-192531351 LGTM. Merging to master. Thanks @JasonMWhite and @koeninger --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as w

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user koeninger commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-191290624 LGTM Thanks for following up on this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If yo

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-191159802 All comments addressed, builds cleanly, all tests passing. GTM? --- If your project is set up for it, you can reply to this email and have your reply appear on GitH

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-191159038 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-191159043 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-191158412 **[Test build #52304 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52304/consoleFull)** for PR 10089 at commit [`a7a0877`](https://g

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-191127051 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-191127059 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-191126457 **[Test build #52299 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52299/consoleFull)** for PR 10089 at commit [`0a78e8d`](https://g

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-191113140 `DirectKafkaStreamSuite` passes all tests for me locally, but the test failure above appeared to be on an outdated sha. Addressed @zsxwing's comments also.

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-191107974 **[Test build #52304 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52304/consoleFull)** for PR 10089 at commit [`a7a0877`](https://gi

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-191081614 **[Test build #52299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52299/consoleFull)** for PR 10089 at commit [`0a78e8d`](https://gi

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user koeninger commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-190304130 @JasonMWhite looks like this failed the "offset recovery" test in DirectKafkaStreamSuite. Are you able to reproduce that test failure locally? --- If your project i

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-186113150 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-186113151 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-186112872 **[Test build #51525 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51525/consoleFull)** for PR 10089 at commit [`f19f746`](https://g

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-186084779 **[Test build #51525 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51525/consoleFull)** for PR 10089 at commit [`f19f746`](https://gi

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-186083123 @JasonMWhite thanks, looks great except some nits. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If yo

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r53426791 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala --- @@ -430,6 +457,27 @@ class DirectKafkaStreamSuite

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r53426422 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaTestUtils.scala --- @@ -152,12 +152,15 @@ private[kafka] class KafkaTestUtils

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-186080968 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-186037636 It looks like this PySpark unit test failure above was prior to my commit to add a backwards-compatible single-argument version as suggested. Could someone kick it

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185545377 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185545374 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185545031 **[Test build #51457 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51457/consoleFull)** for PR 10089 at commit [`7a5dad3`](https://g

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185526582 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185526580 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185524841 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185524838 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185521431 A single-argument version is easy enough, and good to support backwards-compatibility anyway. I haven't been able to get PySpark tests running locally yet, so I'm a

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185519914 **[Test build #51457 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51457/consoleFull)** for PR 10089 at commit [`7a5dad3`](https://gi

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185518134 ``` py4j.Py4JException: Method createTopic([class java.lang.String]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:335

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185517414 Changes addressed, but looks like it's causing PySpark unit tests to fail. Investigating... --- If your project is set up for it, you can reply to this email and h

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r53265282 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala --- @@ -353,10 +353,52 @@ class DirectKafkaStreamS

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r53265248 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala --- @@ -89,23 +89,32 @@ class DirectKafkaInputDSt

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185511634 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185511633 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185511524 **[Test build #51451 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51451/consoleFull)** for PR 10089 at commit [`73e9ae3`](https://g

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r53261182 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala --- @@ -430,6 +471,32 @@ class DirectKafkaStreamSuite

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r53261176 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala --- @@ -430,6 +471,32 @@ class DirectKafkaStreamSuite

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r53260582 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala --- @@ -353,10 +353,52 @@ class DirectKafkaStreamSuite

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r53260346 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala --- @@ -89,23 +89,32 @@ class DirectKafkaInputDStream

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185479065 **[Test build #51451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51451/consoleFull)** for PR 10089 at commit [`73e9ae3`](https://gi

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-185478351 I added unit tests for `maxMessagesPerPartition` to cover 3 edge cases that have been raised here: - when backpressure is disabled, simply uses the `maxRatePerPa

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-184920176 > Sorry, did I put the MiMa exclusion in the wrong section of the file? Then should be in 2.0 section. --- If your project is set up for it, you can reply to t

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-184919738 @JasonMWhite could you add a unit test for `maxMessagesPerPartition`? Since it's protected, you should be able to test it using some faking data. --- If your project i

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r53098287 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala --- @@ -89,23 +89,31 @@ class DirectKafkaInputDStream

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-184916602 Sorry, did I put the MiMa exclusion in the wrong section of the file? --- If your project is set up for it, you can reply to this email and have your reply appear o

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-184912435 **[Test build #51390 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51390/consoleFull)** for PR 10089 at commit [`f7ffd6f`](https://g

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-184912515 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-184912512 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-184907112 **[Test build #51390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51390/consoleFull)** for PR 10089 at commit [`f7ffd6f`](https://gi

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-184905769 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature en

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user koeninger commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-184901629 I think that should be ok. On Tue, Feb 16, 2016 at 4:29 PM, Jason White wrote: > @koeninger I've added the two err

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-184900516 @koeninger I've added the two error messages from Jenkins to the MimaExcludes, under the v2.0 section. There was one from the test function, but this PR also change

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-182008116 Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user koeninger commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-182007914 Sure, it's in project/MimaExcludes.scala. You should be able to match up the problem type and class in the error message from jenkins when adding an appropriat

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-182002607 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-182002605 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-182002520 **[Test build #50983 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50983/consoleFull)** for PR 10089 at commit [`b58d517`](https://g

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-181997614 **[Test build #50983 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50983/consoleFull)** for PR 10089 at commit [`b58d517`](https://gi

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-181991672 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-181990835 I'm not sure how to handle the mima test failure, could you point me to where to add the exclude? I'll rebase and add the exception. --- If your project is set up

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user koeninger commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-181964379 This looked like a good patch, did it just fall through the cracks? The mima test failure was probably just due to the change in signature of KafkaTestUtils.createTo

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-165545678 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-165545581 **[Test build #47938 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47938/consoleFull)** for PR 10089 at commit [`b58d517`](https://g

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-165545679 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-165540920 **[Test build #47938 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47938/consoleFull)** for PR 10089 at commit [`b58d517`](https://gi

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-165539557 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-165536634 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-165536632 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-165531887 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user koeninger commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-165528447 The console output seems like it's available without logging in, and looks like a jenkins issue rather than an actual test failure: GitHub pull request #10089

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-165518544 Is this a CI failure? I don't have access rights to see what happened. I'd like to get this ready to merge, could someone give me a hand? --- If your project is se

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-164539536 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-164539550 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-164527592 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r47363415 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala --- @@ -364,8 +365,8 @@ class DirectKafkaStreamSui

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r47289494 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala --- @@ -364,8 +365,8 @@ class DirectKafkaStreamSuite

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r47286143 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala --- @@ -364,8 +365,8 @@ class DirectKafkaStreamSuite

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r47278574 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala --- @@ -364,8 +365,8 @@ class DirectKafkaStreamSuite

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user mrszg commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-163532836 Original code (before this patch) has serious error - it doesn't respect maxRateLimitPerPartition in case when backpressure rate is smaller than number of partitions. In

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-163427146 Could someone verify the patch please? @tdas perhaps? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user koeninger commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-162562042 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enable

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-162260106 `maxRatePerPartition` now respects the limit set by `maxRateLimitPerPartition`, if it is set. Let me know if you think any additional tests are needed. --- If you

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user koeninger commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-162252793 Sounds good. If you can add that maxRatePerPartition handling this would be ready to go from my point of view On Dec 5, 2015 3:59 PM, "Jason White" wrote:

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-162250749 This patch solved our skew problem. Below is a 15-minute snapshot of our lag earlier this week, showing a single partition getting slowly worse. It would get to abo

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r46763463 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala --- @@ -89,23 +89,29 @@ class DirectKafkaInputDSt

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r46449516 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala --- @@ -36,9 +36,10 @@ import org.scalatest.concur

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r46449571 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala --- @@ -89,23 +89,29 @@ class DirectKafkaInputDSt

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user JasonMWhite commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r46449467 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala --- @@ -364,8 +365,8 @@ class DirectKafkaStreamSui

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r46438227 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala --- @@ -89,23 +89,29 @@ class DirectKafkaInputDStre

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user koeninger commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-161352747 This generally looks sensible to me, would like to see if it solves your issue first. Thanks for working on it. --- If your project is set up for it, you can reply t

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r46431828 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala --- @@ -89,23 +89,29 @@ class DirectKafkaInputDStre

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r46431401 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala --- @@ -364,8 +365,8 @@ class DirectKafkaStreamSuite

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/10089#discussion_r46431493 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala --- @@ -36,9 +36,10 @@ import org.scalatest.concurre

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10089#issuecomment-161188280 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your p

[GitHub] spark pull request: [SPARK-12073] [Streaming] backpressure rate co...

GitHub user JasonMWhite opened a pull request: https://github.com/apache/spark/pull/10089 [SPARK-12073] [Streaming] backpressure rate controller consumes events preferentially from lagg… …ing partitions I'm pretty sure this is the reason we couldn't easily recover from