[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19431 @akonopko Thanks! Sorry, but I just noticed the title of the PR - can you adjust it to match convention, e.g. [SPARK-18580] [DSTREAM][KAFKA] Add spark.streaming.backpressure.initialRate to direct Kafka streams and then I'll get it merged ;) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19431 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88477/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19431 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19431 **[Test build #88477 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88477/testReport)** for PR 19431 at commit [`d11e807`](https://github.com/apache/spark/commit/d11e8078672048693b3538db902a2827d14eeaf5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19431 **[Test build #88477 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88477/testReport)** for PR 19431 at commit [`d11e807`](https://github.com/apache/spark/commit/d11e8078672048693b3538db902a2827d14eeaf5). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19431 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user akonopko commented on the issue: https://github.com/apache/spark/pull/19431 @koeninger resolved the conflict --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19431 @akonopko thanks for this, if you can resolve merge conflict I think we can get this in --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19431 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user omuravskiy commented on the issue: https://github.com/apache/spark/pull/19431 No objections --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19431 @tdas any concerns? If @omuravskiy doesn't express any objections (since these tests are basically taken directly from his linked PR) in the next couple of days, I'm inclined to merge this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19431 @tdas could you take a look at it? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19431
I mean the difference between `test("use backpressure.initialRate with
backpressure")` and `test("backpressure.initialRate should honor
maxRatePerPartition")` are 3 numbers. Wrapping the common code into one
function and making 2 function call would be better.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user akonopko commented on the issue: https://github.com/apache/spark/pull/19431 > Related the doc I thought it's kafka specific but it's not so fine like that Yes, it was implemented only in Kafka Streams but doc doesnt limit usage of this parameter to Kafka > good to merge the common functionalities Not sure I understood you correctly here. You mean in tests ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user akonopko commented on the issue: https://github.com/apache/spark/pull/19431 Latest rate means rate of previous batch. Is it possible that in alive system 0 events were processed? Only if there is no backlog and no new events came during last batch. Completely possible. This happens during first ran. And this parameter should limit it during 1st ran. Quote from docs: This is the initial maximum receiving rate at which each receiver will receive data for the first batch when the backpressure mechanism is enabled. If it happened during system run, for example there is no backlog and no new events came, we still need to limit system rate since with LatestRate = 0 it results in no limit, causing danger of overflowing the system. If somehow cluster was so heavily loaded with other processes that could process 0 events in Spark Streaming, this means that we might have huge backlog after that. Which mean without this fix system has big chance of overflowing --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user akonopko commented on the issue: https://github.com/apache/spark/pull/19431 @gaborgsomogyi `spark.streaming.backpressure.initialRate` is already documented in here: https://spark.apache.org/docs/latest/configuration.html But was mistakenly not included to to direct Kafka Streams --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19431 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19431 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/19431 Until now it was not fully clear documented in which situation does this parameter take effect. I would personally add things into the doc to be a bit more specific. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19431 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19431 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
