[GitHub] spark pull request #19430: Spark 18580
Github user akonopko closed the pull request at: https://github.com/apache/spark/pull/19430 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19430: Spark 18580
GitHub user akonopko reopened a pull request: https://github.com/apache/spark/pull/19430 Spark 18580 ## What changes were proposed in this pull request? Add `spark.streaming.backpressure.initialRate` to direct Kafka Streams for Kafka 0.8 and 0.10 This is required in order to be able to use backpressure with huge lags, which cannot be processed at once. Without this parameter `DirectKafkaInputDStream` with backpressure enabled would try to get all the possible data from Kafka before adjusting consumption rate ## How was this patch tested? - Tests added to `org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala` and `org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala` - Manual tests on YARN cluster You can merge this pull request into a Git repository by running: $ git pull https://github.com/akonopko/spark SPARK-18580 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19430.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19430 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19430: Spark 18580
Github user akonopko closed the pull request at: https://github.com/apache/spark/pull/19430 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19430: Spark 18580
GitHub user akonopko opened a pull request: https://github.com/apache/spark/pull/19430 Spark 18580 ## What changes were proposed in this pull request? Add `spark.streaming.backpressure.initialRate` to direct Kafka Streams for Kafka 0.8 and 0.10 This is required in order to be able to use backpressure with huge lags, which cannot be processed at once. Without this parameter `DirectKafkaInputDStream` with backpressure enabled would try to get all the possible data from Kafka before adjusting consumption rate ## How was this patch tested? - Tests added to `org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala` and `org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala` - Manual tests on YARN cluster You can merge this pull request into a Git repository by running: $ git pull https://github.com/akonopko/spark SPARK-18580 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19430.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19430 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org