[
https://issues.apache.org/jira/browse/SPARK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491087#comment-15491087
]
Cody Koeninger commented on SPARK-17510:
----------------------------------------
I use direct stream for multiple topic jobs where the rate varies by a factor
of 1000x or more across topics, so I don't agree that case was "not really
considered" :)
It sounds more like the issue you're running into is that the amount of work
per item varies widely across topics, not the rate across topics?
Have you actually tried setting max rate at a level that's sufficient just to
prevent crash on initial startup, then letting backpressure take over from
there?
> Set Streaming MaxRate Independently For Multiple Streams
> --------------------------------------------------------
>
> Key: SPARK-17510
> URL: https://issues.apache.org/jira/browse/SPARK-17510
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 2.0.0
> Reporter: Jeff Nadler
>
> We use multiple DStreams coming from different Kafka topics in a Streaming
> application.
> Some settings like maxrate and backpressure enabled/disabled would be better
> passed as config to KafkaUtils.createStream and
> KafkaUtils.createDirectStream, instead of setting them in SparkConf.
> Being able to set a different maxrate for different streams is an important
> requirement for us; we currently work-around the problem by using one
> receiver-based stream and one direct stream.
> We would like to be able to turn on backpressure for only one of the streams
> as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
