Cody Koeninger commented on SPARK-17510:

[~jnadler] See the pull request here https://github.com/apache/spark/pull/15132

Please build and see if it addresses your issue. This will need spark 2.0 and 
kafka 0.10 brokers

If the changes to scaladoc aren't clear enough, this line of testing


shows usage of the new interface.

> Set Streaming MaxRate Independently For Multiple Streams
> --------------------------------------------------------
>                 Key: SPARK-17510
>                 URL: https://issues.apache.org/jira/browse/SPARK-17510
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 2.0.0
>            Reporter: Jeff Nadler
> We use multiple DStreams coming from different Kafka topics in a Streaming 
> application.
> Some settings like maxrate and backpressure enabled/disabled would be better 
> passed as config to KafkaUtils.createStream and 
> KafkaUtils.createDirectStream, instead of setting them in SparkConf.
> Being able to set a different maxrate for different streams is an important 
> requirement for us; we currently work-around the problem by using one 
> receiver-based stream and one direct stream.   
> We would like to be able to turn on backpressure for only one of the streams 
> as well.   

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to