[ https://issues.apache.org/jira/browse/SPARK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491389#comment-15491389 ]
Cody Koeninger commented on SPARK-17510: ---------------------------------------- Just for clarity's sake, compute time is far higher on the stream that is using updateStateByKey? Have you tried mapWIthState? Changing max rate to be per-partition isn't actually a big change in terms of number of lines, the calculations are already done per partition because of backpressure. It's more a question of whether it's worth adding more surface area to the creation api. If I make a branch, are you in a position to test it with a kafka 0.10 cluster, or not? > Set Streaming MaxRate Independently For Multiple Streams > -------------------------------------------------------- > > Key: SPARK-17510 > URL: https://issues.apache.org/jira/browse/SPARK-17510 > Project: Spark > Issue Type: Improvement > Components: Streaming > Affects Versions: 2.0.0 > Reporter: Jeff Nadler > > We use multiple DStreams coming from different Kafka topics in a Streaming > application. > Some settings like maxrate and backpressure enabled/disabled would be better > passed as config to KafkaUtils.createStream and > KafkaUtils.createDirectStream, instead of setting them in SparkConf. > Being able to set a different maxrate for different streams is an important > requirement for us; we currently work-around the problem by using one > receiver-based stream and one direct stream. > We would like to be able to turn on backpressure for only one of the streams > as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org