HeartSaVioR commented on issue #28182: [SPARK-29799][Streaming] add an implemention for issue SPARK-29799. URL: https://github.com/apache/spark/pull/28182#issuecomment-612727225 Adding repartition heavily affects latency if anyone is desired to have lower latency (despite the limitation from the nature of micro-batch), so the issue itself looks valid. The thing is, such functionality has been existing on Spark 2.4.0 (though documentation was missed until Spark 2.4.4, unfortunately), but only available for Structured Streaming. https://spark.apache.org/docs/3.0.0-preview2/structured-streaming-kafka-integration.html https://issues.apache.org/jira/browse/SPARK-23541 https://issues.apache.org/jira/browse/SPARK-28464 https://issues.apache.org/jira/browse/SPARK-30656 I guess the focus has been already moved on Structured Streaming; there're more functionalities which only exist on Structured Streaming and they weren't ported to DStream. If some reviewers see the benefit of porting above issues to DStream then someone could do (I could volunteer too), but until then, I'm not sure there're meaningful reviews here anytime soon.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
