zsxwing opened a new pull request #27388: [SPARK-30656][SS]Support the "minPartitions" option in Kafka batch source and streaming source v1 URL: https://github.com/apache/spark/pull/27388 ### What changes were proposed in this pull request? - Add `minPartitions` support for Kafka Streaming V1 source. - Add `minPartitions` support for Kafka batch V1 and V2 source. - There is lots of refactoring (moving codes to KafkaOffsetReader) to reuse codes. ### Why are the changes needed? Right now, the "minPartitions" option only works in Kafka streaming source v2. It would be great that we can support it in batch and streaming source v1 (v1 is the fallback mode when a user hits a regression in v2) as well. ### Does this PR introduce any user-facing change? Yep. The `minPartitions` options is supported in Kafka batch and streaming queries for both data source V1 and V2. ### How was this patch tested? New unit tests are added to test "minPartitions".
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
