[ https://issues.apache.org/jira/browse/SPARK-33962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
L. C. Hsieh updated SPARK-33962: -------------------------------- Issue Type: Bug (was: Improvement) > Fix incorrect min partition condition in getRanges > -------------------------------------------------- > > Key: SPARK-33962 > URL: https://issues.apache.org/jira/browse/SPARK-33962 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 3.2.0 > Reporter: L. C. Hsieh > Assignee: L. C. Hsieh > Priority: Minor > > When calculating offset ranges, we consider minPartitions configuration. If > minPartitions is not set or is less than or equal the size of given ranges, > it means there are enough partitions at Kafka so we don't need to split > offsets to satisfy min partition requirement. But the current condition is > offsetRanges.size > minPartitions.get and is not correct. Currently getRanges > will split offsets in unnecessary case. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org