[ https://issues.apache.org/jira/browse/SPARK-35239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan reassigned SPARK-35239: ----------------------------------- Assignee: ulysses you > Coalesce shuffle partition should handle empty input RDD > --------------------------------------------------------- > > Key: SPARK-35239 > URL: https://issues.apache.org/jira/browse/SPARK-35239 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.2.0 > Reporter: ulysses you > Assignee: ulysses you > Priority: Minor > > If input RDD partition is empty then the map output statistics will be null. > And if all shuffle stage's input RDD partition is empty, we will skip it and > lose the chance to coalesce partition. > > We can simply create a empty partition for these custom shuffle reader to > reduce the partition number. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org