peter-toth commented on PR #59522: URL: https://github.com/apache/spark/pull/59522#issuecomment-6426181767
Thanks for reworking this into a coalesce below the shuffle, @rekbun. It is the shape I suggested, and it no longer touches the SPJ plans. Looking at it again, I'm not sure the key count is the right target, though: - It restores the task count your connector got before 4.2. That grouping came from the SPJ code, which grouped keyed scans even where nothing used their partitioning. - It has no relation to the data. A key with 100 GB becomes one task, while 10,000 tiny keys stay 10,000 tasks. - Too many small splits isn't specific to keyed scans. Any DSv2 scan runs one task per `InputPartition`, and AQE can't help a leaf stage, since it only works from shuffle statistics. File sources already handle this. `spark.sql.files.maxPartitionBytes` and `spark.sql.files.openCostInBytes` pack files by size, and `spark.sql.files.minPartitionNum` and `spark.sql.files.maxPartitionNum` bound the count. DSv2 has `HasPartitionStatistics.sizeInBytes()` (SPARK-42040) for this kind of decision, but nothing uses it yet. szehon-ho asked about the same thing [there](https://github.com/apache/spark/pull/45314#issuecomment-2021228813). So I'd rather go for a general option, independent of `KeyedPartitioning`: - Pack a DSv2 scan's splits by size when they report `sizeInBytes()`, and cap their count otherwise, like the file source configs do. - Apply it only where nothing uses the scan's partitioning or ordering. That is below a shuffle, or where there is no consumer at all, like a collect, a write or the streamed side of a broadcast join. - Pack at the scan itself. Then a partial aggregate above it combines the splits again, as it did before 4.2. Would that work for your connector, and would you be open to doing it in that form? Do its splits report their size, or would a count cap be enough? @dongjoon-hyun @cloud-fan @viirya @ulysses-you @LuciferYang @pan3793, I'm surprised Spark has no generic answer for small DSv2 splits. Did I miss one, or do you have other ideas? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
