avantgardnerio commented on PR #2106: URL: https://github.com/apache/datafusion-ballista/pull/2106#issuecomment-5025108070
> very high number of output partitions (hundreds/thousands). Let's say there's a company with a 250 node cluster, each with 32 vcores. Currently, with `paritions_per_task = 32`, that would create 8000 sort shuffle output files. Stage N+1 would need to read a small chunk of all 8000 files. Compare that with Hash partitioning, which would output 250 files. I believe this table is correct: <img width="1013" height="672" alt="image" src="https://github.com/user-attachments/assets/ab8ae9ed-6de7-4f97-8a6f-f1ff081fbe8d"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
