EnricoMi commented on PR #48960: URL: https://github.com/apache/spark/pull/48960#issuecomment-2516552322
I did some measurements: with two shuffles and 10.000 partitions I end up having 22,458 folders with 44,912 files and 1.41 GB. With ten sub-directories I get 22 folders with 43,775 files and 1.41 GB. Removing the former takes more than twice as long as removing the latter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
