zuston commented on PR #1660: URL: https://github.com/apache/incubator-uniffle/pull/1660#issuecomment-2072218930
Firstly I think this is meaningful for tez/mr. For spark, the sorted shuffle is still meaningful, like the facebook did similar things in their cosco (facebook internal remote shuffle service), @jerqi could add some contexts for that. > (2) I firstly test in HDD, it is terrible. Then I test in SSD cluster, then better. We can try to merge the block, but need lots of modification. I just wanted to do it quickly, so I reused the original mechanism I'm not sure the current design is available when on producation with too much random io. I prefer using the most general and stable way to optimize this if we have. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
