waitinfuture commented on PR #2373: URL: https://github.com/apache/celeborn/pull/2373#issuecomment-2076301404
> QQ: thinking out aloud, instead of this change - do we want to proactively trigger sort for reducers where we are reading a subset of mapper output (based on `ShufflePartitionsUtil`) ? > > This will help if we are trying to mitigate the impact of reducer read timeouts, etc. It wont bring down the overall load (at worker) though. > > On plus side, it does not suffer from the correctness issues here. > > Thoughts ? Unfortunately we don't know whether a partition split will be read in a map-range fashion until the read request comes : ( BTW, even though we figure out a way, overall sorting some files increases the burden on worker (maybe negligible if worker is under low load) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
