gavin9402 commented on PR #3677: URL: https://github.com/apache/celeborn/pull/3677#issuecomment-4610453440
> This is awesome. I was looking for something similar at work and came across this. Just had a question, my understanding could be wrong here, ShuffleClient c++ implementation is thread safe for calling `read*` and `pushData`. So would it be correct here to say that we can safely do `unsafe impl Sync for ShuffleClient` and change `&mut self` to `&self` for both read* and push_data methods? This could allow sharing the same ShuffleClient object wrapped in Arc to multiple threads safely for concurrent reading and writing. @chandnagautam You're absolutely right. We should make both the read and write APIs parallelizable. The current &mut self signatures force downstream consumers (e.g. Daft) to wrap the client in an Arc<Mutex<_>>, which serializes all concurrent push_data and read_partition calls and defeats multi-partition parallelism. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
