zuston commented on PR #1660: URL: https://github.com/apache/incubator-uniffle/pull/1660#issuecomment-2071345626
Thanks for your work. @zhengchenyu After reviewing the design doc and concrete code briefly, I have some question about this feature. ### Spark client 1. What's the diference of RecordBlob and RecordBuffer? The combine and sort difference is not reflected on the names 2. Why not making the `RMWriteBufferManager` extend the `WriteBufferManager`, `RMWriterBufferManager` can almost replace the WriteBufferManager, we could control whether sort by config. 3. Is the local sort in block level enoguh? Can we make the block bigger if spill to file? ### Shuffle-Server 1. If the merge failed, the reducer should failover to original mechanism 2. The merge process will introduce too much random read. If this happens on the HDD, the whole process is terrible. So the key point is to make the block bigger (Sort merge them before flushing to disk to create a bigger block?) But this looks will break the original blockId mechanism -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
