zuston commented on issue #736: URL: https://github.com/apache/incubator-uniffle/issues/736#issuecomment-1475522464
Add some detailed information about feature of concurrent writing single partition. 1. The reason of this feature is to solve the write lock of writing same partition data. 2. this feature is no cost, which won't use the thread pool to do this, only leveraging the flushing thread pool in `ShuffleFlushManager`, refer to `PooledHdfsShuffleWriteHandler` Based on above description, I think we could do some improvements for this. 1. Make client configuring this to control the concurrency of writing single partition 2. Best effort to reduce file number in case of no race condition, which is important for HDFS performance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@uniffle.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
