zuston commented on code in PR #744:
URL: https://github.com/apache/incubator-uniffle/pull/744#discussion_r1150426734
##########
storage/src/main/java/org/apache/uniffle/storage/handler/impl/PooledHdfsShuffleWriteHandler.java:
##########
@@ -29,13 +30,29 @@
import org.apache.uniffle.storage.handler.api.ShuffleWriteHandler;
import org.apache.uniffle.storage.util.ShuffleStorageUtils;
+/**
+ * The {@link PooledHdfsShuffleWriteHandler} is a wrapper of underlying
multiple
+ * {@link HdfsShuffleWriteHandler} to support concurrency control of writing
single
+ * partition to multi files.
+ *
+ * By leveraging {@link LinkedBlockingDeque}, it will always write the same
file when
+ * no race condition, which is good for reducing file numbers for HDFS.
+ */
public class PooledHdfsShuffleWriteHandler implements ShuffleWriteHandler {
private static final Logger LOGGER =
LoggerFactory.getLogger(PooledHdfsShuffleWriteHandler.class);
- private final BlockingQueue<HdfsShuffleWriteHandler> queue;
+ private final LinkedBlockingDeque<ShuffleWriteHandler> queue;
private final int maxConcurrency;
private final String basePath;
+ // Only for tests
+ @VisibleForTesting
+ public
PooledHdfsShuffleWriteHandler(LinkedBlockingDeque<ShuffleWriteHandler> queue) {
+ this.queue = queue;
+ this.maxConcurrency = queue.size();
Review Comment:
Oh, sorry. I missed this thread.
Got it. I will fix this in another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
