LakshSingla commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1615476668
##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +211,47 @@ private void pushCurrentChunk() throws IOException
{
currentChunk.close();
final Chunk chunk = currentChunk;
- try {
- if (chunk.length() > 0) {
- resultsSize += chunk.length();
+ if (chunk.length() > 0) {
+ try {
+ SEMAPHORE.acquire(); // Acquire a permit from the semaphore
+ pendingFiles.incrementAndGet();
+
+ UPLOAD_EXECUTOR.submit(() -> {
+ try {
+ uploadChunk(chunk);
+ }
+ catch (Exception e) {
+ error = true;
+ LOG.error(e, e.getMessage());
+ throw new RuntimeException(e);
+ }
+ finally {
+ SEMAPHORE.release(); // Release the permit after upload is
completed
Review Comment:
QQ: Wouldn't it be nice if the semaphore was acquired in the thread calling
the release on it, instead of the one helping in scheduling the jobs?
##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -271,50 +313,72 @@ public void close() throws IOException
// This should be emitted as a metric
LOG.info(
"Pushed total [%d] parts containing [%d] bytes in [%d]ms.",
- numChunksPushed,
- resultsSize,
+ numChunksPushed.get(),
+ resultsSize.get(),
pushStopwatch.elapsed(TimeUnit.MILLISECONDS)
);
});
- closer.register(() ->
org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+ try (Closer ignored = closer) {
+ if (!error) {
Review Comment:
Would we ever reach here if error = true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
