kkrugler commented on code in PR #8812:
URL: https://github.com/apache/pinot/pull/8812#discussion_r888291260
##########
pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/src/main/java/org/apache/pinot/plugin/ingestion/batch/standalone/SegmentGenerationJobRunner.java:
##########
@@ -281,7 +317,11 @@ private void submitSegmentGenTask(File localTempDir, URI
inputFileURI, int seqId
_outputDirFS.copyFromLocalFile(localSegmentTarFile,
outputSegmentTarURI);
}
} catch (Exception e) {
- LOGGER.error("Failed to generate Pinot segment for file - {}",
inputFileURI, e);
+ String msg = "Failed to generate Pinot segment for file - " +
inputFileURI.toString();
+ _failure.compareAndSet(null, new RuntimeException(msg, e));
+
+ // We have to decrement the latch by the number of pending tasks.
Review Comment:
1. I haven't seen `shutdownNow()` throwing exceptions due to concurrent task
submission - do you have any refs for that?
2. If we don't call `shutdownNow()`, then all submitted segment generation
tasks will still be executed when the finally{} code block calls
`_executorService.shutdown()`, yes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
