vjagadish1989 commented on a change in pull request #1086: SAMZA-2256:
Hotstandby fix for exception while reading side inputs
URL: https://github.com/apache/samza/pull/1086#discussion_r295430186
##########
File path:
samza-core/src/main/scala/org/apache/samza/storage/ContainerStorageManager.java
##########
@@ -704,44 +718,44 @@ public void run() {
// start the systemConsumers for consuming input
this.sideInputSystemConsumers.start();
- // create a thread for sideInput reads
- Thread readSideInputs = new Thread(() -> {
- while (!shutDownSideInputRead) {
- IncomingMessageEnvelope envelope =
sideInputSystemConsumers.choose(true);
- if (envelope != null) {
-
- if (!envelope.isEndOfStream())
-
sideInputStorageManagers.get(envelope.getSystemStreamPartition()).process(envelope);
- checkSideInputCaughtUp(envelope.getSystemStreamPartition(),
envelope.getOffset(),
- SystemStreamMetadata.OffsetType.NEWEST,
envelope.isEndOfStream());
+ try {
- } else {
- LOG.trace("No incoming message was available");
+ // submit the side input read runnable
+ sideInputsReadExecutor.submit(() -> {
+ try {
+ while (!shouldShutdown) {
+ IncomingMessageEnvelope envelope =
sideInputSystemConsumers.choose(true);
+
+ if (envelope != null) {
+ if (!envelope.isEndOfStream()) {
+
sideInputStorageManagers.get(envelope.getSystemStreamPartition()).process(envelope);
+ }
+
+ checkSideInputCaughtUp(envelope.getSystemStreamPartition(),
envelope.getOffset(),
+ SystemStreamMetadata.OffsetType.NEWEST,
envelope.isEndOfStream());
+ } else {
+ LOG.trace("No incoming message was available");
+ }
+ }
+ } catch (Exception e) {
+ LOG.error("Exception in reading side inputs", e);
+ sideInputException = e;
}
- }
- });
+ });
- readSideInputs.setDaemon(true);
- readSideInputs.setUncaughtExceptionHandler(new
Thread.UncaughtExceptionHandler() {
- @Override
- public void uncaughtException(Thread t, Throwable e) {
- sideInputException = Optional.of(e);
- sideInputsCaughtUp.countDown();
+ // Make the main thread wait until all sideInputs have been caughtup or
an exception was thrown
Review comment:
nit: pick one convention ("side input" vs "sideInput" vs "side-input") and
use it consistently in java-doc comments. currently we have all 3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
