mynameborat commented on a change in pull request #1583:
URL: https://github.com/apache/samza/pull/1583#discussion_r806099549
##########
File path:
samza-core/src/main/java/org/apache/samza/runtime/ContainerLaunchUtil.java
##########
@@ -179,15 +181,26 @@ private static void run(
if (containerRunnerException != null) {
log.error("Container stopped with Exception. Exiting process now.",
containerRunnerException);
- System.exit(1);
+ exitProcess(1);
}
} catch (Throwable e) {
- log.error("Container stopped with Exception. ",
containerRunnerException);
+ log.error("Exiting the process due to {}. \nContainer runner exception:
{}", e, containerRunnerException);
+ exitProcess(1);
} finally {
coordinatorStreamStore.close();
}
}
+ @VisibleForTesting
+ static CoordinatorStreamStore buildCoordinatorStreamStore(Config config,
MetricsRegistryMap metricsRegistryMap) {
+ return new CoordinatorStreamStore(config, metricsRegistryMap);
+ }
+
+ @VisibleForTesting
Review comment:
that is right
##########
File path:
samza-core/src/main/java/org/apache/samza/runtime/ContainerLaunchUtil.java
##########
@@ -179,15 +181,26 @@ private static void run(
if (containerRunnerException != null) {
log.error("Container stopped with Exception. Exiting process now.",
containerRunnerException);
- System.exit(1);
+ exitProcess(1);
}
} catch (Throwable e) {
- log.error("Container stopped with Exception. ",
containerRunnerException);
+ log.error("Exiting the process due to {}. \nContainer runner exception:
{}", e, containerRunnerException);
+ exitProcess(1);
} finally {
coordinatorStreamStore.close();
}
}
+ @VisibleForTesting
+ static CoordinatorStreamStore buildCoordinatorStreamStore(Config config,
MetricsRegistryMap metricsRegistryMap) {
+ return new CoordinatorStreamStore(config, metricsRegistryMap);
+ }
+
+ @VisibleForTesting
Review comment:
I didn't want to go full on refactor this code. For now, I took a stab
to enable unit testing my changes.
Historically, this class has been pain to test and is missing coverage
citing static & hard to mock code paths. We'd need a dedicated refactor and
unit testing effort for this but taking the first step to add one.
##########
File path:
samza-core/src/main/java/org/apache/samza/runtime/ContainerLaunchUtil.java
##########
@@ -179,15 +181,26 @@ private static void run(
if (containerRunnerException != null) {
log.error("Container stopped with Exception. Exiting process now.",
containerRunnerException);
- System.exit(1);
+ exitProcess(1);
}
} catch (Throwable e) {
- log.error("Container stopped with Exception. ",
containerRunnerException);
+ log.error("Exiting the process due to {}. \nContainer runner exception:
{}", e, containerRunnerException);
+ exitProcess(1);
Review comment:
Good catch! I updated the code to make sure we only exit as part of the
finally block.
##########
File path:
samza-core/src/main/java/org/apache/samza/runtime/ContainerLaunchUtil.java
##########
@@ -179,15 +181,26 @@ private static void run(
if (containerRunnerException != null) {
log.error("Container stopped with Exception. Exiting process now.",
containerRunnerException);
- System.exit(1);
+ exitProcess(1);
}
} catch (Throwable e) {
- log.error("Container stopped with Exception. ",
containerRunnerException);
+ log.error("Exiting the process due to {}. \nContainer runner exception:
{}", e, containerRunnerException);
Review comment:
Fixed the log statements. Here is the sample output
```
12:38:29.055 [main] ERROR org.apache.samza.runtime.ContainerLaunchUtil -
Exiting the process due to
java.lang.IllegalArgumentException: ApplicationDescriptorImpl has to be
either TaskApplicationDescriptorImpl or StreamApplicationDescriptorImpl. class
org.apache.samza.application.descriptors.ApplicationDescriptorImpl$$EnhancerByMockitoWithCGLIB$$1d97dd82
is not supported
at
org.apache.samza.task.TaskFactoryUtil.getTaskFactory(TaskFactoryUtil.java:54)
~[samza-core_2.11/:?]
at
org.apache.samza.runtime.ContainerLaunchUtil.run(ContainerLaunchUtil.java:127)
[samza-core_2.11/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_172]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_172]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_172]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172]
at
org.powermock.api.mockito.internal.invocation.MockitoMethodInvocationControl$MockedRealMethod.invoke(MockitoMethodInvocationControl.java:343)
[powermock-api-mockito-1.6.6.jar:?]
at
org.mockito.internal.invocation.realmethod.CleanTraceRealMethod.invoke(CleanTraceRealMethod.java:30)
[mockito-core-1.10.19.jar:?]
at
org.mockito.internal.invocation.InvocationImpl.callRealMethod(InvocationImpl.java:112)
[mockito-core-1.10.19.jar:?]
at
org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:41)
[mockito-core-1.10.19.jar:?]
at
org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
[mockito-core-1.10.19.jar:?]
at
org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
[mockito-core-1.10.19.jar:?]
at
org.powermock.api.mockito.internal.invocation.MockitoMethodInvocationControl.performIntercept(MockitoMethodInvocationControl.java:247)
[powermock-api-mockito-1.6.6.jar:?]
at
org.powermock.api.mockito.internal.invocation.MockitoMethodInvocationControl.invoke(MockitoMethodInvocationControl.java:199)
[powermock-api-mockito-1.6.6.jar:?]
at org.powermock.core.MockGateway.doMethodCall(MockGateway.java:173)
[powermock-core-1.6.6.jar:?]
at org.powermock.core.MockGateway.doMethodCall(MockGateway.java:155)
[powermock-core-1.6.6.jar:?]
at org.powermock.core.MockGateway.methodCall(MockGateway.java:132)
[powermock-core-1.6.6.jar:?]
at
org.apache.samza.runtime.ContainerLaunchUtil.run(ContainerLaunchUtil.java)
[samza-core_2.11/:?]
at
org.apache.samza.runtime.TestContainerLaunchUtil.testRunWithException(TestContainerLaunchUtil.java:64)
[samza-core_2.11/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_172]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_172]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_172]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172]
at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:68)
[junit-4.12.jar:4.12]
at
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:316)
[powermock-module-junit4-1.6.6.jar:?]
at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:89)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:97)
[junit-4.12.jar:4.12]
at
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.executeTest(PowerMockJUnit44RunnerDelegateImpl.java:300)
[powermock-module-junit4-1.6.6.jar:?]
at
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTestInSuper(PowerMockJUnit47RunnerDelegateImpl.java:131)
[powermock-module-junit4-1.6.6.jar:?]
at
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.access$100(PowerMockJUnit47RunnerDelegateImpl.java:59)
[powermock-module-junit4-1.6.6.jar:?]
at
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner$TestExecutorStatement.evaluate(PowerMockJUnit47RunnerDelegateImpl.java:147)
[powermock-module-junit4-1.6.6.jar:?]
at
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.evaluateStatement(PowerMockJUnit47RunnerDelegateImpl.java:107)
[powermock-module-junit4-1.6.6.jar:?]
at
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTest(PowerMockJUnit47RunnerDelegateImpl.java:82)
[powermock-module-junit4-1.6.6.jar:?]
at
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runBeforesThenTestThenAfters(PowerMockJUnit44RunnerDelegateImpl.java:288)
[powermock-module-junit4-1.6.6.jar:?]
at
org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:87)
[junit-4.12.jar:4.12]
at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:50)
[junit-4.12.jar:4.12]
at
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.invokeTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:208)
[powermock-module-junit4-1.6.6.jar:?]
at
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.runMethods(PowerMockJUnit44RunnerDelegateImpl.java:147)
[powermock-module-junit4-1.6.6.jar:?]
at
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$1.run(PowerMockJUnit44RunnerDelegateImpl.java:121)
[powermock-module-junit4-1.6.6.jar:?]
at
org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
[junit-4.12.jar:4.12]
at
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.run(PowerMockJUnit44RunnerDelegateImpl.java:123)
[powermock-module-junit4-1.6.6.jar:?]
at
org.powermock.modules.junit4.common.internal.impl.JUnit4TestSuiteChunkerImpl.run(JUnit4TestSuiteChunkerImpl.java:121)
[powermock-module-junit4-common-1.6.6.jar:?]
at
org.powermock.modules.junit4.common.internal.impl.AbstractCommonPowerMockRunner.run(AbstractCommonPowerMockRunner.java:53)
[powermock-module-junit4-common-1.6.6.jar:?]
at
org.powermock.modules.junit4.PowerMockRunner.run(PowerMockRunner.java:59)
[powermock-module-junit4-1.6.6.jar:?]
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
[junit-4.12.jar:4.12]
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
[junit-rt.jar:?]
at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
[junit-rt.jar:?]
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:221)
[junit-rt.jar:?]
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)
[junit-rt.jar:?]
12:38:29.071 [main] ERROR org.apache.samza.runtime.ContainerLaunchUtil -
Container runner exception:
```
##########
File path:
samza-core/src/test/java/org/apache/samza/runtime/TestContainerLaunchUtil.java
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.samza.runtime;
+
+import java.util.Optional;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import org.apache.samza.application.descriptors.ApplicationDescriptorImpl;
+import org.apache.samza.config.Config;
+import org.apache.samza.coordinator.metadatastore.CoordinatorStreamStore;
+import org.apache.samza.job.model.JobModel;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.powermock.api.mockito.PowerMockito;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
+
+import static org.junit.Assert.*;
+import static org.mockito.Mockito.*;
+
+
+@RunWith(PowerMockRunner.class)
+@PrepareForTest(ContainerLaunchUtil.class)
+public class TestContainerLaunchUtil {
+ private static final String JOB_NAME = "test-job";
+ private static final String JOB_ID = "test-job-i001";
+ private static final String CONTAINER_ID = "test-job-container-0001";
+
+ private static final ApplicationDescriptorImpl APP_DESC =
mock(ApplicationDescriptorImpl.class);
+ private static final JobModel JOB_MODEL = mock(JobModel.class);
+ private static final Config CONFIG = mock(Config.class);
+
+ @Test
+ public void testRunWithException() throws Exception {
+ final CountDownLatch completionLatch = new CountDownLatch(1);
+ PowerMockito.mockStatic(ContainerLaunchUtil.class);
+ PowerMockito.doReturn(mock(CoordinatorStreamStore.class))
+ .when(ContainerLaunchUtil.class, "buildCoordinatorStreamStore",
eq(CONFIG), any());
+ PowerMockito.doAnswer(invocation -> {
+ completionLatch.countDown();
+ return null;
+ }).when(ContainerLaunchUtil.class, "exitProcess", eq(1));
+ PowerMockito.doCallRealMethod()
+ .when(ContainerLaunchUtil.class, "run", eq(APP_DESC), eq(JOB_NAME),
eq(JOB_ID), eq(CONTAINER_ID), any(), any(),
+ eq(JOB_MODEL), eq(CONFIG), any());
+
+ ContainerLaunchUtil.run(APP_DESC, JOB_NAME, JOB_ID, CONTAINER_ID,
Optional.empty(), Optional.empty(), JOB_MODEL,
+ CONFIG, Optional.empty());
+ assertTrue(completionLatch.await(1, TimeUnit.SECONDS));
Review comment:
I don't think `verifyStatic` works as expected here. I noticed the weird
behavior in the event of `doNothing` and invalid mismatch in the arguments for
`exitProcess` method and it succeeded.
I like the fact that it is explicit and runs the test validation this way
and isn't cryptic for readers about what the test does and what it validates.
##########
File path:
samza-core/src/main/java/org/apache/samza/runtime/ContainerLaunchUtil.java
##########
@@ -179,15 +187,33 @@ private static void run(
if (containerRunnerException != null) {
log.error("Container stopped with Exception. Exiting process now.",
containerRunnerException);
- System.exit(1);
+ exitCode = 1;
}
} catch (Throwable e) {
- log.error("Container stopped with Exception. ",
containerRunnerException);
+ /*
+ * Two separate log statements are intended to print the entire stack
trace as part of the logs. Using
+ * single log statement with custom format requires explicitly fetching
stack trace and null checks which makes
+ * the code slightly hard to read in comparison with the current choice.
+ */
+ log.error("Exiting the process due to", e);
+ log.error("Container runner exception: ", containerRunnerException);
+ exitCode = 1;
} finally {
coordinatorStreamStore.close();
+ exitProcess(exitCode);
Review comment:
Agreed. I modified it to just exit only the in event of non-zero exit
code.
However, testing this scenario in unit test is extremely painful in the
current setup and hence I am going to leave that code path where we don't exit
the process untested.
e.g., to get the run method fully working as expected and setting up all
components.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
