phet commented on code in PR #3918:
URL: https://github.com/apache/gobblin/pull/3918#discussion_r1559987681
##########
gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionStoreChangeMonitor.java:
##########
@@ -144,7 +144,12 @@ protected void initializeMonitor() {
}
// TODO: make this multi-threaded to add parallelism
for (DagActionStore.DagAction action : dagActions) {
- handleDagAction(action, true);
+ try {
+ handleDagAction(action, true);
+ } catch (Exception e) {
+ log.warn("Ran into an unexpected error processing DagActionStore
changes during initialization for action {}", action, e);
Review Comment:
`error` level?
##########
gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/runtime/DagActionStoreChangeMonitorTest.java:
##########
@@ -98,6 +118,16 @@ public void setup() {
mockDagActionStoreChangeMonitor.startUpForTest();
}
+ @BeforeClass
+ public void setupTestDb() throws Exception {
+ this.testDb = TestMetastoreDatabaseFactory.get();
+ }
+
+ @AfterClass
+ public void cleanup() throws IOException {
+ this.testDb.close();
Review Comment:
shall we name `setUp()` and `tearDown()` like the other test?
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/utils/FlowCompilationValidationHelper.java:
##########
@@ -136,7 +136,7 @@ public Optional<Dag<JobExecutionPlan>>
validateAndHandleConcurrentExecution(Conf
ConfigurationKeys.FLOW_ALLOW_CONCURRENT_EXECUTION,
String.valueOf(this.isFlowConcurrencyEnabled)));
Dag<JobExecutionPlan> jobExecutionPlanDag =
specCompiler.compileFlow(flowSpec);
- if (jobExecutionPlanDag.isEmpty()) {
+ if (jobExecutionPlanDag == null || jobExecutionPlanDag.isEmpty()) {
Review Comment:
was this NPE always possible, but unhandled, or arises now as a result of
these changes?
##########
gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/runtime/DagActionStoreChangeMonitorTest.java:
##########
@@ -201,6 +231,40 @@ public void testProcessMessageWithDelete() throws
SpecNotFoundException {
verify(mockDagActionStoreChangeMonitor.getFlowCatalog(),
times(0)).getSpecs(any(URI.class));
}
+ @Test
+ public void testStartupSequenceHandlesFailures() throws Exception {
+ Config config = ConfigBuilder.create()
+ .addPrimitive("MysqlDagActionStore." +
ConfigurationKeys.STATE_STORE_DB_URL_KEY, this.testDb.getJdbcUrl())
+ .addPrimitive("MysqlDagActionStore." +
ConfigurationKeys.STATE_STORE_DB_USER_KEY, USER)
+ .addPrimitive("MysqlDagActionStore." +
ConfigurationKeys.STATE_STORE_DB_PASSWORD_KEY, PASSWORD)
+ .addPrimitive("MysqlDagActionStore." +
ConfigurationKeys.STATE_STORE_DB_TABLE_KEY, TABLE)
+ .build();
+ String flowGroup = "testFlowGroup";
+ String flowName = "testFlowName";
+ String jobName = "testJobName";
+ String flowExecutionId = "12345677";
+
+ MysqlDagActionStore mysqlDagActionStore = new MysqlDagActionStore(config);
+ mysqlDagActionStore.addJobDagAction(flowGroup, flowName, flowExecutionId,
jobName, DagActionStore.DagActionType.LAUNCH);
+
+ Config monitorConfig =
ConfigFactory.empty().withValue(ConfigurationKeys.KAFKA_BROKERS,
ConfigValueFactory.fromAnyRef("localhost:0000"))
+
.withValue(Kafka09ConsumerClient.GOBBLIN_CONFIG_VALUE_DESERIALIZER_CLASS_KEY,
ConfigValueFactory.fromAnyRef("org.apache.kafka.common.serialization.ByteArrayDeserializer"))
+ .withValue(ConfigurationKeys.STATE_STORE_ROOT_DIR_KEY,
ConfigValueFactory.fromAnyRef("/tmp/fakeStateStore"))
+ .withValue("zookeeper.connect",
ConfigValueFactory.fromAnyRef("localhost:2121"));
+ DagManager mockDagManager = mock(DagManager.class);
+ FlowCatalog mockFlowCatalog = mock(FlowCatalog.class);
+ Orchestrator mockOrchestrator = mock(Orchestrator.class);
+ // Throw an uncaught exception during startup sequence
+ when(mockFlowCatalog.getSpecs(any(URI.class))).thenThrow(new
RuntimeException("Uncaught exception"));
Review Comment:
should we verify that this method was actually called?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]