[
https://issues.apache.org/jira/browse/GOBBLIN-2124?focusedWorklogId=928601&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-928601
]
ASF GitHub Bot logged work on GOBBLIN-2124:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 03/Aug/24 00:35
Start Date: 03/Aug/24 00:35
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #4016:
URL: https://github.com/apache/gobblin/pull/4016#discussion_r1702418289
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/DagProc.java:
##########
@@ -75,8 +91,19 @@ public final void process(DagManagementStateStore
dagManagementStateStore,
dagProcEngineMetrics.markDagActionsInitialize(getDagActionType(), false);
throw e;
}
- act(dagManagementStateStore, state, dagProcEngineMetrics);
- log.info("{} concluded processing for dagId : {}",
getClass().getSimpleName(), this.dagId);
+ try {
+ act(dagManagementStateStore, state, dagProcEngineMetrics);
+ } catch (Exception e) {
+ if (isTransientException(e)) {
+ log.info("Ignoring transient exception. DagTask {} will conclude and
will not be retried. Exception - {} ",
+ getDagTask(), e);
+
dagManagementStateStore.getDagManagerMetrics().dagProcessingNonRetryableExceptionMeter.mark();
+
dagManagementStateStore.getDagManagerMetrics().dagProcessingExceptionMeter.mark();
Review Comment:
shouldn't this line come before `if (isTransientException(e))` ?
##########
gobblin-utility/src/main/java/org/apache/gobblin/util/ExceptionUtils.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.util;
+
+import java.util.List;
+
+
+public class ExceptionUtils {
+
+ /**
+ * Iterates through the exception change and returns true if it finds
exception that is an instance of any exceptions
Review Comment:
exception "chain"
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcFactory.java:
##########
@@ -48,41 +53,52 @@
@Singleton
public class DagProcFactory implements DagTaskVisitor<DagProc<?>> {
+ private final Config config;
private final FlowCompilationValidationHelper
flowCompilationValidationHelper;
+ private final List<Class<? extends Exception>> nonRetryableExceptions;
@Inject
- public DagProcFactory(FlowCompilationValidationHelper
flowCompilationValidationHelper) {
+ public DagProcFactory(Config config, FlowCompilationValidationHelper
flowCompilationValidationHelper) {
+ this.config = config;
this.flowCompilationValidationHelper = flowCompilationValidationHelper;
+ this.nonRetryableExceptions = ConfigUtils.getStringList(config,
ServiceConfigKeys.DAG_PROC_ENGINE_NON_RETRYABLE_EXCEPTIONS_KEY)
Review Comment:
this doesn't appear used anywhere. did you meant to pass it along to the
`DagProc`s this creates? that would better than providing them the full config.
even better than giving them a `List<Class<? extends Exception>>` would be
to provide each `DagProc` ctor a `Function<Throwable, Boolean>` predicate.
Issue Time Tracking
-------------------
Worklog Id: (was: 928601)
Time Spent: 0.5h (was: 20m)
> ignore non-retryable exceptions in dag proc engine
> --------------------------------------------------
>
> Key: GOBBLIN-2124
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2124
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Arjun Singh Bora
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
