pratapaditya04 commented on code in PR #4197:
URL: https://github.com/apache/gobblin/pull/4197#discussion_r3347320445


##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -970,6 +1090,15 @@ void signalGracefulShutdownAndWaitForTerminal() {
       }
       sendGracefulShutdownSignal(amContainerId.get());
       pollForApplicationCompletionUntil(timeoutMs);
+      // Always kill so the RM cannot re-attempt the AM (no-op if the app 
already finished). If the terminal
+      // outcome wasn't already dispatched by the report-monitor path, this is 
a cancel/forced shutdown ->
+      // surface it as KILLED (the subclass emits JOB_CANCEL). The token, not 
a racy report read, is the gate.
+      LOGGER.info("Graceful-shutdown wait complete; force-killing application 
{} to prevent AM re-attempt", this.applicationId.get());
+      this.yarnClient.killApplication(this.applicationId.get());
+      if (this.terminalHandled.compareAndSet(false, true)) {
+        this.exitCode = 1;
+        onTerminalApplicationStatus(null, FinalApplicationStatus.KILLED);
+      }

Review Comment:
   Good catch. In `11ba7500a`, after `pollForApplicationCompletionUntil(...)` I 
now re-fetch the `ApplicationReport` and, if it is already terminal, dispatch 
the **real** outcome via `handleTerminalAppStatus(...)` (which claims the same 
one-shot `terminalHandled` token) *before* the force-kill. The synthetic 
`KILLED` now only fires when the app is genuinely still running, so a 
self-completed `SUCCEEDED`/`FAILED` is no longer mislabeled. The force-kill 
stays as an idempotent no-op to prevent an AM re-attempt.
   



##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -510,6 +536,64 @@ public void 
handleApplicationReportArrivalEvent(ApplicationReportArrivalEvent ap
     }
   }
 
+  /**
+   * On a terminal {@link ApplicationReport}: update {@link #exitCode} so the 
launcher JVM surfaces the failure
+   * to GGW/Grid Gateway dashboards, then dispatch to the overridable {@link 
#onTerminalApplicationStatus} hook.
+   */
+  @VisibleForTesting
+  void handleTerminalAppStatus(ApplicationReport applicationReport) {
+    if (!this.terminalHandled.compareAndSet(false, true)) {
+      return;
+    }
+    FinalApplicationStatus finalStatus = 
applicationReport.getFinalApplicationStatus();
+    if (finalStatus != FinalApplicationStatus.SUCCEEDED) {
+      this.exitCode = 1;
+    }
+    LOGGER.info("Application reached terminal FinalApplicationStatus {}; 
launcher exitCode={}", finalStatus, this.exitCode);
+    onTerminalApplicationStatus(applicationReport, finalStatus);
+  }
+
+  /**
+   * Terminal-outcome hooks. No-op in OSS (the OSS launcher emits no GTE); a 
subclass (e.g. the GGW/Azkaban-pod
+   * launcher) overrides these to emit the single terminal GTE. {@code 
finalStatus} maps via
+   * {@link #mapFinalAppStatusToEventName}.
+   */
+  protected void onTerminalApplicationStatus(ApplicationReport 
applicationReport, FinalApplicationStatus finalStatus) {
+  }

Review Comment:
   Done in `11ba7500a` — added a `@param applicationReport` note documenting 
that it may be `null` (the forced-kill / cancel path dispatches `KILLED` 
without a report), so overriding subclasses guard before dereferencing.
   



##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/YarnService.java:
##########
@@ -286,6 +290,20 @@ protected void shutDown() throws IOException {
     }
   }
 
+  /**
+   * Derive the {@link FinalApplicationStatus} to report to YARN on 
un-register from the AM's captured Temporal
+   * workflow outcome (see {@link 
GobblinTemporalJobLauncher#getLastTerminalStatus()}). Reporting the true status
+   * — rather than an unconditional SUCCEEDED — lets a launcher polling the 
{@code ApplicationReport} observe job
+   * failures/cancellations end-to-end, and keeps the YARN status consistent 
with the AM JVM exit code. A
+   * {@code null} captured status (no workflow ran) maps to SUCCEEDED. 
Overridable for tests.
+   */
+  protected FinalApplicationStatus getFinalApplicationStatusForUnregister() {
+    io.temporal.api.enums.v1.WorkflowExecutionStatus captured = 
GobblinTemporalJobLauncher.getLastTerminalStatus();
+    FinalApplicationStatus derived = 
GobblinTemporalJobLauncher.mapWorkflowStatusToFinalAppStatus(captured);
+    LOGGER.info("Derived FinalApplicationStatus {} for un-register from 
captured workflow status {}", derived, captured);
+    return derived;
+  }

Review Comment:
   Fixed the Javadoc in `11ba7500a` — a `null` captured status is now 
documented as mapping to `FAILED` (matching 
`mapWorkflowStatusToFinalAppStatus(null)` and the unit tests), not `SUCCEEDED`.
   



##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/YarnService.java:
##########
@@ -262,17 +263,20 @@ protected void shutDown() throws IOException {
       if (!this.containerMap.isEmpty()) {
         synchronized (this.allContainersStopped) {
           try {
-            // Wait 5 minutes for the containers to stop
-            Duration waitTimeout = Duration.ofMinutes(5);
+            // Bounded fallback only: a missed notify must not stall the AM 
un-register. The normal path
+            // is woken immediately by notifyIfAllContainersStopped() as soon 
as the last container stops.
+            Duration waitTimeout = Duration.ofMinutes(2);
             this.allContainersStopped.wait(waitTimeout.toMillis());
-            LOGGER.info("All of the containers have been stopped");
+            LOGGER.info("Finished waiting for containers to stop ({} still 
tracked)", this.containerMap.size());
           } catch (InterruptedException ie) {
             Thread.currentThread().interrupt();
           }
         }
       }

Review Comment:
   Fixed in `11ba7500a` — `shutDown()` now acquires the `allContainersStopped` 
monitor first and loops on `!containerMap.isEmpty()` with a deadline, so an 
already-empty map never waits and a `notifyAll` firing between the old outer 
check and `wait()` can no longer be missed (no full-timeout stall).
   



##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/joblauncher/GobblinTemporalJobLauncher.java:
##########
@@ -97,9 +107,64 @@ public GobblinTemporalJobLauncher(Properties jobProps, Path 
appWorkDir,
 
     // non-null value indicates job has been submitted
     this.workflowId = null;
+    // Reset the process-wide terminal-status cache for this fresh launcher. 
In a real AM JVM only one
+    // launcher ever runs, but tests re-instantiate within the same JVM and 
must not see leaked status.
+    setLastTerminalStatus(null);
     startCancellationExecutor();
   }
 
+  /**
+   * Query Temporal for the current execution status of {@link #workflowId}. 
Returns
+   * {@code WORKFLOW_EXECUTION_STATUS_UNSPECIFIED} as a safe fallback if the 
describe call fails, so callers
+   * downstream emit a JOB_FAILED rather than swallowing the GTE entirely.
+   */
+  private WorkflowExecutionStatus fetchWorkflowStatus() {
+    try {
+      WorkflowStub workflowStub = 
this.client.newUntypedWorkflowStub(this.workflowId);
+      DescribeWorkflowExecutionRequest request = 
DescribeWorkflowExecutionRequest.newBuilder()
+          .setNamespace(this.namespace)
+          .setExecution(workflowStub.getExecution())
+          .build();
+      DescribeWorkflowExecutionResponse response = 
managedWorkflowServiceStubs.getWorkflowServiceStubs()
+          .blockingStub().describeWorkflowExecution(request);
+      return response.getWorkflowExecutionInfo().getStatus();
+    } catch (Exception e) {
+      log.warn("Failed to describe workflow {} for completion GTE; treating as 
UNSPECIFIED (will emit JOB_FAILED)",
+          this.workflowId, e);
+      return WorkflowExecutionStatus.WORKFLOW_EXECUTION_STATUS_UNSPECIFIED;
+    }

Review Comment:
   Thanks — addressed in `11ba7500a`. `fetchWorkflowStatus()` now retries on 
transient gRPC `StatusRuntimeException` codes (`UNAVAILABLE`, 
`DEADLINE_EXCEEDED`, `RESOURCE_EXHAUSTED`) with bounded linear backoff (3 
attempts) before falling back; non-retryable codes fall back immediately.
   
   One inherent caveat: if Temporal is still unreachable after the retries, the 
true status is genuinely unknowable, so we still fall back to `UNSPECIFIED` → 
`JOB_FAILED` rather than emit nothing (leaving GaaS without any terminal state 
would be worse). Happy to tune the attempt count / backoff. Leaving this open 
for your re-review.
   



##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -970,6 +1090,15 @@ void signalGracefulShutdownAndWaitForTerminal() {
       }
       sendGracefulShutdownSignal(amContainerId.get());
       pollForApplicationCompletionUntil(timeoutMs);
+      // Always kill so the RM cannot re-attempt the AM (no-op if the app 
already finished). If the terminal

Review Comment:
   `killApplication` on an already-finished application is idempotent on the RM 
side — it returns without throwing (no `YarnException`), so it is effectively a 
no-op and won't produce error logs; the only line emitted is our own preceding 
`LOGGER.info(... force-killing ...)`. In `11ba7500a` I also now re-check the 
report just before this call and dispatch the real terminal status if the app 
already finished, so for a self-completed app we no longer rely on the kill for 
the outcome at all. Leaving open for your confirmation.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to