Copilot commented on code in PR #4197:
URL: https://github.com/apache/gobblin/pull/4197#discussion_r3338980715


##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -970,6 +1090,15 @@ void signalGracefulShutdownAndWaitForTerminal() {
       }
       sendGracefulShutdownSignal(amContainerId.get());
       pollForApplicationCompletionUntil(timeoutMs);
+      // Always kill so the RM cannot re-attempt the AM (no-op if the app 
already finished). If the terminal
+      // outcome wasn't already dispatched by the report-monitor path, this is 
a cancel/forced shutdown ->
+      // surface it as KILLED (the subclass emits JOB_CANCEL). The token, not 
a racy report read, is the gate.
+      LOGGER.info("Graceful-shutdown wait complete; force-killing application 
{} to prevent AM re-attempt", this.applicationId.get());
+      this.yarnClient.killApplication(this.applicationId.get());
+      if (this.terminalHandled.compareAndSet(false, true)) {
+        this.exitCode = 1;
+        onTerminalApplicationStatus(null, FinalApplicationStatus.KILLED);
+      }

Review Comment:
   The graceful-shutdown path always force-kills the application and (if the 
report-monitor hasn’t already fired) dispatches a synthetic `KILLED` terminal 
status. If the app actually reached a terminal state during 
`pollForApplicationCompletionUntil(...)` (including `SUCCEEDED`), but the 
scheduled report monitor hasn’t processed it yet, this will incorrectly flip 
`exitCode` to 1 and invoke `onTerminalApplicationStatus(..., KILLED)`.
   
   After polling, re-fetch the `ApplicationReport` and, if it’s already 
terminal, dispatch the real final status (and skip the synthetic KILLED path).



##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -510,6 +536,64 @@ public void 
handleApplicationReportArrivalEvent(ApplicationReportArrivalEvent ap
     }
   }
 
+  /**
+   * On a terminal {@link ApplicationReport}: update {@link #exitCode} so the 
launcher JVM surfaces the failure
+   * to GGW/Grid Gateway dashboards, then dispatch to the overridable {@link 
#onTerminalApplicationStatus} hook.
+   */
+  @VisibleForTesting
+  void handleTerminalAppStatus(ApplicationReport applicationReport) {
+    if (!this.terminalHandled.compareAndSet(false, true)) {
+      return;
+    }
+    FinalApplicationStatus finalStatus = 
applicationReport.getFinalApplicationStatus();
+    if (finalStatus != FinalApplicationStatus.SUCCEEDED) {
+      this.exitCode = 1;
+    }
+    LOGGER.info("Application reached terminal FinalApplicationStatus {}; 
launcher exitCode={}", finalStatus, this.exitCode);
+    onTerminalApplicationStatus(applicationReport, finalStatus);
+  }
+
+  /**
+   * Terminal-outcome hooks. No-op in OSS (the OSS launcher emits no GTE); a 
subclass (e.g. the GGW/Azkaban-pod
+   * launcher) overrides these to emit the single terminal GTE. {@code 
finalStatus} maps via
+   * {@link #mapFinalAppStatusToEventName}.
+   */
+  protected void onTerminalApplicationStatus(ApplicationReport 
applicationReport, FinalApplicationStatus finalStatus) {
+  }

Review Comment:
   `onTerminalApplicationStatus` can currently be invoked with a null 
`ApplicationReport` (e.g., the forced-kill path calls 
`onTerminalApplicationStatus(null, KILLED)`). This should be documented 
explicitly so subclasses don’t assume a non-null report and accidentally NPE 
when emitting their terminal event.



##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -1180,6 +1309,19 @@ public void run() {
       }
     });
 
-    gobblinYarnAppLauncher.launch();
+    try {
+      gobblinYarnAppLauncher.launch();
+    } catch (Exception e) {
+      // launch() already invoked handleApplicationLaunchFailure (sets 
exitCode=1, emits terminal GTE) before
+      // rethrowing. Swallow here so we reach the explicit System.exit below 
with the non-zero code.
+      LOGGER.error("GobblinYarnAppLauncher launch failed", e);
+    }
+
+    // Surface AM-level failures (FAILED/KILLED/UNDEFINED 
FinalApplicationStatus, lost-AM-visibility on
+    // exhausted report fetches, or a never-launched application) as a 
non-zero launcher process exit so
+    // GGW/Grid Gateway dashboards see them end-to-end. SUCCEEDED keeps 
exitCode at the field's default of 0.
+    int finalExitCode = gobblinYarnAppLauncher.getExitCode();
+    LOGGER.info("GobblinYarnAppLauncher exiting with code {}", finalExitCode);
+    System.exit(finalExitCode);

Review Comment:
   `main()` calls `System.exit(...)` immediately after `launch()`, which will 
trigger the shutdown hook and stop/kill the YARN application right after 
submission (default `detach.on.exit.enabled=false`). It also prevents the 
report-monitor from ever observing the terminal status, so `exitCode` will 
almost always be 0.
   
   Instead, keep the launcher process alive until it has been stopped by the 
terminal-report path (or exit immediately on launch failure / when detaching), 
then exit with the computed `exitCode`.



##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/YarnService.java:
##########
@@ -262,17 +263,20 @@ protected void shutDown() throws IOException {
       if (!this.containerMap.isEmpty()) {
         synchronized (this.allContainersStopped) {
           try {
-            // Wait 5 minutes for the containers to stop
-            Duration waitTimeout = Duration.ofMinutes(5);
+            // Bounded fallback only: a missed notify must not stall the AM 
un-register. The normal path
+            // is woken immediately by notifyIfAllContainersStopped() as soon 
as the last container stops.
+            Duration waitTimeout = Duration.ofMinutes(2);
             this.allContainersStopped.wait(waitTimeout.toMillis());
-            LOGGER.info("All of the containers have been stopped");
+            LOGGER.info("Finished waiting for containers to stop ({} still 
tracked)", this.containerMap.size());
           } catch (InterruptedException ie) {
             Thread.currentThread().interrupt();
           }
         }
       }

Review Comment:
   `shutDown()` checks `containerMap.isEmpty()` outside the 
`allContainersStopped` monitor, then waits unconditionally once inside. If the 
last container is removed (and `notifyIfAllContainersStopped()` fires) between 
the outer emptiness check and the `wait(...)`, the notify can be missed and 
`shutDown()` will still sleep the full 2-minute timeout.
   
   Re-check the condition after acquiring the monitor (before waiting) so an 
already-empty map never waits.



##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/YarnService.java:
##########
@@ -286,6 +290,20 @@ protected void shutDown() throws IOException {
     }
   }
 
+  /**
+   * Derive the {@link FinalApplicationStatus} to report to YARN on 
un-register from the AM's captured Temporal
+   * workflow outcome (see {@link 
GobblinTemporalJobLauncher#getLastTerminalStatus()}). Reporting the true status
+   * — rather than an unconditional SUCCEEDED — lets a launcher polling the 
{@code ApplicationReport} observe job
+   * failures/cancellations end-to-end, and keeps the YARN status consistent 
with the AM JVM exit code. A
+   * {@code null} captured status (no workflow ran) maps to SUCCEEDED. 
Overridable for tests.
+   */
+  protected FinalApplicationStatus getFinalApplicationStatusForUnregister() {
+    io.temporal.api.enums.v1.WorkflowExecutionStatus captured = 
GobblinTemporalJobLauncher.getLastTerminalStatus();
+    FinalApplicationStatus derived = 
GobblinTemporalJobLauncher.mapWorkflowStatusToFinalAppStatus(captured);
+    LOGGER.info("Derived FinalApplicationStatus {} for un-register from 
captured workflow status {}", derived, captured);
+    return derived;
+  }

Review Comment:
   The Javadoc says a null captured workflow status maps to `SUCCEEDED`, but 
the implementation delegates to 
`GobblinTemporalJobLauncher.mapWorkflowStatusToFinalAppStatus(...)`, which maps 
null to `FAILED` (and tests assert that). The comment should match the actual 
behavior to avoid confusion for callers/maintainers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to