pratapaditya04 commented on code in PR #4197:
URL: https://github.com/apache/gobblin/pull/4197#discussion_r3347366348
##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/joblauncher/GobblinTemporalJobLauncher.java:
##########
@@ -97,9 +107,64 @@ public GobblinTemporalJobLauncher(Properties jobProps, Path
appWorkDir,
// non-null value indicates job has been submitted
this.workflowId = null;
+ // Reset the process-wide terminal-status cache for this fresh launcher.
In a real AM JVM only one
+ // launcher ever runs, but tests re-instantiate within the same JVM and
must not see leaked status.
+ setLastTerminalStatus(null);
startCancellationExecutor();
}
+ /**
+ * Query Temporal for the current execution status of {@link #workflowId}.
Returns
+ * {@code WORKFLOW_EXECUTION_STATUS_UNSPECIFIED} as a safe fallback if the
describe call fails, so callers
+ * downstream emit a JOB_FAILED rather than swallowing the GTE entirely.
+ */
+ private WorkflowExecutionStatus fetchWorkflowStatus() {
+ try {
+ WorkflowStub workflowStub =
this.client.newUntypedWorkflowStub(this.workflowId);
+ DescribeWorkflowExecutionRequest request =
DescribeWorkflowExecutionRequest.newBuilder()
+ .setNamespace(this.namespace)
+ .setExecution(workflowStub.getExecution())
+ .build();
+ DescribeWorkflowExecutionResponse response =
managedWorkflowServiceStubs.getWorkflowServiceStubs()
+ .blockingStub().describeWorkflowExecution(request);
+ return response.getWorkflowExecutionInfo().getStatus();
+ } catch (Exception e) {
+ log.warn("Failed to describe workflow {} for completion GTE; treating as
UNSPECIFIED (will emit JOB_FAILED)",
+ this.workflowId, e);
+ return WorkflowExecutionStatus.WORKFLOW_EXECUTION_STATUS_UNSPECIFIED;
+ }
Review Comment:
Update in `03e7bf9e0`: on a closer look, the Temporal service stubs are
already built with `RpcRetryOptions`
(`TemporalWorkflowClientFactory#buildRpcRetryOptions`, tunable via
`gobblin.temporal.rpc.retry.options.*`), so transient gRPC codes (`UNAVAILABLE`
/ `DEADLINE_EXCEEDED` / `RESOURCE_EXHAUSTED`) are already retried with backoff
at the stub layer **before** this `catch` is ever reached. I've therefore
dropped the manual retry loop + `Thread.sleep` I'd added in the previous commit
and now rely on that existing, configurable retry — the `catch` only handles
the genuinely-unreachable-after-retries fallback to `UNSPECIFIED`. Happy to add
describe-specific tuning if you'd prefer it over the shared stub options.
##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -970,6 +1090,15 @@ void signalGracefulShutdownAndWaitForTerminal() {
}
sendGracefulShutdownSignal(amContainerId.get());
pollForApplicationCompletionUntil(timeoutMs);
+ // Always kill so the RM cannot re-attempt the AM (no-op if the app
already finished). If the terminal
+ // outcome wasn't already dispatched by the report-monitor path, this is
a cancel/forced shutdown ->
+ // surface it as KILLED (the subclass emits JOB_CANCEL). The token, not
a racy report read, is the gate.
+ LOGGER.info("Graceful-shutdown wait complete; force-killing application
{} to prevent AM re-attempt", this.applicationId.get());
+ this.yarnClient.killApplication(this.applicationId.get());
+ if (this.terminalHandled.compareAndSet(false, true)) {
+ this.exitCode = 1;
+ onTerminalApplicationStatus(null, FinalApplicationStatus.KILLED);
+ }
Review Comment:
Follow-up in `03e7bf9e0`: per later feedback, when the re-fetched report
shows the app already terminal we now **skip** `killApplication` entirely and
just dispatch the real status; the force-kill runs only when the app is still
running (or its state couldn't be read).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]