uranusjr commented on code in PR #68048:
URL: https://github.com/apache/airflow/pull/68048#discussion_r3510643084
##########
providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -1230,17 +1264,46 @@ def _poll_k8s_driver_via_api(self) -> str | None:
consecutive_pending * poll_interval,
)
else:
- consecutive_pending = 0
-
- if phase == "Unknown":
- consecutive_unknown += 1
- if consecutive_unknown >= max_consecutive_unknown:
+ phase = pod.status.phase or "Initializing"
+ self.log.info("Application status for %s (phase: %s)",
app_id, phase)
+ if phase == "Succeeded":
+ break
+ if phase == "Failed":
+ container_state = ""
+ if pod.status.container_statuses:
+ cs = pod.status.container_statuses[0]
+ if cs.state and cs.state.terminated:
+ container_state = f"
exit_code={cs.state.terminated.exit_code} reason={cs.state.terminated.reason}"
raise RuntimeError(
- f"Spark application {app_id} reported Unknown
phase "
- f"{consecutive_unknown} times consecutively;
giving up."
+ f"Spark application {app_id} failed
(phase=Failed{container_state})"
)
- else:
- consecutive_unknown = 0
+ if phase == "Pending":
Review Comment:
If I read it correctly, this is never reachable? Line 1222/1256 has `if
phase == "Pending"`, and 1231/1266 is `else:`, which block this is in, so
`phase` can never be `"Pending"` at this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
