JoelEmmanuelCloud commented on code in PR #76410:
URL: https://github.com/apache/airflow/pull/76410#discussion_r4644216861


##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -435,6 +454,44 @@ def __init__(
         self.executors: list[BaseExecutor] = executors if executors else 
ExecutorLoader.init_executors()
         self.executor: BaseExecutor = self.executors[0]
 
+        # Each executor's get_health() runs on its own daemon thread, never on 
the scheduler loop's own
+        # thread: a check that talks to a broker or a cluster control plane 
can be slow or hang, and
+        # must not be able to delay task scheduling. A pool isn't needed since 
each executor is only
+        # ever checked by one in-flight call at a time (see 
_check_executor_health); a daemon thread
+        # also means a hung check can't block process exit the way a 
ThreadPoolExecutor worker would.
+        self._executor_health_check_futures: dict[str, tuple[Future, float]] = 
{}
+        # A probe that ignores the timeout passed to it would otherwise occupy 
its slot forever: once
+        # a pending future has been sitting this many intervals, it's reported 
as stalled instead of
+        # silently skipped on every later tick (see _check_executor_health).
+        self._executor_health_check_stall_after_intervals = 3
+        self._stalled_health_checks: set[str] = set()
+
+        self._executor_health_check_interval = conf.getfloat(
+            "scheduler", "executor_health_check_interval", fallback=30.0
+        )
+        # Must be comfortably below the check interval so a probe that honours 
it always finishes
+        # before the next tick fires; half the interval is the starting point.
+        self._executor_health_check_timeout = 
self._executor_health_check_interval / 2
+
+        # Only executors that actually override get_health() get a Job row at 
all: one that was never
+        # going to report anything isn't a gap in coverage (see 
_configured_executor_keys in
+        # airflow_health.py, which applies this same rule from the 
/monitor/health side). An executor
+        # with no name is also skipped: _executor_health_key has nothing 
stable to key its row under.
+        # issubclass(type(executor), ...), not isinstance(executor, ...): a 
test double standing in
+        # for an executor (a MagicMock, even one speced to BaseExecutor) 
passes isinstance, since mock
+        # fakes that check, but calling a method on it never runs the real 
classmethod body, it just
+        # hands back another mock, which this dict would then read as 
"overrides it". type() bypasses
+        # that trick and correctly reports such a double as not a real 
BaseExecutor subclass.
+        from airflow.executors.base_executor import BaseExecutor as 
_BaseExecutor
+
+        self._executor_health_keys: dict[str, str | None] = {
+            key: executor.team_name
+            for executor in self.executors
+            if issubclass(type(executor), _BaseExecutor)

Review Comment:
   you're right, done. the dict now just calls executor.overrides_get_health() 
directly, and the mocks in tests set overrides_get_health.return_value = False.



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -3912,6 +4190,23 @@ def adopt_or_reset_orphaned_tasks(self, *, session: 
Session = NEW_SESSION) -> in
                         self.log.info("Marked %d SchedulerJob instances as 
failed", num_failed)
                         stats.incr(self.__class__.__name__.lower() + "_end", 
num_failed)
 
+                    # A separate statement, not folded into the SchedulerJob 
one above, so the
+                    # "Marked %d SchedulerJob instances as failed" log line 
and the _end metric keep
+                    # counting only schedulers. These ExecutorJob rows are 
left running with no
+                    # end_date when their owning scheduler is SIGKILLed or 
OOM-killed, since _execute's
+                    # own cleanup (_end_executor_jobs) never gets to run for 
that case; this also
+                    # catches a row a health check still in flight at that 
moment recreates after
+                    # _end_executor_jobs has already closed the others.
+                    session.execute(

Review Comment:
   added one, test_adopt_or_reset_orphaned_tasks_fails_stale_executor_jobs.



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -435,6 +454,44 @@ def __init__(
         self.executors: list[BaseExecutor] = executors if executors else 
ExecutorLoader.init_executors()
         self.executor: BaseExecutor = self.executors[0]
 
+        # Each executor's get_health() runs on its own daemon thread, never on 
the scheduler loop's own

Review Comment:
   trimmed them down.



##########
airflow-core/src/airflow/api/common/airflow_health.py:
##########
@@ -54,13 +58,15 @@ class DetailedHealthStatus(str, Enum):
 
 
 @provide_session
-def get_jobs_health(job_runner_class, *, session: Session = NEW_SESSION) -> 
list[Job]:
+def get_jobs_health(job_runner_class: type | str, *, session: Session = 
NEW_SESSION) -> list[Job]:
     """
-    Return live unfinished jobs and the latest unfinished job, ordered by 
heartbeat.
+    Return live unfinished jobs and the latest unfinished job of the given job 
type, ordered by heartbeat.
 
-    Keep the latest job even when dead so health responses retain its 
heartbeat.
+    Keep the latest job even when dead so health responses retain its 
heartbeat. ``job_runner_class`` is
+    a job runner class that reports its own ``job_type`` (e.g. 
``SchedulerJobRunner``), or a bare
+    job_type string for a job with no runner class of its own, like the 
executor's ``EXECUTOR_JOB_TYPE``.
     """
-    job_type = job_runner_class.job_type
+    job_type = getattr(job_runner_class, "job_type", job_runner_class)

Review Comment:
   done, simpler this way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to