ramitkataria commented on code in PR #76303:
URL: https://github.com/apache/airflow/pull/76303#discussion_r4622721345
##########
airflow-core/src/airflow/models/trigger.py:
##########
@@ -908,17 +908,19 @@ def _(event: BaseTaskEndEvent, *, task_instance:
TaskInstance, session: Session)
callback_type = event.task_instance_state
should_retry = False
dag_bag = DBDagBag()
+ dag = None
- if event.task_instance_state == TaskInstanceState.FAILED:
- # Load the serialized task so retry eligibility matches the normal
task path.
+ if event.task_instance_state in (TaskInstanceState.SUCCESS,
TaskInstanceState.FAILED):
Review Comment:
This now loads and deserializes the Dag for every trigger-ended SUCCESS too;
before it was FAILED only. As far as I can tell the success request is dropped
either way: the sink skips it with routing on, and `_execute_task_callbacks`
returns early for non-failure requests with routing off.
This runs with the flag off as well. Could we keep the load to FAILED? The
`relative_fileloc` fallback below already covers the case where `dag` is `None`.
##########
task-sdk/src/airflow/sdk/api/client.py:
##########
@@ -1204,11 +1208,33 @@ def __init__(self, client: Client):
def run(self, callback_id: uuid.UUID) -> None:
"""Exchange the single-use callback token for an execution token."""
- self.client.patch(f"callbacks/{callback_id}/run")
+ # Retries of this request resend the same attempt ID, so the API
server accepts a retry whose
+ # first response was lost, but not a second delivery of the callback.
+ body = CallbackRunPayload(hostname=get_hostname(), pid=os.getpid(),
attempt_id=uuid.uuid4())
+ try:
+ self.client.patch(f"callbacks/{callback_id}/run",
content=body.model_dump_json())
+ except ServerResponseError as e:
+ if e.response.status_code == HTTPStatus.CONFLICT:
Review Comment:
Dropping a redelivered callback quietly makes sense when the first worker is
still alive. I think there's a gap when it died after the token exchange: the
redelivery is ignored on Celery, and the reaper only looks at `dag_attribute`
rows, so a deadline (`import_path`) callback stays RUNNING and holds its
executor slot until the scheduler restarts.
Before this change the redelivery was recorded as a failure, which settled
the row. This applies with routing off too. Could the reaper's RUNNING check
cover deadline callbacks as well?
##########
task-sdk/src/airflow/sdk/execution_time/callback_supervisor.py:
##########
@@ -534,22 +707,27 @@ def supervise_callback(
exit_code=exit_code,
duration=end - start,
)
- if exit_code == 0:
- terminal_state = CallbackTerminalState.SUCCESS
- else:
+ if process.timed_out:
+ stats.incr("callback.timeouts")
+ terminal_output = (
+ f"Callback timed out after
{process._get_callback_execution_timeout()} seconds"
+ )
+ elif exit_code != 0:
terminal_output = f"Callback subprocess exited with code
{exit_code}"
+ else:
+ terminal_state = CallbackTerminalState.SUCCESS
except Exception as e:
terminal_output = f"Callback supervisor error: {type(e).__name__}:
{e}"
raise
finally:
- try:
- client.callbacks.finish(UUID(id), state=terminal_state,
output=terminal_output)
- except Exception:
- log.exception("Failed to report final callback state",
workload_id=id)
+ _report_callback_state(client, id, terminal_state, terminal_output)
+ # Report first so a slow log upload cannot use up the scheduler's
timeout grace period.
+ if process is not None:
+ process._upload_logs()
if log_path and log_file_descriptor:
log_file_descriptor.close()
- if exit_code != 0:
- raise RuntimeError(f"Callback subprocess exited with code {exit_code}")
+ if terminal_state == CallbackTerminalState.FAILED:
+ raise RuntimeError(terminal_output)
Review Comment:
Potentially out of scope but a failed callback is reported as FAILED and
then we raise here, so the process exits non-zero. A failed task exits 0
(`supervise_task` returns its exit code), and the ECS and Batch executors rely
on that: they read a non-zero exit as a failed launch and resubmit. So a failed
callback is relaunched until the attempts run out, with each relaunch getting a
409 from `/run`.
What if we exit cleanly here like tasks do? Or do you have other ideas? I
can help with a followup PR as well about this
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1481,10 +1565,444 @@ def _enqueue_executor_callbacks(self, session:
Session) -> None:
generator=executor.jwt_generator,
)
- executor.queue_workload(workload, session=session)
- callback.state = CallbackState.QUEUED
+ try:
+ executor.queue_workload(workload, session=session)
+ except NotImplementedError as e:
+ # Some executors run callbacks without declaring it, so
ask the executor itself.
+ self.log.error("Failing callback %s: %s", callback.id, e)
+ callback.state = CallbackState.FAILED
+ callback.output = str(e)
+ else:
+ callback.state = CallbackState.QUEUED
session.add(callback)
+ def _route_dag_processor_callbacks(self, *, session: Session) -> None:
+ """
+ Queue Dag processor callbacks on executors.
+
+ With routing on, callback sinks store Dag processor callbacks as
executor callbacks waiting
+ SCHEDULED (see ``ExecutorCallback.from_dag_processor_request``). A
routed callback is marked
+ QUEUED in the transaction that queues it. One that no executor can run
is failed with the
+ reason, and one that fails to route is tried again in a later loop, up
to
+ ``MAX_ROUTED_CALLBACK_TRIES``.
+
+ Each scheduler admits callbacks per loop into at most half its free
logical slots and half
+ each of its executor instances' free slots, rounded down, with at
least one when a slot is
+ free. This leaves local capacity for tasks when more than one slot is
free; it does not
+ reserve shared workers or limit callback concurrency across
schedulers. Executors send
+ callbacks first, and one left waiting in an executor for a slot would
age toward its start
+ deadline. Callbacks that could be routed but find no free slot wait
for a later loop, while
+ those no executor can run are failed without waiting for one.
+ """
+ free_slots = self._parallelism - sum(executor.slots_occupied for
executor in self.executors)
+ budget = min(self._max_routed_callbacks_per_loop, max(1, free_slots //
2)) if free_slots > 0 else 0
+ executor_budgets = {
+ executor: max(1, executor.slots_available // 2) if
executor.slots_available > 0 else 0
+ for executor in self.executors
+ }
+ now = timezone.utcnow()
+ examined: set[UUID] = set()
+ if budget > 0:
+ # Oldest first, so callbacks are routed in the order they were
requested.
+ oldest = self._select_waiting_callbacks(session=session)
+ for callback in oldest:
+ examined.add(callback.id)
+ if executor := self._settle_waiting_callback(
+ callback,
+ now=now,
+ can_queue=budget > 0,
+ executor_budgets=executor_budgets,
+ session=session,
+ ):
+ budget -= 1
+ executor_budgets[executor] -= 1
+ if len(oldest) < self._max_routed_callbacks_per_loop:
+ # That was every waiting callback no other scheduler is
locking.
+ return
+
+ # After a full oldest window, or none for lack of a free slot, the
next window is checked too,
+ # resuming after the last one checked, so callbacks waiting for a slot
don't keep the rest from
+ # being routed or failed.
+ swept = self._select_waiting_callbacks(
+ session=session, after=self._callback_sweep_cursor,
exclude=examined
+ )
+ self._callback_sweep_cursor = (
+ swept[-1].id if len(swept) == self._max_routed_callbacks_per_loop
else None
+ )
+ for callback in swept:
+ if executor := self._settle_waiting_callback(
+ callback,
+ now=now,
+ can_queue=budget > 0,
+ executor_budgets=executor_budgets,
+ session=session,
+ ):
+ budget -= 1
+ executor_budgets[executor] -= 1
+
+ def _select_waiting_callbacks(
+ self, *, session: Session, after: UUID | None = None, exclude:
set[UUID] | None = None
+ ) -> Sequence[ExecutorCallback]:
+ query = (
+ select(ExecutorCallback)
+ .where(
+ ExecutorCallback.fetch_method ==
CallbackFetchMethod.DAG_ATTRIBUTE,
+ ExecutorCallback.state == CallbackState.SCHEDULED,
+ )
+ .order_by(ExecutorCallback.id)
+ .limit(self._max_routed_callbacks_per_loop)
+ )
+ if after is not None:
+ query = query.where(ExecutorCallback.id > after)
+ if exclude:
+ query = query.where(ExecutorCallback.id.not_in(exclude))
+ return session.scalars(with_row_locks(query, session,
of=ExecutorCallback, skip_locked=True)).all()
+
+ def _settle_waiting_callback(
+ self,
+ callback: ExecutorCallback,
+ *,
+ now: datetime,
+ can_queue: bool,
+ executor_budgets: dict[BaseExecutor, int],
+ session: Session,
+ ) -> BaseExecutor | None:
+ """
+ Route a waiting callback, fail it if no executor can run it, or leave
it waiting.
+
+ A callback left waiting is only looked up again once its executor has
a free slot: waiting costs
+ no queries, and the callback is queued by its current Dag run and team.
+
+ :return: The executor it was queued on, or ``None`` if it wasn't
queued.
+ """
+
+ def has_free_slot(executor: BaseExecutor) -> bool:
+ return can_queue and executor_budgets[executor] > 0 and
executor.slots_available > 0
+
+ try:
+ executor = self._waiting_callback_executors.pop(callback.id, None)
+ if executor is not None and not has_free_slot(executor):
+ self._waiting_callback_executors[callback.id] = executor
+ return None
+ with session.begin_nested():
+ target = self._get_callback_target(callback, session=session)
+ if has_free_slot(target.executor):
+ self._queue_routed_callback(callback, target, now=now,
session=session)
+ return target.executor
+ self._waiting_callback_executors[callback.id] = target.executor
+ except _UnroutableCallbackError as e:
+ self.log.warning("Failing callback %s, which no executor can run:
%s", callback.id, e)
+ callback.state = CallbackState.FAILED
+ callback.output = str(e)
+ except Exception as e:
+ self.log.exception("Failed to route callback %s", callback.id)
+ try_number = callback.data.get(TRY_NUMBER_KEY, 1)
+ if try_number < MAX_ROUTED_CALLBACK_TRIES:
+ callback.data = {**callback.data, TRY_NUMBER_KEY: try_number +
1}
+ else:
+ callback.state = CallbackState.FAILED
+ callback.output = f"Could not be routed in {try_number} tries:
{e}"
+ return None
+
+ def _get_callback_target(self, callback: ExecutorCallback, *, session:
Session) -> _CallbackTarget:
+ """
+ Return where a waiting Dag processor callback runs.
+
+ :raises _UnroutableCallbackError: If no executor can run it.
+ """
+ request = callback.get_callback_request()
+ if request is None:
+ raise _UnroutableCallbackError("It stores no callback request")
+ if get_claiming_importer(request.filepath, request.bundle_name) is not
None:
+ raise _UnroutableCallbackError("Callbacks of Lang-SDK Dags are not
supported")
+
+ executor_name: str | None = None
+ queue: str | None = None
+ executor_config: dict | None = None
+ if isinstance(request, DagCallbackRequest):
Review Comment:
Dag callbacks always go to the default executor and queue, so those workers
need to be able to import every Dag file that has callbacks (for example a Dag
whose tasks run on a queue with extra dependencies). If the parse fails on the
worker, the callback ends up FAILED with no retry. Could the docs call that out
as a requirement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]