ChickenBenny opened a new pull request, #76359:
URL: https://github.com/apache/airflow/pull/76359

   * related: #75387
   * related: #74717
   * related: #76103 (TypeScript SDK)
   
   ---
   
   Follow-up 2 of #75387.
   
   After #75387 the supervisor starts the `execution_timeout` clock for a 
language SDK task as soon as the runtime has the task, using 
`TIRunContext.execution_timeout`. A Python task has no clock until it sends 
`SetExecutionTimeout` right before it runs. The Go runtime never sent it, which 
causes two problems:
   
   1. A Go task that sets no `execution_timeout` doesn't get the worker's 
`[core] default_task_execution_timeout`. Right now it gets the Dag processor's 
default through the serialized Dag, but a follow-up will stop storing that 
default there, and without this change the task would then have no timeout at 
all.
   2. The clock starts a bit earlier than for a Python task.
   
   ### Changes
   
   - When `ti_context.execution_timeout` is unset, the coordinator fills it in 
with the worker's `[core] default_task_execution_timeout`, the same value a 
Python task gets when the worker parses its Dag. It starts the supervisor's 
clock with that value and sends it to the runtime in `StartupDetails`. A `0` 
means no timeout, so it doesn't fall back. Each runtime gets the default 
without reading any config itself: the TypeScript runtime, and a runtime that 
never sends `SetExecutionTimeout`, get it too.
   - Right before the task runs (after XComs are cleared), the Go runtime reads 
the timeout from `ti_context` and does what the Python task runner does:
     - not set, or `0`: sends nothing (no timeout).
     - positive: sends `SetExecutionTimeout`, which restarts the supervisor's 
clock from that point.
     - negative: fails the task without running it.
   
     A failed task retries if `should_retry` is set.
   - The Go runtime still doesn't stop a timed-out task by itself. The 
supervisor sends SIGTERM after the timeout plus `killed_task_cleanup_time`, 
same as before. Stopping the task at the timeout is the Go part of #74717 and 
will be a separate PR.
   - Separate commit: the `iface*` helpers move from `messages.go` to 
`iface.go`, and `ifaceInt` and the new `ifaceFloat` share a generic 
`ifaceNumber`. No behavior change.
   
   ### What's next
   
   A follow-up in core and the standard provider, opened as a draft and merged 
after this PR and #76103, will change what the serialized Dag stores for a stub 
task's `execution_timeout`:
   
   | Stub task sets | Stored | Timeout used |
   |---|---|---|
   | nothing (`default_args` counts as set) | nothing | the worker default, 
filled in by the coordinator |
   | `None` | `0` | none |
   | a `timedelta` | its seconds (same as today) | that value |
   
   It will also drop `execution_timeout` from `client_defaults`. Until then, a 
stub task with `None` is treated the same as one that sets nothing. That 
follow-up has to go in last because it removes the default that language SDK 
tasks get today.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to