jason810496 opened a new pull request, #69688:
URL: https://github.com/apache/airflow/pull/69688
- related: #69520 (discussion on dropping vs. fixing the LogTemplate flow)
## Why
The `LogTemplate` model pins the log id template to each Dag run so task
logs written before a `log_id_template` config change stay readable, but the
Airflow 3 paths of the Elasticsearch and OpenSearch handlers render `log_id`
from the current conf value on both the write (worker) and read (API server)
sides — so changing the config orphans every previously written log document.
## How
- Workers cannot query the metadata DB, so the Dag-run-pinned
`LogTemplate.elasticsearch_id` is delivered to the supervisor via a new
optional `TIRunContext.log_id_template` field, populated by the `ti_run`
Execution API endpoint.
- The API-server read paths (handler `_read`, `get_external_log_url`,
`RemoteLogIO.read`) restore the per-run lookup directly (they have DB access),
falling back to conf when the DB is unreachable or the row is missing — same
pattern the legacy `OsTaskHandler` already used.
- `upload_to_remote` forwards the run context only to handlers whose
`upload` opts into a `ti_context` keyword, so S3/GCS/third-party `RemoteLogIO`
implementations and old-provider × new-sdk combinations are untouched.
- The renderers now always supply the full placeholder set (`logical_date`,
`execution_date`, `data_interval_start/end`) because templates pinned by older
rows (e.g. the pre-2.3 default
`{dag_id}-{task_id}-{logical_date}-{try_number}`) use them and `str.format`
raises `KeyError` on missing names.
## What
- `airflow-core`: add `TIRunContext.log_id_template`; populate it in the
`ti_run` route from `dr.get_log_template()`; add Execution API version
`2026-09-30` with a Cadwyn migration stripping the field for older clients.
- `task-sdk`: retain the `TIRunContext` on `ActivitySubprocess` and pass it
to `upload_to_remote`; regenerate the client datamodels; add the first
in-progress supervisor-schema version file (`schema/versions/v2026_09_30.py`)
plus the regenerated `schema.json` snapshot.
- `providers/elasticsearch` + `providers/opensearch`: pinned-template
resolution on the read paths, `ti_context` opt-in on `RemoteLogIO.upload`
(covers both the `write_stdout`/filebeat and direct-write branches), date
placeholders in the module-level `_render_log_id`.
- Tests: `ti_run` payload/pinning tests, version-negotiation tests for
`2026-09-30`, supervisor retention/forwarding tests, `upload_to_remote`
signature-dispatch tests, and ES/OS pinned-write/pinned-read tests.
Notes for reviewers:
- The `2026-09-30` version date (both bundles) is a placeholder guess at the
3.4.0 release date — happy to rename. The supervisor-schema `versions/`
directory had no in-progress file yet, so this PR creates the first one.
- Deferred follow-ups: the `log_path`/filename write path
(`ExecuteTask.make` still renders from conf), and trigger-log/callback uploads
(no `ti_context`, conf fallback — identical to today's behavior).
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes, with help of Claude Code (Fable 5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
