ashb opened a new pull request, #74868:
URL: https://github.com/apache/airflow/pull/74868
* Resolve a trigger's XCom requests as its deferred task instance
Note: _None_ of this is a security boundary, and should not be treated as
such.
The Execution API resolves XCom reads and writes for a task inside a loop
to the caller's own iteration, using the task identity in the caller's JWT.
Triggers have no such identity: the triggerer talks to an in-process API
that treats every request as the all-zero task instance. A trigger's XCom
requests therefore resolve as an anonymous caller rather than as the
deferred task it runs for, so it can neither read (a 400) nor write the
XComs of a task inside a loop.
Triggers can also share one runner coroutine and one comms channel (the
"shared streams" idea), so only the asyncio task running a trigger's code
knows which trigger it is; a context variable carries that trigger's task
instance id to each request frame. Shared-stream groups are started from
whichever trigger subscribes first but serve several triggers, so they start
with it cleared rather than inheriting that trigger's id. This isn't ideal,
but will do "for now".
The trigger runner process (the one running async code) sets the task
instance id, and since that process runs user code a trigger can claim to be
any task instance. The (in-process) API therefore checks the claim. A write
acts as the task instance only when it is deferred and its trigger is
already assigned to the calling triggerer. A read acts as it whenever that
try is still live, because after a user marks a deferred task failed or
successful the trigger row is gone and the triggerer still runs the
trigger's on_kill(). The most a misbehaving trigger can do is write as
another task that is deferred on the same triggerer, or read as any live
task, and it can already read and write every task's XComs by coordinates
through the in-process API. As stated, this is _not_ a security boundary we
can rely on
Requests that fail the check are rejected rather than falling back to the
anonymous identity, deliberately: a trigger whose task instance is no
longer deferred, or whose trigger was reassigned, should not still be
writing its task's XComs, and falling back would only work for tasks
outside loops. Clearing a deferred task archives its try before on_kill()
runs, so on_kill() after a clear cannot read its task's XComs; a trigger
that needs a job id there should keep it on the trigger, as the in-tree
triggers already do.
Only the in-process API that the triggerer creates knows which triggerer
is calling, so every other API rejects the header rather than trusting it.
A triggerer that eventually talks to a real API server will need its own
credential before this can work there.
Only the XCom routes honour the trigger's task instance id for now.
get_previous_task_instance also resolves relative to the caller, so a
trigger asking for its previous task instance still gets the latest pass
rather than the one at its own loop position. Every other request from the
triggerer keeps its current identity, so access rules and the live-attempt
check on those routes are unchanged.
* fixup! Resolve a trigger's XCom requests as its deferred task instance
Pin the HEAD and DELETE rules in the trigger claim check and assert which
check denied each 403, so loosening the read-method test no longer passes.
* fixup! Resolve a trigger's XCom requests as its deferred task instance
Ignore a trigger failure reported by a triggerer that no longer owns the
trigger. The XCom write check now answers 403 to a stale copy on its
previous owner, which raises before yielding and would otherwise have
Trigger.submit_failure send the task to __fail__ while the new owner is
still running it.
(cherry picked from commit 61f1d4b85e)
Adapted to v3-4-test's CurrentTIToken: the typed execution tokens from the
Dag processor scoped access work (#74370) are not on this branch.
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
