Subham-KRLX opened a new issue, #77116:
URL: https://github.com/apache/airflow/issues/77116
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.5.0 — tested from main, not a released installation. Commit:
dd39c346e693e5711467b98c4e77207e4f16493a
### What happened and how to reproduce it?
A deferrable ExternalTaskSensor can keep waiting after one of its monitored
upstream tasks has failed.
With two upstream tasks, `allowed_states=["success"]` and
`failed_states=["failed"]`:
- One upstream task is failed and the other is successful.
- The non-deferrable sensor raises `ExternalTaskFailedError`.
- The deferrable trigger keeps polling instead of emitting a failed event.
The same problem affects `skipped_states=["skipped"]` when one task is
skipped and the other succeeds.
Confirmed in Breeze using the actual sensor and trigger with mocked
Execution API counts. This was not a full scheduler end-to-end test.
### Minimal reproduction
Run this Python code inside Breeze. The mocked API returns one matching task
for each queried state: one failed task and one successful task. The sleep is
intercepted to stop after the first polling cycle.
```python
import asyncio
from unittest.mock import AsyncMock, patch
from airflow.providers.standard.triggers.external_task import WorkflowTrigger
class PolledAgain(Exception):
pass
async def main():
trigger = WorkflowTrigger(
external_dag_id="upstream",
external_task_ids=["task_a", "task_b"],
run_ids=["manual__repro"],
allowed_states=["success"],
failed_states=["failed"],
)
with (
patch(
"airflow.sdk.execution_time.task_runner.RuntimeTaskInstance.get_ti_count",
autospec=True,
return_value=1,
),
patch(
"airflow.providers.standard.triggers.external_task.asyncio.sleep",
new=AsyncMock(side_effect=PolledAgain),
),
):
events = trigger.run()
try:
print((await anext(events)).payload)
except PolledAgain:
print("Observed: trigger waited again instead of emitting
failed")
finally:
await events.aclose()
asyncio.run(main())
```
Observed output:
```text
Sleeping for 2.0 seconds
Observed: trigger waited again instead of emitting failed
```
Expected: a trigger event with `{"status": "failed"}`.
### What you think should happen instead?
When any monitored task matches failed_states, the deferrable sensor should
fail, matching the non-deferrable sensor. Likewise, any matching skipped state
should produce a skipped event.
WorkflowTrigger._get_count_af_3() currently calculates:
int(count / len(self.external_task_ids))
For one matching task out of two, this becomes int(1 / 2), which is 0. The
trigger therefore misses the partial failure or skip and continues polling.
Source:
https://github.com/apache/airflow/blob/dd39c346e693e5711467b98c4e77207e4f16493a/providers/standard/src/airflow/providers/standard/triggers/external_task.py#L145
### Operating System
Debian GNU/Linux 13 inside a Breeze Docker container on macOS. Python
3.11.18.
### Deployment
Other
### Apache Airflow Provider(s)
standard
### Versions of Apache Airflow Providers
apache-airflow-providers-standard==1.22.0
Installed from the same main checkout.
### Official Helm Chart version
_No response_
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
Breeze CI image built from the referenced checkout. No application source
changes or additional image customizations.
### Anything else?
The reproduction consistently showed:
| Upstream states | Non-deferrable sensor | Deferrable trigger |
|---|---|---|
| failed, success | Fails | Keeps polling |
| skipped, success | Skips | Keeps polling |
| failed, skipped | Fails | Keeps polling |
| failed, failed | Fails | Emits failed |
| success, success | Succeeds | Emits success |
| failed — single monitored task | Fails | Emits failed |
Related older reports #41140 and #47298 concern states not being forwarded
to the trigger. Here the states reach the trigger, but integer truncation
discards a partial match.
---
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]