Subham-KRLX opened a new issue, #77116:
URL: https://github.com/apache/airflow/issues/77116

   ### Under which category would you file this issue?
   
   Providers
   
   ### Apache Airflow version
   
   3.5.0 — tested from main, not a released installation.  Commit: 
dd39c346e693e5711467b98c4e77207e4f16493a
   
   ### What happened and how to reproduce it?
   
   A deferrable ExternalTaskSensor can keep waiting after one of its monitored 
upstream tasks has failed.
   
   With two upstream tasks, `allowed_states=["success"]` and 
`failed_states=["failed"]`:
   
   - One upstream task is failed and the other is successful.
   - The non-deferrable sensor raises `ExternalTaskFailedError`.
   - The deferrable trigger keeps polling instead of emitting a failed event.
   
   The same problem affects `skipped_states=["skipped"]` when one task is 
skipped and the other succeeds.
   
   Confirmed in Breeze using the actual sensor and trigger with mocked 
Execution API counts. This was not a full scheduler end-to-end test.
   
   ### Minimal reproduction
   
   Run this Python code inside Breeze. The mocked API returns one matching task 
for each queried state: one failed task and one successful task. The sleep is 
intercepted to stop after the first polling cycle.
   
   ```python
   import asyncio
   from unittest.mock import AsyncMock, patch
   
   from airflow.providers.standard.triggers.external_task import WorkflowTrigger
   
   
   class PolledAgain(Exception):
       pass
   
   
   async def main():
       trigger = WorkflowTrigger(
           external_dag_id="upstream",
           external_task_ids=["task_a", "task_b"],
           run_ids=["manual__repro"],
           allowed_states=["success"],
           failed_states=["failed"],
       )
   
       with (
           patch(
               
"airflow.sdk.execution_time.task_runner.RuntimeTaskInstance.get_ti_count",
               autospec=True,
               return_value=1,
           ),
           patch(
               
"airflow.providers.standard.triggers.external_task.asyncio.sleep",
               new=AsyncMock(side_effect=PolledAgain),
           ),
       ):
           events = trigger.run()
           try:
               print((await anext(events)).payload)
           except PolledAgain:
               print("Observed: trigger waited again instead of emitting 
failed")
           finally:
               await events.aclose()
   
   
   asyncio.run(main())
   ```
   
   Observed output:
   
   ```text
   Sleeping for 2.0 seconds
   Observed: trigger waited again instead of emitting failed
   ```
   
   Expected: a trigger event with `{"status": "failed"}`.
   
   ### What you think should happen instead?
   
   When any monitored task matches failed_states, the deferrable sensor should 
fail, matching the non-deferrable sensor. Likewise, any matching skipped state 
should produce a skipped event.
   
   WorkflowTrigger._get_count_af_3() currently calculates:
   
       int(count / len(self.external_task_ids))
   
   For one matching task out of two, this becomes int(1 / 2), which is 0. The 
trigger therefore misses the partial failure or skip and continues polling.
   
   Source:
   
https://github.com/apache/airflow/blob/dd39c346e693e5711467b98c4e77207e4f16493a/providers/standard/src/airflow/providers/standard/triggers/external_task.py#L145
   
   ### Operating System
   
   Debian GNU/Linux 13 inside a Breeze Docker container on macOS. Python 
3.11.18.
   
   ### Deployment
   
   Other
   
   ### Apache Airflow Provider(s)
   
   standard
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-standard==1.22.0
   Installed from the same main checkout.
   
   ### Official Helm Chart version
   
   _No response_
   
   ### Kubernetes Version
   
   _No response_
   
   ### Helm Chart configuration
   
   _No response_
   
   ### Docker Image customizations
   
   Breeze CI image built from the referenced checkout. No application source 
changes or additional image customizations.
   
   ### Anything else?
   
   The reproduction consistently showed:
   
   | Upstream states | Non-deferrable sensor | Deferrable trigger |
   |---|---|---|
   | failed, success | Fails | Keeps polling |
   | skipped, success | Skips | Keeps polling |
   | failed, skipped | Fails | Keeps polling |
   | failed, failed | Fails | Emits failed |
   | success, success | Succeeds | Emits success |
   | failed — single monitored task | Fails | Emits failed |
   
   Related older reports #41140 and #47298 concern states not being forwarded 
to the trigger. Here the states reach the trigger, but integer truncation 
discards a partial match.
   
   ---
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to