bnspopi opened a new pull request, #77351:
URL: https://github.com/apache/airflow/pull/77351

   When a mapped task has already expanded and its producer is then re-run and 
returns an empty list, every existing index is marked `REMOVED` and nothing is 
left `SKIPPED`. A first run that maps to no values marks the placeholder 
`SKIPPED` instead, so downstream tasks behave differently after a clear, as 
#76445 lays out: an `all_success` downstream runs, and a 
`none_failed_min_one_success` downstream becomes `upstream_failed` and fails 
the run.
   
   This takes the option suggested in the issue:
   
   - `DagRun._revise_map_indexes_if_mapped` and 
`DagRun._check_for_removed_or_restored_tasks` now mark the lowest existing 
index `SKIPPED` and the others `REMOVED` when the map resolves to no values.
   - `none_failed_min_one_success` leaves removed upstreams out when it checks 
whether every upstream was skipped (`skipped and skipped == upstream - 
removed`). Without this the downstream still became `upstream_failed`, because 
the removed indexes were counted as upstreams that were not skipped. The 
`skipped and` guard keeps the all-removed case as it is today.
   
   This changes behaviour, as the issue notes: after such a re-run, 
`all_success` downstreams that run today are skipped. 
`test_calls_to_verify_integrity_with_mapped_task_zero_length_at_runtime` pinned 
the all-`REMOVED` result and is updated.
   
   Two things reviewers may want to weigh:
   
   - The trigger rule change is not limited to mapped upstreams. Any 
`none_failed_min_one_success` task whose upstreams are a mix of skipped and 
removed, with no success, is now skipped rather than `upstream_failed`.
   - The lowest index is marked `SKIPPED` whatever state it was in, the same 
way the other indexes are marked `REMOVED` whatever state they were in.
   
   Tests, run on Linux with SQLite:
   
   - New `test_mapped_task_mapped_to_zero_is_skipped_on_rerun_too` runs `size 
>> mapped.expand(arg=size.output) >> after` through the scheduling decisions 
for a first run and for a re-run, with `all_success`, 
`none_failed_min_one_success` and `all_done` on `after`, and checks that both 
end the same way and that `verify_integrity` afterwards does not turn the 
skipped instance into a removed one. On `main` the three re-run cases fail.
   - `test_dagrun.py`, `test_trigger_rule_dep.py`, `test_mappedoperator.py`, 
`test_taskinstance.py`, `test_scheduler_job.py`: 1653 passed, 21 skipped, 1 
xfailed, against 1647 passed on `main` with no failures on either.
   
   Not tested: PostgreSQL and MySQL, mapped task groups, and a real scheduler 
run.
   
   @sundeep8967 and @SYED-IRTEZA-HUSSAIN-RIZVI asked about this issue. I did 
not find a PR for it; I am happy to close this one if one of you already has it 
in progress.
   
   related: #76445
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (Claude Code)
   
   Generated-by: Claude Code following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to