rmittal46 opened a new issue, #77256:
URL: https://github.com/apache/airflow/issues/77256

   ### Description
   
   When a Dag is scheduled on a conditional asset expression that nests 
`AssetAll` inside `AssetAny`, e.g.
   
   ```python
   from airflow.sdk import DAG, Asset, AssetAll, AssetAny
   
   with DAG(
       dag_id="consumer",
       schedule=AssetAny(
           AssetAll(Asset("a"), Asset("b")),
           AssetAll(Asset("c"), Asset("d")),
       ),
   ):
       ...
   ```
   
   the scheduler clears **all** `asset_dag_run_queue` rows for the Dag as soon 
as **any** branch is satisfied. Progress towards the branch that did not fire 
is lost.
   
   **Steps to reproduce**
   
   1. Emit an event for `a` (branch `a & b` is now half satisfied; queue = {a}).
   2. Emit events for `c` and `d` → branch `c & d` is satisfied, a Dag run is 
created.
   3. The queue is now empty. The pending `a` is gone (it is attached to the 
run created in step 2 as a triggering event).
   4. Emit an event for `b` → nothing happens. The `a & b` branch needs a 
**new** `a` event, even though `a` was updated after the last run that `a & b` 
actually satisfied.
   
   **Current behaviour:** queued events are cleared per Dag, not per condition 
branch.
   
   **Expected / desired behaviour:** an option to track queued events per 
branch of the condition, so that firing one `AssetAny` branch consumes only the 
events of that branch. Unrelated, partially satisfied `AssetAll` branches keep 
their queued events and can complete later.
   
   ### Use case/motivation
   
   Data platforms commonly model "rebuild X when (source A **and** source B) 
have both refreshed, **or** when (source C **and** source D) have both 
refreshed". Each `AND` group represents a consistent pair of inputs that must 
land together.
   
   With the current behaviour, the `AND` semantics become unreliable as soon as 
another branch fires. The only workaround today is to create an intermediate 
"gate" Dag per `AssetAll` branch, each publishing its own asset, and schedule 
the consumer on `AssetAny(gate_1, gate_2)`. That works, because each Dag has 
its own queue, but it:
   
   - adds extra Dags purely for scheduling bookkeeping
   - makes the asset graph harder to read
   - has to be applied by every user who discovers the behaviour (often only in 
production)
   
   ### Possible approaches
   
   1. **Per-branch consumption (opt-in):** when a branch evaluates true, delete 
only the queue rows for assets referenced by that branch. Rows for other assets 
remain. Could be gated by a Dag-level flag or a parameter on `AssetAny` (e.g. 
`AssetAny(..., consume="branch")`) to keep backward compatibility.
   2. **Documentation:** at minimum, document clearly in the conditional asset 
scheduling docs that the whole queue is cleared when any condition is 
satisfied, and describe the gate-Dag workaround.
   
   Open questions for maintainers:
   - With per-branch consumption, should events consumed by a branch still 
appear as triggering events on runs created by other branches? Probably not, to 
keep the lineage accurate.
   - How should this interact with `max_active_runs` and event coalescing when 
several branches become satisfied in the same scheduler loop?
   
   ### Related issues
   
   - #56050 Most recent Asset events ignored when max_active_runs = 1
   - #55956 Asset event scheduling - introduce max_asset_events parameter
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [I] Conditional asset schedule... rmittal46 (via GitHub) via commits

Reply via email to