GitHub user nord94 created a discussion: Updated datasets for skipped task
Hi comrades! **Problem** A task that declares outlets=[Dataset(...)] only emits a dataset event when it ends in success. If the task ends in skipped, no event is produced, and every consumer DAG scheduled on that dataset stays blocked waiting for an update that will never arrive. This blocks us in a specific case: our ingestion DAGs skip work for sources that haven't changed. A skip there is a normal, expected outcome — the data is current, there is simply nothing new to load. But the downstream datamart DAG can't tell "nothing to ingest" apart from "ingestion hasn't happened yet", so it never triggers, and rarely-updated sources hold the whole datamart hostage. **Two skip paths, both affected** Worker-side skip — the producer task runs and raises AirflowSkipException. Scheduler cascade skip — the producer task never runs at all, because an upstream branch wasn't taken (BranchPythonOperator / ShortCircuitOperator) or its trigger rule wasn't satisfied, and the scheduler marks it skipped directly. **Expected behaviour** When a producer DAG run reaches a terminal state, its declared datasets should be marked updated whether the producing task ended success or skipped. Failures should keep the current behaviour — no event. **Example** dag_a is scheduled on datasets B, C, D, produced by task_b in dag_b, task_c in dag_c, task_d in dag_d respectively. Today: if dag_b runs and task_b skips, B gets no event and dag_a never triggers, even though C and D updated normally. Wanted: once dag_b, dag_c and dag_d have all run, B, C and D are all marked updated and dag_a triggers. GitHub link: https://github.com/apache/airflow/discussions/72639 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
