[GitHub] [airflow] jonasmiederer opened a new issue #10686: Common tasks in downstream of multiple branches always set to skipped due to 'Not Previously Skipped' dependency

[GitBox]

jonasmiederer opened a new issue #10686:
URL: https://github.com/apache/airflow/issues/10686


   **Please note**: This issue is very similar to #7885, but I think it caused 
by the new functionality [Clearing tasks skipped by SkipMixin will skip 
them](https://github.com/apache/airflow/blob/master/UPDATING.md#clearing-tasks-skipped-by-skipmixin-will-skip-them)
 introduced in 1.10.12.
   The problem is the same, but the solution (setting the trigger rule of the 
joined task to `none_failed` as described in the docs) stopped working after 
updating to 1.10.12
   
   **Apache Airflow version**: 1.10.12
   
   **What happened**:
   Two branches join into the same task at one point of their downstream. This 
joint task will set to be skipped due to one of the two branches being skipped.
   Setting the trigger rule of the joined task to `none_failed` used to work 
(the task is executed after the chosen branch was executed), but after updating 
to 1.10.12 the task is always skipped.
   
   <img width="513" alt="Screenshot 2020-09-02 at 09 52 25" 
src="https://user-images.githubusercontent.com/7305005/91954387-05213000-ed02-11ea-9c4d-4f63713a674d.png";>
   
   If the join task was selected, everything is fine:
   <img width="556" alt="Screenshot 2020-09-02 at 09 33 13" 
src="https://user-images.githubusercontent.com/7305005/91954454-1c601d80-ed02-11ea-91d0-0e139499a2b7.png";>
   
   
   **What you expected to happen**:
   The joint task should be executed, although not chosen by the branch 
operator, because it is a downstream task of the chosen branch.
   
   **What do you think went wrong?**:
   If I look into the task instance details, I can see that the task was 
skipped "_because of previous XCom result from parent task branching_", listed 
as the "Not Previously Skipped" dependency. 
   <img width="1336" alt="Screenshot 2020-09-02 at 09 46 24" 
src="https://user-images.githubusercontent.com/7305005/91952324-2d5c5f00-ed01-11ea-90a7-64c463c2300c.png";>
   
   I think that is related to the new functionality [Clearing tasks skipped by 
SkipMixin will skip 
them](https://github.com/apache/airflow/blob/master/UPDATING.md#clearing-tasks-skipped-by-skipmixin-will-skip-them)
 introduced in 1.10.12, but I'm not sure whether this is a bug or I did 
something wrong.
   
   
   **How to reproduce it**:
   
   ```python
   import random
   
   from airflow.models import DAG
   from airflow.operators.dummy_operator import DummyOperator
   from airflow.operators.python_operator import BranchPythonOperator
   from airflow.utils.dates import days_ago
   
   args = {
       'owner': 'Airflow',
       'start_date': days_ago(2),
   }
   
   dag = DAG(
       dag_id='branch_test',
       default_args=args,
       schedule_interval="@daily",
       tags=['example']
   )
   
   run_this_first = DummyOperator(
       task_id='run_this_first',
       dag=dag,
   )
   
   options = ['branch_a']
   
   branching = BranchPythonOperator(
       task_id='branching',
       python_callable=lambda: random.choice(options),
       dag=dag,
   )
   run_this_first >> branching
   
   join = DummyOperator(
       task_id='join',
       trigger_rule='none_failed',
       dag=dag,
   )
   
   for option in options:
       t = DummyOperator(
           task_id=option,
           dag=dag,
       )
   
       dummy_follow = DummyOperator(
           task_id='follow_' + option,
           dag=dag,
       )
   
       branching >> t >> dummy_follow >> join
   
   options.append('join')
   branching >> join
   ```
   
   Has something changed in the functionality how downstream tasks are skipped 
or is this a bug in the new release?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org