[jira] [Assigned] (AIRFLOW-4355) Externally triggered DAG is marked as 'success' even if a task has been 'removed'!
[
https://issues.apache.org/jira/browse/AIRFLOW-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
junhao mai reassigned AIRFLOW-4355:
---
Assignee: junhao mai (was: t oo)
> Externally triggered DAG is marked as 'success' even if a task has been
> 'removed'!
> --
>
> Key: AIRFLOW-4355
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4355
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, DagRun, scheduler
>Affects Versions: 1.10.3
>Reporter: t oo
>Assignee: junhao mai
>Priority: Blocker
> Labels: dynamic
> Fix For: 2.0.0
>
> Attachments: dag_success_even_if_task_removed.png, treeview.png
>
>
> note: all my dags are purely externally triggered
> *Issue:* Dag has 5 parallel tasks that ran successfully and 1 final task that
> somehow got 'removed' state (prior dag runs had 'failed' state) and never ran
> successfully but still the DAG is showing success!
>
> *Command ran* (note that previous commands like airflow trigger_dag -e
> 20190412 qsr_coremytbl were run before and failed for valid reason (ie python
> task failing) ):
> airflow trigger_dag -e 20190412T08:00 qsr_coremytbl --conf '\{"hourstr":"08"}'
>
> *some logs on prior instance of airflow (ec2 was autohealed):*
> [2019-04-18 08:29:40,678] \{logging_mixin.py:95} INFO - [2019-04-18
> 08:29:40,678] {__init__.py:4897} WARNING - Failed to get task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00 [None]>'
> for dag ''. Marking it as removed.
> [2019-04-18 08:29:43,582] \{logging_mixin.py:95} INFO - [2019-04-18
> 08:29:43,582] {__init__.py:4906} INFO - Restoring task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00 [removed]>'
> which was previously removed from DAG ''
> [2019-04-18 08:29:43,618] \{jobs.py:1787} INFO - Creating / updating
> 08:00:00+00:00 [scheduled]> in ORM
> [2019-04-18 08:29:43,676] \{logging_mixin.py:95} INFO - [2019-04-18
> 08:29:43,676] {__init__.py:4897} WARNING - Failed to get task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00
> [scheduled]>' for dag ''. Marking it as removed.
>
> *some logs on newer ec2:*
> [myuser@host logs]$ grep -i hive -R * | sed 's#[0-9]#x#g' | sort | uniq -c |
> grep -v 'airflow-webserver-access.log'
> 2 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl log -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
> 1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl log -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl'), ('format',
> u'json')]
> 1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl rendered -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
> 1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl task -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
> 1 scheduler/latest/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{jobs.py:} INFO - Creating / updating qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl -xx-xx xx:xx:xx+xx:xx
> [scheduled]> in ORM
> 71 scheduler/latest/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{logging_mixin.py:xx} INFO - [-xx-xx xx:xx:xx,xxx] {__init__.py:}
> INFO - Restoring task ' -xx-xx xx:xx:xx+xx:xx [removed]>' which was previously removed from DAG
> ''
> 1 scheduler/-xx-xx/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{jobs.py:} INFO - Creating / updating qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl -xx-xx xx:xx:xx+xx:xx
> [scheduled]> in ORM
> 71 scheduler/-xx-xx/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{logging_mixin.py:xx} INFO - [-xx-xx xx:xx:xx,xxx] {__init__.py:}
> INFO - Restoring task ' -xx-xx xx:xx:xx+xx:xx [removed]>' which was previously removed from DAG
> ''
>
> mysql> *select * from task_instance where task_id like '%REP%';#*
>
> +--+-+++++--++-++--++--++--++--
[jira] [Assigned] (AIRFLOW-4355) Externally triggered DAG is marked as 'success' even if a task has been 'removed'!
[
https://issues.apache.org/jira/browse/AIRFLOW-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bjorn Olsen reassigned AIRFLOW-4355:
Assignee: Bjorn Olsen
> Externally triggered DAG is marked as 'success' even if a task has been
> 'removed'!
> --
>
> Key: AIRFLOW-4355
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4355
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, DagRun, scheduler
>Affects Versions: 1.10.3
>Reporter: t oo
>Assignee: Bjorn Olsen
>Priority: Blocker
> Labels: dynamic
> Fix For: 2.0.0
>
> Attachments: dag_success_even_if_task_removed.png, treeview.png
>
>
> note: all my dags are purely externally triggered
> *Issue:* Dag has 5 parallel tasks that ran successfully and 1 final task that
> somehow got 'removed' state (prior dag runs had 'failed' state) and never ran
> successfully but still the DAG is showing success!
>
> *Command ran* (note that previous commands like airflow trigger_dag -e
> 20190412 qsr_coremytbl were run before and failed for valid reason (ie python
> task failing) ):
> airflow trigger_dag -e 20190412T08:00 qsr_coremytbl --conf '\{"hourstr":"08"}'
>
> *some logs on prior instance of airflow (ec2 was autohealed):*
> [2019-04-18 08:29:40,678] \{logging_mixin.py:95} INFO - [2019-04-18
> 08:29:40,678] {__init__.py:4897} WARNING - Failed to get task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00 [None]>'
> for dag ''. Marking it as removed.
> [2019-04-18 08:29:43,582] \{logging_mixin.py:95} INFO - [2019-04-18
> 08:29:43,582] {__init__.py:4906} INFO - Restoring task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00 [removed]>'
> which was previously removed from DAG ''
> [2019-04-18 08:29:43,618] \{jobs.py:1787} INFO - Creating / updating
> 08:00:00+00:00 [scheduled]> in ORM
> [2019-04-18 08:29:43,676] \{logging_mixin.py:95} INFO - [2019-04-18
> 08:29:43,676] {__init__.py:4897} WARNING - Failed to get task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00
> [scheduled]>' for dag ''. Marking it as removed.
>
> *some logs on newer ec2:*
> [myuser@host logs]$ grep -i hive -R * | sed 's#[0-9]#x#g' | sort | uniq -c |
> grep -v 'airflow-webserver-access.log'
> 2 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl log -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
> 1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl log -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl'), ('format',
> u'json')]
> 1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl rendered -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
> 1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl task -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
> 1 scheduler/latest/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{jobs.py:} INFO - Creating / updating qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl -xx-xx xx:xx:xx+xx:xx
> [scheduled]> in ORM
> 71 scheduler/latest/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{logging_mixin.py:xx} INFO - [-xx-xx xx:xx:xx,xxx] {__init__.py:}
> INFO - Restoring task ' -xx-xx xx:xx:xx+xx:xx [removed]>' which was previously removed from DAG
> ''
> 1 scheduler/-xx-xx/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{jobs.py:} INFO - Creating / updating qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl -xx-xx xx:xx:xx+xx:xx
> [scheduled]> in ORM
> 71 scheduler/-xx-xx/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{logging_mixin.py:xx} INFO - [-xx-xx xx:xx:xx,xxx] {__init__.py:}
> INFO - Restoring task ' -xx-xx xx:xx:xx+xx:xx [removed]>' which was previously removed from DAG
> ''
>
> mysql> *select * from task_instance where task_id like '%REP%';#*
>
> +--+-+++++--++-++--++--++--++++-
