[
https://issues.apache.org/jira/browse/AIRFLOW-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891767#comment-15891767
]
Bolke de Bruin commented on AIRFLOW-932:
----------------------------------------
I just ran this on my AIRFLOW-910 branch and I am not seeing this behaviour. I
did change your execution date to 2016 as 2017 won't run.
{code}
MariaDB [airflow]> select * from task_instance;
+----------------------------+----------------------------------+----------------------------+----------------------------+----------------------------+----------+---------+------------+--------------------------+----------+--------+------+---------+-----------------+----------------+-------------+-------+
| task_id | dag_id |
execution_date | start_date | end_date
| duration | state | try_number | hostname | unixname |
job_id | pool | queue | priority_weight | operator | queued_dttm | pid
|
+----------------------------+----------------------------------+----------------------------+----------------------------+----------------------------+----------+---------+------------+--------------------------+----------+--------+------+---------+-----------------+----------------+-------------+-------+
| create_and_transfer_roster | teztg | 2016-06-14
00:00:00.000000 | 2017-03-02 08:58:35.867219 | 2017-03-02 08:58:46.868470 |
11.0013 | success | 1 | bolkes-macbook-pro.local | bolke | 2 |
NULL | default | 1 | SubDagOperator | NULL | 46856 |
| create_roster | teztg.create_and_transfer_roster | 2016-06-14
00:00:00.000000 | 2017-03-02 08:58:42.681159 | 2017-03-02 08:58:42.702009 |
0.02085 | success | 1 | bolkes-macbook-pro.local | bolke | 4 |
NULL | default | 1 | PythonOperator | NULL | 46862 |
+----------------------------+----------------------------------+----------------------------+----------------------------+----------------------------+----------+---------+------------+--------------------------+----------+--------+------+---------+-----------------+----------------+-------------+-------+
{code}
{code}
[2017-03-02 08:58:29,039] {jobs.py:1756} INFO - Checking run <DagRun teztg @
2016-06-14 00:00:00: backfill_2016-06-14T00:00:00, externally triggered: False>
[2017-03-02 08:58:29,044] {models.py:1125} INFO - Dependencies all met for
<TaskInstance: teztg.create_and_transfer_roster 2016-06-14 00:00:00 [None]>
[2017-03-02 08:58:29,047] {base_executor.py:50} INFO - Adding to queue: airflow
run teztg create_and_transfer_roster 2016-06-14T00:00:00 --local -sd
DAGS_FOLDER/task_removed_bug.py
[2017-03-02 08:58:33,985] {jobs.py:1949} INFO - [backfill progress] | dag run 1
of 1 | tasks waiting: 1 | succeeded: 0 | kicked_off: 1 | failed: 0 | skipped: 0
| deadlocked: 0
[2017-03-02 08:58:33,987] {local_executor.py:45} INFO - LocalWorker running
airflow run teztg create_and_transfer_roster 2016-06-14T00:00:00 --local -sd
DAGS_FOLDER/task_removed_bug.py
[2017-03-02 08:58:33,989] {models.py:1125} INFO - Dependencies all met for
<TaskInstance: teztg.create_and_transfer_roster 2016-06-14 00:00:00 [scheduled]>
/Users/bolke/Documents/dev/airflow_env/lib/python2.7/site-packages/airflow-1.9.0.dev0+apache.incubating-py2.7.egg/airflow/configuration.py:548:
DeprecationWarning: This method will be removed in future versions. Use
'parser.read_file()' instead.
self.readfp(StringIO.StringIO(string))
[2017-03-02 08:58:34,561] {__init__.py:56} INFO - Using executor LocalExecutor
Logging into:
/Users/bolke/airflow/logs/teztg/create_and_transfer_roster/2016-06-14T00:00:00
[2017-03-02 08:58:38,987] {jobs.py:1949} INFO - [backfill progress] | dag run 1
of 1 | tasks waiting: 1 | succeeded: 0 | kicked_off: 1 | failed: 0 | skipped: 0
| deadlocked: 0
[2017-03-02 08:58:38,990] {models.py:1119} INFO - Dependencies not met for
<TaskInstance: teztg.create_and_transfer_roster 2016-06-14 00:00:00 [running]>,
dependency 'Task Instance State' FAILED: Task is in the 'running' state which
is not a valid state for execution. The task must be cleared in order to be run.
[2017-03-02 08:58:38,990] {models.py:1119} INFO - Dependencies not met for
<TaskInstance: teztg.create_and_transfer_roster 2016-06-14 00:00:00 [running]>,
dependency 'Task Instance Not Already Running' FAILED: Task is already running,
it started on 2017-03-02 08:58:35.867219.
[2017-03-02 08:58:43,994] {jobs.py:1949} INFO - [backfill progress] | dag run 1
of 1 | tasks waiting: 1 | succeeded: 0 | kicked_off: 1 | failed: 0 | skipped: 0
| deadlocked: 0
[2017-03-02 08:58:43,997] {models.py:1119} INFO - Dependencies not met for
<TaskInstance: teztg.create_and_transfer_roster 2016-06-14 00:00:00 [running]>,
dependency 'Task Instance State' FAILED: Task is in the 'running' state which
is not a valid state for execution. The task must be cleared in order to be run.
[2017-03-02 08:58:43,997] {models.py:1119} INFO - Dependencies not met for
<TaskInstance: teztg.create_and_transfer_roster 2016-06-14 00:00:00 [running]>,
dependency 'Task Instance Not Already Running' FAILED: Task is already running,
it started on 2017-03-02 08:58:35.867219.
[2017-03-02 08:58:49,001] {jobs.py:1949} INFO - [backfill progress] | dag run 1
of 1 | tasks waiting: 1 | succeeded: 0 | kicked_off: 1 | failed: 0 | skipped: 0
| deadlocked: 0
[2017-03-02 08:58:54,007] {jobs.py:1849} WARNING - (u'teztg',
u'create_and_transfer_roster', datetime.datetime(2016, 6, 14, 0, 0)) state
success not in tasks_to_run=[]
[2017-03-02 08:58:54,007] {jobs.py:1949} INFO - [backfill progress] | dag run 1
of 1 | tasks waiting: 0 | succeeded: 1 | kicked_off: 1 | failed: 0 | skipped: 0
| deadlocked: 0
[2017-03-02 08:58:54,009] {models.py:3984} INFO - Updating state for <DagRun
teztg @ 2016-06-14 00:00:00: backfill_2016-06-14T00:00:00, externally
triggered: False> considering 1 task(s)
[2017-03-02 08:58:54,011] {models.py:4030} INFO - Marking run <DagRun teztg @
2016-06-14 00:00:00: backfill_2016-06-14T00:00:00, externally triggered: False>
successful
[2017-03-02 08:58:54,028] {jobs.py:2001} INFO - Backfill done. Exiting.
{code}
Can this be related to your local patches?
> Backfills delete existing task instances and mark them as removed
> -----------------------------------------------------------------
>
> Key: AIRFLOW-932
> URL: https://issues.apache.org/jira/browse/AIRFLOW-932
> Project: Apache Airflow
> Issue Type: Sub-task
> Components: backfill
> Reporter: Dan Davydov
> Priority: Blocker
>
> I'm still investigating.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
