[ 
https://issues.apache.org/jira/browse/AIRFLOW-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122365#comment-16122365
 ] 

Lily Chang commented on AIRFLOW-1482:
-------------------------------------

We ran into the same "Executor reports task instance finished (XXX) although 
the task says its XXX. Was the task killed externally?" using airflow 1.8.0, 
celery 4.0.2, redis==2.10.5. Jobs scheduled, backfilled, or via airflow run can 
fail but if I keep rerun the jobs, that might get through at some point. Tried 
resetdb and still not working. Any idea what could cause this problem?

> Error when try to backfill the example_trigger_controller_dag
> -------------------------------------------------------------
>
>                 Key: AIRFLOW-1482
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1482
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: backfill
>    Affects Versions: 1.8.1, 1.8.2
>         Environment: Ubuntu: 16.04
> Python: 2.7
> CeleryExecutor
> Broker: Redis
>            Reporter: Timothee N
>            Priority: Blocker
>         Attachments: airflow_1.png, airflow_2.png, airflow_3.png
>
>
> Hello,
> Running a backfill command for the 
> {noformat}example_trigger_controller_dag{noformat} example dag, result in the 
> failed task {noformat}test_trigger_dagrun{noformat}
> It seems to me that the problem comes from the TriggerDagRunOperator in the 
> example_trigger_controller_dag ?
> Backfill command: {noformat}airflow backfill -s 2017-07-10 -e 2017-07-13 
> --pool backfill example_trigger_controller_dag{noformat}
> Tested in 1.8.1 and 1.8.2rc1
> Here is the output log from the backfill command :
> {noformat}
> [2017-08-02 13:53:00,844] {__init__.py:57} INFO - Using executor 
> CeleryExecutor
> [2017-08-02 13:53:00,888] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2017-08-02 13:53:00,902] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> /var/lib/airflow/local/lib/python2.7/site-packages/airflow/www/app.py:23: 
> FlaskWTFDeprecationWarning: "flask_wtf.CsrfProtect" has been renamed to 
> "CSRFProtect" and will be removed in 1.0.
>   csrf = CsrfProtect()
> [2017-08-02 13:53:01,033] {models.py:168} INFO - Filling up the DagBag from 
> /var/lib/airflow/dags
> [2017-08-02 13:53:01,332] {models.py:1128} INFO - Dependencies all met for 
> <TaskInstance: example_trigger_controller_dag.test_trigger_dagrun 2017-07-10 
> 00:00:00 [scheduled]>
> [2017-08-02 13:53:01,337] {base_executor.py:50} INFO - Adding to queue: 
> airflow run example_trigger_controller_dag test_trigger_dagrun 
> 2017-07-10T00:00:00 --pickle 1 --local --pool backfill
> [2017-08-02 13:53:06,267] {celery_executor.py:81} INFO - [celery] queuing 
> (u'example_trigger_controller_dag', u'test_trigger_dagrun', 
> datetime.datetime(2017, 7, 10, 0, 0)) through celery, queue=default
> [2017-08-02 13:53:06,330] {models.py:4164} INFO - Updating state for <DagRun 
> example_trigger_controller_dag @ 2017-07-10 00:00:00: 
> backfill_2017-07-10T00:00:00, externally triggered: False> considering 1 
> task(s)
> [2017-08-02 13:53:06,334] {jobs.py:2020} INFO - [backfill progress] | 
> finished run 0 of 1 | tasks waiting: 0 | succeeded: 0 | kicked_off: 1 | 
> failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
> [2017-08-02 13:53:11,273] {jobs.py:1743} ERROR - Executor reports task 
> instance <TaskInstance: example_trigger_controller_dag.test_trigger_dagrun 
> 2017-07-10 00:00:00 [queued]> finished (failed) although the task says its 
> queued. Was the task killed externally?
> [2017-08-02 13:53:11,273] {models.py:1433} ERROR - Executor reports task 
> instance <TaskInstance: example_trigger_controller_dag.test_trigger_dagrun 
> 2017-07-10 00:00:00 [queued]> finished (failed) although the task says its 
> queued. Was the task killed externally?
> None
> [2017-08-02 13:53:11,273] {models.py:1457} INFO - Marking task as FAILED.
> [2017-08-02 13:53:11,279] {models.py:1478} ERROR - Executor reports task 
> instance <TaskInstance: example_trigger_controller_dag.test_trigger_dagrun 
> 2017-07-10 00:00:00 [queued]> finished (failed) although the task says its 
> queued. Was the task killed externally?
> [2017-08-02 13:53:11,281] {jobs.py:1694} ERROR - Task instance <TaskInstance: 
> example_trigger_controller_dag.test_trigger_dagrun 2017-07-10 00:00:00 
> [failed]> failed
> [2017-08-02 13:53:11,283] {models.py:4164} INFO - Updating state for <DagRun 
> example_trigger_controller_dag @ 2017-07-10 00:00:00: 
> backfill_2017-07-10T00:00:00, externally triggered: False> considering 1 
> task(s)
> [2017-08-02 13:53:11,285] {models.py:4204} INFO - Marking run <DagRun 
> example_trigger_controller_dag @ 2017-07-10 00:00:00: 
> backfill_2017-07-10T00:00:00, externally triggered: False> failed
> [2017-08-02 13:53:11,298] {jobs.py:2020} INFO - [backfill progress] | 
> finished run 1 of 1 | tasks waiting: 0 | succeeded: 0 | kicked_off: 0 | 
> failed: 1 | skipped: 0 | deadlocked: 0 | not ready: 0
> Traceback (most recent call last):
>   File "/var/lib/airflow/bin/airflow", line 28, in <module>
>     args.func(args)
>   File 
> "/var/lib/airflow/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 
> 167, in backfill
>     pool=args.pool)
>   File 
> "/var/lib/airflow/local/lib/python2.7/site-packages/airflow/models.py", line 
> 3373, in run
>     job.run()
>   File "/var/lib/airflow/local/lib/python2.7/site-packages/airflow/jobs.py", 
> line 201, in run
>     self._execute()
>   File "/var/lib/airflow/local/lib/python2.7/site-packages/airflow/jobs.py", 
> line 2063, in _execute
>     raise AirflowException(err)
> airflow.exceptions.AirflowException: 
> ---------------------------------------------------
> Some task instances failed:
> set([(u'example_trigger_controller_dag', u'test_trigger_dagrun', 
> datetime.datetime(2017, 7, 10, 0, 0))])
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to