Victor created AIRFLOW-3374:
-------------------------------
Summary: KubernetesPodOperator gets stuck on failure
Key: AIRFLOW-3374
URL: https://issues.apache.org/jira/browse/AIRFLOW-3374
Project: Apache Airflow
Issue Type: Bug
Components: kubernetes
Affects Versions: 1.10.1
Reporter: Victor
I am running Airflow 1.10.1rc2 on kubernetes with the LocalExecutor and DAGs
using the KubernetesPodOperator and when the execution fail (in this case it is
because kubernetes can't download the image), the logs tell me that it failed
but the task stays marked as running and nothing happens.
Also the pod doesn't get deleted so I suppose something is happening in
launch.run_pod…
This is the DAG operator call:
{quote}{{with DAG(}}
{{ 'demo',}}
{{ default_args=default_args,}}
{{ # since we always take the latest upload available,}}
{{ # we don't have to catchup (i.e., execute every run between start_date and
now)}}
{{ # but run only once.}}
{{ catchup=False,}}
{{ # for now, we don't schedule the DAG daily}}
{{ schedule_interval=None}}
{{) as dag:}}
{{ datapipe = kubernetes_pod_operator.KubernetesPodOperator(}}
{{ task_id='datapipe',}}
{{ name='datapipe',}}
{{ namespace='default',}}
{{ image='my-image:master',}}
{{ cmds=['python3'],}}
{{ arguments=['-m', 'datapipe'],}}
{{ in_cluster=True,}}
{{ is_delete_operator_pod=True,}}
{{ # TODO add image_pull_secrets (in 1.10.2, see
https://github.com/apache/incubator-airflow/pull/4188)}}
{{ )}}{quote}
Those are the logs:
{quote}*** Reading local file:
/airflow/logs/demo/datapipe/2018-11-20T15:51:31.604882+00:00/1.log
[2018-11-20 15:51:35,483] \{models.py:1361} INFO - Dependencies all met for
<TaskInstance: demo.datapipe 2018-11-20T15:51:31.604882+00:00 [queued]>
[2018-11-20 15:51:35,497] \{models.py:1361} INFO - Dependencies all met for
<TaskInstance: demo.datapipe 2018-11-20T15:51:31.604882+00:00 [queued]>
[2018-11-20 15:51:35,497] \{models.py:1573} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------
[2018-11-20 15:51:35,534] \{models.py:1595} INFO - Executing
<Task(KubernetesPodOperator): datapipe> on 2018-11-20T15:51:31.604882+00:00
[2018-11-20 15:51:35,535] \{base_task_runner.py:118} INFO - Running: ['bash',
'-c', 'airflow run demo datapipe 2018-11-20T15:51:31.604882+00:00 --job_id 3
--raw -sd /usr/local/airflow/dags_volume/..data/demo.py --cfg_path
/tmp/tmpsf0htmc0']
[2018-11-20 15:51:36,799] \{base_task_runner.py:101} INFO - Job 3: Subtask
datapipe [2018-11-20 15:51:36,795] \{settings.py:174} INFO -
setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2018-11-20 15:51:37,425] \{base_task_runner.py:101} INFO - Job 3: Subtask
datapipe [2018-11-20 15:51:37,424] \{__init__.py:51} INFO - Using executor
LocalExecutor
[2018-11-20 15:51:37,779] \{base_task_runner.py:101} INFO - Job 3: Subtask
datapipe [2018-11-20 15:51:37,779] \{models.py:271} INFO - Filling up the
DagBag from /usr/local/airflow/dags_volume/..data/demo.py
[2018-11-20 15:51:38,379] \{base_task_runner.py:101} INFO - Job 3: Subtask
datapipe [2018-11-20 15:51:38,378] \{cli.py:484} INFO - Running <TaskInstance:
demo.datapipe 2018-11-20T15:51:31.604882+00:00 [running]> on host
infra-airflow-6d78c56489-r9trl
[2018-11-20 15:51:38,452] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:38,451] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:39,458] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:39,458] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:40,467] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:40,466] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:41,473] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:41,473] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:42,479] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:42,479] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:43,485] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:43,485] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:44,492] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:44,492] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:45,498] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:45,498] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:46,505] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:46,505] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:47,511] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:47,511] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:48,635] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:48,635] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:49,642] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:49,642] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:50,649] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:50,649] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:51,656] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:51,656] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:52,665] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:52,664] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:53,679] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:53,675] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:54,739] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:54,739] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:55,747] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:55,746] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:56,822] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:56,822] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:57,829] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:57,829] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:58,841] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:58,841] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:51:59,848] \{logging_mixin.py:95} INFO - [2018-11-20
15:51:59,848] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:00,856] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:00,856] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:01,864] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:01,864] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:02,871] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:02,871] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:03,878] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:03,878] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:04,883] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:04,883] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:05,890] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:05,889] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:06,897] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:06,897] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:07,908] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:07,908] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:08,920] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:08,919] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:09,925] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:09,925] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:10,942] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:10,942] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:11,948] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:11,948] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:12,955] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:12,955] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:13,961] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:13,961] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:14,968] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:14,968] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:15,975] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:15,974] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:16,980] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:16,980] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:17,986] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:17,986] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:18,996] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:18,996] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:20,003] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:20,003] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:21,012] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:21,012] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:22,019] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:22,019] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:23,027] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:23,027] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:24,034] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:24,034] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:25,072] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:25,072] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:26,078] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:26,078] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:27,085] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:27,085] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:28,090] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:28,090] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:29,101] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:29,101] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:30,107] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:30,107] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:31,114] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:31,114] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:32,120] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:32,120] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:33,127] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:33,127] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:34,134] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:34,134] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:35,141] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:35,141] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:36,149] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:36,149] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:37,156] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:37,156] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:38,164] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:38,163] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:39,175] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:39,175] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:40,182] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:40,182] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:41,189] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:41,189] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:42,196] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:42,195] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:43,202] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:43,202] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:44,211] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:44,211] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:45,218] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:45,217] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:46,224] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:46,224] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:47,230] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:47,230] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:48,236] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:48,236] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:49,246] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:49,246] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:50,253] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:50,253] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:51,259] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:51,259] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:52,266] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:52,265] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:53,272] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:53,272] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:54,277] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:54,277] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:55,284] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:55,284] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:56,291] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:56,291] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:57,298] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:57,298] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:58,305] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:58,305] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:52:59,316] \{logging_mixin.py:95} INFO - [2018-11-20
15:52:59,315] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:00,322] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:00,322] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:01,329] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:01,329] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:02,336] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:02,336] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:03,343] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:03,343] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:04,351] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:04,351] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:05,358] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:05,357] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:06,364] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:06,363] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:07,369] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:07,369] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:08,375] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:08,375] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:09,386] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:09,385] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:10,431] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:10,431] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:11,438] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:11,438] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:12,446] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:12,445] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:13,456] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:13,456] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:14,464] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:14,464] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:15,471] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:15,470] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:16,478] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:16,478] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:17,533] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:17,532] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:18,610] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:18,610] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:19,928] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:19,927] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:20,981] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:20,981] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:21,988] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:21,988] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:22,994] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:22,994] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:24,002] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:24,002] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:25,007] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:25,007] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:26,015] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:26,015] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:27,021] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:27,020] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:28,027] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:28,027] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:29,034] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:29,034] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:30,046] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:30,046] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:31,052] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:31,052] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:32,059] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:32,059] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:33,064] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:33,064] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:34,077] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:34,077] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:35,087] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:35,086] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:36,092] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:36,092] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:37,098] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:37,098] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:38,105] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:38,105] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:39,112] \{logging_mixin.py:95} INFO - [2018-11-20
15:53:39,112] \{pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
event of type Pending
[2018-11-20 15:53:39,129] \{models.py:1760} ERROR - Pod Launching failed: Pod
took too long to start
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
line 123, in execute
get_logs=self.get_logs)
File
"/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 86, in run_pod
raise AirflowException("Pod took too long to start")
airflow.exceptions.AirflowException: Pod took too long to start
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1659,
in _run_raw_task
result = task_copy.execute(context=context)
File
"/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
line 135, in execute
raise AirflowException('Pod Launching failed: \{error}'.format(error=ex))
airflow.exceptions.AirflowException: Pod Launching failed: Pod took too long to
start
[2018-11-20 15:53:39,134] \{models.py:1791} INFO - Marking task as FAILED.
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
