[ https://issues.apache.org/jira/browse/AIRFLOW-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16694389#comment-16694389 ]
Victor commented on AIRFLOW-3374: --------------------------------- For the record, this is not a deterministic error, because today, the task was marked as failed when this situation happened. > KubernetesPodOperator gets stuck on failure > ------------------------------------------- > > Key: AIRFLOW-3374 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3374 > Project: Apache Airflow > Issue Type: Bug > Components: kubernetes > Affects Versions: 1.10.1 > Reporter: Victor > Priority: Major > > I am running Airflow 1.10.1rc2 on kubernetes with the LocalExecutor and DAGs > using the KubernetesPodOperator and when the execution fail (in this case it > is because kubernetes can't download the image), the logs tell me that it > failed but the task stays marked as running and nothing happens. > Also the pod doesn't get deleted so I suppose something is happening in > launch.run_pod… > This is the DAG operator call: > {code:java} > with DAG( > 'demo', > default_args=default_args, > # since we always take the latest upload available, > # we don't have to catchup (i.e., execute every run between start_date > and now) > # but run only once. > catchup=False, > # for now, we don't schedule the DAG daily > schedule_interval=None > ) as dag: > datapipe = kubernetes_pod_operator.KubernetesPodOperator( > task_id='datapipe', > name='datapipe', > namespace='default', > image='my-image:master', > cmds=['python3'], > arguments=['-m', 'fb'], > in_cluster=True, > is_delete_operator_pod=True, > # TODO add image_pull_secrets (in 1.10.2, see > https://github.com/apache/incubator-airflow/pull/4188) > ) > {code} > Those are the logs: > {noformat} > *** Reading local file: > /airflow/logs/demo/datapipe/2018-11-20T15:51:31.604882+00:00/1.log > [2018-11-20 15:51:35,483] {models.py:1361} INFO - Dependencies all met for > <TaskInstance: demo.datapipe 2018-11-20T15:51:31.604882+00:00 [queued]> > [2018-11-20 15:51:35,497] {models.py:1361} INFO - Dependencies all met for > <TaskInstance: demo.datapipe 2018-11-20T15:51:31.604882+00:00 [queued]> > [2018-11-20 15:51:35,497] {models.py:1573} INFO - > -------------------------------------------------------------------------------- > Starting attempt 1 of 1 > -------------------------------------------------------------------------------- > [2018-11-20 15:51:35,534] {models.py:1595} INFO - Executing > <Task(KubernetesPodOperator): datapipe> on 2018-11-20T15:51:31.604882+00:00 > [2018-11-20 15:51:35,535] {base_task_runner.py:118} INFO - Running: ['bash', > '-c', 'airflow run demo datapipe 2018-11-20T15:51:31.604882+00:00 --job_id 3 > --raw -sd /usr/local/airflow/dags_volume/..data/demo.py --cfg_path > /tmp/tmpsf0htmc0'] > [2018-11-20 15:51:36,799] {base_task_runner.py:101} INFO - Job 3: Subtask > datapipe [2018-11-20 15:51:36,795] {settings.py:174} INFO - > setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800 > [2018-11-20 15:51:37,425] {base_task_runner.py:101} INFO - Job 3: Subtask > datapipe [2018-11-20 15:51:37,424] {__init__.py:51} INFO - Using executor > LocalExecutor > [2018-11-20 15:51:37,779] {base_task_runner.py:101} INFO - Job 3: Subtask > datapipe [2018-11-20 15:51:37,779] {models.py:271} INFO - Filling up the > DagBag from /usr/local/airflow/dags_volume/..data/demo.py > [2018-11-20 15:51:38,379] {base_task_runner.py:101} INFO - Job 3: Subtask > datapipe [2018-11-20 15:51:38,378] {cli.py:484} INFO - Running <TaskInstance: > demo.datapipe 2018-11-20T15:51:31.604882+00:00 [running]> on host > infra-airflow-6d78c56489-r9trl > [2018-11-20 15:51:38,452] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:38,451] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:39,458] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:39,458] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:40,467] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:40,466] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:41,473] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:41,473] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:42,479] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:42,479] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:43,485] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:43,485] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:44,492] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:44,492] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:45,498] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:45,498] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:46,505] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:46,505] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:47,511] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:47,511] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:48,635] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:48,635] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:49,642] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:49,642] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:50,649] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:50,649] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:51,656] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:51,656] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:52,665] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:52,664] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:53,679] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:53,675] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:54,739] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:54,739] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:55,747] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:55,746] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:56,822] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:56,822] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:57,829] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:57,829] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:58,841] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:58,841] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:59,848] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:59,848] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:00,856] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:00,856] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:01,864] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:01,864] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:02,871] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:02,871] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:03,878] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:03,878] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:04,883] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:04,883] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:05,890] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:05,889] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:06,897] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:06,897] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:07,908] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:07,908] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:08,920] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:08,919] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:09,925] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:09,925] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:10,942] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:10,942] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:11,948] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:11,948] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:12,955] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:12,955] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:13,961] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:13,961] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:14,968] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:14,968] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:15,975] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:15,974] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:16,980] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:16,980] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:17,986] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:17,986] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:18,996] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:18,996] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:20,003] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:20,003] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:21,012] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:21,012] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:22,019] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:22,019] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:23,027] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:23,027] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:24,034] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:24,034] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:25,072] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:25,072] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:26,078] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:26,078] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:27,085] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:27,085] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:28,090] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:28,090] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:29,101] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:29,101] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:30,107] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:30,107] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:31,114] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:31,114] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:32,120] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:32,120] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:33,127] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:33,127] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:34,134] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:34,134] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:35,141] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:35,141] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:36,149] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:36,149] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:37,156] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:37,156] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:38,164] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:38,163] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:39,175] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:39,175] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:40,182] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:40,182] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:41,189] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:41,189] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:42,196] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:42,195] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:43,202] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:43,202] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:44,211] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:44,211] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:45,218] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:45,217] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:46,224] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:46,224] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:47,230] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:47,230] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:48,236] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:48,236] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:49,246] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:49,246] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:50,253] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:50,253] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:51,259] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:51,259] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:52,266] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:52,265] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:53,272] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:53,272] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:54,277] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:54,277] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:55,284] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:55,284] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:56,291] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:56,291] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:57,298] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:57,298] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:58,305] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:58,305] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:59,316] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:59,315] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:00,322] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:00,322] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:01,329] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:01,329] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:02,336] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:02,336] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:03,343] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:03,343] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:04,351] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:04,351] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:05,358] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:05,357] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:06,364] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:06,363] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:07,369] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:07,369] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:08,375] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:08,375] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:09,386] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:09,385] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:10,431] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:10,431] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:11,438] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:11,438] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:12,446] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:12,445] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:13,456] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:13,456] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:14,464] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:14,464] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:15,471] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:15,470] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:16,478] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:16,478] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:17,533] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:17,532] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:18,610] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:18,610] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:19,928] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:19,927] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:20,981] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:20,981] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:21,988] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:21,988] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:22,994] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:22,994] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:24,002] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:24,002] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:25,007] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:25,007] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:26,015] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:26,015] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:27,021] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:27,020] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:28,027] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:28,027] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:29,034] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:29,034] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:30,046] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:30,046] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:31,052] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:31,052] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:32,059] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:32,059] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:33,064] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:33,064] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:34,077] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:34,077] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:35,087] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:35,086] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:36,092] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:36,092] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:37,098] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:37,098] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:38,105] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:38,105] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:39,112] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:39,112] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:39,129] {models.py:1760} ERROR - Pod Launching failed: Pod > took too long to start > Traceback (most recent call last): > File > "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", > line 123, in execute > get_logs=self.get_logs) > File > "/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py", > line 86, in run_pod > raise AirflowException("Pod took too long to start") > airflow.exceptions.AirflowException: Pod took too long to start > During handling of the above exception, another exception occurred: > Traceback (most recent call last): > File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1659, > in _run_raw_task > result = task_copy.execute(context=context) > File > "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", > line 135, in execute > raise AirflowException('Pod Launching failed: {error}'.format(error=ex)) > airflow.exceptions.AirflowException: Pod Launching failed: Pod took too long > to start > [2018-11-20 15:53:39,134] {models.py:1791} INFO - Marking task as > FAILED.{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)