[ https://issues.apache.org/jira/browse/AIRFLOW-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Victor updated AIRFLOW-3374: ---------------------------- Description: I am running Airflow 1.10.1rc2 on kubernetes with the LocalExecutor and DAGs using the KubernetesPodOperator and when the execution fail (in this case it is because kubernetes can't download the image), the logs tell me that it failed but the task stays marked as running and nothing happens. Also the pod doesn't get deleted so I suppose something is happening in launch.run_pod… This is the DAG operator call: {code:java} with DAG( 'demo', default_args=default_args, # since we always take the latest upload available, # we don't have to catchup (i.e., execute every run between start_date and now) # but run only once. catchup=False, # for now, we don't schedule the DAG daily schedule_interval=None ) as dag: datapipe = kubernetes_pod_operator.KubernetesPodOperator( task_id='datapipe', name='datapipe', namespace='default', image='my-image:master', cmds=['python3'], arguments=['-m', 'fb'], in_cluster=True, is_delete_operator_pod=True, # TODO add image_pull_secrets (in 1.10.2, see https://github.com/apache/incubator-airflow/pull/4188) ) {code} Those are the logs: {noformat} *** Reading local file: /airflow/logs/demo/datapipe/2018-11-20T15:51:31.604882+00:00/1.log [2018-11-20 15:51:35,483] {models.py:1361} INFO - Dependencies all met for <TaskInstance: demo.datapipe 2018-11-20T15:51:31.604882+00:00 [queued]> [2018-11-20 15:51:35,497] {models.py:1361} INFO - Dependencies all met for <TaskInstance: demo.datapipe 2018-11-20T15:51:31.604882+00:00 [queued]> [2018-11-20 15:51:35,497] {models.py:1573} INFO - -------------------------------------------------------------------------------- Starting attempt 1 of 1 -------------------------------------------------------------------------------- [2018-11-20 15:51:35,534] {models.py:1595} INFO - Executing <Task(KubernetesPodOperator): datapipe> on 2018-11-20T15:51:31.604882+00:00 [2018-11-20 15:51:35,535] {base_task_runner.py:118} INFO - Running: ['bash', '-c', 'airflow run demo datapipe 2018-11-20T15:51:31.604882+00:00 --job_id 3 --raw -sd /usr/local/airflow/dags_volume/..data/demo.py --cfg_path /tmp/tmpsf0htmc0'] [2018-11-20 15:51:36,799] {base_task_runner.py:101} INFO - Job 3: Subtask datapipe [2018-11-20 15:51:36,795] {settings.py:174} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800 [2018-11-20 15:51:37,425] {base_task_runner.py:101} INFO - Job 3: Subtask datapipe [2018-11-20 15:51:37,424] {__init__.py:51} INFO - Using executor LocalExecutor [2018-11-20 15:51:37,779] {base_task_runner.py:101} INFO - Job 3: Subtask datapipe [2018-11-20 15:51:37,779] {models.py:271} INFO - Filling up the DagBag from /usr/local/airflow/dags_volume/..data/demo.py [2018-11-20 15:51:38,379] {base_task_runner.py:101} INFO - Job 3: Subtask datapipe [2018-11-20 15:51:38,378] {cli.py:484} INFO - Running <TaskInstance: demo.datapipe 2018-11-20T15:51:31.604882+00:00 [running]> on host infra-airflow-6d78c56489-r9trl [2018-11-20 15:51:38,452] {logging_mixin.py:95} INFO - [2018-11-20 15:51:38,451] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:39,458] {logging_mixin.py:95} INFO - [2018-11-20 15:51:39,458] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:40,467] {logging_mixin.py:95} INFO - [2018-11-20 15:51:40,466] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:41,473] {logging_mixin.py:95} INFO - [2018-11-20 15:51:41,473] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:42,479] {logging_mixin.py:95} INFO - [2018-11-20 15:51:42,479] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:43,485] {logging_mixin.py:95} INFO - [2018-11-20 15:51:43,485] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:44,492] {logging_mixin.py:95} INFO - [2018-11-20 15:51:44,492] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:45,498] {logging_mixin.py:95} INFO - [2018-11-20 15:51:45,498] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:46,505] {logging_mixin.py:95} INFO - [2018-11-20 15:51:46,505] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:47,511] {logging_mixin.py:95} INFO - [2018-11-20 15:51:47,511] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:48,635] {logging_mixin.py:95} INFO - [2018-11-20 15:51:48,635] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:49,642] {logging_mixin.py:95} INFO - [2018-11-20 15:51:49,642] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:50,649] {logging_mixin.py:95} INFO - [2018-11-20 15:51:50,649] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:51,656] {logging_mixin.py:95} INFO - [2018-11-20 15:51:51,656] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:52,665] {logging_mixin.py:95} INFO - [2018-11-20 15:51:52,664] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:53,679] {logging_mixin.py:95} INFO - [2018-11-20 15:51:53,675] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:54,739] {logging_mixin.py:95} INFO - [2018-11-20 15:51:54,739] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:55,747] {logging_mixin.py:95} INFO - [2018-11-20 15:51:55,746] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:56,822] {logging_mixin.py:95} INFO - [2018-11-20 15:51:56,822] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:57,829] {logging_mixin.py:95} INFO - [2018-11-20 15:51:57,829] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:58,841] {logging_mixin.py:95} INFO - [2018-11-20 15:51:58,841] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:59,848] {logging_mixin.py:95} INFO - [2018-11-20 15:51:59,848] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:00,856] {logging_mixin.py:95} INFO - [2018-11-20 15:52:00,856] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:01,864] {logging_mixin.py:95} INFO - [2018-11-20 15:52:01,864] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:02,871] {logging_mixin.py:95} INFO - [2018-11-20 15:52:02,871] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:03,878] {logging_mixin.py:95} INFO - [2018-11-20 15:52:03,878] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:04,883] {logging_mixin.py:95} INFO - [2018-11-20 15:52:04,883] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:05,890] {logging_mixin.py:95} INFO - [2018-11-20 15:52:05,889] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:06,897] {logging_mixin.py:95} INFO - [2018-11-20 15:52:06,897] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:07,908] {logging_mixin.py:95} INFO - [2018-11-20 15:52:07,908] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:08,920] {logging_mixin.py:95} INFO - [2018-11-20 15:52:08,919] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:09,925] {logging_mixin.py:95} INFO - [2018-11-20 15:52:09,925] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:10,942] {logging_mixin.py:95} INFO - [2018-11-20 15:52:10,942] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:11,948] {logging_mixin.py:95} INFO - [2018-11-20 15:52:11,948] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:12,955] {logging_mixin.py:95} INFO - [2018-11-20 15:52:12,955] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:13,961] {logging_mixin.py:95} INFO - [2018-11-20 15:52:13,961] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:14,968] {logging_mixin.py:95} INFO - [2018-11-20 15:52:14,968] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:15,975] {logging_mixin.py:95} INFO - [2018-11-20 15:52:15,974] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:16,980] {logging_mixin.py:95} INFO - [2018-11-20 15:52:16,980] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:17,986] {logging_mixin.py:95} INFO - [2018-11-20 15:52:17,986] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:18,996] {logging_mixin.py:95} INFO - [2018-11-20 15:52:18,996] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:20,003] {logging_mixin.py:95} INFO - [2018-11-20 15:52:20,003] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:21,012] {logging_mixin.py:95} INFO - [2018-11-20 15:52:21,012] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:22,019] {logging_mixin.py:95} INFO - [2018-11-20 15:52:22,019] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:23,027] {logging_mixin.py:95} INFO - [2018-11-20 15:52:23,027] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:24,034] {logging_mixin.py:95} INFO - [2018-11-20 15:52:24,034] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:25,072] {logging_mixin.py:95} INFO - [2018-11-20 15:52:25,072] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:26,078] {logging_mixin.py:95} INFO - [2018-11-20 15:52:26,078] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:27,085] {logging_mixin.py:95} INFO - [2018-11-20 15:52:27,085] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:28,090] {logging_mixin.py:95} INFO - [2018-11-20 15:52:28,090] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:29,101] {logging_mixin.py:95} INFO - [2018-11-20 15:52:29,101] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:30,107] {logging_mixin.py:95} INFO - [2018-11-20 15:52:30,107] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:31,114] {logging_mixin.py:95} INFO - [2018-11-20 15:52:31,114] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:32,120] {logging_mixin.py:95} INFO - [2018-11-20 15:52:32,120] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:33,127] {logging_mixin.py:95} INFO - [2018-11-20 15:52:33,127] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:34,134] {logging_mixin.py:95} INFO - [2018-11-20 15:52:34,134] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:35,141] {logging_mixin.py:95} INFO - [2018-11-20 15:52:35,141] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:36,149] {logging_mixin.py:95} INFO - [2018-11-20 15:52:36,149] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:37,156] {logging_mixin.py:95} INFO - [2018-11-20 15:52:37,156] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:38,164] {logging_mixin.py:95} INFO - [2018-11-20 15:52:38,163] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:39,175] {logging_mixin.py:95} INFO - [2018-11-20 15:52:39,175] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:40,182] {logging_mixin.py:95} INFO - [2018-11-20 15:52:40,182] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:41,189] {logging_mixin.py:95} INFO - [2018-11-20 15:52:41,189] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:42,196] {logging_mixin.py:95} INFO - [2018-11-20 15:52:42,195] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:43,202] {logging_mixin.py:95} INFO - [2018-11-20 15:52:43,202] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:44,211] {logging_mixin.py:95} INFO - [2018-11-20 15:52:44,211] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:45,218] {logging_mixin.py:95} INFO - [2018-11-20 15:52:45,217] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:46,224] {logging_mixin.py:95} INFO - [2018-11-20 15:52:46,224] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:47,230] {logging_mixin.py:95} INFO - [2018-11-20 15:52:47,230] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:48,236] {logging_mixin.py:95} INFO - [2018-11-20 15:52:48,236] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:49,246] {logging_mixin.py:95} INFO - [2018-11-20 15:52:49,246] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:50,253] {logging_mixin.py:95} INFO - [2018-11-20 15:52:50,253] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:51,259] {logging_mixin.py:95} INFO - [2018-11-20 15:52:51,259] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:52,266] {logging_mixin.py:95} INFO - [2018-11-20 15:52:52,265] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:53,272] {logging_mixin.py:95} INFO - [2018-11-20 15:52:53,272] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:54,277] {logging_mixin.py:95} INFO - [2018-11-20 15:52:54,277] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:55,284] {logging_mixin.py:95} INFO - [2018-11-20 15:52:55,284] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:56,291] {logging_mixin.py:95} INFO - [2018-11-20 15:52:56,291] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:57,298] {logging_mixin.py:95} INFO - [2018-11-20 15:52:57,298] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:58,305] {logging_mixin.py:95} INFO - [2018-11-20 15:52:58,305] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:59,316] {logging_mixin.py:95} INFO - [2018-11-20 15:52:59,315] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:00,322] {logging_mixin.py:95} INFO - [2018-11-20 15:53:00,322] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:01,329] {logging_mixin.py:95} INFO - [2018-11-20 15:53:01,329] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:02,336] {logging_mixin.py:95} INFO - [2018-11-20 15:53:02,336] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:03,343] {logging_mixin.py:95} INFO - [2018-11-20 15:53:03,343] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:04,351] {logging_mixin.py:95} INFO - [2018-11-20 15:53:04,351] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:05,358] {logging_mixin.py:95} INFO - [2018-11-20 15:53:05,357] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:06,364] {logging_mixin.py:95} INFO - [2018-11-20 15:53:06,363] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:07,369] {logging_mixin.py:95} INFO - [2018-11-20 15:53:07,369] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:08,375] {logging_mixin.py:95} INFO - [2018-11-20 15:53:08,375] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:09,386] {logging_mixin.py:95} INFO - [2018-11-20 15:53:09,385] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:10,431] {logging_mixin.py:95} INFO - [2018-11-20 15:53:10,431] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:11,438] {logging_mixin.py:95} INFO - [2018-11-20 15:53:11,438] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:12,446] {logging_mixin.py:95} INFO - [2018-11-20 15:53:12,445] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:13,456] {logging_mixin.py:95} INFO - [2018-11-20 15:53:13,456] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:14,464] {logging_mixin.py:95} INFO - [2018-11-20 15:53:14,464] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:15,471] {logging_mixin.py:95} INFO - [2018-11-20 15:53:15,470] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:16,478] {logging_mixin.py:95} INFO - [2018-11-20 15:53:16,478] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:17,533] {logging_mixin.py:95} INFO - [2018-11-20 15:53:17,532] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:18,610] {logging_mixin.py:95} INFO - [2018-11-20 15:53:18,610] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:19,928] {logging_mixin.py:95} INFO - [2018-11-20 15:53:19,927] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:20,981] {logging_mixin.py:95} INFO - [2018-11-20 15:53:20,981] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:21,988] {logging_mixin.py:95} INFO - [2018-11-20 15:53:21,988] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:22,994] {logging_mixin.py:95} INFO - [2018-11-20 15:53:22,994] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:24,002] {logging_mixin.py:95} INFO - [2018-11-20 15:53:24,002] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:25,007] {logging_mixin.py:95} INFO - [2018-11-20 15:53:25,007] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:26,015] {logging_mixin.py:95} INFO - [2018-11-20 15:53:26,015] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:27,021] {logging_mixin.py:95} INFO - [2018-11-20 15:53:27,020] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:28,027] {logging_mixin.py:95} INFO - [2018-11-20 15:53:28,027] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:29,034] {logging_mixin.py:95} INFO - [2018-11-20 15:53:29,034] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:30,046] {logging_mixin.py:95} INFO - [2018-11-20 15:53:30,046] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:31,052] {logging_mixin.py:95} INFO - [2018-11-20 15:53:31,052] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:32,059] {logging_mixin.py:95} INFO - [2018-11-20 15:53:32,059] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:33,064] {logging_mixin.py:95} INFO - [2018-11-20 15:53:33,064] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:34,077] {logging_mixin.py:95} INFO - [2018-11-20 15:53:34,077] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:35,087] {logging_mixin.py:95} INFO - [2018-11-20 15:53:35,086] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:36,092] {logging_mixin.py:95} INFO - [2018-11-20 15:53:36,092] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:37,098] {logging_mixin.py:95} INFO - [2018-11-20 15:53:37,098] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:38,105] {logging_mixin.py:95} INFO - [2018-11-20 15:53:38,105] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:39,112] {logging_mixin.py:95} INFO - [2018-11-20 15:53:39,112] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:39,129] {models.py:1760} ERROR - Pod Launching failed: Pod took too long to start Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 123, in execute get_logs=self.get_logs) File "/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 86, in run_pod raise AirflowException("Pod took too long to start") airflow.exceptions.AirflowException: Pod took too long to start During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1659, in _run_raw_task result = task_copy.execute(context=context) File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 135, in execute raise AirflowException('Pod Launching failed: {error}'.format(error=ex)) airflow.exceptions.AirflowException: Pod Launching failed: Pod took too long to start [2018-11-20 15:53:39,134] {models.py:1791} INFO - Marking task as FAILED.{noformat} was: I am running Airflow 1.10.1rc2 on kubernetes with the LocalExecutor and DAGs using the KubernetesPodOperator and when the execution fail (in this case it is because kubernetes can't download the image), the logs tell me that it failed but the task stays marked as running and nothing happens. Also the pod doesn't get deleted so I suppose something is happening in launch.run_pod… This is the DAG operator call: {code:java} with DAG( 'demo', default_args=default_args, # since we always take the latest upload available, # we don't have to catchup (i.e., execute every run between start_date and now) # but run only once. catchup=False, # for now, we don't schedule the DAG daily schedule_interval=None ) as dag: datapipe = kubernetes_pod_operator.KubernetesPodOperator( task_id='datapipe', name='datapipe', namespace='default', image='registry.gitlab.com/brennus-analytics/ps-fb/datapipe:master', cmds=['python3'], arguments=['-m', 'fb'], in_cluster=True, is_delete_operator_pod=True, # TODO add image_pull_secrets (in 1.10.2, see https://github.com/apache/incubator-airflow/pull/4188) ) {code} Those are the logs: {noformat} *** Reading local file: /airflow/logs/demo/datapipe/2018-11-20T15:51:31.604882+00:00/1.log [2018-11-20 15:51:35,483] {models.py:1361} INFO - Dependencies all met for <TaskInstance: demo.datapipe 2018-11-20T15:51:31.604882+00:00 [queued]> [2018-11-20 15:51:35,497] {models.py:1361} INFO - Dependencies all met for <TaskInstance: demo.datapipe 2018-11-20T15:51:31.604882+00:00 [queued]> [2018-11-20 15:51:35,497] {models.py:1573} INFO - -------------------------------------------------------------------------------- Starting attempt 1 of 1 -------------------------------------------------------------------------------- [2018-11-20 15:51:35,534] {models.py:1595} INFO - Executing <Task(KubernetesPodOperator): datapipe> on 2018-11-20T15:51:31.604882+00:00 [2018-11-20 15:51:35,535] {base_task_runner.py:118} INFO - Running: ['bash', '-c', 'airflow run demo datapipe 2018-11-20T15:51:31.604882+00:00 --job_id 3 --raw -sd /usr/local/airflow/dags_volume/..data/demo.py --cfg_path /tmp/tmpsf0htmc0'] [2018-11-20 15:51:36,799] {base_task_runner.py:101} INFO - Job 3: Subtask datapipe [2018-11-20 15:51:36,795] {settings.py:174} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800 [2018-11-20 15:51:37,425] {base_task_runner.py:101} INFO - Job 3: Subtask datapipe [2018-11-20 15:51:37,424] {__init__.py:51} INFO - Using executor LocalExecutor [2018-11-20 15:51:37,779] {base_task_runner.py:101} INFO - Job 3: Subtask datapipe [2018-11-20 15:51:37,779] {models.py:271} INFO - Filling up the DagBag from /usr/local/airflow/dags_volume/..data/demo.py [2018-11-20 15:51:38,379] {base_task_runner.py:101} INFO - Job 3: Subtask datapipe [2018-11-20 15:51:38,378] {cli.py:484} INFO - Running <TaskInstance: demo.datapipe 2018-11-20T15:51:31.604882+00:00 [running]> on host infra-airflow-6d78c56489-r9trl [2018-11-20 15:51:38,452] {logging_mixin.py:95} INFO - [2018-11-20 15:51:38,451] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:39,458] {logging_mixin.py:95} INFO - [2018-11-20 15:51:39,458] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:40,467] {logging_mixin.py:95} INFO - [2018-11-20 15:51:40,466] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:41,473] {logging_mixin.py:95} INFO - [2018-11-20 15:51:41,473] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:42,479] {logging_mixin.py:95} INFO - [2018-11-20 15:51:42,479] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:43,485] {logging_mixin.py:95} INFO - [2018-11-20 15:51:43,485] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:44,492] {logging_mixin.py:95} INFO - [2018-11-20 15:51:44,492] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:45,498] {logging_mixin.py:95} INFO - [2018-11-20 15:51:45,498] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:46,505] {logging_mixin.py:95} INFO - [2018-11-20 15:51:46,505] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:47,511] {logging_mixin.py:95} INFO - [2018-11-20 15:51:47,511] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:48,635] {logging_mixin.py:95} INFO - [2018-11-20 15:51:48,635] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:49,642] {logging_mixin.py:95} INFO - [2018-11-20 15:51:49,642] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:50,649] {logging_mixin.py:95} INFO - [2018-11-20 15:51:50,649] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:51,656] {logging_mixin.py:95} INFO - [2018-11-20 15:51:51,656] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:52,665] {logging_mixin.py:95} INFO - [2018-11-20 15:51:52,664] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:53,679] {logging_mixin.py:95} INFO - [2018-11-20 15:51:53,675] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:54,739] {logging_mixin.py:95} INFO - [2018-11-20 15:51:54,739] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:55,747] {logging_mixin.py:95} INFO - [2018-11-20 15:51:55,746] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:56,822] {logging_mixin.py:95} INFO - [2018-11-20 15:51:56,822] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:57,829] {logging_mixin.py:95} INFO - [2018-11-20 15:51:57,829] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:58,841] {logging_mixin.py:95} INFO - [2018-11-20 15:51:58,841] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:51:59,848] {logging_mixin.py:95} INFO - [2018-11-20 15:51:59,848] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:00,856] {logging_mixin.py:95} INFO - [2018-11-20 15:52:00,856] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:01,864] {logging_mixin.py:95} INFO - [2018-11-20 15:52:01,864] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:02,871] {logging_mixin.py:95} INFO - [2018-11-20 15:52:02,871] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:03,878] {logging_mixin.py:95} INFO - [2018-11-20 15:52:03,878] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:04,883] {logging_mixin.py:95} INFO - [2018-11-20 15:52:04,883] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:05,890] {logging_mixin.py:95} INFO - [2018-11-20 15:52:05,889] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:06,897] {logging_mixin.py:95} INFO - [2018-11-20 15:52:06,897] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:07,908] {logging_mixin.py:95} INFO - [2018-11-20 15:52:07,908] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:08,920] {logging_mixin.py:95} INFO - [2018-11-20 15:52:08,919] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:09,925] {logging_mixin.py:95} INFO - [2018-11-20 15:52:09,925] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:10,942] {logging_mixin.py:95} INFO - [2018-11-20 15:52:10,942] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:11,948] {logging_mixin.py:95} INFO - [2018-11-20 15:52:11,948] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:12,955] {logging_mixin.py:95} INFO - [2018-11-20 15:52:12,955] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:13,961] {logging_mixin.py:95} INFO - [2018-11-20 15:52:13,961] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:14,968] {logging_mixin.py:95} INFO - [2018-11-20 15:52:14,968] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:15,975] {logging_mixin.py:95} INFO - [2018-11-20 15:52:15,974] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:16,980] {logging_mixin.py:95} INFO - [2018-11-20 15:52:16,980] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:17,986] {logging_mixin.py:95} INFO - [2018-11-20 15:52:17,986] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:18,996] {logging_mixin.py:95} INFO - [2018-11-20 15:52:18,996] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:20,003] {logging_mixin.py:95} INFO - [2018-11-20 15:52:20,003] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:21,012] {logging_mixin.py:95} INFO - [2018-11-20 15:52:21,012] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:22,019] {logging_mixin.py:95} INFO - [2018-11-20 15:52:22,019] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:23,027] {logging_mixin.py:95} INFO - [2018-11-20 15:52:23,027] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:24,034] {logging_mixin.py:95} INFO - [2018-11-20 15:52:24,034] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:25,072] {logging_mixin.py:95} INFO - [2018-11-20 15:52:25,072] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:26,078] {logging_mixin.py:95} INFO - [2018-11-20 15:52:26,078] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:27,085] {logging_mixin.py:95} INFO - [2018-11-20 15:52:27,085] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:28,090] {logging_mixin.py:95} INFO - [2018-11-20 15:52:28,090] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:29,101] {logging_mixin.py:95} INFO - [2018-11-20 15:52:29,101] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:30,107] {logging_mixin.py:95} INFO - [2018-11-20 15:52:30,107] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:31,114] {logging_mixin.py:95} INFO - [2018-11-20 15:52:31,114] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:32,120] {logging_mixin.py:95} INFO - [2018-11-20 15:52:32,120] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:33,127] {logging_mixin.py:95} INFO - [2018-11-20 15:52:33,127] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:34,134] {logging_mixin.py:95} INFO - [2018-11-20 15:52:34,134] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:35,141] {logging_mixin.py:95} INFO - [2018-11-20 15:52:35,141] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:36,149] {logging_mixin.py:95} INFO - [2018-11-20 15:52:36,149] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:37,156] {logging_mixin.py:95} INFO - [2018-11-20 15:52:37,156] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:38,164] {logging_mixin.py:95} INFO - [2018-11-20 15:52:38,163] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:39,175] {logging_mixin.py:95} INFO - [2018-11-20 15:52:39,175] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:40,182] {logging_mixin.py:95} INFO - [2018-11-20 15:52:40,182] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:41,189] {logging_mixin.py:95} INFO - [2018-11-20 15:52:41,189] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:42,196] {logging_mixin.py:95} INFO - [2018-11-20 15:52:42,195] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:43,202] {logging_mixin.py:95} INFO - [2018-11-20 15:52:43,202] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:44,211] {logging_mixin.py:95} INFO - [2018-11-20 15:52:44,211] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:45,218] {logging_mixin.py:95} INFO - [2018-11-20 15:52:45,217] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:46,224] {logging_mixin.py:95} INFO - [2018-11-20 15:52:46,224] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:47,230] {logging_mixin.py:95} INFO - [2018-11-20 15:52:47,230] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:48,236] {logging_mixin.py:95} INFO - [2018-11-20 15:52:48,236] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:49,246] {logging_mixin.py:95} INFO - [2018-11-20 15:52:49,246] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:50,253] {logging_mixin.py:95} INFO - [2018-11-20 15:52:50,253] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:51,259] {logging_mixin.py:95} INFO - [2018-11-20 15:52:51,259] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:52,266] {logging_mixin.py:95} INFO - [2018-11-20 15:52:52,265] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:53,272] {logging_mixin.py:95} INFO - [2018-11-20 15:52:53,272] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:54,277] {logging_mixin.py:95} INFO - [2018-11-20 15:52:54,277] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:55,284] {logging_mixin.py:95} INFO - [2018-11-20 15:52:55,284] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:56,291] {logging_mixin.py:95} INFO - [2018-11-20 15:52:56,291] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:57,298] {logging_mixin.py:95} INFO - [2018-11-20 15:52:57,298] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:58,305] {logging_mixin.py:95} INFO - [2018-11-20 15:52:58,305] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:52:59,316] {logging_mixin.py:95} INFO - [2018-11-20 15:52:59,315] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:00,322] {logging_mixin.py:95} INFO - [2018-11-20 15:53:00,322] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:01,329] {logging_mixin.py:95} INFO - [2018-11-20 15:53:01,329] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:02,336] {logging_mixin.py:95} INFO - [2018-11-20 15:53:02,336] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:03,343] {logging_mixin.py:95} INFO - [2018-11-20 15:53:03,343] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:04,351] {logging_mixin.py:95} INFO - [2018-11-20 15:53:04,351] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:05,358] {logging_mixin.py:95} INFO - [2018-11-20 15:53:05,357] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:06,364] {logging_mixin.py:95} INFO - [2018-11-20 15:53:06,363] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:07,369] {logging_mixin.py:95} INFO - [2018-11-20 15:53:07,369] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:08,375] {logging_mixin.py:95} INFO - [2018-11-20 15:53:08,375] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:09,386] {logging_mixin.py:95} INFO - [2018-11-20 15:53:09,385] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:10,431] {logging_mixin.py:95} INFO - [2018-11-20 15:53:10,431] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:11,438] {logging_mixin.py:95} INFO - [2018-11-20 15:53:11,438] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:12,446] {logging_mixin.py:95} INFO - [2018-11-20 15:53:12,445] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:13,456] {logging_mixin.py:95} INFO - [2018-11-20 15:53:13,456] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:14,464] {logging_mixin.py:95} INFO - [2018-11-20 15:53:14,464] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:15,471] {logging_mixin.py:95} INFO - [2018-11-20 15:53:15,470] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:16,478] {logging_mixin.py:95} INFO - [2018-11-20 15:53:16,478] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:17,533] {logging_mixin.py:95} INFO - [2018-11-20 15:53:17,532] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:18,610] {logging_mixin.py:95} INFO - [2018-11-20 15:53:18,610] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:19,928] {logging_mixin.py:95} INFO - [2018-11-20 15:53:19,927] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:20,981] {logging_mixin.py:95} INFO - [2018-11-20 15:53:20,981] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:21,988] {logging_mixin.py:95} INFO - [2018-11-20 15:53:21,988] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:22,994] {logging_mixin.py:95} INFO - [2018-11-20 15:53:22,994] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:24,002] {logging_mixin.py:95} INFO - [2018-11-20 15:53:24,002] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:25,007] {logging_mixin.py:95} INFO - [2018-11-20 15:53:25,007] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:26,015] {logging_mixin.py:95} INFO - [2018-11-20 15:53:26,015] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:27,021] {logging_mixin.py:95} INFO - [2018-11-20 15:53:27,020] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:28,027] {logging_mixin.py:95} INFO - [2018-11-20 15:53:28,027] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:29,034] {logging_mixin.py:95} INFO - [2018-11-20 15:53:29,034] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:30,046] {logging_mixin.py:95} INFO - [2018-11-20 15:53:30,046] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:31,052] {logging_mixin.py:95} INFO - [2018-11-20 15:53:31,052] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:32,059] {logging_mixin.py:95} INFO - [2018-11-20 15:53:32,059] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:33,064] {logging_mixin.py:95} INFO - [2018-11-20 15:53:33,064] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:34,077] {logging_mixin.py:95} INFO - [2018-11-20 15:53:34,077] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:35,087] {logging_mixin.py:95} INFO - [2018-11-20 15:53:35,086] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:36,092] {logging_mixin.py:95} INFO - [2018-11-20 15:53:36,092] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:37,098] {logging_mixin.py:95} INFO - [2018-11-20 15:53:37,098] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:38,105] {logging_mixin.py:95} INFO - [2018-11-20 15:53:38,105] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:39,112] {logging_mixin.py:95} INFO - [2018-11-20 15:53:39,112] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an event of type Pending [2018-11-20 15:53:39,129] {models.py:1760} ERROR - Pod Launching failed: Pod took too long to start Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 123, in execute get_logs=self.get_logs) File "/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 86, in run_pod raise AirflowException("Pod took too long to start") airflow.exceptions.AirflowException: Pod took too long to start During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1659, in _run_raw_task result = task_copy.execute(context=context) File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 135, in execute raise AirflowException('Pod Launching failed: {error}'.format(error=ex)) airflow.exceptions.AirflowException: Pod Launching failed: Pod took too long to start [2018-11-20 15:53:39,134] {models.py:1791} INFO - Marking task as FAILED.{noformat} > KubernetesPodOperator gets stuck on failure > ------------------------------------------- > > Key: AIRFLOW-3374 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3374 > Project: Apache Airflow > Issue Type: Bug > Components: kubernetes > Affects Versions: 1.10.1 > Reporter: Victor > Priority: Major > > I am running Airflow 1.10.1rc2 on kubernetes with the LocalExecutor and DAGs > using the KubernetesPodOperator and when the execution fail (in this case it > is because kubernetes can't download the image), the logs tell me that it > failed but the task stays marked as running and nothing happens. > Also the pod doesn't get deleted so I suppose something is happening in > launch.run_pod… > This is the DAG operator call: > {code:java} > with DAG( > 'demo', > default_args=default_args, > # since we always take the latest upload available, > # we don't have to catchup (i.e., execute every run between start_date > and now) > # but run only once. > catchup=False, > # for now, we don't schedule the DAG daily > schedule_interval=None > ) as dag: > datapipe = kubernetes_pod_operator.KubernetesPodOperator( > task_id='datapipe', > name='datapipe', > namespace='default', > image='my-image:master', > cmds=['python3'], > arguments=['-m', 'fb'], > in_cluster=True, > is_delete_operator_pod=True, > # TODO add image_pull_secrets (in 1.10.2, see > https://github.com/apache/incubator-airflow/pull/4188) > ) > {code} > Those are the logs: > {noformat} > *** Reading local file: > /airflow/logs/demo/datapipe/2018-11-20T15:51:31.604882+00:00/1.log > [2018-11-20 15:51:35,483] {models.py:1361} INFO - Dependencies all met for > <TaskInstance: demo.datapipe 2018-11-20T15:51:31.604882+00:00 [queued]> > [2018-11-20 15:51:35,497] {models.py:1361} INFO - Dependencies all met for > <TaskInstance: demo.datapipe 2018-11-20T15:51:31.604882+00:00 [queued]> > [2018-11-20 15:51:35,497] {models.py:1573} INFO - > -------------------------------------------------------------------------------- > Starting attempt 1 of 1 > -------------------------------------------------------------------------------- > [2018-11-20 15:51:35,534] {models.py:1595} INFO - Executing > <Task(KubernetesPodOperator): datapipe> on 2018-11-20T15:51:31.604882+00:00 > [2018-11-20 15:51:35,535] {base_task_runner.py:118} INFO - Running: ['bash', > '-c', 'airflow run demo datapipe 2018-11-20T15:51:31.604882+00:00 --job_id 3 > --raw -sd /usr/local/airflow/dags_volume/..data/demo.py --cfg_path > /tmp/tmpsf0htmc0'] > [2018-11-20 15:51:36,799] {base_task_runner.py:101} INFO - Job 3: Subtask > datapipe [2018-11-20 15:51:36,795] {settings.py:174} INFO - > setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800 > [2018-11-20 15:51:37,425] {base_task_runner.py:101} INFO - Job 3: Subtask > datapipe [2018-11-20 15:51:37,424] {__init__.py:51} INFO - Using executor > LocalExecutor > [2018-11-20 15:51:37,779] {base_task_runner.py:101} INFO - Job 3: Subtask > datapipe [2018-11-20 15:51:37,779] {models.py:271} INFO - Filling up the > DagBag from /usr/local/airflow/dags_volume/..data/demo.py > [2018-11-20 15:51:38,379] {base_task_runner.py:101} INFO - Job 3: Subtask > datapipe [2018-11-20 15:51:38,378] {cli.py:484} INFO - Running <TaskInstance: > demo.datapipe 2018-11-20T15:51:31.604882+00:00 [running]> on host > infra-airflow-6d78c56489-r9trl > [2018-11-20 15:51:38,452] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:38,451] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:39,458] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:39,458] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:40,467] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:40,466] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:41,473] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:41,473] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:42,479] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:42,479] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:43,485] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:43,485] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:44,492] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:44,492] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:45,498] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:45,498] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:46,505] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:46,505] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:47,511] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:47,511] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:48,635] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:48,635] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:49,642] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:49,642] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:50,649] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:50,649] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:51,656] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:51,656] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:52,665] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:52,664] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:53,679] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:53,675] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:54,739] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:54,739] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:55,747] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:55,746] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:56,822] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:56,822] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:57,829] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:57,829] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:58,841] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:58,841] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:51:59,848] {logging_mixin.py:95} INFO - [2018-11-20 > 15:51:59,848] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:00,856] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:00,856] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:01,864] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:01,864] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:02,871] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:02,871] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:03,878] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:03,878] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:04,883] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:04,883] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:05,890] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:05,889] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:06,897] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:06,897] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:07,908] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:07,908] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:08,920] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:08,919] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:09,925] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:09,925] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:10,942] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:10,942] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:11,948] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:11,948] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:12,955] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:12,955] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:13,961] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:13,961] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:14,968] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:14,968] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:15,975] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:15,974] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:16,980] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:16,980] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:17,986] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:17,986] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:18,996] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:18,996] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:20,003] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:20,003] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:21,012] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:21,012] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:22,019] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:22,019] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:23,027] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:23,027] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:24,034] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:24,034] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:25,072] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:25,072] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:26,078] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:26,078] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:27,085] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:27,085] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:28,090] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:28,090] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:29,101] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:29,101] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:30,107] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:30,107] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:31,114] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:31,114] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:32,120] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:32,120] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:33,127] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:33,127] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:34,134] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:34,134] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:35,141] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:35,141] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:36,149] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:36,149] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:37,156] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:37,156] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:38,164] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:38,163] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:39,175] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:39,175] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:40,182] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:40,182] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:41,189] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:41,189] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:42,196] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:42,195] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:43,202] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:43,202] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:44,211] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:44,211] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:45,218] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:45,217] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:46,224] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:46,224] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:47,230] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:47,230] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:48,236] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:48,236] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:49,246] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:49,246] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:50,253] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:50,253] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:51,259] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:51,259] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:52,266] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:52,265] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:53,272] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:53,272] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:54,277] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:54,277] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:55,284] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:55,284] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:56,291] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:56,291] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:57,298] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:57,298] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:58,305] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:58,305] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:52:59,316] {logging_mixin.py:95} INFO - [2018-11-20 > 15:52:59,315] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:00,322] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:00,322] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:01,329] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:01,329] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:02,336] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:02,336] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:03,343] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:03,343] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:04,351] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:04,351] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:05,358] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:05,357] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:06,364] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:06,363] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:07,369] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:07,369] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:08,375] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:08,375] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:09,386] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:09,385] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:10,431] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:10,431] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:11,438] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:11,438] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:12,446] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:12,445] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:13,456] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:13,456] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:14,464] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:14,464] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:15,471] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:15,470] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:16,478] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:16,478] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:17,533] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:17,532] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:18,610] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:18,610] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:19,928] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:19,927] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:20,981] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:20,981] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:21,988] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:21,988] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:22,994] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:22,994] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:24,002] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:24,002] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:25,007] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:25,007] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:26,015] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:26,015] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:27,021] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:27,020] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:28,027] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:28,027] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:29,034] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:29,034] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:30,046] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:30,046] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:31,052] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:31,052] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:32,059] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:32,059] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:33,064] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:33,064] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:34,077] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:34,077] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:35,087] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:35,086] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:36,092] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:36,092] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:37,098] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:37,098] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:38,105] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:38,105] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:39,112] {logging_mixin.py:95} INFO - [2018-11-20 > 15:53:39,112] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an > event of type Pending > [2018-11-20 15:53:39,129] {models.py:1760} ERROR - Pod Launching failed: Pod > took too long to start > Traceback (most recent call last): > File > "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", > line 123, in execute > get_logs=self.get_logs) > File > "/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py", > line 86, in run_pod > raise AirflowException("Pod took too long to start") > airflow.exceptions.AirflowException: Pod took too long to start > During handling of the above exception, another exception occurred: > Traceback (most recent call last): > File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1659, > in _run_raw_task > result = task_copy.execute(context=context) > File > "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", > line 135, in execute > raise AirflowException('Pod Launching failed: {error}'.format(error=ex)) > airflow.exceptions.AirflowException: Pod Launching failed: Pod took too long > to start > [2018-11-20 15:53:39,134] {models.py:1791} INFO - Marking task as > FAILED.{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)