[
https://issues.apache.org/jira/browse/AIRFLOW-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16694436#comment-16694436
]
Victor commented on AIRFLOW-3374:
---------------------------------
Actually, I don't think it is related to KubernetesPodOperator but maybe to the
scheduler?
> KubernetesPodOperator gets stuck on failure
> -------------------------------------------
>
> Key: AIRFLOW-3374
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3374
> Project: Apache Airflow
> Issue Type: Bug
> Components: kubernetes
> Affects Versions: 1.10.1
> Reporter: Victor
> Priority: Major
>
> I am running Airflow 1.10.1rc2 on kubernetes with the LocalExecutor and DAGs
> using the KubernetesPodOperator and when the execution fail (in this case it
> is because kubernetes can't download the image), the logs tell me that it
> failed but the task stays marked as running and nothing happens.
> Also the pod doesn't get deleted so I suppose something is happening in
> launch.run_pod…
> This is the DAG operator call:
> {code:java}
> with DAG(
> 'demo',
> default_args=default_args,
> # since we always take the latest upload available,
> # we don't have to catchup (i.e., execute every run between start_date
> and now)
> # but run only once.
> catchup=False,
> # for now, we don't schedule the DAG daily
> schedule_interval=None
> ) as dag:
> datapipe = kubernetes_pod_operator.KubernetesPodOperator(
> task_id='datapipe',
> name='datapipe',
> namespace='default',
> image='my-image:master',
> cmds=['python3'],
> arguments=['-m', 'fb'],
> in_cluster=True,
> is_delete_operator_pod=True,
> # TODO add image_pull_secrets (in 1.10.2, see
> https://github.com/apache/incubator-airflow/pull/4188)
> )
> {code}
> Those are the logs:
> {noformat}
> *** Reading local file:
> /airflow/logs/demo/datapipe/2018-11-20T15:51:31.604882+00:00/1.log
> [2018-11-20 15:51:35,483] {models.py:1361} INFO - Dependencies all met for
> <TaskInstance: demo.datapipe 2018-11-20T15:51:31.604882+00:00 [queued]>
> [2018-11-20 15:51:35,497] {models.py:1361} INFO - Dependencies all met for
> <TaskInstance: demo.datapipe 2018-11-20T15:51:31.604882+00:00 [queued]>
> [2018-11-20 15:51:35,497] {models.py:1573} INFO -
> --------------------------------------------------------------------------------
> Starting attempt 1 of 1
> --------------------------------------------------------------------------------
> [2018-11-20 15:51:35,534] {models.py:1595} INFO - Executing
> <Task(KubernetesPodOperator): datapipe> on 2018-11-20T15:51:31.604882+00:00
> [2018-11-20 15:51:35,535] {base_task_runner.py:118} INFO - Running: ['bash',
> '-c', 'airflow run demo datapipe 2018-11-20T15:51:31.604882+00:00 --job_id 3
> --raw -sd /usr/local/airflow/dags_volume/..data/demo.py --cfg_path
> /tmp/tmpsf0htmc0']
> [2018-11-20 15:51:36,799] {base_task_runner.py:101} INFO - Job 3: Subtask
> datapipe [2018-11-20 15:51:36,795] {settings.py:174} INFO -
> setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
> [2018-11-20 15:51:37,425] {base_task_runner.py:101} INFO - Job 3: Subtask
> datapipe [2018-11-20 15:51:37,424] {__init__.py:51} INFO - Using executor
> LocalExecutor
> [2018-11-20 15:51:37,779] {base_task_runner.py:101} INFO - Job 3: Subtask
> datapipe [2018-11-20 15:51:37,779] {models.py:271} INFO - Filling up the
> DagBag from /usr/local/airflow/dags_volume/..data/demo.py
> [2018-11-20 15:51:38,379] {base_task_runner.py:101} INFO - Job 3: Subtask
> datapipe [2018-11-20 15:51:38,378] {cli.py:484} INFO - Running <TaskInstance:
> demo.datapipe 2018-11-20T15:51:31.604882+00:00 [running]> on host
> infra-airflow-6d78c56489-r9trl
> [2018-11-20 15:51:38,452] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:38,451] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:39,458] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:39,458] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:40,467] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:40,466] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:41,473] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:41,473] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:42,479] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:42,479] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:43,485] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:43,485] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:44,492] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:44,492] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:45,498] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:45,498] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:46,505] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:46,505] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:47,511] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:47,511] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:48,635] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:48,635] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:49,642] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:49,642] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:50,649] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:50,649] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:51,656] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:51,656] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:52,665] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:52,664] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:53,679] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:53,675] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:54,739] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:54,739] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:55,747] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:55,746] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:56,822] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:56,822] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:57,829] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:57,829] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:58,841] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:58,841] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:51:59,848] {logging_mixin.py:95} INFO - [2018-11-20
> 15:51:59,848] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:00,856] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:00,856] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:01,864] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:01,864] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:02,871] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:02,871] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:03,878] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:03,878] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:04,883] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:04,883] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:05,890] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:05,889] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:06,897] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:06,897] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:07,908] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:07,908] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:08,920] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:08,919] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:09,925] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:09,925] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:10,942] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:10,942] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:11,948] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:11,948] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:12,955] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:12,955] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:13,961] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:13,961] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:14,968] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:14,968] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:15,975] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:15,974] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:16,980] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:16,980] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:17,986] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:17,986] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:18,996] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:18,996] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:20,003] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:20,003] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:21,012] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:21,012] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:22,019] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:22,019] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:23,027] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:23,027] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:24,034] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:24,034] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:25,072] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:25,072] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:26,078] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:26,078] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:27,085] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:27,085] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:28,090] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:28,090] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:29,101] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:29,101] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:30,107] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:30,107] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:31,114] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:31,114] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:32,120] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:32,120] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:33,127] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:33,127] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:34,134] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:34,134] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:35,141] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:35,141] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:36,149] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:36,149] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:37,156] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:37,156] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:38,164] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:38,163] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:39,175] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:39,175] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:40,182] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:40,182] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:41,189] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:41,189] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:42,196] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:42,195] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:43,202] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:43,202] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:44,211] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:44,211] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:45,218] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:45,217] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:46,224] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:46,224] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:47,230] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:47,230] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:48,236] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:48,236] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:49,246] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:49,246] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:50,253] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:50,253] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:51,259] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:51,259] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:52,266] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:52,265] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:53,272] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:53,272] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:54,277] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:54,277] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:55,284] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:55,284] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:56,291] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:56,291] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:57,298] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:57,298] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:58,305] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:58,305] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:52:59,316] {logging_mixin.py:95} INFO - [2018-11-20
> 15:52:59,315] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:00,322] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:00,322] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:01,329] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:01,329] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:02,336] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:02,336] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:03,343] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:03,343] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:04,351] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:04,351] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:05,358] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:05,357] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:06,364] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:06,363] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:07,369] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:07,369] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:08,375] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:08,375] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:09,386] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:09,385] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:10,431] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:10,431] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:11,438] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:11,438] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:12,446] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:12,445] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:13,456] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:13,456] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:14,464] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:14,464] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:15,471] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:15,470] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:16,478] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:16,478] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:17,533] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:17,532] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:18,610] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:18,610] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:19,928] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:19,927] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:20,981] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:20,981] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:21,988] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:21,988] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:22,994] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:22,994] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:24,002] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:24,002] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:25,007] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:25,007] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:26,015] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:26,015] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:27,021] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:27,020] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:28,027] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:28,027] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:29,034] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:29,034] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:30,046] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:30,046] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:31,052] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:31,052] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:32,059] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:32,059] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:33,064] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:33,064] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:34,077] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:34,077] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:35,087] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:35,086] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:36,092] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:36,092] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:37,098] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:37,098] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:38,105] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:38,105] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:39,112] {logging_mixin.py:95} INFO - [2018-11-20
> 15:53:39,112] {pod_launcher.py:121} INFO - Event: datapipe-2a76c2a2 had an
> event of type Pending
> [2018-11-20 15:53:39,129] {models.py:1760} ERROR - Pod Launching failed: Pod
> took too long to start
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 123, in execute
> get_logs=self.get_logs)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 86, in run_pod
> raise AirflowException("Pod took too long to start")
> airflow.exceptions.AirflowException: Pod took too long to start
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1659,
> in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 135, in execute
> raise AirflowException('Pod Launching failed: {error}'.format(error=ex))
> airflow.exceptions.AirflowException: Pod Launching failed: Pod took too long
> to start
> [2018-11-20 15:53:39,134] {models.py:1791} INFO - Marking task as
> FAILED.{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
