bparhy commented on issue #14590:
URL: https://github.com/apache/airflow/issues/14590#issuecomment-799516277
I just see in my scheduler logs. Please let me know if this helps. The task
just immediately fails and there is no logs. The pod just goes to ERROR state.
[2021-03-15 15:24:05,573] {kubernetes_executor.py:853} INFO - Deleted pod:
('test_dmart_bi_case_dly', 'proc_start', datetime.datetime(2021, 3, 3, 19, 47,
58, 415513, tzinfo=tzlocal()), 1) in namespace XXX
[2021-03-15 15:24:23,584] {scheduler_job.py:946} INFO - 1 tasks up for
execution:
<TaskInstance: hello_world.hello_task 2021-03-15 15:21:40.190964+00:00
[scheduled]>
[2021-03-15 15:24:23,591] {scheduler_job.py:980} INFO - Figuring out tasks
to run in Pool(name=default_pool) with 128 open slots and 1 task instances
ready to be queued
[2021-03-15 15:24:23,592] {scheduler_job.py:1008} INFO - DAG hello_world has
0/16 running and queued tasks
[2021-03-15 15:24:23,598] {scheduler_job.py:1068} INFO - Setting the
following tasks to queued state:
<TaskInstance: hello_world.hello_task 2021-03-15 15:21:40.190964+00:00
[scheduled]>
[2021-03-15 15:24:23,607] {scheduler_job.py:1142} INFO - Setting the
following 1 tasks to queued state:
<TaskInstance: hello_world.hello_task 2021-03-15 15:21:40.190964+00:00
[queued]>
[2021-03-15 15:24:23,608] {scheduler_job.py:1178} INFO - Sending
('hello_world', 'hello_task', datetime.datetime(2021, 3, 15, 15, 21, 40,
190964, tzinfo=<TimezoneInfo [UTC, GMT, +00:00:00, STD]>), 1) to executor with
priority 1 and queue default
[2021-03-15 15:24:23,608] {base_executor.py:58} INFO - Adding to queue:
['airflow', 'run', 'hello_world', 'hello_task',
'2021-03-15T15:21:40.190964+00:00', '--local', '--pool', 'default_pool', '-sd',
'/usr/local/airflow/dags/network/hello_world.py']
[2021-03-15 15:24:23,609] {kubernetes_executor.py:793} INFO - Add task
('hello_world', 'hello_task', datetime.datetime(2021, 3, 15, 15, 21, 40,
190964, tzinfo=<TimezoneInfo [UTC, GMT, +00:00:00, STD]>), 1) with command
['airflow', 'run', 'hello_world', 'hello_task',
'2021-03-15T15:21:40.190964+00:00', '--local', '--pool', 'default_pool', '-sd',
'/usr/local/airflow/dags/network/hello_world.py'] with executor_config
{'KubernetesExecutor': {'request_cpu': '200m', 'limit_cpu': '400m',
'request_memory': '558Mi', 'limit_memory': '600Mi', 'image_pull_policy':
'IfNotPresent'}}
[2021-03-15 15:24:23,613] {kubernetes_executor.py:429} INFO - Kubernetes job
is (('hello_world', 'hello_task', datetime.datetime(2021, 3, 15, 15, 21, 40,
190964, tzinfo=<TimezoneInfo [UTC, GMT, +00:00:00, STD]>), 1), ['airflow',
'run', 'hello_world', 'hello_task', '2021-03-15T15:21:40.190964+00:00',
'--local', '--pool', 'default_pool', '-sd',
'/usr/local/airflow/dags/network/hello_world.py'], {'api_version': 'v1',
'kind': 'Pod',
'metadata': {'annotations': None,
'cluster_name': None,
'creation_timestamp': None,
'deletion_grace_period_seconds': None,
'deletion_timestamp': None,
'finalizers': None,
'generate_name': None,
'generation': None,
'initializers': None,
'labels': None,
'managed_fields': None,
'name': None,
'namespace': None,
'owner_references': None,
'resource_version': None,
'self_link': None,
'uid': None},
'spec': {'active_deadline_seconds': None,
'affinity': None,
'automount_service_account_token': None,
'containers': [{'args': [],
'command': [],
'env': [],
'env_from': [],
'image': None,
'image_pull_policy': 'IfNotPresent',
'lifecycle': None,
'liveness_probe': None,
'name': 'base',
'ports': [],
'readiness_probe': None,
'resources': {'limits': {'cpu': '400m',
'memory': '600Mi'},
'requests': {'cpu': '200m',
'memory': '558Mi'}},
'security_context': None,
'stdin': None,
'stdin_once': None,
'termination_message_path': None,
'termination_message_policy': None,
'tty': None,
'volume_devices': None,
'volume_mounts': [],
'working_dir': None}],
'dns_config': None,
'dns_policy': None,
'enable_service_links': None,
'host_aliases': None,
'host_ipc': None,
'host_network': False,
'host_pid': None,
'hostname': None,
'image_pull_secrets': [],
'init_containers': None,
'node_name': None,
'node_selector': None,
'preemption_policy': None,
'priority': None,
'priority_class_name': None,
'readiness_gates': None,
'restart_policy': None,
'runtime_class_name': None,
'scheduler_name': None,
'security_context': None,
'service_account': None,
'service_account_name': None,
'share_process_namespace': None,
'subdomain': None,
'termination_grace_period_seconds': None,
'tolerations': None,
'volumes': []},
'status': None})
/usr/local/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py:331:
DeprecationWarning: Using `airflow.contrib.kubernetes.pod.Pod` is deprecated.
Please use `k8s.V1Pod`.
security_context=_extract_security_context(pod.spec.security_context)
/usr/local/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py:77:
DeprecationWarning: Using `airflow.contrib.kubernetes.pod.Pod` is deprecated.
Please use `k8s.V1Pod` instead.
pod = self._mutate_pod_backcompat(pod)
[2021-03-15 15:24:23,659] {kubernetes_executor.py:327} INFO - Event:
helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 had an event of type ADDED
[2021-03-15 15:24:23,659] {kubernetes_executor.py:369} INFO - Event:
helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 Pending
[2021-03-15 15:24:23,674] {kubernetes_executor.py:327} INFO - Event:
helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 had an event of type
MODIFIED
[2021-03-15 15:24:23,674] {kubernetes_executor.py:369} INFO - Event:
helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 Pending
[2021-03-15 15:24:23,696] {kubernetes_executor.py:327} INFO - Event:
helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 had an event of type
MODIFIED
[2021-03-15 15:24:23,697] {kubernetes_executor.py:369} INFO - Event:
helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 Pending
[2021-03-15 15:24:25,226] {kubernetes_executor.py:327} INFO - Event:
helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 had an event of type
MODIFIED
[2021-03-15 15:24:25,226] {kubernetes_executor.py:377} INFO - Event:
helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 is Running
[2021-03-15 15:24:27,353] {kubernetes_executor.py:327} INFO - Event:
helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 had an event of type
MODIFIED
[2021-03-15 15:24:27,353] {kubernetes_executor.py:371} INFO - Event:
helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 Failed
[2021-03-15 15:24:27,566] {kubernetes_executor.py:500} INFO - Attempting to
finish pod; pod_id: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5;
state: failed; labels: {'airflow-worker':
'38cee1a9-9e0a-406a-a3d9-d7132b1769ff', 'airflow_version': '1.10.14', 'dag_id':
'hello_world', 'execution_date': '2021-03-15T15_21_40.190964_plus_00_00',
'kubernetes_executor': 'True', 'task_id': 'hello_task', 'try_number': '1'}
[2021-03-15 15:24:27,572] {kubernetes_executor.py:600} INFO - Found matching
task hello_world-hello_task (2021-03-15 15:21:40.190964+00:00) with current
state of queued
[2021-03-15 15:24:27,574] {kubernetes_executor.py:814} INFO - Changing state
of (('hello_world', 'hello_task', datetime.datetime(2021, 3, 15, 15, 21, 40,
190964, tzinfo=tzlocal()), 1), 'failed',
'helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5', 'xxx', '634337857') to
failed
[2021-03-15 15:24:27,589] {kubernetes_executor.py:327} INFO - Event:
helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 had an event of type
MODIFIED
[2021-03-15 15:24:27,589] {kubernetes_executor.py:371} INFO - Event:
helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 Failed
[2021-03-15 15:24:27,592] {kubernetes_executor.py:853} INFO - Deleted pod:
('hello_world', 'hello_task', datetime.datetime(2021, 3, 15, 15, 21, 40,
190964, tzinfo=tzlocal()), 1) in namespace xxx
[2021-03-15 15:24:27,596] {kubernetes_executor.py:327} INFO - Event:
helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 had an event of type
DELETED
[2021-03-15 15:24:27,597] {kubernetes_executor.py:371} INFO - Event:
helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 Failed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
