Sunny Gupta created AIRFLOW-3253:
------------------------------------
Summary: KubernetesPodOperator Unauthorized Code 401
Key: AIRFLOW-3253
URL: https://issues.apache.org/jira/browse/AIRFLOW-3253
Project: Apache Airflow
Issue Type: Bug
Components: authentication, gcp, kubernetes
Affects Versions: 1.10.0
Reporter: Sunny Gupta
apache-airflow==1.10.0
kubernetes==7.0.0 (Tried)
kubernetes==8.0.0b1 (Tried)
Everytime after couple of successful scheduled runs, some runs failed and throw
below error.
Error looks related to k8s authorization and it seems like a pattern in my
case, everytime expiry comes near, job fails and after new expiry updates it
runs for a while and fails.
Above speculation could be wrong, need help to fix this issue. I am running one
sample python hello DAG and planning to move production workload but this is
blocker for me.
Tried :(
* ~/.kube folder clearing and regenerate token by `gcloud container clusters
get-credentials ***` even tried setting as cron to force update tokens.
* Tried kubernetes==7.0.0 to latest beta version.
Below is my kubectl config. When I run *kubectl* cli to do GET ops on pods,
nodes resources,no issues.
{code:java}
$ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://XX.XX.XX.XX
name: gke_us-central1-b_dev-kube-cluster
contexts:
- context:
cluster: gke_us-central1-b_dev-kube-cluster
user: gke_us-central1-b_dev-kube-cluster
name: gke_us-central1-b_dev-kube-cluster
current-context: gke_us-central1-b_dev-kube-cluster
kind: Config
preferences: {}
users:
- name: gke_us-central1-b_dev-kube-cluster
user:
auth-provider:
config:
access-token: ya29.c.****TOKEN****5EREdigv
cmd-args: config config-helper --format=json
cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
expiry: 2018-10-24T20:54:37Z
expiry-key: '{.credential.token_expiry}'
token-key: '{.credential.access_token}'
name: gcp
{code}
In an hour, running every */5 min, 2-3 jobs fails. with below error.
{code:java}
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT',
'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d28a6bfeed0',
'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type':
'application/json'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}{code}
{code:java}
// complete logs
====================================================
*** Log file does not exist:
/root/airflow/logs/pyk8s.v3/python-hello/2018-10-24T06:16:00+00:00/1.log
*** Fetching from:
http://aflow-worker.internal:8793/log/pyk8s.v3/python-hello/2018-10-24T06:16:00+00:00/1.log
[2018-10-24 06:20:02,947] {models.py:1335} INFO - Dependencies all met for
<TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [queued]>
[2018-10-24 06:20:02,952] {models.py:1335} INFO - Dependencies all met for
<TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [queued]>
[2018-10-24 06:20:02,952] {models.py:1547} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------
[2018-10-24 06:20:02,966] {models.py:1569} INFO - Executing
<Task(KubernetesPodOperator): python-hello> on 2018-10-24T06:16:00+00:00
[2018-10-24 06:20:02,967] {base_task_runner.py:124} INFO - Running: ['bash',
'-c', 'airflow run pyk8s.v3 python-hello 2018-10-24T06:16:00+00:00 --job_id 354
--raw -sd DAGS_FOLDER/pyk8s.v3.py --cfg_path /tmp/tmpf0saygt7']
[2018-10-24 06:20:03,405] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello [2018-10-24 06:20:03,404] {settings.py:174} INFO -
setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2018-10-24 06:20:03,808] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello [2018-10-24 06:20:03,807] {__init__.py:51} INFO - Using executor
CeleryExecutor
[2018-10-24 06:20:03,970] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello [2018-10-24 06:20:03,970] {models.py:258} INFO - Filling up the
DagBag from /root/airflow/dags/pyk8s.v3.py
[2018-10-24 06:20:04,255] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello [2018-10-24 06:20:04,255] {cli.py:492} INFO - Running
<TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [running]> on
host aflow-worker.c.internal
[2018-10-24 06:20:04,337] {logging_mixin.py:95} INFO - [2018-10-24
06:20:04,332] {pod_launcher.py:58} ERROR - Exception when attempting to create
Namespaced Pod.
Traceback (most recent call last):
File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 55, in run_pod_async
resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6057, in create_namespaced_pod
(data) = self.create_namespaced_pod_with_http_info(namespace, body,
**kwargs)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6142, in create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 321, in call_api
_return_http_data_only, collection_formats, _preload_content,
_request_timeout)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 155, in __call_api
_request_timeout=_request_timeout)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 364, in request
body=body)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 266, in POST
body=body)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT',
'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d45a6bfeed0',
'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type':
'application/json'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[2018-10-24 06:20:04,337] {logging_mixin.py:95} INFO -
[2018-10-24 06:20:04,337] {models.py:1736} ERROR - (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT',
'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d45a6bfeed0',
'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type':
'application/json'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
Traceback (most recent call last):
File "/root/airflow/venv/lib/python3.5/site-packages/airflow/models.py", line
1633, in _run_raw_task
result = task_copy.execute(context=context)
File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
line 115, in execute
get_logs=self.get_logs)
File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 71, in run_pod
resp = self.run_pod_async(pod)
File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 55, in run_pod_async
resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6057, in create_namespaced_pod
(data) = self.create_namespaced_pod_with_http_info(namespace, body,
**kwargs)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6142, in create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 321, in call_api
_return_http_data_only, collection_formats, _preload_content,
_request_timeout)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 155, in __call_api
_request_timeout=_request_timeout)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 364, in request
body=body)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 266, in POST
body=body)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT',
'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d45a6bfeed0',
'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type':
'application/json'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[2018-10-24 06:20:04,339] {models.py:1764} INFO - Marking task as FAILED.
[2018-10-24 06:20:04,493] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello Traceback (most recent call last):
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File "/root/airflow/venv/bin/airflow", line 32, in <module>
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello args.func(args)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/utils/cli.py", line 74,
in wrapper
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello return f(*args, **kwargs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/bin/cli.py", line 498,
in run
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello _run(args, dag, ti)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/bin/cli.py", line 402,
in _run
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello pool=args.pool,
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/utils/db.py", line 74,
in wrapper
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello return func(*args, **kwargs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/models.py", line 1633,
in _run_raw_task
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello result = task_copy.execute(context=context)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
line 115, in execute
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello get_logs=self.get_logs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 71, in run_pod
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello resp = self.run_pod_async(pod)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 55, in run_pod_async
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello resp = self._client.create_namespaced_pod(body=req,
namespace=pod.namespace)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6057, in create_namespaced_pod
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello (data) = self.create_namespaced_pod_with_http_info(namespace,
body, **kwargs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6142, in create_namespaced_pod_with_http_info
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello collection_formats=collection_formats)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 321, in call_api
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello _return_http_data_only, collection_formats, _preload_content,
_request_timeout)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 155, in __call_api
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello _request_timeout=_request_timeout)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 364, in request
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello body=body)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 266, in POST
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello body=body)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 222, in request
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello raise ApiException(http_resp=r)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello kubernetes.client.rest.ApiException: (401)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello Reason: Unauthorized
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018
06:20:04 GMT', 'Content-Length': '129', 'Audit-Id':
'89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic
realm="kubernetes-master"', 'Content-Type': 'application/json'})
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello
[2018-10-24 06:20:07,938] {logging_mixin.py:95} INFO - [2018-10-24
06:20:07,937] {jobs.py:2612} INFO - Task exited with return code 1
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
