[
https://issues.apache.org/jira/browse/AIRFLOW-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunny Gupta updated AIRFLOW-3253:
---------------------------------
Attachment: Screenshot from 2018-10-25 02-08-28.png
Description:
apache-airflow==1.10.0
kubernetes==7.0.0 (Tried)
kubernetes==8.0.0b1 (Tried)
Everytime after couple of successful scheduled runs, some runs failed and throw
below error.
Error looks related to k8s authorization and it seems like a pattern in my
case, everytime expiry comes near, job fails and after new expiry updates it
runs for a while and fails.
!Screenshot from 2018-10-25 02-08-28.png!
Above speculation could be wrong, need help to fix this issue. I am running one
sample python hello DAG and planning to move production workload but this is
blocker for me.
Tried :(
* ~/.kube folder clearing and regenerate token by `gcloud container clusters
get-credentials ***` even tried setting as cron to force update tokens.
* Tried kubernetes==7.0.0 to latest beta version.
Below is my kubectl config. When I run *kubectl* cli to do GET ops on pods,
nodes resources,no issues.
{code:java}
$ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://XX.XX.XX.XX
name: gke_us-central1-b_dev-kube-cluster
contexts:
- context:
cluster: gke_us-central1-b_dev-kube-cluster
user: gke_us-central1-b_dev-kube-cluster
name: gke_us-central1-b_dev-kube-cluster
current-context: gke_us-central1-b_dev-kube-cluster
kind: Config
preferences: {}
users:
- name: gke_us-central1-b_dev-kube-cluster
user:
auth-provider:
config:
access-token: ya29.c.****TOKEN****5EREdigv
cmd-args: config config-helper --format=json
cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
expiry: 2018-10-24T20:54:37Z
expiry-key: '{.credential.token_expiry}'
token-key: '{.credential.access_token}'
name: gcp
{code}
In an hour, running every */5 min, 2-3 jobs fails. with below error.
{code:java}
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT',
'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d28a6bfeed0',
'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type':
'application/json'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}{code}
{code:java}
// complete logs
====================================================
*** Log file does not exist:
/root/airflow/logs/pyk8s.v3/python-hello/2018-10-24T06:16:00+00:00/1.log
*** Fetching from:
http://aflow-worker.internal:8793/log/pyk8s.v3/python-hello/2018-10-24T06:16:00+00:00/1.log
[2018-10-24 06:20:02,947] {models.py:1335} INFO - Dependencies all met for
<TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [queued]>
[2018-10-24 06:20:02,952] {models.py:1335} INFO - Dependencies all met for
<TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [queued]>
[2018-10-24 06:20:02,952] {models.py:1547} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------
[2018-10-24 06:20:02,966] {models.py:1569} INFO - Executing
<Task(KubernetesPodOperator): python-hello> on 2018-10-24T06:16:00+00:00
[2018-10-24 06:20:02,967] {base_task_runner.py:124} INFO - Running: ['bash',
'-c', 'airflow run pyk8s.v3 python-hello 2018-10-24T06:16:00+00:00 --job_id 354
--raw -sd DAGS_FOLDER/pyk8s.v3.py --cfg_path /tmp/tmpf0saygt7']
[2018-10-24 06:20:03,405] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello [2018-10-24 06:20:03,404] {settings.py:174} INFO -
setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2018-10-24 06:20:03,808] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello [2018-10-24 06:20:03,807] {__init__.py:51} INFO - Using executor
CeleryExecutor
[2018-10-24 06:20:03,970] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello [2018-10-24 06:20:03,970] {models.py:258} INFO - Filling up the
DagBag from /root/airflow/dags/pyk8s.v3.py
[2018-10-24 06:20:04,255] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello [2018-10-24 06:20:04,255] {cli.py:492} INFO - Running
<TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [running]> on
host aflow-worker.c.internal
[2018-10-24 06:20:04,337] {logging_mixin.py:95} INFO - [2018-10-24
06:20:04,332] {pod_launcher.py:58} ERROR - Exception when attempting to create
Namespaced Pod.
Traceback (most recent call last):
File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 55, in run_pod_async
resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6057, in create_namespaced_pod
(data) = self.create_namespaced_pod_with_http_info(namespace, body,
**kwargs)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6142, in create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 321, in call_api
_return_http_data_only, collection_formats, _preload_content,
_request_timeout)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 155, in __call_api
_request_timeout=_request_timeout)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 364, in request
body=body)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 266, in POST
body=body)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT',
'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d45a6bfeed0',
'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type':
'application/json'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[2018-10-24 06:20:04,337] {logging_mixin.py:95} INFO -
[2018-10-24 06:20:04,337] {models.py:1736} ERROR - (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT',
'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d45a6bfeed0',
'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type':
'application/json'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
Traceback (most recent call last):
File "/root/airflow/venv/lib/python3.5/site-packages/airflow/models.py", line
1633, in _run_raw_task
result = task_copy.execute(context=context)
File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
line 115, in execute
get_logs=self.get_logs)
File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 71, in run_pod
resp = self.run_pod_async(pod)
File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 55, in run_pod_async
resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6057, in create_namespaced_pod
(data) = self.create_namespaced_pod_with_http_info(namespace, body,
**kwargs)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6142, in create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 321, in call_api
_return_http_data_only, collection_formats, _preload_content,
_request_timeout)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 155, in __call_api
_request_timeout=_request_timeout)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 364, in request
body=body)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 266, in POST
body=body)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT',
'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d45a6bfeed0',
'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type':
'application/json'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[2018-10-24 06:20:04,339] {models.py:1764} INFO - Marking task as FAILED.
[2018-10-24 06:20:04,493] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello Traceback (most recent call last):
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File "/root/airflow/venv/bin/airflow", line 32, in <module>
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello args.func(args)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/utils/cli.py", line 74,
in wrapper
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello return f(*args, **kwargs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/bin/cli.py", line 498,
in run
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello _run(args, dag, ti)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/bin/cli.py", line 402,
in _run
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello pool=args.pool,
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/utils/db.py", line 74,
in wrapper
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello return func(*args, **kwargs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/models.py", line 1633,
in _run_raw_task
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello result = task_copy.execute(context=context)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
line 115, in execute
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello get_logs=self.get_logs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 71, in run_pod
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello resp = self.run_pod_async(pod)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 55, in run_pod_async
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello resp = self._client.create_namespaced_pod(body=req,
namespace=pod.namespace)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6057, in create_namespaced_pod
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello (data) = self.create_namespaced_pod_with_http_info(namespace,
body, **kwargs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6142, in create_namespaced_pod_with_http_info
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello collection_formats=collection_formats)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 321, in call_api
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello _return_http_data_only, collection_formats, _preload_content,
_request_timeout)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 155, in __call_api
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello _request_timeout=_request_timeout)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 364, in request
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello body=body)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 266, in POST
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello body=body)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 222, in request
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello raise ApiException(http_resp=r)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello kubernetes.client.rest.ApiException: (401)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello Reason: Unauthorized
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018
06:20:04 GMT', 'Content-Length': '129', 'Audit-Id':
'89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic
realm="kubernetes-master"', 'Content-Type': 'application/json'})
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello
[2018-10-24 06:20:07,938] {logging_mixin.py:95} INFO - [2018-10-24
06:20:07,937] {jobs.py:2612} INFO - Task exited with return code 1
{code}
was:
apache-airflow==1.10.0
kubernetes==7.0.0 (Tried)
kubernetes==8.0.0b1 (Tried)
Everytime after couple of successful scheduled runs, some runs failed and throw
below error.
Error looks related to k8s authorization and it seems like a pattern in my
case, everytime expiry comes near, job fails and after new expiry updates it
runs for a while and fails.
Above speculation could be wrong, need help to fix this issue. I am running one
sample python hello DAG and planning to move production workload but this is
blocker for me.
Tried :(
* ~/.kube folder clearing and regenerate token by `gcloud container clusters
get-credentials ***` even tried setting as cron to force update tokens.
* Tried kubernetes==7.0.0 to latest beta version.
Below is my kubectl config. When I run *kubectl* cli to do GET ops on pods,
nodes resources,no issues.
{code:java}
$ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://XX.XX.XX.XX
name: gke_us-central1-b_dev-kube-cluster
contexts:
- context:
cluster: gke_us-central1-b_dev-kube-cluster
user: gke_us-central1-b_dev-kube-cluster
name: gke_us-central1-b_dev-kube-cluster
current-context: gke_us-central1-b_dev-kube-cluster
kind: Config
preferences: {}
users:
- name: gke_us-central1-b_dev-kube-cluster
user:
auth-provider:
config:
access-token: ya29.c.****TOKEN****5EREdigv
cmd-args: config config-helper --format=json
cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
expiry: 2018-10-24T20:54:37Z
expiry-key: '{.credential.token_expiry}'
token-key: '{.credential.access_token}'
name: gcp
{code}
In an hour, running every */5 min, 2-3 jobs fails. with below error.
{code:java}
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT',
'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d28a6bfeed0',
'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type':
'application/json'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}{code}
{code:java}
// complete logs
====================================================
*** Log file does not exist:
/root/airflow/logs/pyk8s.v3/python-hello/2018-10-24T06:16:00+00:00/1.log
*** Fetching from:
http://aflow-worker.internal:8793/log/pyk8s.v3/python-hello/2018-10-24T06:16:00+00:00/1.log
[2018-10-24 06:20:02,947] {models.py:1335} INFO - Dependencies all met for
<TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [queued]>
[2018-10-24 06:20:02,952] {models.py:1335} INFO - Dependencies all met for
<TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [queued]>
[2018-10-24 06:20:02,952] {models.py:1547} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------
[2018-10-24 06:20:02,966] {models.py:1569} INFO - Executing
<Task(KubernetesPodOperator): python-hello> on 2018-10-24T06:16:00+00:00
[2018-10-24 06:20:02,967] {base_task_runner.py:124} INFO - Running: ['bash',
'-c', 'airflow run pyk8s.v3 python-hello 2018-10-24T06:16:00+00:00 --job_id 354
--raw -sd DAGS_FOLDER/pyk8s.v3.py --cfg_path /tmp/tmpf0saygt7']
[2018-10-24 06:20:03,405] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello [2018-10-24 06:20:03,404] {settings.py:174} INFO -
setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2018-10-24 06:20:03,808] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello [2018-10-24 06:20:03,807] {__init__.py:51} INFO - Using executor
CeleryExecutor
[2018-10-24 06:20:03,970] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello [2018-10-24 06:20:03,970] {models.py:258} INFO - Filling up the
DagBag from /root/airflow/dags/pyk8s.v3.py
[2018-10-24 06:20:04,255] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello [2018-10-24 06:20:04,255] {cli.py:492} INFO - Running
<TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [running]> on
host aflow-worker.c.internal
[2018-10-24 06:20:04,337] {logging_mixin.py:95} INFO - [2018-10-24
06:20:04,332] {pod_launcher.py:58} ERROR - Exception when attempting to create
Namespaced Pod.
Traceback (most recent call last):
File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 55, in run_pod_async
resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6057, in create_namespaced_pod
(data) = self.create_namespaced_pod_with_http_info(namespace, body,
**kwargs)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6142, in create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 321, in call_api
_return_http_data_only, collection_formats, _preload_content,
_request_timeout)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 155, in __call_api
_request_timeout=_request_timeout)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 364, in request
body=body)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 266, in POST
body=body)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT',
'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d45a6bfeed0',
'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type':
'application/json'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[2018-10-24 06:20:04,337] {logging_mixin.py:95} INFO -
[2018-10-24 06:20:04,337] {models.py:1736} ERROR - (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT',
'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d45a6bfeed0',
'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type':
'application/json'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
Traceback (most recent call last):
File "/root/airflow/venv/lib/python3.5/site-packages/airflow/models.py", line
1633, in _run_raw_task
result = task_copy.execute(context=context)
File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
line 115, in execute
get_logs=self.get_logs)
File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 71, in run_pod
resp = self.run_pod_async(pod)
File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 55, in run_pod_async
resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6057, in create_namespaced_pod
(data) = self.create_namespaced_pod_with_http_info(namespace, body,
**kwargs)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6142, in create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 321, in call_api
_return_http_data_only, collection_formats, _preload_content,
_request_timeout)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 155, in __call_api
_request_timeout=_request_timeout)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 364, in request
body=body)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 266, in POST
body=body)
File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04 GMT',
'Content-Length': '129', 'Audit-Id': '89dcda61-a60f-4b23-85d6-9d45a6bfeed0',
'Www-Authenticate': 'Basic realm="kubernetes-master"', 'Content-Type':
'application/json'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[2018-10-24 06:20:04,339] {models.py:1764} INFO - Marking task as FAILED.
[2018-10-24 06:20:04,493] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello Traceback (most recent call last):
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File "/root/airflow/venv/bin/airflow", line 32, in <module>
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello args.func(args)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/utils/cli.py", line 74,
in wrapper
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello return f(*args, **kwargs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/bin/cli.py", line 498,
in run
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello _run(args, dag, ti)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/bin/cli.py", line 402,
in _run
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello pool=args.pool,
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/utils/db.py", line 74,
in wrapper
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello return func(*args, **kwargs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/models.py", line 1633,
in _run_raw_task
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello result = task_copy.execute(context=context)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
line 115, in execute
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello get_logs=self.get_logs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 71, in run_pod
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello resp = self.run_pod_async(pod)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 55, in run_pod_async
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello resp = self._client.create_namespaced_pod(body=req,
namespace=pod.namespace)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6057, in create_namespaced_pod
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello (data) = self.create_namespaced_pod_with_http_info(namespace,
body, **kwargs)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
line 6142, in create_namespaced_pod_with_http_info
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello collection_formats=collection_formats)
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 321, in call_api
[2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello _return_http_data_only, collection_formats, _preload_content,
_request_timeout)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 155, in __call_api
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello _request_timeout=_request_timeout)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
line 364, in request
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello body=body)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 266, in POST
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello body=body)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello File
"/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
line 222, in request
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello raise ApiException(http_resp=r)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello kubernetes.client.rest.ApiException: (401)
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello Reason: Unauthorized
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018
06:20:04 GMT', 'Content-Length': '129', 'Audit-Id':
'89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic
realm="kubernetes-master"', 'Content-Type': 'application/json'})
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello
[2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
python-hello
[2018-10-24 06:20:07,938] {logging_mixin.py:95} INFO - [2018-10-24
06:20:07,937] {jobs.py:2612} INFO - Task exited with return code 1
{code}
> KubernetesPodOperator Unauthorized Code 401
> -------------------------------------------
>
> Key: AIRFLOW-3253
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3253
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication, gcp, kubernetes
> Affects Versions: 1.10.0
> Reporter: Sunny Gupta
> Priority: Minor
> Attachments: Screenshot from 2018-10-25 02-08-28.png
>
>
> apache-airflow==1.10.0
> kubernetes==7.0.0 (Tried)
> kubernetes==8.0.0b1 (Tried)
>
> Everytime after couple of successful scheduled runs, some runs failed and
> throw below error.
> Error looks related to k8s authorization and it seems like a pattern in my
> case, everytime expiry comes near, job fails and after new expiry updates it
> runs for a while and fails.
> !Screenshot from 2018-10-25 02-08-28.png!
> Above speculation could be wrong, need help to fix this issue. I am running
> one sample python hello DAG and planning to move production workload but this
> is blocker for me.
> Tried :(
> * ~/.kube folder clearing and regenerate token by `gcloud container clusters
> get-credentials ***` even tried setting as cron to force update tokens.
> * Tried kubernetes==7.0.0 to latest beta version.
> Below is my kubectl config. When I run *kubectl* cli to do GET ops on pods,
> nodes resources,no issues.
>
> {code:java}
> $ kubectl config view
> apiVersion: v1
> clusters:
> - cluster:
> certificate-authority-data: DATA+OMITTED
> server: https://XX.XX.XX.XX
> name: gke_us-central1-b_dev-kube-cluster
> contexts:
> - context:
> cluster: gke_us-central1-b_dev-kube-cluster
> user: gke_us-central1-b_dev-kube-cluster
> name: gke_us-central1-b_dev-kube-cluster
> current-context: gke_us-central1-b_dev-kube-cluster
> kind: Config
> preferences: {}
> users:
> - name: gke_us-central1-b_dev-kube-cluster
> user:
> auth-provider:
> config:
> access-token: ya29.c.****TOKEN****5EREdigv
> cmd-args: config config-helper --format=json
> cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
> expiry: 2018-10-24T20:54:37Z
> expiry-key: '{.credential.token_expiry}'
> token-key: '{.credential.access_token}'
> name: gcp
> {code}
>
>
> In an hour, running every */5 min, 2-3 jobs fails. with below error.
>
> {code:java}
> kubernetes.client.rest.ApiException: (401)
> Reason: Unauthorized
> HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04
> GMT', 'Content-Length': '129', 'Audit-Id':
> '89dcda61-a60f-4b23-85d6-9d28a6bfeed0', 'Www-Authenticate': 'Basic
> realm="kubernetes-master"', 'Content-Type': 'application/json'})
> HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}{code}
>
>
> {code:java}
> // complete logs
> ====================================================
> *** Log file does not exist:
> /root/airflow/logs/pyk8s.v3/python-hello/2018-10-24T06:16:00+00:00/1.log
> *** Fetching from:
> http://aflow-worker.internal:8793/log/pyk8s.v3/python-hello/2018-10-24T06:16:00+00:00/1.log
> [2018-10-24 06:20:02,947] {models.py:1335} INFO - Dependencies all met for
> <TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [queued]>
> [2018-10-24 06:20:02,952] {models.py:1335} INFO - Dependencies all met for
> <TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [queued]>
> [2018-10-24 06:20:02,952] {models.py:1547} INFO -
> --------------------------------------------------------------------------------
> Starting attempt 1 of 1
> --------------------------------------------------------------------------------
> [2018-10-24 06:20:02,966] {models.py:1569} INFO - Executing
> <Task(KubernetesPodOperator): python-hello> on 2018-10-24T06:16:00+00:00
> [2018-10-24 06:20:02,967] {base_task_runner.py:124} INFO - Running: ['bash',
> '-c', 'airflow run pyk8s.v3 python-hello 2018-10-24T06:16:00+00:00 --job_id
> 354 --raw -sd DAGS_FOLDER/pyk8s.v3.py --cfg_path /tmp/tmpf0saygt7']
> [2018-10-24 06:20:03,405] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello [2018-10-24 06:20:03,404] {settings.py:174} INFO -
> setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
> [2018-10-24 06:20:03,808] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello [2018-10-24 06:20:03,807] {__init__.py:51} INFO - Using executor
> CeleryExecutor
> [2018-10-24 06:20:03,970] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello [2018-10-24 06:20:03,970] {models.py:258} INFO - Filling up the
> DagBag from /root/airflow/dags/pyk8s.v3.py
> [2018-10-24 06:20:04,255] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello [2018-10-24 06:20:04,255] {cli.py:492} INFO - Running
> <TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [running]> on
> host aflow-worker.c.internal
> [2018-10-24 06:20:04,337] {logging_mixin.py:95} INFO - [2018-10-24
> 06:20:04,332] {pod_launcher.py:58} ERROR - Exception when attempting to
> create Namespaced Pod.
> Traceback (most recent call last):
> File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 55, in run_pod_async
> resp = self._client.create_namespaced_pod(body=req,
> namespace=pod.namespace)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6057, in create_namespaced_pod
> (data) = self.create_namespaced_pod_with_http_info(namespace, body,
> **kwargs)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6142, in create_namespaced_pod_with_http_info
> collection_formats=collection_formats)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 321, in call_api
> _return_http_data_only, collection_formats, _preload_content,
> _request_timeout)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 155, in __call_api
> _request_timeout=_request_timeout)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 364, in request
> body=body)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
> line 266, in POST
> body=body)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
> line 222, in request
> raise ApiException(http_resp=r)
> kubernetes.client.rest.ApiException: (401)
> Reason: Unauthorized
> HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04
> GMT', 'Content-Length': '129', 'Audit-Id':
> '89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic
> realm="kubernetes-master"', 'Content-Type': 'application/json'})
> HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
> [2018-10-24 06:20:04,337] {logging_mixin.py:95} INFO -
> [2018-10-24 06:20:04,337] {models.py:1736} ERROR - (401)
> Reason: Unauthorized
> HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04
> GMT', 'Content-Length': '129', 'Audit-Id':
> '89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic
> realm="kubernetes-master"', 'Content-Type': 'application/json'})
> HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
> Traceback (most recent call last):
> File "/root/airflow/venv/lib/python3.5/site-packages/airflow/models.py",
> line 1633, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 115, in execute
> get_logs=self.get_logs)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 71, in run_pod
> resp = self.run_pod_async(pod)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 55, in run_pod_async
> resp = self._client.create_namespaced_pod(body=req,
> namespace=pod.namespace)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6057, in create_namespaced_pod
> (data) = self.create_namespaced_pod_with_http_info(namespace, body,
> **kwargs)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6142, in create_namespaced_pod_with_http_info
> collection_formats=collection_formats)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 321, in call_api
> _return_http_data_only, collection_formats, _preload_content,
> _request_timeout)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 155, in __call_api
> _request_timeout=_request_timeout)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 364, in request
> body=body)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
> line 266, in POST
> body=body)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
> line 222, in request
> raise ApiException(http_resp=r)
> kubernetes.client.rest.ApiException: (401)
> Reason: Unauthorized
> HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04
> GMT', 'Content-Length': '129', 'Audit-Id':
> '89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic
> realm="kubernetes-master"', 'Content-Type': 'application/json'})
> HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
> [2018-10-24 06:20:04,339] {models.py:1764} INFO - Marking task as FAILED.
> [2018-10-24 06:20:04,493] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello Traceback (most recent call last):
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File "/root/airflow/venv/bin/airflow", line 32, in <module>
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello args.func(args)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/utils/cli.py", line
> 74, in wrapper
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello return f(*args, **kwargs)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/bin/cli.py", line
> 498, in run
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello _run(args, dag, ti)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/bin/cli.py", line
> 402, in _run
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello pool=args.pool,
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/utils/db.py", line
> 74, in wrapper
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello return func(*args, **kwargs)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/models.py", line
> 1633, in _run_raw_task
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello result = task_copy.execute(context=context)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 115, in execute
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello get_logs=self.get_logs)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 71, in run_pod
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello resp = self.run_pod_async(pod)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 55, in run_pod_async
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello resp = self._client.create_namespaced_pod(body=req,
> namespace=pod.namespace)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6057, in create_namespaced_pod
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello (data) =
> self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6142, in create_namespaced_pod_with_http_info
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello collection_formats=collection_formats)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 321, in call_api
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello _return_http_data_only, collection_formats,
> _preload_content, _request_timeout)
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 155, in __call_api
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello _request_timeout=_request_timeout)
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 364, in request
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello body=body)
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
> line 266, in POST
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello body=body)
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
> line 222, in request
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello raise ApiException(http_resp=r)
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello kubernetes.client.rest.ApiException: (401)
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello Reason: Unauthorized
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018
> 06:20:04 GMT', 'Content-Length': '129', 'Audit-Id':
> '89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic
> realm="kubernetes-master"', 'Content-Type': 'application/json'})
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello
> [2018-10-24 06:20:07,938] {logging_mixin.py:95} INFO - [2018-10-24
> 06:20:07,937] {jobs.py:2612} INFO - Task exited with return code 1
> {code}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
