[
https://issues.apache.org/jira/browse/AIRFLOW-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663257#comment-16663257
]
Sunny Gupta commented on AIRFLOW-3253:
--------------------------------------
Thanks [~TrevorEdwards]. Good to know its an acknowledged issue. Waiting for
PR-92 to get merge in main.
Its irrelevant to this issue but I am struggling to get POD ID from object
(*pyhello* in this case) of KubernetesPodOperator. How to get the exact pod
name going to spun in k8s?
I am looking for this :
[https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/kubernetes/pod_generator.py#L152]
I wanted to clean the pod after successful instance run, so downstream task is
kind of BashOperator running *kubectl delete -l lkey=lvalue* but I am afraid,
if labels are not unique and being shared by another pod instance, it might
kill it as well. So, I wanted to target the exact pod name for deletion. OR is
there any better/recommended way to deal with this?
Thanks.
{code:java}
pyhello = KubernetesPodOperator(namespace='default',
image="python:3.6",
cmds=["python","-c"],
arguments=["print('hello world')"],
labels={"foo": "bar"},
name="python-hello",
task_id="python-hello",
get_logs=True,
dag=dag
)
{code}
> KubernetesPodOperator Unauthorized Code 401
> -------------------------------------------
>
> Key: AIRFLOW-3253
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3253
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication, gcp, kubernetes
> Affects Versions: 1.10.0
> Reporter: Sunny Gupta
> Priority: Minor
> Attachments: Screenshot from 2018-10-25 02-08-28.png
>
>
> apache-airflow==1.10.0
> kubernetes==7.0.0 (Tried)
> kubernetes==8.0.0b1 (Tried)
>
> Everytime after couple of successful scheduled runs, some runs failed and
> throw below error.
> Error looks related to k8s authorization and it seems like a pattern in my
> case, everytime expiry comes near, job fails and after new expiry updates it
> runs for a while and fails.
> !Screenshot from 2018-10-25 02-08-28.png!
> Above speculation could be wrong, need help to fix this issue. I am running
> one sample python hello DAG and planning to move production workload but this
> is blocker for me.
> Tried :(
> * ~/.kube folder clearing and regenerate token by `gcloud container clusters
> get-credentials ***` even tried setting as cron to force update tokens.
> * Tried kubernetes==7.0.0 to latest beta version.
> Below is my kubectl config. When I run *kubectl* cli to do GET ops on pods,
> nodes resources,no issues.
>
> {code:java}
> $ kubectl config view
> apiVersion: v1
> clusters:
> - cluster:
> certificate-authority-data: DATA+OMITTED
> server: https://XX.XX.XX.XX
> name: gke_us-central1-b_dev-kube-cluster
> contexts:
> - context:
> cluster: gke_us-central1-b_dev-kube-cluster
> user: gke_us-central1-b_dev-kube-cluster
> name: gke_us-central1-b_dev-kube-cluster
> current-context: gke_us-central1-b_dev-kube-cluster
> kind: Config
> preferences: {}
> users:
> - name: gke_us-central1-b_dev-kube-cluster
> user:
> auth-provider:
> config:
> access-token: ya29.c.****TOKEN****5EREdigv
> cmd-args: config config-helper --format=json
> cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
> expiry: 2018-10-24T20:54:37Z
> expiry-key: '{.credential.token_expiry}'
> token-key: '{.credential.access_token}'
> name: gcp
> {code}
>
>
> In an hour, running every */5 min, 2-3 jobs fails. with below error.
>
> {code:java}
> kubernetes.client.rest.ApiException: (401)
> Reason: Unauthorized
> HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04
> GMT', 'Content-Length': '129', 'Audit-Id':
> '89dcda61-a60f-4b23-85d6-9d28a6bfeed0', 'Www-Authenticate': 'Basic
> realm="kubernetes-master"', 'Content-Type': 'application/json'})
> HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}{code}
>
>
> {code:java}
> // complete logs
> ====================================================
> *** Log file does not exist:
> /root/airflow/logs/pyk8s.v3/python-hello/2018-10-24T06:16:00+00:00/1.log
> *** Fetching from:
> http://aflow-worker.internal:8793/log/pyk8s.v3/python-hello/2018-10-24T06:16:00+00:00/1.log
> [2018-10-24 06:20:02,947] {models.py:1335} INFO - Dependencies all met for
> <TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [queued]>
> [2018-10-24 06:20:02,952] {models.py:1335} INFO - Dependencies all met for
> <TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [queued]>
> [2018-10-24 06:20:02,952] {models.py:1547} INFO -
> --------------------------------------------------------------------------------
> Starting attempt 1 of 1
> --------------------------------------------------------------------------------
> [2018-10-24 06:20:02,966] {models.py:1569} INFO - Executing
> <Task(KubernetesPodOperator): python-hello> on 2018-10-24T06:16:00+00:00
> [2018-10-24 06:20:02,967] {base_task_runner.py:124} INFO - Running: ['bash',
> '-c', 'airflow run pyk8s.v3 python-hello 2018-10-24T06:16:00+00:00 --job_id
> 354 --raw -sd DAGS_FOLDER/pyk8s.v3.py --cfg_path /tmp/tmpf0saygt7']
> [2018-10-24 06:20:03,405] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello [2018-10-24 06:20:03,404] {settings.py:174} INFO -
> setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
> [2018-10-24 06:20:03,808] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello [2018-10-24 06:20:03,807] {__init__.py:51} INFO - Using executor
> CeleryExecutor
> [2018-10-24 06:20:03,970] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello [2018-10-24 06:20:03,970] {models.py:258} INFO - Filling up the
> DagBag from /root/airflow/dags/pyk8s.v3.py
> [2018-10-24 06:20:04,255] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello [2018-10-24 06:20:04,255] {cli.py:492} INFO - Running
> <TaskInstance: pyk8s.v3.python-hello 2018-10-24T06:16:00+00:00 [running]> on
> host aflow-worker.c.internal
> [2018-10-24 06:20:04,337] {logging_mixin.py:95} INFO - [2018-10-24
> 06:20:04,332] {pod_launcher.py:58} ERROR - Exception when attempting to
> create Namespaced Pod.
> Traceback (most recent call last):
> File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 55, in run_pod_async
> resp = self._client.create_namespaced_pod(body=req,
> namespace=pod.namespace)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6057, in create_namespaced_pod
> (data) = self.create_namespaced_pod_with_http_info(namespace, body,
> **kwargs)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6142, in create_namespaced_pod_with_http_info
> collection_formats=collection_formats)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 321, in call_api
> _return_http_data_only, collection_formats, _preload_content,
> _request_timeout)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 155, in __call_api
> _request_timeout=_request_timeout)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 364, in request
> body=body)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
> line 266, in POST
> body=body)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
> line 222, in request
> raise ApiException(http_resp=r)
> kubernetes.client.rest.ApiException: (401)
> Reason: Unauthorized
> HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04
> GMT', 'Content-Length': '129', 'Audit-Id':
> '89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic
> realm="kubernetes-master"', 'Content-Type': 'application/json'})
> HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
> [2018-10-24 06:20:04,337] {logging_mixin.py:95} INFO -
> [2018-10-24 06:20:04,337] {models.py:1736} ERROR - (401)
> Reason: Unauthorized
> HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04
> GMT', 'Content-Length': '129', 'Audit-Id':
> '89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic
> realm="kubernetes-master"', 'Content-Type': 'application/json'})
> HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
> Traceback (most recent call last):
> File "/root/airflow/venv/lib/python3.5/site-packages/airflow/models.py",
> line 1633, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 115, in execute
> get_logs=self.get_logs)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 71, in run_pod
> resp = self.run_pod_async(pod)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 55, in run_pod_async
> resp = self._client.create_namespaced_pod(body=req,
> namespace=pod.namespace)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6057, in create_namespaced_pod
> (data) = self.create_namespaced_pod_with_http_info(namespace, body,
> **kwargs)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6142, in create_namespaced_pod_with_http_info
> collection_formats=collection_formats)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 321, in call_api
> _return_http_data_only, collection_formats, _preload_content,
> _request_timeout)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 155, in __call_api
> _request_timeout=_request_timeout)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 364, in request
> body=body)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
> line 266, in POST
> body=body)
> File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
> line 222, in request
> raise ApiException(http_resp=r)
> kubernetes.client.rest.ApiException: (401)
> Reason: Unauthorized
> HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018 06:20:04
> GMT', 'Content-Length': '129', 'Audit-Id':
> '89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic
> realm="kubernetes-master"', 'Content-Type': 'application/json'})
> HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
> [2018-10-24 06:20:04,339] {models.py:1764} INFO - Marking task as FAILED.
> [2018-10-24 06:20:04,493] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello Traceback (most recent call last):
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File "/root/airflow/venv/bin/airflow", line 32, in <module>
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello args.func(args)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/utils/cli.py", line
> 74, in wrapper
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello return f(*args, **kwargs)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/bin/cli.py", line
> 498, in run
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello _run(args, dag, ti)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/bin/cli.py", line
> 402, in _run
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello pool=args.pool,
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/utils/db.py", line
> 74, in wrapper
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello return func(*args, **kwargs)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/models.py", line
> 1633, in _run_raw_task
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello result = task_copy.execute(context=context)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 115, in execute
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello get_logs=self.get_logs)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 71, in run_pod
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello resp = self.run_pod_async(pod)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 55, in run_pod_async
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello resp = self._client.create_namespaced_pod(body=req,
> namespace=pod.namespace)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6057, in create_namespaced_pod
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello (data) =
> self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6142, in create_namespaced_pod_with_http_info
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello collection_formats=collection_formats)
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 321, in call_api
> [2018-10-24 06:20:04,494] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello _return_http_data_only, collection_formats,
> _preload_content, _request_timeout)
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 155, in __call_api
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello _request_timeout=_request_timeout)
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/api_client.py",
> line 364, in request
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello body=body)
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
> line 266, in POST
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello body=body)
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello File
> "/root/airflow/venv/lib/python3.5/site-packages/kubernetes/client/rest.py",
> line 222, in request
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello raise ApiException(http_resp=r)
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello kubernetes.client.rest.ApiException: (401)
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello Reason: Unauthorized
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 24 Oct 2018
> 06:20:04 GMT', 'Content-Length': '129', 'Audit-Id':
> '89dcda61-a60f-4b23-85d6-9d45a6bfeed0', 'Www-Authenticate': 'Basic
> realm="kubernetes-master"', 'Content-Type': 'application/json'})
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello
> [2018-10-24 06:20:04,495] {base_task_runner.py:107} INFO - Job 354: Subtask
> python-hello
> [2018-10-24 06:20:07,938] {logging_mixin.py:95} INFO - [2018-10-24
> 06:20:07,937] {jobs.py:2612} INFO - Task exited with return code 1
> {code}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
