Jon Davies created AIRFLOW-2955:
-----------------------------------
Summary: Kubernetes pod operator: Unable to set requests/limits on
task pods
Key: AIRFLOW-2955
URL: https://issues.apache.org/jira/browse/AIRFLOW-2955
Project: Apache Airflow
Issue Type: Bug
Reporter: Jon Davies
When I try and set a resource limit/request on a DAG task with the
KubernetesPodOperator, I get:
{code:java}
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task
Traceback (most recent call last):
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task
File "/usr/local/bin/airflow", line 32, in <module>
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task
args.func(args)
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task
File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 74,
in wrapper
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task
return f(*args, **kwargs)
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task
File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 498, in
run
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task
_run(args, dag, ti)
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task
File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 402, in
_run
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task
pool=args.pool,
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in
wrapper
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task
return func(*args, **kwargs)
[2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task
File "/usr/local/lib/python3.7/site-packages/airflow/models.py", line 1633, in
_run_raw_task
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task
result = task_copy.execute(context=context)
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task
File
"/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
line 115, in execute
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task
get_logs=self.get_logs)
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task
File
"/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 71, in run_pod
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task
resp = self.run_pod_async(pod)
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task
File
"/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
line 52, in run_pod_async
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task
req = self.kube_req_factory.create(pod)
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task
File
"/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/kubernetes_request_factory/pod_request_factory.py",
line 56, in create
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task
self.extract_resources(pod, req)
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task
File
"/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/kubernetes_request_factory/kubernetes_request_factory.py",
line 160, in extract_resources
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task
if not pod.resources or pod.resources.is_empty_resource_request():
[2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task
AttributeError: 'dict' object has no attribute 'is_empty_resource_request'
{code}
...setting
https://github.com/apache/incubator-airflow/blob/fc10f7e0a04145a0b2f31f8d0990bbe900b4e8a2/airflow/example_dags/example_kubernetes_executor.py#L66
works, however that only adjusts the metadata for the worker pod.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
