[
https://issues.apache.org/jira/browse/AIRFLOW-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836530#comment-16836530
]
ASF GitHub Bot commented on AIRFLOW-2955:
-----------------------------------------
dimberman commented on pull request #4551: [AIRFLOW-2955] Fix kubernetes pod
operator to set requests and limits on task pods
URL: https://github.com/apache/airflow/pull/4551
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Kubernetes pod operator: Unable to set requests/limits on task pods
> -------------------------------------------------------------------
>
> Key: AIRFLOW-2955
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2955
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Jon Davies
> Assignee: Summit Suen
> Priority: Major
>
> When I try and set a resource limit/request on a DAG task with the
> KubernetesPodOperator as follows:
> {code:java}
> resources={"limit_cpu": 1, "request_cpu": 1},
> {code}
> ...I get:
> {code:java}
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask
> task Traceback (most recent call last):
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask
> task File "/usr/local/bin/airflow", line 32, in <module>
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask
> task args.func(args)
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask
> task File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py",
> line 74, in wrapper
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask
> task return f(*args, **kwargs)
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask
> task File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line
> 498, in run
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask
> task _run(args, dag, ti)
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask
> task File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line
> 402, in _run
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask
> task pool=args.pool,
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask
> task File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py",
> line 74, in wrapper
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask
> task return func(*args, **kwargs)
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask
> task File "/usr/local/lib/python3.7/site-packages/airflow/models.py", line
> 1633, in _run_raw_task
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask
> task result = task_copy.execute(context=context)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask
> task File
> "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 115, in execute
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask
> task get_logs=self.get_logs)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask
> task File
> "/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 71, in run_pod
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask
> task resp = self.run_pod_async(pod)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask
> task File
> "/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 52, in run_pod_async
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask
> task req = self.kube_req_factory.create(pod)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask
> task File
> "/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/kubernetes_request_factory/pod_request_factory.py",
> line 56, in create
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask
> task self.extract_resources(pod, req)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask
> task File
> "/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/kubernetes_request_factory/kubernetes_request_factory.py",
> line 160, in extract_resources
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask
> task if not pod.resources or pod.resources.is_empty_resource_request():
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask
> task AttributeError: 'dict' object has no attribute
> 'is_empty_resource_request'
> {code}
> ...setting
> https://github.com/apache/incubator-airflow/blob/fc10f7e0a04145a0b2f31f8d0990bbe900b4e8a2/airflow/example_dags/example_kubernetes_executor.py#L66
> works, however that only adjusts the metadata for the worker pod and not the
> pod ultimately used for the task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
