ecerulm opened a new issue #16563:
URL: https://github.com/apache/airflow/issues/16563
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the
following questions.
Don't worry if they're not all applicable; just try to include what you can
:-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Description**
<!-- A short description of your feature -->
Currently, if you want to change the kubernetes resources allocated to an
operator you need to something like
```
from kubernetes.client import models as k8s
task1 = PythonOperator(
task_id="id1",
python_callable=mycallable,
dag=dag1,
executor_config={
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(
limits={
"memory": "6Gi",
"cpu": "1",
}
)
)
]
)
)
},
```
which works, but I think that is very verbose and for a (in my opinion)
common case. I would prefer a more direct syntax like
```
task1 = PythonOperator(
task_id="id1",
python_callable=mycallable,
dag=dag1,
executor_config={
"pod_limits_memory": "6Gi",
"pod_limits_cpu": "1",
}
```
Alternatively, maybe a documentation change to
*
https://github.com/apache/airflow/blob/main/docs/apache-airflow/executor/kubernetes.rst
*
https://github.com/apache/airflow/blob/main/airflow/example_dags/example_kubernetes_executor_config.py
would make this (IMHO common) task easier
**Use case / motivation**
<!-- What do you want to happen?
Rather than telling us how you might implement this solution, try to take a
step back and describe what you are trying to achieve.
-->
It think the figuring out the `executor_config` > `pod_override` and the
`kubernetes.client.models` api just to increase the memory limit for a tasks
was kind of hard. I guess a lot of people would benefit from
* exposing a simpler way to achieve it or
* documenting this specific use case in the kubernetes executor documentation
**Are you willing to submit a PR?**
<!--- We accept contributions! -->
**Related Issues**
<!-- Is there currently another issue associated with this? -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
