hterik opened a new issue #19673:
URL: https://github.com/apache/airflow/issues/19673
### Description
We have some use cases which require fairly complex and dynamic
configuration of Kubernets pods. For example configuring the nodeSelector VM
size based on task inputs. Also juggling of Volumes, and in particular
persistent volume claims. Imagine tasks allocating and preparing volumes which
get passed on as input to other tasks.
The mechanism for this today in airflow is
* `pod_template_file` - static and global yaml configuration affecting all
tasks
* `pod_mutation_hook` - Sortof dynamic, can run code and has full access to
V1Pod object, but is not aware of which Dag, DagRun, Task or TaskInstance is
about to be run, can not access the `get_current_context`.
* `executor_config.pod_override` - Static per task configuration. Limited
to what `pod_generator.reconcile_xxx` supports, which is a bit limited and can
have a some unexpected behavior at times. Not possible to read any DagRun or
TaskInstance inputs.
Would it be possible to add a new more rich pod mutation function with
following additions:
* Without the reconcile-limitations. (like pod_mutation_hook)
* Aware which dag and task is currently executing. (like pod_override)
* Can read runtime information from the DagRun and TaskInstance. Most
importantly `dag_run.conf` and task inputs, eg XCcom return_value from upstream
tasks, or perhaps even any xcom value.
### Use case/motivation
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
