Digoya opened a new issue, #35803:
URL: https://github.com/apache/airflow/issues/35803
### Description
Allow to set pool_slots for a task based on the result of another task.
One task may require different amount of resources for runtime depending on
its inputs. However we can assume approximate resource requirements for a task
based on its initial parameters. It would be nice to be able to dynamically set
`pool_slots` based on the result of another task task.
```python
@dag
def example_dag():
@task
def get_slots_required():
from lib import estimate_cost
data = get_data()
return 3 if estimate_cost(data) > 15 else 1
@task
def heavy_or_not_task():
...
slots = get_slots_required()
heavy_or_not_task.override(pool_slots=slots)()
dag_instance = example_dag()
```
### Use case/motivation
This feature expands capabilities of pool system in Airflow allowing more
fine grained control over resources making system more stable and efficient.
### Related issues
https://github.com/apache/airflow/issues/33657
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
