jens-scheffler-bosch opened a new issue, #32452:
URL: https://github.com/apache/airflow/issues/32452

   ### Apache Airflow version
   
   main (development)
   
   ### What happened
   
   I was noting the `task_instance_mutation_hook` feature and followed the 
example in 
https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/cluster-policies.html#task-instance-mutation
 to see how this works to "dynamically adjust the queue of a task instance.
   Reason originally was that I noticed that the queue attribute does not 
support being added to `template_fields`, but it seems using 
`task_instance_mutation_hook` we can achieve a bit of flexibility to queuing. 
So far this "works" but I saw a few inconsistencies:
   
   * When using `task_instance_mutation_hook` the UI in DAGs->Grid->Task 
Details->More Details always shows the task definition value of `queue` and not 
the mutated value, which actually is stored in DB
   * Moreover I realized when navigating through the tasks in Grid (w/o 
starting a DAG or changing completed tasks) the pure view of the Grid was 
overwriting values in the DB, thus completed tasks stored in DB were 
over-written with task standard values - so even if a task was in the past 
completed in a mutated queue, after clicking around the information was reset 
in DB. A bit strange that "just" navigating in UI writes back data into 
`task_instance` DB table.
   
   ### What you think should happen instead
   
   When I use the `task_instance_mutation_hook` and actually mutated task is in 
DB, then the UI should display the correct state as well as when navigating 
through the UI data should not be changed.
   
   ### How to reproduce
   
   * Implement a `task_instance_mutation_hook` like described in 
https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/cluster-policies.html#task-instance-mutation.
   * Run one DAG that should be affected by the hook - I used 
example_bash_operator (Note, add a celery worker or the changed qeue name to an 
existing worker to get the run completed!)
   * Check values in DB, e.g. `SELECT task_id, queue FROM task_instance WHERE 
dag_id='example_bash_operator';` - see the queue is mutated
   * Check task instance detail in the UI. UI shows the task default queue value
   * Navigate through different DAG runs in Grid view, check status by clicking 
on various tasks and also the task you checked earlier
   * Check values again in DB, e.g. `SELECT task_id, queue FROM task_instance 
WHERE dag_id='example_bash_operator';` - see the queue value is reset to task 
standard definition.
   
   ### Operating System
   
   Ubuntu 20.04 / Breeze Dev setup in Py 3.8 Container
   
   ### Versions of Apache Airflow Providers
   
   No specific setup, execution in Breeze from latest main
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   Development setup via `breeze start-airflow...`
   Python 3.8
   Postgres
   Celery worker
   
   ### Anything else
   
   I tried to debug down to understand the root cause of the inconsistentcy but 
was not sure if this is "by design", a "flaw" or just "a bug". Before opening a 
PR I need to understand how it is desired.
   
   I nailed down the root to be 
airflow/models/taskinstance.py:refresh_from_task() which is called when loading.
   
   Open question from my side:
   * Is it "intended" that the queue field is volatile (and it is stored in DB 
just for the work distribution)? == This is a feature, don't trust task 
mutation.
   * Is it a inconsistency how from multiple sides the task instances are 
instantiated and the mutation code is not called consistently across all 
places? --> Fix would be to call mutation code in all places consistently
   * Is the queue in DB the authoritative value? Then the refresh_from_task() 
might not be called to update queue except in the initial creation. --> Fix 
would be selectively copying queue field
   * How does the problem relate to the "feature" I see in the code with 
`pool_override` - would it be consistent to make queue also officially 
override-able?
   * How about other fields stored in `task_instance` table in DB: `pool_slots` 
and `priority_weight` - do we need to apply the same fix for these as well?
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to