potiuk commented on code in PR #33355:
URL: https://github.com/apache/airflow/pull/33355#discussion_r1324030019
##########
airflow/operators/python.py:
##########
@@ -606,7 +614,60 @@ def _prepare_venv(self, venv_path: Path) -> None:
index_urls=self.index_urls,
)
+ def _calculate_cache_hash(self) -> str:
Review Comment:
The key part here is that "depedencies, python version, venv path" are all
part of the DAG file definition. And that one might (and will) change
independently from the tasks being run. There is currently no mechanism in
airflow to assure that the task "instance" is using particular version of the
DAG Python code. It might change any time and which version of the Python code
will be used, depends only on the timing of executing the task on the worker
(and how the worker is syncing the DAG code).
When you are backfilling for example, you are all but guaranteed that the
same tasks will run for different intervals one after another and if the
definition of the venv parametrs changes in the meantime and get synced to the
worker, some of the tasks will attempt they PVO runs with "before" and some
with "after" the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
