uranusjr commented on code in PR #33355:
URL: https://github.com/apache/airflow/pull/33355#discussion_r1324035463
##########
airflow/operators/python.py:
##########
@@ -606,7 +614,60 @@ def _prepare_venv(self, venv_path: Path) -> None:
index_urls=self.index_urls,
)
+ def _calculate_cache_hash(self) -> str:
Review Comment:
As Airflow is implemented now backfill tasks actually can only use the
latest version of the defined task. There are possible race conditions but
honestly those generally would cause worse problems elsewhere than the cached
virtual environment because Airflow does not have access to older DAG code.
And OK, if you worry about race condition, it’s of course still fine to put
a hash in the virtual environment path. But even with that, having all
parameters recorded in the marker file, and using that for comparison, is still
better than relying on the hash string IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
