potiuk commented on code in PR #59183: URL: https://github.com/apache/airflow/pull/59183#discussion_r2608538775
########## airflow-core/src/airflow/assets/manager.py: ########## @@ -54,6 +55,59 @@ log = structlog.get_logger(__name__) +@contextmanager +def _aquire_apdr_lock( + *, + session: Session, + dag_id: str, + partition_key: str, + asset_id: int, + max_retries: int = 10, + retry_delay: float = 0.05, +): + """ + Context manager to acquire a lock for AssetPartitionDagRun creation. + + - SQLite: uses AssetPartitionDagRunMutexLock table as row-level lock is not supported + - Postgres/MySQL: uses row-level lock on AssetModel. + """ + if get_dialect_name(session) == "sqlite": + from airflow.models.asset import AssetPartitionDagRunMutexLock + + for _ in range(max_retries): + try: + mutex = AssetPartitionDagRunMutexLock(target_dag_id=dag_id, partition_key=partition_key) + session.add(mutex) + session.flush() + try: + yield # mutex acquired + finally: + session.delete(mutex) Review Comment: One caveat here. I had a second thought about it and I did a bit reading. I think this will not work because of default sqlite transaction isolation in WAL mode for parallel processes accessing sqlite - because they might mot share locks. We are using WAL mode of sqlite (Write Ahead Log) - which I think always wok in "read committed" transaction isolation mode and not share locks. This means that when you insert row and do not commit it (which happens here), paralllel insert will still work - if one of them gets committed (which will not happen here as we always delete the row before commit in finally) - then the other insert will fail on commit (but this one will not happen either). Effectively such lock does not lock anything - both inserts will pass - one will not wait for the other. https://sqlite.org/isolation.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
