vincbeck commented on code in PR #44456:
URL: https://github.com/apache/airflow/pull/44456#discussion_r1876183983
##########
airflow/dag_processing/collection.py:
##########
@@ -435,14 +456,17 @@ def add_asset_trigger_references(
refs_to_add: dict[tuple[str, str], set[str]] = {}
refs_to_remove: dict[tuple[str, str], set[str]] = {}
triggers: dict[str, BaseTrigger] = {}
+ active_assets = _find_active_assets(self.assets.keys(),
session=session)
Review Comment:
I am not a sqlalchemy expert so I would not be surprised if a better version
of this query exists :). Let's look at it.
```
return {
(asset_model.name, asset_model.uri)
for asset_model in session.scalars(
select(AssetModel)
.join(AssetActive, (AssetActive.name == AssetModel.name) &
(AssetActive.uri == AssetModel.uri))
.where(tuple_(AssetActive.name,
AssetActive.uri).in_(name_uri_assets))
.where(AssetModel.consuming_dags.any(DagScheduleAssetReference.dag_id.in_(active_dags)))
.options(
joinedload(AssetModel.consuming_dags).joinedload(DagScheduleAssetReference.dag),
)
).unique()
}
```
Among a list of assets, I want to retrieve those that are active. By active
I mean:
- Exist in DAGs (have not been removed). This is done with the line
`.join(AssetActive, (AssetActive.name == AssetModel.name) & (AssetActive.uri ==
AssetModel.uri))`
- Check only for a given list of assets. This is done with this line `
.where(tuple_(AssetActive.name, AssetActive.uri).in_(name_uri_assets))`. The PK
of an asset is a couple (name, uri).
- Assets used to schedule DAGs. This is done with
`.where(AssetModel.consuming_dags.any(DagScheduleAssetReference.dag_id.in_(active_dags)))`
I am happy to refactor it if you think there is a way to optimize it.
Unfortunately my knowledge of SQLAlchemy is pretty basic so I dont think I'll
be able to come up with something better (I was already pleased to have
something working :)).
> Should any of those relations have fks that are missing?
Most of them have PK. There is a PK in `AssetActive` on the couple (name,
uri). `AssetModel.consuming_dags` is using the magic `relationship` which also
uses PKs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]