uranusjr commented on code in PR #42343:
URL: https://github.com/apache/airflow/pull/42343#discussion_r1774261268
##########
airflow/datasets/manager.py:
##########
@@ -58,12 +58,55 @@ class DatasetManager(LoggingMixin):
def __init__(self, **kwargs):
super().__init__(**kwargs)
- def create_datasets(self, dataset_models: list[DatasetModel], session:
Session) -> None:
+ def create_datasets(self, datasets: list[Dataset], *, session: Session) ->
list[DatasetModel]:
"""Create new datasets."""
- for dataset_model in dataset_models:
- session.add(dataset_model)
- for dataset_model in dataset_models:
- self.notify_dataset_created(dataset=Dataset(uri=dataset_model.uri,
extra=dataset_model.extra))
+
+ def _add_one(dataset: Dataset) -> DatasetModel:
+ model = DatasetModel.from_public(dataset)
+ session.add(model)
+ return model
+
+ models = [_add_one(d) for d in datasets]
+ for dataset in datasets:
+ self.notify_dataset_created(dataset=dataset)
Review Comment:
On second though, maybe not…? Flushing does not actually make the objects
visible (because transaction) so the listener hook still cannot “see” the
object in the database (because it does not have access to the session used to
insert the object). So the additional flush in the middle of the function does
not actually affect anything in the listener. I’ll combine the loops. In a
follow-up PR I’ll maybe also refactor `register_dataset_change`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
