dstandish commented on code in PR #24908:
URL: https://github.com/apache/airflow/pull/24908#discussion_r917273468
##########
airflow/models/dataset.py:
##########
@@ -199,3 +199,86 @@ def __repr__(self):
for attr in [x.name for x in self.__mapper__.primary_key]:
args.append(f"{attr}={getattr(self, attr)!r}")
return f"{self.__class__.__name__}({', '.join(args)})"
+
+
+class DatasetEvent(Base):
+ """
+ A table to store datasets events.
+
+ :param dataset_id: reference to Dataset record
+ :param extra: JSON field for arbitrary extra info
+ :param source_task_id: the task_id of the TI which updated the dataset
+ :param source_dag_id: the dag_id of the TI which updated the dataset
+ :param source_run_id: the run_id of the TI which updated the dataset
+ :param source_map_index: the map_index of the TI which updated the dataset
+
+ We use relationships instead of foreign keys so that dataset events are
not deleted even
+ if the foreign key object is.
+ """
+
+ id = Column(Integer, primary_key=True, autoincrement=True)
+ dataset_id = Column(Integer, nullable=False)
+ extra = Column(ExtendedJSON, nullable=True)
+ source_task_id = Column(StringID(), nullable=True)
+ source_dag_id = Column(StringID(), nullable=True)
+ source_run_id = Column(StringID(), nullable=True)
+ source_map_index = Column(Integer, nullable=True,
server_default=text("-1"))
Review Comment:
And I guess, the idea is just to make it clear, when looking at the dataset
event, and when we see the references to a dag_id, for instance, it's clear
we're referring to "the dag that wrote to the dataset".
E.g. when you are in the downstream TI that is processing the dataset
update, and you are looking at this dataset event object (e.g. to see what was
done to the dataset), and you see dag or task references on it -- will it be
clear enough that those references mean "this is the dag / task that updated
the dataset".
Probably so but let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
