iting0321 commented on PR #4826:
URL: https://github.com/apache/polaris/pull/4826#issuecomment-5004735687
Hi @dimas-b, I changed
```java
replaceDatasetEdges(List<LineageEdge> edges, Instant eventTime)
```
to
```java
replaceDatasetEdges(
List<LineageDataset> targetDatasets,
List<LineageEdge> edges,
Instant eventTime)
```
The reason is that OpenLineage allows events like:
```text
inputs = []
outputs = [C]
```
In that case, the derived dataset edges are empty:
```text
edges = []
```
However, this is different from an event with no asserted output snapshot.
It means that `C` is the target dataset, and its current upstream snapshot is
empty. In other words, any existing upstream edges into `C` should be removed.
With only `edges`, the persistence layer cannot distinguish between:
```text
targetDatasets = [C], edges = []
```
meaning “replace C’s upstream snapshot with an empty set,” and:
```text
targetDatasets = [], edges = []
```
meaning no target snapshot was asserted.
So `targetDatasets` defines the replacement scope, while `edges` defines the
replacement content. Thanks!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
