uranusjr commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687564898
##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
yield k, v
seen.add(k)
+ def iter_dag_deps(self, *, source: str, target: str) ->
Iterator[DagDependency]:
+ """
+ Iterate dataset, dataset aliases and their resolved datasets as dag
dependency.
+
+ :meta private:
+ """
+ dag_deps: set[DagDependency] = set()
Review Comment:
For DatasetAll and DatasetAny specifically, I don’t think there’s a
situation we need to keep the original user input. I’d even argue it’s an
oversight I did not implement dedup in `__and__` and `__or__` in the initial
release.
There are cases where we can’t dedup in all/any but need further cleanup in
the UI though, say `("a" | "b") & ("a" | "c")`. This cannot be simplified, but
we only want one line to `a` instead of two. But that’s a UI consideration that
should only be done either in the API layer or frontend (React).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
