dakshin-k commented on issue #30974:
URL: https://github.com/apache/airflow/issues/30974#issuecomment-1529325649
Hi, just to provide a bit more info on one of the use cases that inspired
this issue:
* We have two datasets, say A and B, both of which are provided by third
parties (and therefore fail occasionally)
* A is updated daily, B is updated weekly
* Dataset C is effectively a join of A and B
* Whenever A is updated, C needs to be updated
* We take the newly available A, and the _latest available_ version of B
to compute C
Our current plan to handle this is to mark only A as an inlet to C, and
handle the dependency on B within the DAG itself. Disadvantage would be that B
doesn't show up in the Datasets view at all, which we're not too happy about -
A and B are both crucial to the system, it's just that B has a higher TTL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
