Lee-W commented on PR #63848: URL: https://github.com/apache/airflow/pull/63848#issuecomment-4116476271
> I have a use case question. Say I have a downstream workflow that wants to react to partitioned upstreams, but doesn’t really care about the particular partitions (maybe to do some global calculations or cache busting etc). Is there a way to do that, or is the downstream forced to have some kind of partition even if its tasks (or even further downstream workflows that depend on its asset events) just ignore it? I think we'll need some kind of partition for that > Or maybe the design question is more fundamental. An asset event means the asset has changed in some way, and there are really two ways you may want to be notified when change happens: Only on a “full” change (e.g. file replacement, deletion), or on any change. The former kind shouldn’t trigger on a partitioned update, but the latter would want to. Is there a way to distinguish the two? In the current design, we only have full/non-partitioned and partitioned. We do not yet have a way to listen to both. > When should this distinguishing logic happen? On the upstream (when an asset event is emitted), or the downstream (when listening to an asset)? If we were to have this, I think it should be handled in the downstream instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
