Lee-W commented on PR #70225: URL: https://github.com/apache/airflow/pull/70225#issuecomment-5104970292
Thanks for answering in details. But I'm still not convinced this should be a task-level thing. 1. Why couldn't we just split them into two Dags and let the downstream one be asset-scheduled? If a Dag needs both a time and an asset schedule, we also have `AssetOrTimeSchedule`. And for the "wait for N" cases, we have `RollupMapper` with a `WaitPolicy` (`MinimumCount` / `WaitForAll`). 2. And why do we need `extra` here? `partition_key` is the field we designed to be the discriminator — it's the indexed one, while `extra` is free-form metadata the scheduler doesn't index. If a producer needs to distinguish events by something, that something belongs in the partition key, and a custom `PartitionMapper` can decode it into whatever shape the downstream Dag needs. But my main concern is that this dependency is not known by Airflow. We can't schedule on it, and it doesn't show up in the lineage. The whole point of assets is that declaring a data dependency makes it visible to the platform. That works against what the feature is for. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
