nathadfield commented on PR #70225: URL: https://github.com/apache/airflow/pull/70225#issuecomment-5176492049
@Lee-W We hit this exact tension migrating some pipelines to partitioned assets, a concrete data point. Take a simple linear DAG: task A runs, then task B needs an asset produced by a different DAG before it can proceed, then task C runs. Today that's A -> ExternalTaskSensor -> B -> C, and A runs immediately regardless of whether the other DAG's data is ready. Forcing that dependency onto the DAG's own schedule would delay A too, a real regression, since A never needed it. Splitting A into its own DAG doesn't help either: A, B, and C are one logical unit of work that should be run, observed, and retried together, and splitting them apart only serves the scheduler's DAG-level-only asset constraint, not any real independence between them. A task-level sensor between A and B is the right fit, and this shape isn't rare. On the visibility concern: could the sensor also set inlets for the asset(s) it waits on? inlets/outlets already drive the lineage graph independent of scheduling, so the dependency stays visible to the platform without tying it to what triggers the DAG run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
