cmarteepants commented on issue #76979: URL: https://github.com/apache/airflow/issues/76979#issuecomment-6833027935
We have a user hitting the first case in production on 3.3.0. They added a new upstream to an existing partitioned consumer, which is a pretty normal pipeline change, and the in-flight partition never ran. The most painful aspect of this is that there's no indication something is wrong since a run is just not created. They ended up building their own monitoring to flag asset-triggered Dags that haven't run in the expected window and fix each one by hand. Republishing the event works, but then they have to go clean up all the other consumers of that asset. +1 to a consumer-scoped publish like `consumer_dag_id`. That would at least turn their manual fix into an API call. That said, I think detection is the bigger gap. Even with scoped recovery, you still don’t know a partition is stuck unless you’ve built your own monitoring. I’ll open a separate issue for that so this one stays focused on recovery. cc @Lee-W and @hussein-awala since this is close to #76463 #76706 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
