Bowrna commented on issue #36516: URL: https://github.com/apache/airflow/issues/36516#issuecomment-2014663145
> > One possible option could be to publish the DAG state metric. Consumers can monitor the DAG's last state from the metrics and create alerts around it. However, this option doesn't provide a reaction to the state change. > > Correct - my initial proposal in the discussion is active checking via API - metrics is almost the same but we would have to have _something_ gathering and publishing the metrics regularly (still you'd have to monitor the metrics and react). In order to implement this one you'd have to have something regularly checking state of DAGs. > > One of the options would be to build it in DAGFileProcessor. It already parses all dags regularly, so if we store state observed during last parsing, DAGFileProcessor could fire a notification when the state changed from last parsing. A bit racy approach and would have to be carefully done to account for potential state changes while the file is being parsed, but generally doable I think. Approach: 1. Checking in the DAGFileProcessor about the state of the DAG and on noticing change of the state, hook a notification and maintain a state that its notified along with the time at which its notified( or say time at which the state change is seen). - this is active verifying in the DAGFileProcessor side and its like polling technique 2. ListenerPlugin approach - Watching when the DB gets a state change message and hooking a notification as a result of it. Though this seems like cleaner approach, have to check in depth on which one would be feasible. Let me make a elaborate plan and present it with the findings I have collected based on the above 2 approach -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
