slords commented on issue #41101: URL: https://github.com/apache/airflow/issues/41101#issuecomment-2263043273
The issue is that DAGs are triggering even though not all of the datasets have been updated. Sometimes the DAGs trigger successfully, but other times they don't. This makes it really hard to figure out what's wrong. To fix this, we need to find the logs that show why the DAG was scheduled even though not all the datasets had updates. MWAA doesn't let you set the debug flag on any log, only INFO. I'm working with AWS to see if there is an internal way to turn on DEBUG so we can get to these logs. You don't need to know about my setup to see what the problem is. I've attached some screenshots that clearly show which datasets are required and that, on two of the five occasions that the DAG ran, it didn't have all the required datasets available. I understand how open source works and I know that most people won't raise bugs. But, I've been contributing to open source projects for years, and I firmly believe that any issue I can help fix makes the product better for everyone. Filing a bug report is a good way to get an issue identified and hopefully resolved. It's also possible that others are experiencing the same problem but haven't said anything yet. If they see this bug report, they can confirm the issue and maybe even provide some additional information that will help us find a solution. I was disappointed by the unhelpful and dismissive attitude that came across in your response. Not everyone has the same level of technical expertise or resources, and it can be frustrating to be made to feel like a nuisance. Just because I don't have all the answers doesn't mean my bug report isn't valid. I'm providing as much information and context as I can, and I'm happy to cooperate with anyone who wants to help troubleshoot the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
