potiuk commented on issue #39332: URL: https://github.com/apache/airflow/issues/39332#issuecomment-2508942907
Just one comment here. I think it's worth looking at the description of the issue first and understand that - not all issues are the same even if log messages are similar. This isssue has nothing to do with regular ways how airflow dag file processor runs in production. The "--num-runs" flag is really a "test" flag that should not be used in production, really. It's a remnant of Airlfow 1 where we had to recycle scheduler - and you are not supposed to use it in production. It's not the intention of this flag (and yes it's behaviour should like be fixed - or more likely we should remove this flag completely because yes @kosteev is right that this has those kinds of effects he mentioned here). I added a detailed explanation in https://github.com/apache/airflow/discussions/44495#discussioncomment-11421723 to explain what and how you can take look at to see the root cause of the problem. I would love if everyone who has similar issue follows it and post their finding on what kind of problem they found with their system and how they fixed it. I don't believe there is a consistent issue in Airflow, it's more set of not following the best practices, hiding problems with the way how DAGs are written, or synced or some similar issues. But having a result of the investigation and possible issues that users could have, following my outline and "investigation scenario" that I put there might help us to see if we can add better guidelines or maybe even better warnings/messages for similar reasons: So if I might ask you: * @vghar-bh * @raphaelauv * @AlexHam * @rtrindvg Please follow the https://github.com/apache/airflow/discussions/44495#discussioncomment-11421723 and post THERE (this is a different issue altogether as I explained above) results of your investigation following my guidelines. Thanks in advance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
