potiuk commented on PR #41048: URL: https://github.com/apache/airflow/pull/41048#issuecomment-2257395873
This is a very bad idea IMHO. You should fix your database to not to close the connections and not implement "blanket" retries. This might leave your data inconsistent and if you read carefully the stackoverflow question you will see that this is absolutely not recommended by one of the sqlalchemy creators. Just to explicitly quote it here: > hi there - SQLAlchemy author here. I've just been pointed to this recipe. I would highly recommend against using a pattern such as the above. "retry" for connectivity should be performed at the top level of a transaction only, and that's what the pool pre_ping feature is for. if you lose a connection in the middle of a transaction, you need to re-run your whole operation. explicit is better than implicit. – [zzzeek](https://stackoverflow.com/users/34549/zzzeek) [CommentedOct 19, 2020 at 14:48](https://stackoverflow.com/questions/53287215/retry-failed-sqlalchemy-queries#comment113928088_53300049) And this is exactly what we do in Airflow - we have a number of cases where we explicity retry a transaction on failure - we even have special decorator for that - when we know we can retry a DB transaction safely - following this advice. By doing it you might end up with a number of problems - that will be impossible to diagnose and you wil be basically on your own scratching your head when it happens (please don't reach out for maintainers help in this case, because we won't be able to help you - other than telling the same - fix your database stability. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
