dstandish commented on code in PR #28440:
URL: https://github.com/apache/airflow/pull/28440#discussion_r1064944840
##########
airflow/cli/commands/task_command.py:
##########
@@ -390,11 +400,19 @@ def task_run(args, dag=None):
log.info("Running %s on host %s", ti, hostname)
+ # IMPORTANT, have to re-configure ORM with the NullPool, otherwise, each
"run" command may leave
+ # behind multiple open sleeping connections while heartbeating, which could
+ # easily exceed the database connection limit when
+ # processing hundreds of simultaneous tasks.
+ # this should be last thing before running, to reduce likelihood of an
open session
+ # which can cause trouble if running process in a fork.
+ settings.reconfigure_orm(disable_connection_pool=True)
Review Comment:
i don't *think* this will fail if we remove this change. you could call it
a drive by and i can remove if you like.
task runner tests failed with this change. then i was like "oh why is it
just throwing away whole log config, let's try and not do that". then after
leaving the FTH intact for that test, that i discovered there was an extraneous
session being created in created in _render_filename. as part of
troubleshooting the failures and discovering that though, also saw that this
reconfigure is happening earlier than it probably ought to. should be done as
late as possible i think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
