Colin Watson has proposed merging ~cjwatson/launchpad:buildd-manager-file-limit into launchpad:master.
Commit message: Update buildd-manager's open file descriptor limit Requested reviews: Launchpad code reviewers (launchpad-reviewers) For more details, see: https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/413048 We added 30 new builders to Launchpad's production build farm today (for a total of 348), after which we started seeing "Too many open files" from buildd-manager. This is because the value we set for `RLIMIT_NOFILE` was tuned for the old in-process download strategy, which we replaced in July 2020. Re-tune it for the new process-pool-based download strategy. I also updated the description and default for `config.builddmaster.download_connections`, which were similarly based on the old in-process download strategy. The value on production was already set to 128. -- Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:buildd-manager-file-limit into launchpad:master.
diff --git a/daemons/buildd-manager.tac b/daemons/buildd-manager.tac index 2612d9a..015b877 100644 --- a/daemons/buildd-manager.tac +++ b/daemons/buildd-manager.tac @@ -27,9 +27,12 @@ dbconfig.override(dbuser='buildd_manager', isolation_level='read_committed') # Should be removed from callsites verified to not need it. set_immediate_mail_delivery(True) -# Allow generous slack for database connections, idle download connections, -# etc. -soft_nofile = config.builddmaster.download_connections + 1024 +# ampoule uses five file descriptors per subprocess (i.e. +# 5 * config.builddmaster.download_connections); we also need at least three +# per active builder for resuming virtualized builders or making XML-RPC +# calls, and we also need to allow slack for odds and ends like database +# connections. +soft_nofile = 5 * config.builddmaster.download_connections + 2048 _, hard_nofile = resource.getrlimit(resource.RLIMIT_NOFILE) resource.setrlimit(resource.RLIMIT_NOFILE, (soft_nofile, hard_nofile)) diff --git a/lib/lp/services/config/schema-lazr.conf b/lib/lp/services/config/schema-lazr.conf index 73bfb70..ef17be9 100644 --- a/lib/lp/services/config/schema-lazr.conf +++ b/lib/lp/services/config/schema-lazr.conf @@ -67,9 +67,9 @@ socket_timeout: 40 # datatype: integer virtualized_socket_timeout: 30 -# The maximum number of file download connections that may be open -# across all builders. -download_connections: 2048 +# The maximum number of child processes to run for downloading files from +# builders. +download_connections: 128 # Activate the Build Notification system. # datatype: boolean
_______________________________________________ Mailing list: https://launchpad.net/~launchpad-reviewers Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-reviewers More help : https://help.launchpad.net/ListHelp
