Alexandros Kosiaris has submitted this change and it was merged.

Change subject: nodepool: lower throttling rate to OpenStack API
......................................................................


nodepool: lower throttling rate to OpenStack API

Whenever CI is busy, I have noticed instances taking up to 90 seconds to
be available when the ideal time is 40 seconds. The investigation on
T146813 showed that it is purely due to Nodepool throttling queries made
to the OpenStack API.

We previously dropped an unneeded queries (on each deletion, nodepool
listed the tenant floating IPs which we do not use). We still have a
plateau of tasks processing though.

The contention can be seen on the Nodepool Grafana board:
https://grafana.wikimedia.org/dashboard/db/nodepool

"Instance launch time" spikes way above the usual 40 seconds

"Tasks per minute for All providers" shows a plateau at 7 or 8 tasks per
minutes. That correspond to 1/rate or 7.5 tasks/minutes.

Lowering the rate from 8 to the arbitrary 6 offers 30% speed up at 10
tasks/minutes.  Will get us the instances ready slightly faster.

Change-Id: Ia2d303e16c071c542534148dab0f1f286218ead9
---
M modules/nodepool/templates/nodepool.yaml.erb
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Alexandros Kosiaris: Verified; Looks good to me, approved



diff --git a/modules/nodepool/templates/nodepool.yaml.erb 
b/modules/nodepool/templates/nodepool.yaml.erb
index 4d45cd6..f1927fe 100644
--- a/modules/nodepool/templates/nodepool.yaml.erb
+++ b/modules/nodepool/templates/nodepool.yaml.erb
@@ -77,7 +77,7 @@
     api-timeout: 60  # seconds
     boot-timeout: 300  # seconds
     max-servers: 12
-    rate: 8  # seconds
+    rate: 6  # seconds
     # 'eqiad.wmflabs' is magically added by wmflabs
     template-hostname: '{image.name}-{timestamp}'
     images:

-- 
To view, visit https://gerrit.wikimedia.org/r/315214
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia2d303e16c071c542534148dab0f1f286218ead9
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Hashar <has...@free.fr>
Gerrit-Reviewer: Alexandros Kosiaris <akosia...@wikimedia.org>
Gerrit-Reviewer: Andrew Bogott <abog...@wikimedia.org>
Gerrit-Reviewer: Chasemp <r...@wikimedia.org>
Gerrit-Reviewer: jenkins-bot <>

_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

Reply via email to