Hashar has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/315214

Change subject: nodepool: lower throttling rate to OpenStack API
......................................................................

nodepool: lower throttling rate to OpenStack API

Whenever CI is busy, I have noticed instances taking up to 90 seconds to
be available when the ideal time is 40 seconds. The investigation on
T146813 showed that it is purely due to Nodepool throttling queries made
to the OpenStack API.

We previously dropped an unneeded queries (on each deletion, nodepool
listed the tenant floating IPs which we do not use). We still have a
plateau of tasks processing though.

The contention can be seen on the Nodepool Grafana board:
https://grafana.wikimedia.org/dashboard/db/nodepool

"Instance launch time" spikes way above the usual 40 seconds

"Tasks per minute for All providers" shows a plateau at 7 or 8 tasks per
minutes. That correspond to 1/rate or 7.5 tasks/minutes.

Lowering the rate from 8 to the arbitrary 6 offers 30% speed up at 10
tasks/minutes.  Will get us the instances ready slightly faster.

Change-Id: Ia2d303e16c071c542534148dab0f1f286218ead9
---
M modules/nodepool/templates/nodepool.yaml.erb
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/14/315214/1

diff --git a/modules/nodepool/templates/nodepool.yaml.erb 
b/modules/nodepool/templates/nodepool.yaml.erb
index 4d45cd6..f1927fe 100644
--- a/modules/nodepool/templates/nodepool.yaml.erb
+++ b/modules/nodepool/templates/nodepool.yaml.erb
@@ -77,7 +77,7 @@
     api-timeout: 60  # seconds
     boot-timeout: 300  # seconds
     max-servers: 12
-    rate: 8  # seconds
+    rate: 6  # seconds
     # 'eqiad.wmflabs' is magically added by wmflabs
     template-hostname: '{image.name}-{timestamp}'
     images:

-- 
To view, visit https://gerrit.wikimedia.org/r/315214
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia2d303e16c071c542534148dab0f1f286218ead9
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Hashar <has...@free.fr>

_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

Reply via email to