Andrew Bogott has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/385939 )
Change subject: nfs: increase timeouts when checking mount availability
......................................................................
nfs: increase timeouts when checking mount availability
Puppet continues to flap occasionally on hosts that use nfs,
typically because a high IO load is causing the mounts
to timeout (which causes puppet to error out).
In many cases this IO slowness is a feature rather than a bug
(it's caused by per-instance throttling that we've put in on
purpose to prevent NFS overload). So, let's just be patient
and try not to alert when things are slow.
Puppet has a full 30 minutes to complete a run so I wouldn't
mind increasing these even further, to 2 or even 5 minutes.
Change-Id: I2ccab28115fde1987e2781d99ed100e7ef43646a
---
M modules/labstore/manifests/nfs_mount.pp
M modules/toollabs/manifests/init.pp
2 files changed, 3 insertions(+), 3 deletions(-)
Approvals:
Andrew Bogott: Looks good to me, approved
Rush: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/modules/labstore/manifests/nfs_mount.pp
b/modules/labstore/manifests/nfs_mount.pp
index 5ee1bad..6fc0af6 100644
--- a/modules/labstore/manifests/nfs_mount.pp
+++ b/modules/labstore/manifests/nfs_mount.pp
@@ -179,8 +179,8 @@
# Puppet will normally get stuck and freeze raising load and
effectively
# failing to run
exec { "create-${mount_path}":
- command => "/usr/bin/timeout -k 5s 10s /bin/mkdir ${mount_path}",
- unless => "/usr/bin/timeout -k 5s 30s /usr/bin/test -d
${mount_path}",
+ command => "/usr/bin/timeout -k 5s 20s /bin/mkdir ${mount_path}",
+ unless => "/usr/bin/timeout -k 5s 60s /usr/bin/test -d
${mount_path}",
logoutput => true,
require => Mount[$mount_path],
}
diff --git a/modules/toollabs/manifests/init.pp
b/modules/toollabs/manifests/init.pp
index e859898..146df24 100644
--- a/modules/toollabs/manifests/init.pp
+++ b/modules/toollabs/manifests/init.pp
@@ -34,7 +34,7 @@
exec {'ensure-grid-is-on-NFS':
command => '/bin/false',
- unless => "/usr/bin/timeout -k 5s 30s /usr/bin/test -e
${project_path}/herald",
+ unless => "/usr/bin/timeout -k 5s 60s /usr/bin/test -e
${project_path}/herald",
}
file { $sysdir:
--
To view, visit https://gerrit.wikimedia.org/r/385939
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I2ccab28115fde1987e2781d99ed100e7ef43646a
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Andrew Bogott <[email protected]>
Gerrit-Reviewer: Andrew Bogott <[email protected]>
Gerrit-Reviewer: Chasemp <[email protected]>
Gerrit-Reviewer: Madhuvishy <[email protected]>
Gerrit-Reviewer: Merlijn van Deen <[email protected]>
Gerrit-Reviewer: Rush <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
