Yuvipanda has submitted this change and it was merged.
Change subject: labs: Centralize config of which projects have NFS enabled
......................................................................
labs: Centralize config of which projects have NFS enabled
This introduces:
- A config file that is the canonical source of truth about
which projects have which NFS mounts enabled.
- A puppet function (nfs_volume_mounted) which can be called
to check if a particular volume should be mounted in a
particular project.
This replaces the current setup, where the data is contained in
LDAP (set via 'configure' in wikitech) and in hiera (to set
which volumes puppet will actually try to mount).
This also will introduce policy changes:
- New projects will not get NFS by default anymore
- Opting into NFS requires that the project owner pokes
someone from the Ops team and explains their rationale.
This should be reflected in the instructions for requesting
a new project at T76375
Follow up changes:
- Update manage-nfs-volumes deamon to read from this config
file than from LDAP
- Continue auditing projects, removing NFS when not needed
(See T102240)
Bug: T102403
Change-Id: I79b71ccd1b1b7c31aa28d590dcc82a9332a28928
---
M manifests/role/labs.pp
A modules/openstack/files/nfs-mounts-config.yaml
A modules/openstack/lib/puppet/parser/functions/mount_nfs_volume.rb
3 files changed, 821 insertions(+), 8 deletions(-)
Approvals:
Yuvipanda: Verified; Looks good to me, approved
diff --git a/manifests/role/labs.pp b/manifests/role/labs.pp
index 8e5fb87..99e2756 100644
--- a/manifests/role/labs.pp
+++ b/manifests/role/labs.pp
@@ -35,13 +35,11 @@
ensure => present,
}
- $nfs_mounts = hiera('nfs_mounts')
-
$nfs_opts = 'vers=4,bg,hard,intr,sec=sys,proto=tcp,port=0,noatime,nofsc'
$nfs_server = 'labstore.svc.eqiad.wmnet'
$dumps_server = 'labstore1003.eqiad.wmnet'
- if $nfs_mounts['home'] {
+ if mount_nfs_volume($::instanceproject, 'home') {
mount { '/home':
ensure => mounted,
atboot => true,
@@ -52,7 +50,7 @@
}
}
- if $nfs_mounts['project'] or $nfs_mounts['scratch'] {
+ if mount_nfs_volume($::instanceproject, 'project') or
mount_nfs_volume($::instanceproject, 'scratch') {
# Directory for data mounts
file { '/data':
ensure => directory,
@@ -62,7 +60,7 @@
}
}
- if $nfs_mounts['project'] {
+ if mount_nfs_volume($::instanceproject, 'project') {
file { '/data/project':
ensure => directory,
require => File['/data'],
@@ -78,7 +76,7 @@
}
}
- if $nfs_mounts['scratch'] {
+ if mount_nfs_volume($::instanceproject, 'scratch') {
file { '/data/scratch':
ensure => directory,
require => File['/data'],
@@ -95,7 +93,7 @@
}
# Only create if we need /public/dumps or /public/keys
- if $nfs_mounts['dumps'] or os_version('ubuntu <= precise') {
+ if mount_nfs_volume($::instanceproject, 'dumps') or os_version('ubuntu <=
precise') {
# Directory for public (readonly) mounts
file { '/public':
ensure => directory,
@@ -105,7 +103,7 @@
}
}
- if $nfs_mounts['dumps'] {
+ if mount_nfs_volume($::instanceproject, 'dumps') {
file { '/public/dumps':
ensure => directory,
require => File['/public'],
diff --git a/modules/openstack/files/nfs-mounts-config.yaml
b/modules/openstack/files/nfs-mounts-config.yaml
new file mode 100644
index 0000000..31d6858
--- /dev/null
+++ b/modules/openstack/files/nfs-mounts-config.yaml
@@ -0,0 +1,785 @@
+account-creation-assistance:
+ dumps: false
+ home: false
+ project: true
+ scratch: false
+akosiaristests:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+analytics:
+ dumps: false
+ home: false
+ project: true
+ scratch: false
+bastion:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+bots:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+butterfly:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+catgraph:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+cephtest:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+chasetest:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+collection-alt-renderer:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+commonsarchive:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+contintcloud:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+contributors:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+conventionextension:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+crisiswiki:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+cvn:
+ dumps: false
+ home: false
+ project: true
+ scratch: false
+cvresearch:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+deployment-prep:
+ dumps: false
+ home: false
+ project: true
+ scratch: false
+design:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+developer-doc:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+dns:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+dumps:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+dwl:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+editor-engagement:
+ dumps: false
+ home: false
+ project: true
+ scratch: false
+embed-sandbox:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+etcd:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+extdist:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+fastcci:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+fatg:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+full-text-reference-tool:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+fundraising:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+gather:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+gerrit:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+glam:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+globaleducation:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+grantreview:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+graphite:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+gsoc2014-fonttailor-demo:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+hat-imagescalers:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+hhvm:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+huggle:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+icinga:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+importarticles:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+integration:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+ircnotifier:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+jawiki:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+jupyter:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+language:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+logstash:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+mailman:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+maps:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+maps-team:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+marathon:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+math:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+mediahandler-tests:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+mediawiki-api:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+mediawiki-core-team:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+mediawiki-dev:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+mediawiki-verp:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+megacron:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+mobile:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+mobile-smoketests:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+monitoring:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+multimedia:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+mwoffliner:
+ dumps: false
+ home: false
+ project: false
+ scratch: true
+mwreview:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+netflow:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+newsletter:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+nginx:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+oa-signalling:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+ogvjs-integration:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+opengrok:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+openid:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+openstack:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+orgcharts:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+osm4wiki:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+osmit:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+otrs:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+oxygenguide:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+packaging:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+pagemigration:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+patchtest:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+pdbhandler:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+performance:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+persistence:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+persona:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+phabricator:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+phragile:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+piwik:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+planet:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+project-proxy:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+propertysuggester:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+pubsubhubbub:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+puppet:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+puppet-ca-replacement:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+puppet-cleanup:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+puppet3-diffs:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+quality-assurance:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+quarry:
+ dumps: false
+ home: false
+ project: true
+ scratch: false
+rdfiodev:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+reportcard:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+revscoring:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+safesandbox:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+scrumbugz:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+search:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+security-tools:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+sensu:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+services:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+services-testbed:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+shinken:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+shiny-r:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+signwriting:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+snuggle:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+social-tools:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+spam-honeypot:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+staging:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+structured-wikiquote:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+sugarcrm:
+ dumps: false
+ home: false
+ project: true
+ scratch: false
+swift:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+test-twemproxy:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+testlabs:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+tools:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+toolsbeta:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+toolserver-legacy:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+tor:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+translatesvg:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+trebuchet:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+ttmserver:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+uploadwizard-osm-embedding:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+utrs:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+varnish:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+ve-languagetool:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+video:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+visualeditor:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+wdq-mm:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+webplatform:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+wikibrain:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+wikidata-build:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+wikidata-dev:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+wikidata-page-banner:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+wikidata-quality:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+wikidata-query:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+wikidata-topicmaps:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+wikidumpparse:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+wikimania-support:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+wikisource-dev:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+wikisource-tools:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+wikispy:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+wikistats:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+wikistream:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+wikiviajesve:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+wikivoyage:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+wildcat:
+ dumps: true
+ home: true
+ project: true
+ scratch: true
+wlmjudging:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+wlmjurytool:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
+wmt:
+ dumps: false
+ home: false
+ project: true
+ scratch: false
+xtools:
+ dumps: false
+ home: false
+ project: false
+ scratch: false
diff --git a/modules/openstack/lib/puppet/parser/functions/mount_nfs_volume.rb
b/modules/openstack/lib/puppet/parser/functions/mount_nfs_volume.rb
new file mode 100644
index 0000000..c51b416
--- /dev/null
+++ b/modules/openstack/lib/puppet/parser/functions/mount_nfs_volume.rb
@@ -0,0 +1,30 @@
+# == Function: mount_nfs_volume( project, mount )
+#
+# Copyright (c) 2015 Wikimedia Foundation Inc.
+#
+# Returns true if the mount should be mounted in
+# instance of the project
+#
+# Reads this information from openstack/files/nfs-mounts-config.yaml
+# in the openstack module of operations/puppet.git
+module Puppet::Parser::Functions
+ @@labs_nfs_config_touched = nil
+ @@labs_nfs_config = nil
+ newfunction(:mount_nfs_volume, :type => :rvalue, :arity => 2) do |args|
+ module_path = function_get_module_path(['openstack'])
+ path = "#{module_path}/files/nfs-mounts-config.yaml"
+ mtime = File.stat(path).mtime
+ if @@labs_nfs_config_touched.nil? || mtime != @@labs_nfs_config_touched
+ @@labs_nfs_config = function_loadyaml([path])
+ @@labs_nfs_config_touched = mtime
+ end
+ config = @@labs_nfs_config
+ project = args[0]
+ mount = args[1]
+ if config.has_key? project and config[project].has_key? mount
+ config[project][mount]
+ else
+ false
+ end
+ end
+end
--
To view, visit https://gerrit.wikimedia.org/r/218637
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I79b71ccd1b1b7c31aa28d590dcc82a9332a28928
Gerrit-PatchSet: 22
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Yuvipanda <[email protected]>
Gerrit-Reviewer: Andrew Bogott <[email protected]>
Gerrit-Reviewer: Faidon Liambotis <[email protected]>
Gerrit-Reviewer: Giuseppe Lavagetto <[email protected]>
Gerrit-Reviewer: Mark Bergsma <[email protected]>
Gerrit-Reviewer: Mobrovac <[email protected]>
Gerrit-Reviewer: Ryan Lane <[email protected]>
Gerrit-Reviewer: Tim Landscheidt <[email protected]>
Gerrit-Reviewer: Yuvipanda <[email protected]>
Gerrit-Reviewer: coren <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
