jenkins-bot has submitted this change and it was merged.
Change subject: Introduce foreachwikiwithextension
......................................................................
Introduce foreachwikiwithextension
Follow up to I2ad8cc3. Now that it is possible for a wiki to opt-out of
enabling a globally installed extension we need a way to also exclude
those wikis from `foreachwiki` scripts. In the Wikimedia production
environment, this would be done using `foreachwikiindblist` and manually
curated dblist files. In MediaWiki-Vagrant we'd like thing to be a bit
more automated. This change introduces a new `wikihasextension` helper
script that can generate a dblist of wikis with a given extension
enabled. This is paired with the new `foreachwikiwithextension` script
which is a `foreachwiki` work-alike that takes an extension,
a maintenance script and optional additional arguments.
The extensions active for a wiki are tracked in a new
`$wmvActiveExtensions` global variable as part of the actions of the
mediawiki::extension Puppet module. Existing uses of `foreachwiki` that
depend on a particular extension being installed have been updated to
use `foreachwikiwithextension` instead.
All of this will fix problems found when combining role::centralauth and
role::ldapauth in a single MediaWiki-Vagrant VM.
Bug: T128501
Change-Id: I7b2bf3d25ec148a0e949159178bf2616b5739c82
---
M puppet/modules/mediawiki/manifests/multiwiki.pp
M puppet/modules/mediawiki/templates/extension.php.erb
M puppet/modules/mediawiki/templates/multiwiki/CommonSettings.php.erb
M puppet/modules/mediawiki/templates/multiwiki/LoadWgConf.php.erb
M puppet/modules/mediawiki/templates/multiwiki/foreachwiki.erb
A puppet/modules/mediawiki/templates/multiwiki/foreachwikiwithextension.erb
A puppet/modules/mediawiki/templates/multiwiki/wikihasextension.erb
M puppet/modules/mediawiki/templates/wiki/dbConf.php.erb
M puppet/modules/role/manifests/antispoof.pp
M puppet/modules/role/manifests/centralauth.pp
M puppet/modules/role/manifests/cirrussearch.pp
M puppet/modules/role/manifests/commons.pp
M puppet/modules/role/manifests/echo.pp
M puppet/modules/role/manifests/geodata_elastic.pp
M puppet/modules/role/manifests/wikidata.pp
M
puppet/modules/role/templates/centralauth/is-centralauth-migratePass0-needed.bash.erb
A puppet/modules/role/templates/cirrussearch/build_search_index.erb
A
puppet/modules/role/templates/cirrussearch/is-cirrussearch-forceindex-needed.erb
18 files changed, 167 insertions(+), 22 deletions(-)
Approvals:
Anomie: Looks good to me, but someone else must approve
Dduvall: Looks good to me, approved
jenkins-bot: Verified
diff --git a/puppet/modules/mediawiki/manifests/multiwiki.pp
b/puppet/modules/mediawiki/manifests/multiwiki.pp
index cbb76f4..73fe929 100644
--- a/puppet/modules/mediawiki/manifests/multiwiki.pp
+++ b/puppet/modules/mediawiki/manifests/multiwiki.pp
@@ -248,6 +248,22 @@
content => template('mediawiki/multiwiki/foreachwiki.erb'),
}
+ file { '/usr/local/bin/foreachwikiwithextension':
+ ensure => present,
+ owner => 'root',
+ group => 'root',
+ mode => '0755',
+ content =>
template('mediawiki/multiwiki/foreachwikiwithextension.erb'),
+ }
+
+ file { '/usr/local/bin/wikihasextension':
+ ensure => present,
+ owner => 'root',
+ group => 'root',
+ mode => '0755',
+ content => template('mediawiki/multiwiki/wikihasextension.erb'),
+ }
+
file { '/usr/local/bin/sql':
ensure => link,
target => '/usr/bin/mysql',
diff --git a/puppet/modules/mediawiki/templates/extension.php.erb
b/puppet/modules/mediawiki/templates/extension.php.erb
index e648e74..bd5a956 100644
--- a/puppet/modules/mediawiki/templates/extension.php.erb
+++ b/puppet/modules/mediawiki/templates/extension.php.erb
@@ -13,3 +13,6 @@
include_once "$IP/extensions/<%= @ext_name %>/<%=
@ext_entrypoint %>";
}
+ // Keep track of what extensions are loaded to make `wikihasextension`
work
+ // Why in the hell doesn't MediaWiki already do this for us?
+ $wmvActiveExtensions[] = '<%= @ext_name %>';
diff --git
a/puppet/modules/mediawiki/templates/multiwiki/CommonSettings.php.erb
b/puppet/modules/mediawiki/templates/multiwiki/CommonSettings.php.erb
index d7c518f..e310212 100644
--- a/puppet/modules/mediawiki/templates/multiwiki/CommonSettings.php.erb
+++ b/puppet/modules/mediawiki/templates/multiwiki/CommonSettings.php.erb
@@ -60,6 +60,13 @@
$wgServer = preg_replace( '#^[^\/]+#', '', WebRequest::detectServer() );
}
+/**
+ * @var array $wmvActiveExtensions List of extensions enabled for the wiki
+ * Populated by settings.d files generated by ::mediawiki::extension and
+ * used to power the `wikihasextension` script.
+ */
+$wmvActiveExtensions = array();
+
foreach(
array_merge(
// Settings from default wiki
diff --git a/puppet/modules/mediawiki/templates/multiwiki/LoadWgConf.php.erb
b/puppet/modules/mediawiki/templates/multiwiki/LoadWgConf.php.erb
index 1198878..1d5083a 100644
--- a/puppet/modules/mediawiki/templates/multiwiki/LoadWgConf.php.erb
+++ b/puppet/modules/mediawiki/templates/multiwiki/LoadWgConf.php.erb
@@ -2,8 +2,14 @@
// This file is managed by Puppet.
// See puppet/modules/mediawiki/templates/multiwiki/LoadWgConf.php.erb
-$wgCentralAuthAutoLoginWikis = array();
+// Globals that are populated by *dbConf.php
+/** @var array $wgLocalDatabases List of all wikis in this wiki farm */
$wgLocalDatabases = array();
+/** @var array $wgCentralAuthAutoLoginWikis Dict of wiki_name => db_name */
+$wgCentralAuthAutoLoginWikis = array();
+/** @var array $wgMediawikiRoot Dict of db_name => src_dir */
+$wgMediawikiRoot = array();
+
foreach ( glob( '<%= @wiki_priority_dir %>/*dbConf.php' ) as $file ) {
// Use require rather than require_once as something in the request may
have
// loaded the dbConf files previously.
diff --git a/puppet/modules/mediawiki/templates/multiwiki/foreachwiki.erb
b/puppet/modules/mediawiki/templates/multiwiki/foreachwiki.erb
index 8b30d83..fa0a0a4 100755
--- a/puppet/modules/mediawiki/templates/multiwiki/foreachwiki.erb
+++ b/puppet/modules/mediawiki/templates/multiwiki/foreachwiki.erb
@@ -1,18 +1,34 @@
#!/usr/bin/env bash
+# Run an mwscript command against all wikis.
# Ensure that the script is run as the www-data user
[[ $(whoami) = www-data ]] || exec sudo -u www-data -- "$0" "$@"
+function usage() {
+ echo "Usage: $(basename $0) <command>..." >&2
+ echo " command: command and optional arguments" >&2
+ echo >&2
+ exit
+}
+
+if [[ $# -lt 1 ]]; then
+ usage
+fi
+
CMD=$1
shift
+
+if [[ "$CMD" == "-h" || "$CMD" == "--help" ]]; then
+ usage
+fi
FAILURES=0
for db in $(alldbs); do
echo -----------------------------------------------------------------
echo $db
echo -----------------------------------------------------------------
- mwscript "$CMD" --wiki=$db "$@" |
- sed -u "s/^/$db: /"
+ /usr/local/bin/mwscript "$CMD" --wiki=$db "$@" |
+ /bin/sed -u -e "s/^/${db}: /"
if [[ ${PIPESTATUS[0]} -ne 0 ]]; then
((FAILURES++))
diff --git
a/puppet/modules/mediawiki/templates/multiwiki/foreachwikiwithextension.erb
b/puppet/modules/mediawiki/templates/multiwiki/foreachwikiwithextension.erb
new file mode 100755
index 0000000..5ea3fd7
--- /dev/null
+++ b/puppet/modules/mediawiki/templates/multiwiki/foreachwikiwithextension.erb
@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+# Run an mwscript command against all wikis that have a given extension
+# enabled.
+#
+# Example:
+# foreachwikiwithextension CentralAuth \
+# extensions/CentralAuth/maintenance/migratePass0.php
+
+# Ensure that the script is run as the www-data user
+[[ $(whoami) = www-data ]] || exec sudo -u www-data -- "$0" "$@"
+
+function usage() {
+ echo "Usage: $(basename $0) <extension> <command>..." >&2
+ echo " extension: MediaWiki extension name" >&2
+ echo " command: command and optional arguments" >&2
+ echo >&2
+ exit
+}
+
+if [[ $# -lt 2 ]]; then
+ usage
+fi
+
+EXT=$1
+CMD=$2
+shift
+shift
+
+if [[ "$EXT" == "-h" || "$EXT" == "--help" ]]; then
+ usage
+fi
+
+FAILURES=0
+for db in $(/usr/local/bin/wikihasextension "$EXT"); do
+ echo -----------------------------------------------------------------
+ echo $db
+ echo -----------------------------------------------------------------
+ /usr/local/bin/mwscript "$CMD" --wiki="$db" "$@" |
+ /bin/sed -u -e "s/^/${db}: /"
+
+ if [[ ${PIPESTATUS[0]} -ne 0 ]]; then
+ ((FAILURES++))
+ fi
+done
+
+if [[ $FAILURES -ne 0 ]]; then
+ echo "$FAILURES wikis failed." >&2
+ exit 1
+fi
diff --git a/puppet/modules/mediawiki/templates/multiwiki/wikihasextension.erb
b/puppet/modules/mediawiki/templates/multiwiki/wikihasextension.erb
new file mode 100755
index 0000000..304470e
--- /dev/null
+++ b/puppet/modules/mediawiki/templates/multiwiki/wikihasextension.erb
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Find all local wikis that have a given extension enabled.
+# This can be used like a dblist to find a collection of wikis that should
+# have some maintenance script run.
+
+# Ensure that the script is run as the www-data user
+[[ $(whoami) = www-data ]] || exec sudo -u www-data -- "$0" "$@"
+
+function usage() {
+ echo "Usage: $(basename $0) <extension>" >&2
+ echo " extension: MediaWiki extension name" >&2
+ echo >&2
+ exit
+}
+
+if [[ $# -lt 1 ]]; then
+ usage
+fi
+
+EXT=$1
+shift
+
+if [[ "$EXT" == "-h" || "$EXT" == "--help" ]]; then
+ usage
+fi
+
+for db in $(alldbs); do
+ echo "if (in_array('${EXT}',\$wmvActiveExtensions)){echo \"${db}\n\";}" |
+ /usr/local/bin/mwscript eval.php --wiki=${db} |
+ /bin/sed '/^\s*$/d'
+done
diff --git a/puppet/modules/mediawiki/templates/wiki/dbConf.php.erb
b/puppet/modules/mediawiki/templates/wiki/dbConf.php.erb
index 03ffab0..611a0bd 100644
--- a/puppet/modules/mediawiki/templates/wiki/dbConf.php.erb
+++ b/puppet/modules/mediawiki/templates/wiki/dbConf.php.erb
@@ -1,5 +1,7 @@
<?php
// This file is managed by Puppet.
+// See modules/mediawiki/templates/wiki/dbConf.php.erb
+
$wgLocalDatabases[] = '<%= @db_name %>';
$wgCentralAuthAutoLoginWikis['<%= @wiki_name %>'] = '<%= @db_name %>';
$wgMediawikiRoot['<%= @db_name %>'] = '<%= @src_dir %>';
diff --git a/puppet/modules/role/manifests/antispoof.pp
b/puppet/modules/role/manifests/antispoof.pp
index dcdc03d..b6f1006 100644
--- a/puppet/modules/role/manifests/antispoof.pp
+++ b/puppet/modules/role/manifests/antispoof.pp
@@ -8,7 +8,7 @@
}
mediawiki::maintenance { 'populate_spoofuser':
- command => '/usr/local/bin/foreachwiki
extensions/AntiSpoof/maintenance/batchAntiSpoof.php',
+ command => '/usr/local/bin/foreachwikiwithextension AntiSpoof
extensions/AntiSpoof/maintenance/batchAntiSpoof.php',
refreshonly => true,
require => Mediawiki::Extension['AntiSpoof'],
subscribe => Exec['update_all_databases'],
diff --git a/puppet/modules/role/manifests/centralauth.pp
b/puppet/modules/role/manifests/centralauth.pp
index 6b5a042..4b2382b 100644
--- a/puppet/modules/role/manifests/centralauth.pp
+++ b/puppet/modules/role/manifests/centralauth.pp
@@ -110,7 +110,7 @@
# about Admin users on all wikis. If someone changes Wiki[admin_user],
# this will run every time.
mediawiki::maintenance { 'Pass 0 of CentralAuth':
- command => '/usr/local/bin/foreachwiki
extensions/CentralAuth/maintenance/migratePass0.php',
+ command => '/usr/local/bin/foreachwikiwithextension CentralAuth
extensions/CentralAuth/maintenance/migratePass0.php',
onlyif => '/usr/local/bin/is-centralauth-migratePass0-needed',
require => [
File['/usr/local/bin/is-centralauth-migratePass0-needed'],
diff --git a/puppet/modules/role/manifests/cirrussearch.pp
b/puppet/modules/role/manifests/cirrussearch.pp
index edb9ef2..60166e5 100644
--- a/puppet/modules/role/manifests/cirrussearch.pp
+++ b/puppet/modules/role/manifests/cirrussearch.pp
@@ -87,13 +87,17 @@
values => template('elasticsearch/CirrusSearch-commons.php.erb'),
}
+ file { '/usr/local/bin/is-cirrussearch-forceindex-needed':
+ ensure => present,
+ owner => 'root',
+ group => 'root',
+ mode => '0755',
+ content =>
template('role/cirrussearch/is-cirrussearch-forceindex-needed.erb'),
+ }
+
mediawiki::maintenance { 'build_search_index':
- command => '/usr/local/bin/foreachwiki
extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --startOver &&
/usr/local/bin/foreachwiki
extensions/CirrusSearch/maintenance/forceSearchIndex.php',
- onlyif => '/usr/local/bin/mwscript
extensions/CirrusSearch/maintenance/cirrusNeedsToBeBuilt.php --quiet',
- require => [
- Class['::mediawiki::multiwiki'],
- Mediawiki::Extension['CirrusSearch'],
- Exec['update_all_databases'],
- ]
+ command => template('role/cirrussearch/build_search_index.erb'),
+ onlyif => '/usr/local/bin/is-cirrussearch-forceindex-needed',
+ require => File['/usr/local/bin/is-cirrussearch-forceindex-needed'],
}
}
diff --git a/puppet/modules/role/manifests/commons.pp
b/puppet/modules/role/manifests/commons.pp
index e0266ce..211763d 100644
--- a/puppet/modules/role/manifests/commons.pp
+++ b/puppet/modules/role/manifests/commons.pp
@@ -38,7 +38,7 @@
}
mediawiki::maintenance { 'refresh globalusage table':
- command => '/usr/local/bin/foreachwiki
extensions/GlobalUsage/refreshGlobalimagelinks.php --pages
existing,nonexisting',
+ command => '/usr/local/bin/foreachwikiwithextension GlobalUsage
extensions/GlobalUsage/refreshGlobalimagelinks.php --pages
existing,nonexisting',
cwd => $::mediawiki::dir,
require => Mediawiki::Extension['GlobalUsage'],
}
diff --git a/puppet/modules/role/manifests/echo.pp
b/puppet/modules/role/manifests/echo.pp
index ad1618f..bdf89ca 100644
--- a/puppet/modules/role/manifests/echo.pp
+++ b/puppet/modules/role/manifests/echo.pp
@@ -32,7 +32,7 @@
}
mediawiki::maintenance { 'backfill echo_unread_wikis':
- command => '/usr/local/bin/foreachwiki
extensions/Echo/maintenance/backfillUnreadWikis.php',
+ command => '/usr/local/bin/foreachwikiwithextension Echo
extensions/Echo/maintenance/backfillUnreadWikis.php',
refreshonly => true,
}
diff --git a/puppet/modules/role/manifests/geodata_elastic.pp
b/puppet/modules/role/manifests/geodata_elastic.pp
index 1a90173..219f0e2 100644
--- a/puppet/modules/role/manifests/geodata_elastic.pp
+++ b/puppet/modules/role/manifests/geodata_elastic.pp
@@ -14,7 +14,7 @@
}
mediawiki::maintenance { 'force geodata index':
- command => '/usr/local/bin/foreachwiki
extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php',
+ command => '/usr/local/bin/foreachwikiwithextension CirrusSearch
extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php',
refreshonly => true,
}
}
diff --git a/puppet/modules/role/manifests/wikidata.pp
b/puppet/modules/role/manifests/wikidata.pp
index e58ac9c..498921d 100644
--- a/puppet/modules/role/manifests/wikidata.pp
+++ b/puppet/modules/role/manifests/wikidata.pp
@@ -27,16 +27,12 @@
command => '/usr/bin/git remote set-url origin
https://gerrit.wikimedia.org/r/wikidata/build-resources',
unless => "/usr/bin/git remote -v | grep -q
'https://gerrit.wikimedia.org/r/wikidata/build-resources'",
cwd => "${::mediawiki::dir}/extensions/WikidataBuildResources",
- require => Mediawiki::Extension[ 'WikidataBuildResources' ],
+ require => Mediawiki::Extension['WikidataBuildResources'],
}
mediawiki::maintenance { 'wikidata-populate-site-tables':
- command => "/usr/local/bin/foreachwiki
extensions/WikidataBuildResources/extensions/Wikibase/lib/maintenance/populateSitesTable.php
--load-from
http://en${mediawiki::multiwiki::base_domain}${::port_fragment}/w/api.php";,
+ command => "/usr/local/bin/foreachwikiwithextension
WikidataBuildResources
extensions/WikidataBuildResources/extensions/Wikibase/lib/maintenance/populateSitesTable.php
--load-from
http://en${mediawiki::multiwiki::base_domain}${::port_fragment}/w/api.php";,
refreshonly => true,
- require => [
- Class['::mediawiki::multiwiki'],
- Mediawiki::Extension['WikidataBuildResources'],
- ],
}
Mediawiki::Wiki<| |> ~>
Mediawiki::Maintenance['wikidata-populate-site-tables']
diff --git
a/puppet/modules/role/templates/centralauth/is-centralauth-migratePass0-needed.bash.erb
b/puppet/modules/role/templates/centralauth/is-centralauth-migratePass0-needed.bash.erb
index fa863ea..1dd0b2d 100644
---
a/puppet/modules/role/templates/centralauth/is-centralauth-migratePass0-needed.bash.erb
+++
b/puppet/modules/role/templates/centralauth/is-centralauth-migratePass0-needed.bash.erb
@@ -2,7 +2,7 @@
# Exits with status 0 if migratePass0.php needs to run because
# not all wikis are in localnames.
-alldbs_out=$(alldbs)
+alldbs_out=$(/usr/local/bin/wikihasextension CentralAuth)
expected_count=$(echo "$alldbs_out"|wc -l)
alldbs_comma=$(echo "$alldbs_out"|xargs|sed "s/ /', '/g")
diff --git a/puppet/modules/role/templates/cirrussearch/build_search_index.erb
b/puppet/modules/role/templates/cirrussearch/build_search_index.erb
new file mode 100644
index 0000000..2d43406
--- /dev/null
+++ b/puppet/modules/role/templates/cirrussearch/build_search_index.erb
@@ -0,0 +1 @@
+/usr/local/bin/foreachwikiwithextension CirrusSearch
extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --startOver &&
/usr/local/bin/foreachwikiwithextension CirrusSearch
extensions/CirrusSearch/maintenance/forceSearchIndex.php
diff --git
a/puppet/modules/role/templates/cirrussearch/is-cirrussearch-forceindex-needed.erb
b/puppet/modules/role/templates/cirrussearch/is-cirrussearch-forceindex-needed.erb
new file mode 100644
index 0000000..1e92da4
--- /dev/null
+++
b/puppet/modules/role/templates/cirrussearch/is-cirrussearch-forceindex-needed.erb
@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+# Exits with status 0 if CirrusSearch isn't setup for a wiki.
+# Designed to be used as an 'onlyif' command in a Puppet exec.
+
+CHECK=extensions/CirrusSearch/maintenance/cirrusNeedsToBeBuilt.php
+
+for db in $(/usr/local/bin/wikihasextension CirrusSearch); do
+ # When cirrusNeedsToBeBuilt.php exits successsfully it means that the wiki
+ # needs to be setup.
+ /usr/local/bin/mwscript ${CHECK} --wiki=${db} --quiet && exit 0
+done
+
+# If we didn't exit early then all wikis are setup for Cirrus.
+exit 1
--
To view, visit https://gerrit.wikimedia.org/r/278536
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I7b2bf3d25ec148a0e949159178bf2616b5739c82
Gerrit-PatchSet: 5
Gerrit-Project: mediawiki/vagrant
Gerrit-Branch: master
Gerrit-Owner: BryanDavis <[email protected]>
Gerrit-Reviewer: Anomie <[email protected]>
Gerrit-Reviewer: BryanDavis <[email protected]>
Gerrit-Reviewer: Dduvall <[email protected]>
Gerrit-Reviewer: Gergő Tisza <[email protected]>
Gerrit-Reviewer: Mattflaschen <[email protected]>
Gerrit-Reviewer: Mobrovac <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
