Smalyshev has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/392736 )
Change subject: Create script for automatic reload of categories
......................................................................
Create script for automatic reload of categories
Bug: T173772
Change-Id: I96e0863aff73a3cece8c93c8f03ee4901759b7b1
---
M modules/wdqs/manifests/gui.pp
A modules/wdqs/templates/cron/reloadCategories.sh.erb
M modules/wdqs/templates/vars.yaml.erb
3 files changed, 39 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/36/392736/1
diff --git a/modules/wdqs/manifests/gui.pp b/modules/wdqs/manifests/gui.pp
index 9ed953a..44c2ed8 100644
--- a/modules/wdqs/manifests/gui.pp
+++ b/modules/wdqs/manifests/gui.pp
@@ -39,4 +39,13 @@
group => 'wikidev',
mode => '0775',
}
+
+ file { '/usr/local/bin/reloadCategories.sh':
+ ensure => present,
+ content => template('cron/reloadCategories.sh.erb'),
+ owner => 'root',
+ group => 'root',
+ mode => '0755',
+ }
+
}
diff --git a/modules/wdqs/templates/cron/reloadCategories.sh.erb
b/modules/wdqs/templates/cron/reloadCategories.sh.erb
new file mode 100755
index 0000000..7cf8c3d
--- /dev/null
+++ b/modules/wdqs/templates/cron/reloadCategories.sh.erb
@@ -0,0 +1,29 @@
+#!/bin/bash
+# This script is reloading categories into a new namespace
+# NOTE: This should be run under user that has rights to
+# sudo service nginx restart
+if [ -r /etc/wdqs/vars.sh ]; then
+ . /etc/wdqs/vars.sh
+fi
+
+DEPLOY_DIR=${DEPLOY_DIR:-"<%= @package_dir %>"}
+DATA_DIR=${DATA_DIR:-"<%= @data_dir %>"}
+ALIAS_FILE="<%= @alias_map %>"
+
+today=$(date -u +'%Y%m%d')
+newNamespace="categories{$today}"
+# Drop old dumps
+rm -f "${DATA_DIR}/*-categories.ttl.gz"
+# Drop old logs
+rm -f "${DATA_DIR}/categories*.log"
+cd $DEPLOY_DIR
+bash createNamespace.sh $newNamespace || exit 1
+# Load the data
+bash forAllCategoryWikis.sh loadCategoryDump.sh $newNamespace >>
"${DATA_DIR}/${newNamespace}.log"
+# Get old namespace
+oldNamespace=$(cat $ALIAS_FILE | grep categories | cut -d' ' -f2)
+# Switch the map
+echo "categories ${newNamespace}" > $ALIAS_FILE
+sudo service nginx restart
+# Drop old namespace
+curl -s -X DELETE http://localhost:9999/bigdata/namespace/${oldNamespace}
\ No newline at end of file
diff --git a/modules/wdqs/templates/vars.yaml.erb
b/modules/wdqs/templates/vars.yaml.erb
index 3ba5b15..21c6a1f 100644
--- a/modules/wdqs/templates/vars.yaml.erb
+++ b/modules/wdqs/templates/vars.yaml.erb
@@ -1,3 +1,4 @@
---
data_dir: <%= @data_dir %>
endpoint: <%= @endpoint %>
+package_dir: <%= @package_dir %>
--
To view, visit https://gerrit.wikimedia.org/r/392736
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I96e0863aff73a3cece8c93c8f03ee4901759b7b1
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Smalyshev <smalys...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
