[MediaWiki-commits] [Gerrit] Labs: Script to back labstore filesystems up - change (operations/puppet)
coren has submitted this change and it was merged.
Change subject: Labs: Script to back labstore filesystems up
..
Labs: Script to back labstore filesystems up
This will create the snapshot as needed to make a time-
consistent copy.
TODO: clean snapshots up as free space is becoming low or
they are becoming full.
Bug: T105027
Change-Id: I078179f84a323957a4124f502aea3073d5c993b5
---
A hieradata/role/common/labstore/fileserver.yaml
A modules/labstore/files/replication-rsync.conf
A modules/labstore/files/storage-replicate
M modules/labstore/manifests/fileserver.pp
4 files changed, 294 insertions(+), 2 deletions(-)
Approvals:
coren: Looks good to me, approved
jenkins-bot: Verified
diff --git a/hieradata/role/common/labstore/fileserver.yaml
b/hieradata/role/common/labstore/fileserver.yaml
new file mode 100644
index 000..dab033b
--- /dev/null
+++ b/hieradata/role/common/labstore/fileserver.yaml
@@ -0,0 +1,3 @@
+# Paramiko needs to ssh into these for replication/backups
+ssh::server::disable_nist_kex: false
+ssh::server::explicit_macs: false
diff --git a/modules/labstore/files/replication-rsync.conf
b/modules/labstore/files/replication-rsync.conf
new file mode 100644
index 000..fb94d9a
--- /dev/null
+++ b/modules/labstore/files/replication-rsync.conf
@@ -0,0 +1,10 @@
+# Do not back log and default output files up
+# (they tend to grow a lot, and are not valuable
+# enough to keep for DR purposes)
+- /tools/**/*.log
+- /tools/**/*.err
+- /tools/**/*.out
+# Not relevant to rsync
+- /lost+found
+# Allow endusers to filter their own backups, too
+: .nobackup
diff --git a/modules/labstore/files/storage-replicate
b/modules/labstore/files/storage-replicate
new file mode 100755
index 000..62333a2
--- /dev/null
+++ b/modules/labstore/files/storage-replicate
@@ -0,0 +1,264 @@
+#! /usr/bin/python3
+# -*- coding: utf-8 -*-
+#
+# Copyright © 2015 Marc-André Pelletier mpellet...@wikimedia.org
+#
+# Permission to use, copy, modify, and/or distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED AS IS AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+#
+##
+## THIS FILE IS MANAGED BY PUPPET
+##
+## Source: modules/labstore/storage-replicate
+## From: tbd
+##
+
+##
+## storage-replicate
+##
+## usage: storage-replicate mountpoint host dest
+##
+## Replicates the directory at mountpoint (which must have a
+## volume mounted) to the destination host, at mountpoint
+## dest. A snapshot of the source will be taken (and kept)
+## and a temporary snapshot of the destination will be taken
+## before the rsync proper (so that there exists a consistent
+## snapshot at all times).
+##
+## This script provides for locking to avoid more than one
+## replication taking place at a time and making a mess of things.
+## The lock directory is also where the source snapshot
+## will be mounted.
+##
+
+import argparse
+import re
+import datetime
+import subprocess
+import sys
+import logging
+import logging.handlers
+import os
+import paramiko
+from shlex import quote
+
+
+class RuntimeError(Exception):
+def __init__(self, ctx, err):
+self.ctx = ctx
+self.err = err
+
+def __str__(self):
+if self.ctx.host:
+return '[%s] %s' % (self.ctx.host, self.err)
+return '[local] ' + repr(self.err)
+
+
+class Context:
+This provides a (trivial) abstraction for executing
+commands and reading files either locally (via subprocess
+and open) or remotely (via paramiko) such that the same
+interface can be used for both.
+
+def __init__(self, host):
+if host:
+self.host = host
+self.client = paramiko.SSHClient()
+self.client.load_system_host_keys()
+self.client.connect(hostname = host, key_filename =
'/root/.ssh/id_labstore')
+else:
+self.host = None
+self.client = None
+
+def read(self, path):
+if self.host:
+(out, err) = self.run('/bin/cat', path)
+if err and err != :
+ raise RuntimeError(self, err)
+
+return out.splitlines()
+
+else:
+with open(path, 'r') as fd:
+return fd.readlines()
+
+def run(self, *cmd):
+This executes the specified command either as a subprocess
+(for local contexts) or via SSH (for remote
[MediaWiki-commits] [Gerrit] Labs: Script to back labstore filesystems up - change (operations/puppet)
coren has uploaded a new change for review.
https://gerrit.wikimedia.org/r/224064
Change subject: Labs: Script to back labstore filesystems up
..
Labs: Script to back labstore filesystems up
This will create the snapshot as needed to make a time-
consistent copy.
TODO: clean snapshots up as free space is becoming low or
they are becoming full.
A possibility to consider is to have the script /also/
create a safety snapshot at the destination before starting the
rsync, but it's not clear if introducing more knowledge of
the destination layout here is wise.
Bug: T105027
Change-Id: I078179f84a323957a4124f502aea3073d5c993b5
---
A modules/labstore/files/replication-rsync.conf
A modules/labstore/files/storage-replicate
M modules/labstore/manifests/fileserver.pp
3 files changed, 196 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/64/224064/1
diff --git a/modules/labstore/files/replication-rsync.conf
b/modules/labstore/files/replication-rsync.conf
new file mode 100644
index 000..fb94d9a
--- /dev/null
+++ b/modules/labstore/files/replication-rsync.conf
@@ -0,0 +1,10 @@
+# Do not back log and default output files up
+# (they tend to grow a lot, and are not valuable
+# enough to keep for DR purposes)
+- /tools/**/*.log
+- /tools/**/*.err
+- /tools/**/*.out
+# Not relevant to rsync
+- /lost+found
+# Allow endusers to filter their own backups, too
+: .nobackup
diff --git a/modules/labstore/files/storage-replicate
b/modules/labstore/files/storage-replicate
new file mode 100755
index 000..7989461
--- /dev/null
+++ b/modules/labstore/files/storage-replicate
@@ -0,0 +1,169 @@
+#! /usr/bin/python
+# -*- coding: utf-8 -*-
+#
+# Copyright © 2015 Marc-André Pelletier mpellet...@wikimedia.org
+#
+# Permission to use, copy, modify, and/or distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED AS IS AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+#
+##
+## THIS FILE IS MANAGED BY PUPPET
+##
+## Source: modules/labstore/storage-replicate
+## From: tbd
+##
+
+##
+## storage-replicate
+##
+## usage: storage-replicate mountpoint dest
+##
+## Replicates the directory at mountpoint (which must have a
+## volume mounted) to the destination specified
+## by dest. The actual copy is done with rsync, so any value
+## acceptable to rsync as destination can be used here. A
+## snapshot will be taken, but not discarded at the end.
+##
+## This script provides for locking to avoid more than one
+## replication taking place at a time and making a mess of things.
+## The presence of a file named 'skipped' in the $lockdir means
+## that the replication is running late, with each line in the
+## file being a skipped attempt because a prior rsync was already
+## taking place. It may be wise to make an icinga check for
+## the presence and number of lines in this file.
+##
+## This script is intended to be run (daily?) by cron, but it
+## it safe to invoke manually - it will do nothing if there is
+## already an rsync in progress.
+##
+
+import argparse
+import re
+import datetime
+import subprocess
+import sys
+import logging
+import logging.handlers
+import os
+
+def system(*cmd):
+sub = subprocess.Popen(list(cmd), stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
+(out, err) = sub.communicate()
+if sub.returncode:
+err = err.splitlines(False)[0].strip()
+if not err or err=='':
+if sub.returncode 0:
+err = killed by signal %d % -sub.returncode
+else:
+err = exited with %d % sub.returncode
+return (None, err)
+return (out, None)
+
+class Lockdir:
+def __init__(self, path):
+self.path = path
+self.mountpoint = %s/snapshot % path
+self.err = None
+
+def __enter__(self):
+try:
+os.mkdir(self.path, 0700)
+os.mkdir(self.mountpoint, 0700)
+except OSError as e:
+self.err = unable to create lock directory %s: %s % (self.path,
e.strerror)
+return self
+
+def __exit__(self, e1, e2, e3):
+(out, err) = system('/bin/umount', '-fl', self.mountpoint)
+(out, err) = system('/bin/rm', '-rf', self.path);
+return None
+
+syslog = logging.getLogger('storage-replicate')
+handler = logging.handlers.SysLogHandler(address =
