ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/86677
Change subject: wikiqueries.py db user/passwd fixups, add docs
......................................................................
wikiqueries.py db user/passwd fixups, add docs
* added short README for the script, another for the config file
* script now can get db user/password from mw adminsettings or
other php file
Change-Id: I4b2f3cc1802eadaefe192f6682d673789249cf0b
---
A xmldumps-backup/wikiqueries/README.config
A xmldumps-backup/wikiqueries/README.txt
M xmldumps-backup/wikiqueries/wikiqueries.py
3 files changed, 72 insertions(+), 5 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/dumps
refs/changes/77/86677/1
diff --git a/xmldumps-backup/wikiqueries/README.config
b/xmldumps-backup/wikiqueries/README.config
new file mode 100644
index 0000000..1dec47b
--- /dev/null
+++ b/xmldumps-backup/wikiqueries/README.config
@@ -0,0 +1,34 @@
+By default, all configuration options are read from the file "dumpincr.conf"
in the current directory.
+A different filename may be specified at run time.
+
+The following configuration options are accepted:
+
+In the "wiki" section,
+mediawiki -- full path to the directory of the MediaWiki installation
+allwikislist -- full path to a list of all projects to be dumped, as they
appear in MySql
+privatewikislist -- full path to a list of all projects that are private and
hence should not be dumped, if any
+closedwikislist -- full path to a list of all projects that are closed and
hence should not be dumped, if any
+adminsettings -- path relative to the directory specified for 'mediawiki',
to the php file where $wgDBadminuser
+ and $wgDBadminpassword or $wgDBuser and $wgDBpassword are
defined, typically
+ AdminSettings.php or LocalSettings.php
+
+In the "output" section,
+wikiqueriesdir -- full path to the top level directory where the query
result files will be written
+temp -- full path to a directory which is used to the generation
of temporary files
+fileperms -- read and write permissions that will be assigned to
created files; this is in octal four-digit
+ format, for example 0644
+
+In the "database" section,
+user -- the name of a database user with read access to all tables in the
databases
+ which will be dumped
+password -- the password for the above user
+These can be omitted if you set 'adminsettings' in the "wiki" section as
described above.
+
+In the "tools" section,
+php -- the full path to the php command
+mysql -- the full path to the mysql command
+gzip -- the full path to the gzip command
+bzip2 -- the full path to the bzip2 command
+
+In the "query" section,
+queryfile -- the full path to the file containing the mysql query to
run on all wikis
diff --git a/xmldumps-backup/wikiqueries/README.txt
b/xmldumps-backup/wikiqueries/README.txt
new file mode 100644
index 0000000..0058f99
--- /dev/null
+++ b/xmldumps-backup/wikiqueries/README.txt
@@ -0,0 +1,25 @@
+wikiqueries.py is a short little script that will run the same sql query
+ovr all the wikis in your wikifarm, writing the output in files with
+a standard naming scheme provided by the user. You might use this to
+generate a daily list of all page titles in the main namespace, of
+all media files uploaded in the last 24 hours, or whatever else you
+find useful.
+
+Installation
+
+You need to set up the config file first. See README.config for details.
+
+Put wikiquery.py, your config file, and a copy of WikiDump.py (from the
+XML data dumps repo where you should have gotten a copy of this file as well)
+in a directory from which the script will run.
+
+Try a test: run
+
+python wikiqueries.py --verbose --configfile path-to-your-config-here \
+ --query 'select page_title from page where page_namespace=0;' wikidbname
+
+where wiki-dbname should be the name of one wiki database in your farm.
+
+Does it run successfully? If so you could try leaving off the wikidbname and
seeing if it
+runs across all the wikis in your farm.
+
diff --git a/xmldumps-backup/wikiqueries/wikiqueries.py
b/xmldumps-backup/wikiqueries/wikiqueries.py
index 5a9a59e..6c9c189 100644
--- a/xmldumps-backup/wikiqueries/wikiqueries.py
+++ b/xmldumps-backup/wikiqueries/wikiqueries.py
@@ -6,6 +6,7 @@
import re
import sys
import ConfigParser
+import WikiDump
import subprocess
import socket
import time
@@ -37,7 +38,7 @@
def getFileName(self):
return fileNameFormat.format(w=self.wikiName, d=self.date)
-class Config(object):
+class Config(WikiDump.Config):
def __init__(self, configFile=False):
self.projectName = False
@@ -58,8 +59,7 @@
"temp":"/wikiqueries/temp",
"fileperms": "0640",
#"database": {
- "user": "root",
- "password": "",
+ # these are now set in getDbUserAndPassword() if needed
#"tools": {
"php": "/bin/php",
"gzip": "/usr/bin/gzip",
@@ -82,6 +82,7 @@
raise ConfigParser.NoOptionError('wiki','mediawiki')
self.parseConfFile()
+ self.getDbUserAndPassword() # get from MW adminsettings file if not
set in conf file
def parseConfFile(self):
self.mediawiki = self.conf.get("wiki", "mediawiki")
@@ -96,10 +97,17 @@
self.fileperms = self.conf.get("output", "fileperms")
self.fileperms = int(self.fileperms,0)
+ self.wikiDir = self.mediawiki # the parent class methods want this
+ self.dbUser = None
+ self.dbPassword = None
if not self.conf.has_section('database'):
self.conf.add_section('database')
- self.dbUser = self.conf.get("database", "user")
- self.dbPassword = self.conf.get("database", "password")
+ if self.conf.has_option('database', 'user'):
+ self.dbUser = self.conf.get("database", "user")
+ if self.conf.has_option('database', 'password'):
+ self.dbPassword = self.conf.get("database", "password")
+ sys.stderr.write("here we have %s and %s\n" % (self.dbUser,
self.dbPassword))
+ self.getDbUserAndPassword() # get from MW adminsettings file if not
set in conf file
if not self.conf.has_section('tools'):
self.conf.add_section('tools')
--
To view, visit https://gerrit.wikimedia.org/r/86677
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4b2f3cc1802eadaefe192f6682d673789249cf0b
Gerrit-PatchSet: 1
Gerrit-Project: operations/dumps
Gerrit-Branch: ariel
Gerrit-Owner: ArielGlenn <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
