Ori.livneh has uploaded a new change for review.
https://gerrit.wikimedia.org/r/61279
Change subject: Default MariaDB character encoding for EL data: utf8 -> utf8mb4
......................................................................
Default MariaDB character encoding for EL data: utf8 -> utf8mb4
This change sets the default character encoding for MySQL / MariaDB
EventLogging data to 'utf8mb4' (was: 'utf8'), adding support for characters
above the Base Multilingual Plane. Deployment will require manual migration of
existing data in the database.
One of the consequences of this migration is that the previous default size for
string columns is not longer appropriate, since the columns it generates are
not indexable by InnoDB, which will not index columns beyond 767 bytes. This
change therefore amends the default size to be 191, which is the maximum size a
utf8mb4 string column can be and still remain indexable.
Finally, as a way of not being blocked on deployment of I8fdcc046d, this change
adds a live hack that substitutes 'utf8mb4' for 'utf8' in database connection
strings. The hack can be removed once I8fdcc046d is deployed.
FIXME: Database setup instructions and minimum requirements should be
documented.
Change-Id: Ia94f2c2155de5fb9031a8164306720e06455cced
---
M server/bin/json2sql
M server/eventlogging/jrm.py
2 files changed, 12 insertions(+), 3 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/EventLogging
refs/changes/79/61279/1
diff --git a/server/bin/json2sql b/server/bin/json2sql
index 3238739..d8cb7c8 100755
--- a/server/bin/json2sql
+++ b/server/bin/json2sql
@@ -42,6 +42,10 @@
logging.basicConfig(stream=sys.stderr, level=args.loglevel)
logging.getLogger('sqlalchemy.engine').setLevel(args.loglevel)
+# XXX(ori, 28-Apr-2013): this coerces 'utf8' client encoding to 'utf8mb4';
+# required only until change XXX is merged.
+if db.endswith('utf8'):
+ db = db + 'mb4'
meta = sqlalchemy.MetaData(args.db)
sub = eventlogging.zmq_subscribe(args.input, sid=args.sid, json=True)
diff --git a/server/eventlogging/jrm.py b/server/eventlogging/jrm.py
index ece81ec..fd6efad 100644
--- a/server/eventlogging/jrm.py
+++ b/server/eventlogging/jrm.py
@@ -34,7 +34,7 @@
#: A dictionary mapping database engine names to table defaults.
ENGINE_TABLE_OPTIONS = {
'mysql': {
- 'mysql_charset': 'utf8',
+ 'mysql_charset': 'utf8mb4',
'mysql_engine': 'InnoDB'
}
}
@@ -63,8 +63,13 @@
return datetime.datetime.strptime(value, MEDIAWIKI_TIMESTAMP)
+#: Maximum length for string and string-like types. Because InnoDB limits index
+#: columns to 767 bytes, the maximum length for a utf8mb4 column (which
+#: reserves up to four bytes per character) is 191 (191 * 4 = 764).
+STRING_MAX_LEN = 191
+
#: Default table column definition, to be overridden by mappers below.
-COLUMN_DEFAULTS = {'type_': sqlalchemy.Unicode(255)}
+COLUMN_DEFAULTS = {'type_': sqlalchemy.Unicode(STRING_MAX_LEN)}
#: Mapping of JSON Schema attributes to valid values. Each value maps to
#: a dictionary of options. The options are compounded into a single
@@ -81,7 +86,7 @@
'boolean': {'type_': sqlalchemy.Boolean},
'integer': {'type_': sqlalchemy.Integer},
'number': {'type_': sqlalchemy.Float},
- 'string': {'type_': sqlalchemy.Unicode(255)},
+ 'string': {'type_': sqlalchemy.Unicode(STRING_MAX_LEN)},
}),
('format', {
'utc-millisec': {'type_': MediaWikiTimestamp, 'index': True},
--
To view, visit https://gerrit.wikimedia.org/r/61279
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia94f2c2155de5fb9031a8164306720e06455cced
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/EventLogging
Gerrit-Branch: master
Gerrit-Owner: Ori.livneh <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
