Nuria has submitted this change and it was merged.
Change subject: Adds alembic migrations
......................................................................
Adds alembic migrations
Instructions for using alembic can be found in database_migrations/README
Instructions for setting up have been updated in the main README.md
Card: analytics 1431
Change-Id: Ibd7671094502159b52adb597935123014073aeca
---
M README.md
A alembic.ini
A database_migrations/README
A database_migrations/env.py
A database_migrations/script.py.mako
A database_migrations/versions/492fe78451c6_initial_revision.py
M requirements.txt
A scripts/99_alembic_setup
M wikimetrics/database.py
9 files changed, 298 insertions(+), 10 deletions(-)
Approvals:
Nuria: Verified; Looks good to me, approved
diff --git a/README.md b/README.md
index d9f8e30..08ab125 100644
--- a/README.md
+++ b/README.md
@@ -16,7 +16,7 @@
$ sudo pip install -e .
````
-Now you need to set up your MySQL databases. You just need empty databases
because SQLAlchemy will create the tables it needs:
+Now you need to set up your MySQL databases. We will first create empty
databases and then populate them using alembic.
````
$ sudo mysql -p < scripts/00_create_wikimetrics_db
@@ -24,6 +24,19 @@
$ sudo mysql -p < scripts/02_create_dewiki_db
````
+And finally, just upgrade the schema to the latest version:
+
+````
+$ alembic upgrade head
+````
+
+If the `alembic upgrade head` command gives you an error that tables already
exist, that means you already had a wikimetrics database set up. It's ok, we
got you covered. Just run the following script and re-run `alembic upgrade
head`:
+
+````
+$ mysql -uwikimetrics -pwikimetrics < scripts/99_alembic_setup
+````
+
+
Wikimetrics has over 90% unit test coverage. We use Nose to write unit and
integration tests to achieve this, and you should too.
````
diff --git a/alembic.ini b/alembic.ini
new file mode 100644
index 0000000..d63c1b5
--- /dev/null
+++ b/alembic.ini
@@ -0,0 +1,54 @@
+# A generic, single database configuration.
+
+[alembic]
+# path to migration scripts
+script_location = database_migrations
+
+# template used to generate migration files
+# file_template = %%(rev)s_%%(slug)s
+
+# max length of characters to apply to the
+# "slug" field
+#truncate_slug_length = 40
+
+# set to 'true' to run the environment during
+# the 'revision' command, regardless of autogenerate
+# revision_environment = false
+
+sqlalchemy.url = not-used
+
+
+# Logging configuration
+[loggers]
+keys = root,sqlalchemy,alembic
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARN
+handlers = console
+qualname =
+
+[logger_sqlalchemy]
+level = WARN
+handlers =
+qualname = sqlalchemy.engine
+
+[logger_alembic]
+level = INFO
+handlers =
+qualname = alembic
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S
diff --git a/database_migrations/README b/database_migrations/README
new file mode 100644
index 0000000..8130d9d
--- /dev/null
+++ b/database_migrations/README
@@ -0,0 +1,12 @@
+For setup instructions, see the main README.md file of this repository.
+
+Alembic is configured to handle database migrations for the wikimetrics
database. Autogeneration of migrations is supported. The general workflow is:
+
+````
+$ # make some changes to the models in wikimetrics.models
+$ alembic revision --autogenerate -m "Short description of your change"
+$ # confirm the file added to database_migrations/versions/ does what you want
+$ alembic upgrade head
+````
+
+At this point your local database is up to date with your models and the
proper alembic scripts have been added. From a database point of view, you're
ready to submit your change.
diff --git a/database_migrations/env.py b/database_migrations/env.py
new file mode 100644
index 0000000..a6613f7
--- /dev/null
+++ b/database_migrations/env.py
@@ -0,0 +1,68 @@
+from __future__ import with_statement
+from alembic import context
+from sqlalchemy import engine_from_config, pool
+from logging.config import fileConfig
+
+from wikimetrics.configurables import db
+
+# this is the Alembic Config object, which provides
+# access to the values within the .ini file in use.
+config = context.config
+
+# Interpret the config file for Python logging.
+# This line sets up loggers
+fileConfig(config.config_file_name)
+
+# add your model's MetaData object here for 'autogenerate' support
+# initialize metadata
+db.get_session()
+# set metadata
+target_metadata = db.WikimetricsBase.metadata
+
+# other values from the config, defined by the needs of env.py,
+# can be acquired:
+# my_important_option = config.get_main_option("my_important_option")
+# ... etc.
+
+
+def run_migrations_offline():
+ """Run migrations in 'offline' mode.
+
+ This configures the context with just a URL
+ and not an Engine, though an Engine is acceptable
+ here as well. By skipping the Engine creation
+ we don't even need a DBAPI to be available.
+
+ Calls to context.execute() here emit the given string to the
+ script output.
+
+ """
+ url = db.get_engine().url
+ context.configure(url=url)
+
+ with context.begin_transaction():
+ context.run_migrations()
+
+
+def run_migrations_online():
+ """Run migrations in 'online' mode.
+
+ In this scenario we need to create an Engine
+ and associate a connection with the context.
+
+ """
+ engine = db.get_engine()
+
+ connection = engine.connect()
+ context.configure(connection=connection, target_metadata=target_metadata)
+
+ try:
+ with context.begin_transaction():
+ context.run_migrations()
+ finally:
+ connection.close()
+
+if context.is_offline_mode():
+ run_migrations_offline()
+else:
+ run_migrations_online()
diff --git a/database_migrations/script.py.mako
b/database_migrations/script.py.mako
new file mode 100644
index 0000000..9570201
--- /dev/null
+++ b/database_migrations/script.py.mako
@@ -0,0 +1,22 @@
+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision}
+Create Date: ${create_date}
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = ${repr(up_revision)}
+down_revision = ${repr(down_revision)}
+
+from alembic import op
+import sqlalchemy as sa
+${imports if imports else ""}
+
+def upgrade():
+ ${upgrades if upgrades else "pass"}
+
+
+def downgrade():
+ ${downgrades if downgrades else "pass"}
diff --git a/database_migrations/versions/492fe78451c6_initial_revision.py
b/database_migrations/versions/492fe78451c6_initial_revision.py
new file mode 100644
index 0000000..7eee239
--- /dev/null
+++ b/database_migrations/versions/492fe78451c6_initial_revision.py
@@ -0,0 +1,100 @@
+"""Initial Revision
+
+Revision ID: 492fe78451c6
+Revises: None
+Create Date: 2014-02-04 13:11:45.727956
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '492fe78451c6'
+down_revision = None
+
+from alembic import op
+import sqlalchemy as sa
+
+
+def upgrade():
+ ### commands auto generated by Alembic - please adjust! ###
+ op.create_table(
+ 'user',
+ sa.Column('id', sa.Integer(), nullable=False),
+ sa.Column('username', sa.String(length=50), nullable=True),
+ sa.Column('email', sa.String(length=254), nullable=True),
+ sa.Column('role', sa.String(length=50), nullable=True),
+ sa.Column('google_id', sa.String(length=254), nullable=True),
+ sa.Column('meta_mw_id', sa.String(length=254), nullable=True),
+ sa.Column('authenticated', sa.Boolean(), nullable=True),
+ sa.Column('active', sa.Boolean(), nullable=True),
+ sa.PrimaryKeyConstraint('id')
+ )
+ op.create_table(
+ 'report',
+ sa.Column('id', sa.Integer(), nullable=False),
+ sa.Column('created', sa.DateTime(), nullable=True),
+ sa.Column('user_id', sa.Integer(), nullable=True),
+ sa.Column('queue_result_key', sa.String(length=50), nullable=True),
+ sa.Column('result_key', sa.String(length=50), nullable=True),
+ sa.Column('status', sa.String(length=50), nullable=True),
+ sa.Column('name', sa.String(length=2000), nullable=True),
+ sa.Column('show_in_ui', sa.Boolean(), nullable=True),
+ sa.Column('parameters', sa.String(length=4000), nullable=True),
+ sa.PrimaryKeyConstraint('id')
+ )
+ op.create_table(
+ 'wiki_user',
+ sa.Column('id', sa.Integer(), nullable=False),
+ sa.Column('mediawiki_username', sa.String(length=255), nullable=True),
+ sa.Column('mediawiki_userid', sa.Integer(), nullable=True),
+ sa.Column('project', sa.String(length=45), nullable=True),
+ sa.Column('valid', sa.Boolean(), nullable=True),
+ sa.Column('reason_invalid', sa.String(length=200), nullable=True),
+ sa.Column('validating_cohort', sa.Integer(), nullable=True),
+ sa.PrimaryKeyConstraint('id')
+ )
+ op.create_table(
+ 'cohort',
+ sa.Column('id', sa.Integer(), nullable=False),
+ sa.Column('name', sa.String(length=50), nullable=True),
+ sa.Column('description', sa.String(length=254), nullable=True),
+ sa.Column('default_project', sa.String(length=50), nullable=True),
+ sa.Column('created', sa.DateTime(), nullable=True),
+ sa.Column('changed', sa.DateTime(), nullable=True),
+ sa.Column('enabled', sa.Boolean(), nullable=True),
+ sa.Column('public', sa.Boolean(), nullable=True),
+ sa.Column('validated', sa.Boolean(), nullable=True),
+ sa.Column('validate_as_user_ids', sa.Boolean(), nullable=True),
+ sa.Column('validation_queue_key', sa.String(length=50), nullable=True),
+ sa.PrimaryKeyConstraint('id')
+ )
+ op.create_table(
+ 'cohort_wiki_user',
+ sa.Column('id', sa.Integer(), nullable=False),
+ sa.Column('wiki_user_id', sa.Integer(), nullable=True),
+ sa.Column('cohort_id', sa.Integer(), nullable=True),
+ sa.ForeignKeyConstraint(['cohort_id'], ['cohort.id'], ),
+ sa.ForeignKeyConstraint(['wiki_user_id'], ['wiki_user.id'], ),
+ sa.PrimaryKeyConstraint('id')
+ )
+ op.create_table(
+ 'cohort_user',
+ sa.Column('id', sa.Integer(), nullable=False),
+ sa.Column('user_id', sa.Integer(), nullable=True),
+ sa.Column('cohort_id', sa.Integer(), nullable=True),
+ sa.Column('role', sa.String(length=45), nullable=True),
+ sa.ForeignKeyConstraint(['cohort_id'], ['cohort.id'], ),
+ sa.ForeignKeyConstraint(['user_id'], ['user.id'], ),
+ sa.PrimaryKeyConstraint('id')
+ )
+ ### end Alembic commands ###
+
+
+def downgrade():
+ ### commands auto generated by Alembic - please adjust! ###
+ op.drop_table('cohort_user')
+ op.drop_table('cohort_wiki_user')
+ op.drop_table('cohort')
+ op.drop_table('wiki_user')
+ op.drop_table('report')
+ op.drop_table('user')
+ ### end Alembic commands ###
diff --git a/requirements.txt b/requirements.txt
index a0a83b6..40bd80d 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -14,3 +14,4 @@
PyYAML==3.10
PyJWT==0.1.6
python-dateutil==2.2
+alembic==0.6.3
diff --git a/scripts/99_alembic_setup b/scripts/99_alembic_setup
new file mode 100644
index 0000000..535b280
--- /dev/null
+++ b/scripts/99_alembic_setup
@@ -0,0 +1,10 @@
+use wikimetrics;
+
+ CREATE TABLE IF NOT EXISTS `alembic_version` (
+ `version_num` varchar(32) NOT NULL
+ ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
+
+ INSERT INTO alembic_version (version_num)
+ SELECT '492fe78451c6' as version_num
+ FROM (SELECT count(*) count FROM alembic_version) as c
+ WHERE c.count = 0;
diff --git a/wikimetrics/database.py b/wikimetrics/database.py
index ff11bf7..c702518 100644
--- a/wikimetrics/database.py
+++ b/wikimetrics/database.py
@@ -80,6 +80,21 @@
self.mediawiki_sessionmakers = {}
self.project_host_map = self.get_project_host_map(usecache=True)
+ def get_engine(self):
+ """
+ Create a sqlalchemy engine for the wikimetrics database.
+
+ Returns:
+ new or cached sqlalchemy engine connected to the wikimetrics
database.
+ """
+ if self.wikimetrics_engine is None:
+ self.wikimetrics_engine = create_engine(
+ self.config['WIKIMETRICS_ENGINE_URL'],
+ echo=self.config['SQL_ECHO'],
+ )
+
+ return self.wikimetrics_engine
+
def get_session(self):
"""
On the first run, instantiates the Wikimetrics session maker
@@ -89,18 +104,11 @@
Returns:
new sqlalchemy session open to the wikimetrics database
"""
- if not self.wikimetrics_sessionmaker:
- self.wikimetrics_engine = create_engine(
- self.config['WIKIMETRICS_ENGINE_URL'],
- echo=self.config['SQL_ECHO'],
- )
+ if self.wikimetrics_sessionmaker is None:
+ self.get_engine()
# This import is necessary here so that
# WikimetricsBase knows about all its children.
import wikimetrics.models
- self.WikimetricsBase.metadata.create_all(
- self.wikimetrics_engine,
- checkfirst=True
- )
self.wikimetrics_sessionmaker =
sessionmaker(self.wikimetrics_engine)
return self.wikimetrics_sessionmaker()
--
To view, visit https://gerrit.wikimedia.org/r/111220
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibd7671094502159b52adb597935123014073aeca
Gerrit-PatchSet: 4
Gerrit-Project: analytics/wikimetrics
Gerrit-Branch: master
Gerrit-Owner: Milimetric <[email protected]>
Gerrit-Reviewer: Csalvia <[email protected]>
Gerrit-Reviewer: Milimetric <[email protected]>
Gerrit-Reviewer: Nuria <[email protected]>
Gerrit-Reviewer: Ottomata <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
