jenkins-bot has submitted this change and it was merged. Change subject: Setup local development environment for ErfgoedBot ......................................................................
Setup local development environment for ErfgoedBot So far, we did not have much choice to test the Python bots (especially harvesting) than to merge and deploy. This introduces a full local development environment for ErfgoedBot (especially harvesting), using docker-compose. - Abstract the docker-compose setup in a base file - Add a Dockerfile to build a database with only empty tables, by contrast to the one using a full dump - Add Dockerfile based on Python to run the bot - Add a new docker-compose file, which uses the empty database, the web application and the Python bot. - Add an override config file for the database_config.yml config file - Fix imports of checkers in converters.py to simple import (as erfgoedbot is not necessarily considered as a package) Change-Id: I1204b1ee768aa7839ab23784c200143d95d23728 --- M README.md A conf/Dockerfile.bot R conf/Dockerfile.dbdump A conf/Dockerfile.dbempty A conf/database_config.dev.yml A docker-compose-base.yml A docker-compose-bot.yml M docker-compose.yml M erfgoedbot/converters.py 9 files changed, 95 insertions(+), 15 deletions(-) Approvals: Lokal Profil: Looks good to me, but someone else must approve Jean-Frédéric: Looks good to me, approved jenkins-bot: Verified diff --git a/README.md b/README.md index 7e66f7c..dc64c24 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,14 @@ To hack on it, use [tox](https://tox.readthedocs.io) to run the tests +To spin-up a development environement: + +``` +python erfgoedbot/monument_tables.py +docker-compose -f docker-compose-bot.yml up -d +docker-compose -f docker-compose-bot.yml run --rm bot python erfgoedbot/update_id_dump.py +``` + Monuments Database and API -------------------------- @@ -22,6 +30,6 @@ To hack on it, use [Composer](https://getcomposer.org/) to run PHP tests and [docker-compose](https://docs.docker.com/compose/) to spin-up a local development environment. ``` -sh bin/download_monuments_database_dump.sh +./bin/download_monuments_database_dump.sh docker-compose up -d ``` diff --git a/conf/Dockerfile.bot b/conf/Dockerfile.bot new file mode 100644 index 0000000..986010c --- /dev/null +++ b/conf/Dockerfile.bot @@ -0,0 +1,4 @@ +FROM python:2.7 +ADD requirements.txt requirements.txt +RUN pip install -r requirements.txt +WORKDIR /code diff --git a/conf/Dockerfile.db b/conf/Dockerfile.dbdump similarity index 100% rename from conf/Dockerfile.db rename to conf/Dockerfile.dbdump diff --git a/conf/Dockerfile.dbempty b/conf/Dockerfile.dbempty new file mode 100644 index 0000000..240a934 --- /dev/null +++ b/conf/Dockerfile.dbempty @@ -0,0 +1,7 @@ +FROM mysql:5.5.49 + +# Create all database tables +COPY ./erfgoedbot/sql/*.sql /docker-entrypoint-initdb.d/ + +# Remove the connect to database at the top of each SQL file +RUN sed -i 's/^connect.*$//' docker-entrypoint-initdb.d/* diff --git a/conf/database_config.dev.yml b/conf/database_config.dev.yml new file mode 100644 index 0000000..4ad3e96 --- /dev/null +++ b/conf/database_config.dev.yml @@ -0,0 +1,10 @@ +--- +monuments_db: + server: db + db_name: s51138__heritage_p + username: heritage + password: password + +commons_db: + server: commonswiki.labsdb + db_name: commonswiki_p diff --git a/docker-compose-base.yml b/docker-compose-base.yml new file mode 100644 index 0000000..141be9a --- /dev/null +++ b/docker-compose-base.yml @@ -0,0 +1,35 @@ +web: + build: . + dockerfile: conf/Dockerfile.web + ports: + - "8000:80" + +db_dump: + build: ./conf/ + dockerfile: Dockerfile.dbdump + volumes: + - ./mysql:/etc/mysql/conf.d + environment: + MYSQL_ROOT_PASSWORD: root_password + MYSQL_DATABASE: s51138__heritage_p + MYSQL_USER: heritage + MYSQL_PASSWORD: password + +db_empty: + build: . + dockerfile: conf/Dockerfile.dbempty + volumes: + - ./mysql:/etc/mysql/conf.d + environment: + MYSQL_ROOT_PASSWORD: root_password + MYSQL_DATABASE: s51138__heritage_p + MYSQL_USER: heritage + MYSQL_PASSWORD: password + +bot: + build: . + dockerfile: conf/Dockerfile.bot + volumes: + - .:/code + environment: + PYWIKIBOT2_NO_USER_CONFIG: 1 diff --git a/docker-compose-bot.yml b/docker-compose-bot.yml new file mode 100644 index 0000000..af463b0 --- /dev/null +++ b/docker-compose-bot.yml @@ -0,0 +1,22 @@ +bot: + extends: + file: docker-compose-base.yml + service: bot + links: + - db + volumes: + - "./conf/database_config.dev.yml:/code/erfgoedbot/database_config.yml:ro" + +db: + extends: + file: docker-compose-base.yml + service: db_empty + +web: + extends: + file: docker-compose-base.yml + service: web + links: + - db + volumes: + - .:/var/www/ diff --git a/docker-compose.yml b/docker-compose.yml index 3a44c6e..f178c0d 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,8 +1,7 @@ web: - build: . - dockerfile: conf/Dockerfile.web - ports: - - "8000:80" + extends: + file: docker-compose-base.yml + service: web links: - db volumes: @@ -15,13 +14,8 @@ - ./toolbox:/var/www/html/toolbox - ./toolinfo.json:/var/www/html/toolinfo.json - ./tools:/var/www/html/tools + db: - build: ./conf - dockerfile: Dockerfile.db - volumes: - - ./mysql:/etc/mysql/conf.d - environment: - MYSQL_ROOT_PASSWORD: root_password - MYSQL_DATABASE: s51138__heritage_p - MYSQL_USER: heritage - MYSQL_PASSWORD: password + extends: + file: docker-compose-base.yml + service: db_dump diff --git a/erfgoedbot/converters.py b/erfgoedbot/converters.py index cba1004..92abc44 100644 --- a/erfgoedbot/converters.py +++ b/erfgoedbot/converters.py @@ -4,7 +4,7 @@ """Conversion methods""" import re -from erfgoedbot.checkers import is_int +from checkers import is_int def CH1903Converter(x, y): -- To view, visit https://gerrit.wikimedia.org/r/303498 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I1204b1ee768aa7839ab23784c200143d95d23728 Gerrit-PatchSet: 6 Gerrit-Project: labs/tools/heritage Gerrit-Branch: master Gerrit-Owner: Jean-Frédéric <jeanfrederic.w...@gmail.com> Gerrit-Reviewer: Jean-Frédéric <jeanfrederic.w...@gmail.com> Gerrit-Reviewer: Lokal Profil <lokal.pro...@gmail.com> Gerrit-Reviewer: Multichill <maar...@mdammers.nl> Gerrit-Reviewer: jenkins-bot <> _______________________________________________ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
