Ottomata has uploaded a new change for review. https://gerrit.wikimedia.org/r/261648
Change subject: Add EventLogging dockerization ...................................................................... Add EventLogging dockerization Change-Id: I4bd66929d11dd6d21db7b70c632a592a5e9a0595 --- A docker/Dockerfile A docker/README.md A docker/debian-backports.list A docker/dev/Dockerfile A docker/run A docker/service/Dockerfile A docker/service/topics.yaml A docker/wikimedia.key A docker/wikimedia.list 9 files changed, 284 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/eventlogging refs/changes/48/261648/1 diff --git a/docker/Dockerfile b/docker/Dockerfile new file mode 100644 index 0000000..2268392 --- /dev/null +++ b/docker/Dockerfile @@ -0,0 +1,41 @@ +FROM debian:jessie +MAINTAINER Andrew Otto <[email protected]> + +COPY debian-backports.list /etc/apt/sources.list.d/debian-backports.list +COPY wikimedia.list /etc/apt/sources.list.d/wikimedia.list +COPY wikimedia.key /tmp/wikimedia.key + +RUN set -x; \ + apt-key add /tmp/wikimedia.key \ + && apt-get update + +RUN set -x; \ + apt-get install -y --no-install-recommends \ + git \ + ca-certificates \ + python-dateutil \ + python-jsonschema \ + python-mysqldb \ + python-sqlalchemy \ + python-kafka \ + python-zmq \ + python-six \ + python-statsd \ + python-tornado \ + python-sprockets-mixins-statsd \ + python-yaml + +RUN set -x; \ + mkdir -p /usr/src \ + && git clone https://gerrit.wikimedia.org/r/p/eventlogging \ + /usr/src/eventlogging \ + && cd /usr/src/eventlogging \ + && git submodule update --init + +ENV PYTHONPATH="/usr/src/eventlogging" +ENV PATH="$PATH:/usr/src/eventlogging/bin" + +# Default to InnoDB in case MySQL writer is used. +ENV EVENTLOGGING_MYSQL_ENGINE=InnoDB + +ENTRYPOINT ["/bin/bash"] diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 0000000..ea66a02 --- /dev/null +++ b/docker/README.md @@ -0,0 +1,106 @@ +# EventLogging Docker images + +Each Dockerfile in this directory tree can be used to run an eventlogging +component. + +The docker/run script is a helper wrapper around docker commands to make +running a simple EventLogging docker image easy. + +``` +$ docker/run --help +Usage: docker/run [<component>] ... + +Builds and run an EventLogging docker container. + +If <component> is not given, then a shell in the base container +with the codebase checkout out from upstream master at +/usr/src/eventlogging will be started. + +If <component> is dev, then a shell in the dev container with the +local codebase mounted at /srv/eventlogging will be started. + +Otherwise, one of the docker subdirectories will be started. + +Anything after <component> will be passed as the arguments +to docker run, so you may use CMD overrides to pass +additional arguments to ENTRYPOINTS. E.g. +docker/run service file:///tmp/out.log +``` + +``` +$ docker/run dev +Building base EventLogging image... +docker build --tag wikimedia/eventlogging /Users/otto/Projects/wm/analytics/eventlogging/docker +Sending build context to Docker daemon 18.43 kB +Step 1 : FROM debian:jessie + +... + +Removing intermediate container da54f5652e68 +Successfully built 83b6700005d5 +docker build --tag wikimedia/eventlogging-dev /Users/otto/Projects/wm/analytics/eventlogging/docker/dev +Sending build context to Docker daemon 2.048 kB +Step 1 : FROM wikimedia/eventlogging + ---> 83b6700005d5 +Step 2 : ENV PYTHONPATH "/srv/eventlogging" + ---> Running in b20677d9bd16 + ---> cfac280615fe +Removing intermediate container b20677d9bd16 +Step 3 : ENV PATH "/srv/eventlogging/bin:$PATH" + ---> Running in 116b722ac0e9 + ---> 50257e9379f6 +Removing intermediate container 116b722ac0e9 +Successfully built 50257e9379f6 +docker run -i -t -v /Users/otto/Projects/wm/analytics/eventlogging:/srv/eventlogging wikimedia/eventlogging-dev +root@e5167bd2a4c3:/# + +``` +# Images + +## wikimedia/eventlogging - docker/Dockerfile + +This is the base EventLogging docker image. Other specific component images are built from this. You can use this image if you want to work with eventlogging code or run + +``` +$ docker build --tag wikimedia/eventlogging ./docker +... +$ docker run -i -t --name eventlogging-base wikimedia/eventlogging + +root@a97ab96fae69:/# eventlogging-consumer -h +... +``` + +## wikimedia/eventlogging-dev - docker/Dockerfile-dev +This is the development version of the base EventLogging docker images. +Instead of cloning master from upstream, it expects you to mount +your working copy of the EventLogging codebase at /srv/eventlogging +when running this image. + +``` +$ docker build --tag wikimedia/eventlogging-dev ./docker/dev +... +$ docker run -i -t --name eventlogging-dev -v $(pwd):/srv/eventlogging wikimedia/eventlogging-dev + +root@925aff049d11:/# which eventlogging-consumer +/srv/eventlogging/bin/eventlogging-consumer +root@925aff049d11:/# eventlogging-consumer -h +... +``` + +## wikimedia/eventlogging-service - docker/service/Dockerfile + +``` +$ docker build --tag wikimedia/eventlogging-service ./docker +... +$ docker run -p 8085:8085 --name eventlogging-service-stdout wikimedia/eventlogging-serivice +2015-12-30 18:26:44,052 (MainThread) Loading local schemas from /usr/src/eventlogging/config/schemas/jsonschema +2015-12-30 18:26:44,054 (MainThread) Loading schema from file:///usr/src/eventlogging/config/schemas/jsonschema/mediawiki/revision_visibility_set/1.yaml +2015-12-30 18:26:44,075 (MainThread) Loading schema from file:///usr/src/eventlogging/config/schemas/jsonschema/mediawiki/page_delete/1.yaml +2015-12-30 18:26:44,097 (MainThread) Loading schema from file:///usr/src/eventlogging/config/schemas/jsonschema/mediawiki/page_restore/1.yaml +2015-12-30 18:26:44,114 (MainThread) Loading schema from file:///usr/src/eventlogging/config/schemas/jsonschema/mediawiki/page_edit/1.yaml +2015-12-30 18:26:44,136 (MainThread) Loading schema from file:///usr/src/eventlogging/config/schemas/jsonschema/mediawiki/page_move/1.yaml +2015-12-30 18:26:44,157 (MainThread) Publishing valid JSON events to stdout://. + +# In another shell: +$ curl -X POST -d @page_edit_record.json -H 'Content-Type: application/json' http://$(docker-machine ip default):8085/v1/events +``` diff --git a/docker/debian-backports.list b/docker/debian-backports.list new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docker/debian-backports.list diff --git a/docker/dev/Dockerfile b/docker/dev/Dockerfile new file mode 100644 index 0000000..8feaeb7 --- /dev/null +++ b/docker/dev/Dockerfile @@ -0,0 +1,10 @@ +FROM wikimedia/eventlogging + +# Make sure you mount your local eventlogging codebase at +# /srv/eventlogging when running: +# docker run -i -t -v $(pwd):/srv/eventlogging wikimedia/eventlogging:dev + +# Change PYTHONPATH to use the development codebase, and +# put development executables ahead of other PATHS +ENV PYTHONPATH="/srv/eventlogging" +ENV PATH="/srv/eventlogging/bin:$PATH" diff --git a/docker/run b/docker/run new file mode 100755 index 0000000..2205c9b --- /dev/null +++ b/docker/run @@ -0,0 +1,62 @@ +#!/bin/bash + +# Builds and runs an EventLogging container + +if [ "${1}" == '--help' ]; then + echo " +Usage: docker/run [<component>] ... + +Builds and run an EventLogging docker container. + +If <component> is not given, then a shell in the base container +with the codebase checkout out from upstream master at +/usr/src/eventlogging will be started. + +If <component> is dev, then a shell in the dev container with the +local codebase mounted at /srv/eventlogging will be started. + +Otherwise, one of the docker subdirectories will be started. + +Anything after <component> will be passed as the arguments +to docker run, so you may use CMD overrides to pass +additional arguments to ENTRYPOINTS. E.g. +docker/run service file:///tmp/out.log +" + exit 0 +fi + + +component="${1}" +docker_dir=$(pwd)/docker/${component} +image_name="wikimedia/eventlogging" +docker_args='' + +# make sure the base container is built +docker images wikimedia/eventlogging | grep -q 'wikimedia/eventlogging' +if [ $? -ne 0 ]; then + echo "Building base EventLogging image..." + echo docker build --tag 'wikimedia/eventlogging' $(pwd)/docker + docker build --tag 'wikimedia/eventlogging' $(pwd)/docker +fi + +if [ "${component}" != '' ]; then + image_name="wikimedia/eventlogging-${component}" + shift +else + shift + shift; +fi + +# component special cases +if [ "${component}" == 'dev' ]; then + # mount local codebase if + docker_args="-v $(pwd):/srv/eventlogging" +elif [ "${component}" == 'service' ]; then + docker_args='-p 8085:8085' +fi + +# Build and run the image +echo docker build --tag "${image_name}" "${docker_dir}" +docker build --tag "${image_name}" "${docker_dir}" +echo docker run -i -t ${docker_args} "${image_name}" $@ +docker run -i -t ${docker_args} "${image_name}" $@ diff --git a/docker/service/Dockerfile b/docker/service/Dockerfile new file mode 100644 index 0000000..66119dd --- /dev/null +++ b/docker/service/Dockerfile @@ -0,0 +1,14 @@ +FROM wikimedia/eventlogging + +COPY topics.yaml /etc/topics.yaml + +ENTRYPOINT [ \ + "/usr/bin/python", \ + "-OO", \ + "/usr/src/eventlogging/bin/eventlogging-service", \ + "--topic-config", \ + "/etc/topics.yaml", \ + "--schemas-path", \ + "/usr/src/eventlogging/config/schemas/jsonschema" \ +] +CMD ["stdout://"] diff --git a/docker/service/topics.yaml b/docker/service/topics.yaml new file mode 100644 index 0000000..0fe7c6d --- /dev/null +++ b/docker/service/topics.yaml @@ -0,0 +1,19 @@ +# +# Topic config file. This file specifies which schemas are allowed to +# be produced to topics. +# + +mediawiki.page_edit: + schema_name: page_edit + +mediawiki.page_delete: + schema_name: page_delete + +mediawiki.page_move: + schema_name: page_move + +mediawiki.page_restore: + schema_name: page_restore + +mediawiki.revision_visibility_set: + schema_name: revision_visibility_set diff --git a/docker/wikimedia.key b/docker/wikimedia.key new file mode 100644 index 0000000..6195c1d --- /dev/null +++ b/docker/wikimedia.key @@ -0,0 +1,30 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- +Version: GnuPG v1.4.11 (GNU/Linux) + +mQGiBEUtBY8RBACJTZdWEBZHlibArWM1HrX5rcPCb+o2nmeTfNrtMpmVbkmi9vBE +VmIDnjc+VQlJNoiBOKMhAhRSO0rIwEbOTewiQSPERfsClGpv0ikb3kQVFls5HpfZ +49u9EAERRez+P2VJUH7CBmigKdxtKRGM5aLI+eOLUl+lZUn4NU6BsQOUGwCgtLiL +I+8DSNkoiV40UR3uFsS9KLMD/30Lth9A9JgwrDTFl8rlNxq3Eluulv0+2MYoDutW +2p384vJ8Vil+x1GPzZXT1NVHCPdJMXqfnUl33XkPJEFSJ3B1WhwU3muItPoM+GKM +cnJMn2rYJa6Fae7UZy8iRJwSuqSg4mGNa900m/izyYoijJzl1u4HtZhbV++lgubO +j+YfA/4sz68H/ZQZwG+d8X/xTgZ3+9qekqGFgxdGTICtiD7IRPPaQ7EUWOBml6tn +SHfd0TBkCKtfFkr6+rA3ZJ5pyo3OwO2yUAvlBOPeaX4ZKTl7+8lG9kqqGIBm/iZy +bHC75DF506Zm4IiesAXRmRqfB8gReOHEJybZkaCg8FZqhdGErLQ8V2lraW1lZGlh +IEFyY2hpdmUgQXV0b21hdGljIFNpZ25pbmcgS2V5IDxyb290QHdpa2ltZWRpYS5v +cmc+iF8EExECACAFAkUtBY8CGwMGCwkIBwMCBBUCCAMEFgIDAQIeAQIXgAAKCRAJ +29n5P2zUSmaGAJjipA+xkWInJJHCCcoJrf7rBzqEAJ9OEsJuxbBOvOJBovwpWtNh +goMcubkCDQRFLQWdEAgAvEAe6PnzhGdOC3ZYIeJalQyBEelRZiEdzjtdojTNEf29 +6J75O8QqjQt/pyOZ9w/DCiy81dym3GlXeS61tcfNdSBMXqGtgXAskLV1djz0U7SU +89MiwjrSiYhYRYNSQcrshVDjzpHkj8HY0gwNyN5yZ1xnZ9/WG46Kay6quHQbfKn7 +Egxs6ONJJaW7H1MM2cPzsJuzk1aXq4PJOFHgDo9J2j+nGVgk8XdGqgk5t0we69Oh +YXlxUTjgOE+XMxk4PEOFDjk0pTmxOUMP0b08dpvJf652O4jpnylBIiT9ZxRadENM +zmeBT//sJJIleYDh2a1xeDDQwzRig1swFnfYeuEugwADBQf/b7xdqYrLZYqtJVLO +fgh3HOJ605KNYlyreKj67x04fy8lIhrkp3wraVTN74+jObNhJTq3VesUoPPgJqRi +sABCwbGQeKriz7NUAflBliVapPjSd7qD696zO+wQd03z8wJecdxAcmw89+8jyHWV +bgSf3Thy0pfCDBOZL5ApDzPp/zveTAJJdl9xJ+kQA9g4kXIbqdsv0ytqfT56CAOC +vBIJ7JuzIz8eKZ5LlPoGosU5C6TPwlHwfrh1ttD5/LdoSbcz1ThCM3Q9nasvmQjQ +EGZteBiJH8UogRLTsqbJCtQM6aQL8J/+bWjSrmPdCp2z/dzFTgtga4DKcXiSYo+U +JVwilYhJBBgRAgAJBQJFLQWdAhsMAAoJEAnb2fk/bNRK/HEAn0ud2S4zsHv4Ayzp +QqdXQFnLYQ6mAJ9LlSuxDwXm+ln+7o++xUBMQCKJ7g== +=XekF +-----END PGP PUBLIC KEY BLOCK----- \ No newline at end of file diff --git a/docker/wikimedia.list b/docker/wikimedia.list new file mode 100644 index 0000000..505f00e --- /dev/null +++ b/docker/wikimedia.list @@ -0,0 +1,2 @@ +deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports thirdparty +deb-src http://apt.wikimedia.org/wikimedia jessie-wikimedia main backports thirdparty -- To view, visit https://gerrit.wikimedia.org/r/261648 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I4bd66929d11dd6d21db7b70c632a592a5e9a0595 Gerrit-PatchSet: 1 Gerrit-Project: eventlogging Gerrit-Branch: master Gerrit-Owner: Ottomata <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
