RATIS-372. Docker containers and docker-compose orchestration for the logservice
Signed-off-by: Rajeshbabu Chintaguntla <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/incubator-ratis/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-ratis/commit/29850741 Tree: http://git-wip-us.apache.org/repos/asf/incubator-ratis/tree/29850741 Diff: http://git-wip-us.apache.org/repos/asf/incubator-ratis/diff/29850741 Branch: refs/heads/master Commit: 29850741e635d31e49274dec610f106c001b11f0 Parents: 52c1091 Author: Josh Elser <[email protected]> Authored: Sun Dec 23 23:46:00 2018 -0500 Committer: Josh Elser <[email protected]> Committed: Thu Jan 3 15:41:24 2019 -0500 ---------------------------------------------------------------------- ratis-logservice/Dockerfile | 19 ++++++++ ratis-logservice/README.md | 24 +++++----- ratis-logservice/docker-compose.yml | 59 +++++++++++++++++++++++++ ratis-logservice/src/assembly/assembly.xml | 16 +++++++ 4 files changed, 108 insertions(+), 10 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/29850741/ratis-logservice/Dockerfile ---------------------------------------------------------------------- diff --git a/ratis-logservice/Dockerfile b/ratis-logservice/Dockerfile new file mode 100644 index 0000000..244c402 --- /dev/null +++ b/ratis-logservice/Dockerfile @@ -0,0 +1,19 @@ +FROM openjdk:8-jdk-alpine + +###### +RUN apk update && apk upgrade + +ENV BASEDIR /opt/ratis-logservice/ + +# Our ratis-logservice tarball +ARG BINARY +ARG VERSION + +# Extract, symlink, and remove the ratis-logservice tarball +COPY $BINARY $BASEDIR +RUN tar xf ${BASEDIR}/ratis-logservice-${VERSION}-bin.tar.gz -C $BASEDIR \ + && ln -s ${BASEDIR}/ratis-logservice-${VERSION} ${BASEDIR}/current \ + && rm ${BASEDIR}/ratis-logservice-${VERSION}-bin.tar.gz + +# Use the nice symlink we made as our "starting directory" +WORKDIR $BASEDIR/current http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/29850741/ratis-logservice/README.md ---------------------------------------------------------------------- diff --git a/ratis-logservice/README.md b/ratis-logservice/README.md index 190f478..8dc9e9f 100644 --- a/ratis-logservice/README.md +++ b/ratis-logservice/README.md @@ -1,18 +1,22 @@ # Ratis LogService +The Ratis LogService is an distributed, log implementation built on top of Apache Ratis. The LogService provides the +ability to create named, durable, append-only data structures with the ability to perform linear reads. -## Example shell +## Launching the LogService -Build: -```bash -$ mvn install -``` +The LogService is compose of two Ratis quorums: the metadata quorum and the log quorum. These can be launched manually +or via docker-compose. -Change to logservice dectory: +First, the project must be built: ```bash -$ cd ratis-log-service +$ mvn clean package -DskipTests ``` +Then, the service can be launched. + +### Manual + Launch Metadata daemons: ```bash $ mvn exec:java -Dexec.mainClass=org.apache.ratis.logservice.server.MetadataServer -Dexec.args="-p 9991 -d $HOME/logservice1 -h localhost -q localhost:9991,localhost:9992,localhost:9993" @@ -38,7 +42,7 @@ $ mvn exec:java -Dexec.mainClass=org.apache.ratis.logservice.server.LogServer -D $ mvn exec:java -Dexec.mainClass=org.apache.ratis.logservice.server.LogServer -Dexec.args="-p 9953 -d $HOME/worker3 -h localhost -q localhost:9991,localhost:9992,localhost:9993" ``` -Now, we have started three daemons which can service a single LogStream. They will rep to the Metadata q, +Now, we have started three daemons which can service a single LogStream. They will register to the Metadata quorum, and the Metadata quorum will choose three of them to form a RAFT quorum to "host" a single Log. Note: the `q` option here references to the Metadata quorum, not the worker quorum as is the case for the Metadata daemons. @@ -59,7 +63,7 @@ $ docker-compose up Then, a client container can be launched to connect to the cluster: ```bash -$ mvn exec:java -Dexec.mainClass=org.apache.ratis.logservice.shell.LogServiceShell -Dexec.args="-q localhost:9990,localhost:9991,localhost:9992" +$ docker run --rm --network ratis-logservice_default -it ratis-logservice java -cp "/opt/ratis-logservice/current/conf:/opt/ratis-logservice/current/lib/*" org.apache.ratis.logservice.shell.LogServiceShell -q master1.logservice.ratis.org:9999,master2.logservice.ratis.org:9999,master3.logservice.ratis.org:9999 ``` -This command will launch an interactive shell that you can use to interact with the system. +Take care that the correct network is provided to the LogServiceShell command. http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/29850741/ratis-logservice/docker-compose.yml ---------------------------------------------------------------------- diff --git a/ratis-logservice/docker-compose.yml b/ratis-logservice/docker-compose.yml new file mode 100644 index 0000000..b82873b --- /dev/null +++ b/ratis-logservice/docker-compose.yml @@ -0,0 +1,59 @@ +version: '3' + +# We already have a default network using the bridge adapter, so we don't need to redefine it. + +services: + master_1: + image: ratis-logservice:latest + command: "java -cp 'conf:lib/*' org.apache.ratis.logservice.server.MetadataServer -h master1.logservice.ratis.org -p 9999 -d /tmp/logservice-metadata -q master1.logservice.ratis.org:9999,master2.logservice.ratis.org:9999,master3.logservice.ratis.org:9999" + ports: + - 9999 + networks: + default: + aliases: + - master1.logservice.ratis.org + master_2: + image: ratis-logservice:latest + command: "java -cp 'conf:lib/*' org.apache.ratis.logservice.server.MetadataServer -h master2.logservice.ratis.org -p 9999 -d /tmp/logservice-metadata -q master1.logservice.ratis.org:9999,master2.logservice.ratis.org:9999,master3.logservice.ratis.org:9999" + ports: + - 9999 + networks: + default: + aliases: + - master2.logservice.ratis.org + master_3: + image: ratis-logservice:latest + command: "java -cp 'conf:lib/*' org.apache.ratis.logservice.server.MetadataServer -h master3.logservice.ratis.org -p 9999 -d /tmp/logservice-metadata -q master1.logservice.ratis.org:9999,master2.logservice.ratis.org:9999,master3.logservice.ratis.org:9999" + ports: + - 9999 + networks: + default: + aliases: + - master3.logservice.ratis.org + worker_1: + image: ratis-logservice:latest + command: "java -cp 'conf:lib/*' org.apache.ratis.logservice.server.LogServer -p 9999 -h worker1.logservice.ratis.org -d /tmp/logservice-worker -q master1.logservice.ratis.org:9999,master2.logservice.ratis.org:9999,master3.logservice.ratis.org:9999" + ports: + - 9999 + networks: + default: + aliases: + - worker1.logservice.ratis.org + worker_2: + image: ratis-logservice:latest + command: "java -cp 'conf:lib/*' org.apache.ratis.logservice.server.LogServer -p 9999 -h worker2.logservice.ratis.org -d /tmp/logservice-worker -q master1.logservice.ratis.org:9999,master2.logservice.ratis.org:9999,master3.logservice.ratis.org:9999" + ports: + - 9999 + networks: + default: + aliases: + - worker2.logservice.ratis.org + worker_3: + image: ratis-logservice:latest + command: "java -cp 'conf:lib/*' org.apache.ratis.logservice.server.LogServer -p 9999 -h worker3.logservice.ratis.org -d /tmp/logservice-worker -q master1.logservice.ratis.org:9999,master2.logservice.ratis.org:9999,master3.logservice.ratis.org:9999" + ports: + - 9999 + networks: + default: + aliases: + - worker3.logservice.ratis.org http://git-wip-us.apache.org/repos/asf/incubator-ratis/blob/29850741/ratis-logservice/src/assembly/assembly.xml ---------------------------------------------------------------------- diff --git a/ratis-logservice/src/assembly/assembly.xml b/ratis-logservice/src/assembly/assembly.xml index 77d1f94..c524c96 100644 --- a/ratis-logservice/src/assembly/assembly.xml +++ b/ratis-logservice/src/assembly/assembly.xml @@ -13,4 +13,20 @@ </includes> </dependencySet> </dependencySets> + <fileSets> + <fileSet> + <includes> + <include>README.md</include> + </includes> + </fileSet> + <fileSet> + <!-- Source directory, keeps 'src/test/resources' out of 'conf/' --> + <directory>src/test/resources</directory> + <!-- Destination directory --> + <outputDirectory>conf</outputDirectory> + <includes> + <include>log4j.properties</include> + </includes> + </fileSet> + </fileSets> </assembly>
