Repository: marmotta Updated Branches: refs/heads/develop dd1994a92 -> 27889eb17
MARMOTTA-621: latest changes to get the ostrich running in docker Project: http://git-wip-us.apache.org/repos/asf/marmotta/repo Commit: http://git-wip-us.apache.org/repos/asf/marmotta/commit/02d04c42 Tree: http://git-wip-us.apache.org/repos/asf/marmotta/tree/02d04c42 Diff: http://git-wip-us.apache.org/repos/asf/marmotta/diff/02d04c42 Branch: refs/heads/develop Commit: 02d04c4276e703196384a79c9e339d624b0139af Parents: dd1994a Author: Sergio Fernández <[email protected]> Authored: Wed Dec 16 13:20:52 2015 +0100 Committer: Sergio Fernández <[email protected]> Committed: Wed Dec 16 13:20:52 2015 +0100 ---------------------------------------------------------------------- libraries/ostrich/Dockerfile | 76 ++++++++++++++++++++++ libraries/ostrich/README.md | 101 ++++++++++++++++++++++++++++++ libraries/ostrich/backend/Dockerfile | 73 --------------------- libraries/ostrich/backend/README.md | 99 ----------------------------- 4 files changed, 177 insertions(+), 172 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/marmotta/blob/02d04c42/libraries/ostrich/Dockerfile ---------------------------------------------------------------------- diff --git a/libraries/ostrich/Dockerfile b/libraries/ostrich/Dockerfile new file mode 100644 index 0000000..887684d --- /dev/null +++ b/libraries/ostrich/Dockerfile @@ -0,0 +1,76 @@ +# Dockerfile for Apache Marmotta Ostrich Triplestore + +FROM debian:stretch +MAINTAINER Sergio Fernández <[email protected]> + +ADD . /src +WORKDIR /src + +# configuration +ENV DEBIAN_FRONTEND noninteractive +ENV OSTRICH_PATH /opt/ostrich +ENV DB_PATH /data/ostrich +ENV DB_PORT 10000 + +# base environment +RUN apt-get update -qq \ + && apt-get install -qq -y \ + locales \ + apt-utils \ + git \ + && localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 +ENV LANG en_US.utf8 + +RUN apt-get update -qq \ + && apt-get install -y \ + build-essential \ + autoconf \ + libtool \ + cmake \ + libc++-dev \ + python-all-dev \ + python-virtualenv \ + && apt-get install -y \ + libraptor2-dev \ + librasqal3-dev \ + libgoogle-glog-dev \ + libgflags-dev \ + libleveldb-dev \ + zlib1g-dev \ + libgflags-dev \ + libgtest-dev \ + libboost-all-dev \ + libgoogle-perftools-dev + +RUN git clone https://github.com/grpc/grpc.git \ + && cd grpc \ + && git submodule update --init --recursive \ + && make \ + && cd third_party/protobuf \ + && make install \ + && cd ../.. \ + && make install \ + && cd + +# build +RUN cd backend \ + && mkdir build \ + && cd build \ + && cmake -D CMAKE_BUILD_TYPE=Release .. \ + && make \ + && cd +RUN mkdir -p ${DB_PATH} +RUN mkdir ${OSTRICH_PATH} +RUN cp -r /src/backend/build/* ${OSTRICH_PATH}/ + +# clean +RUN apt-get -y clean -qq \ + && apt-get -y autoclean -qq \ + && apt-get -y autoremove -qq \ + && rm -rf /var/lib/apt/lists/* \ + && rm -rf /src + +EXPOSE 10000 + +CMD ${OSTRICH_PATH}/persistence/marmotta_persistence -db ${DB_PATH} -port ${DB_PORT} + http://git-wip-us.apache.org/repos/asf/marmotta/blob/02d04c42/libraries/ostrich/README.md ---------------------------------------------------------------------- diff --git a/libraries/ostrich/README.md b/libraries/ostrich/README.md new file mode 100644 index 0000000..b94fdf4 --- /dev/null +++ b/libraries/ostrich/README.md @@ -0,0 +1,101 @@ +# Apache Marmotta Ostrich Backend + +This repository implements an experimental high-performance backend for Apache Marmotta +using LevelDB as storage and gRPC as communication channel between the Java frontend +and the C++ backend. + +If it proves to be useful, the repository will eventually be merged into the main +development branch of Apache Marmotta + +## Dependencies (C++) + +To compile the C++ backend, you need to have the following dependencies installed: + + * libraptor (used for parsing/serializing in C++) + * librasqal (used for server-side SPARQL evaluation) + * libglog (logging) + * libgflags (command line arguments) + * libleveldb (database backend) + * libgrpc (gRPC runtime) + * libprotobuf (messaging, data model) + +With the exception of libgrpc and libprotobuf, all libraries are available in Linux repositories. +Debian: + + apt-get install libraptor2-dev librasqal3-dev libgoogle-glog-dev libgflags-dev libleveldb-dev + +The backend uses the new Proto 3 format and the gRPC SDK. These need to be installed separately; +please follow the instructions at [https://github.com/grpc/grpc](https://github.com/grpc/grpc/blob/master/INSTALL). + + +## Compilation (C++) + +The backend uses cmake to compile the modules. Create a new directory `build`, run cmake, and run make: + + cd backend + mkdir build && cd build + cmake .. + make + cd .. + +## Compilation (Java) + +The frontend is compiled with Maven and depends on many Apache Marmotta modules to work. Build it with + + mvn clean install + +## Running C++ Backend + +Start the backend from the cmake build directory as follows: + + ./backend/build/persistence/marmotta_persistence -db /path/to/database -port 10000 + +The binary accepts many different options. Please see `--help` for details. + +## Running Docker + +The C++ backend can be ran in the provided Docker image. Just build it: + + docker build -t apachemarmotta/ostrich . + +Then you can run Ostrich as a container: + + docker run -t -d -p 10000:10000 apachemarmotta/ostrich + +connecting normally to `localhost:10000`. + +## Running Sharding + +The repository contains an experimental implementation of a sharding server that proxies and +distributes requests based on a hash calculation over statements. In heavy load environments, +this is potentially much faster than running a single persistence backend. The setup requires +several persistence backends (shards) and a sharding proxy. To experiment, you can start these +on the same machine as follows: + + ./backend/build/persistence/marmotta_persistence -db /path/to/shard1 -port 10001 + ./backend/build/persistence/marmotta_persistence -db /path/to/shard2 -port 10002 + ./sharding/marmotta_sharding --port 10000 --backends localhost:10001,localhost:10002 + +You can then access the sharding server through Marmotta like the persistence server. Running all instances +on the same host is only useful for testing. In production environments, you would of course run all three +(or more) instances on different hosts. Note that the number and order of backends should not change once +data has been imported, because otherwise the hashing algorithm will do the wrong thing. + +## Running Apache Marmotta + +There is a `ostrich` Maven profile to run the webapp launcher: + + cd launchers/marmotta-webapp + mvn tomcat7:run -Postrich + +Afterwards, point your browser to [localhost:8080](http://localhost:8080/). + +## Command Line Client + +A C++ command line client is available for very fast bulk imports and simple queries. To import +a large turtle file, run: + + ./client/marmotta_client --format=turtle import file.ttl + +The client connects by default to `localhost:10000` (change with `--host` and `--port` flags). + http://git-wip-us.apache.org/repos/asf/marmotta/blob/02d04c42/libraries/ostrich/backend/Dockerfile ---------------------------------------------------------------------- diff --git a/libraries/ostrich/backend/Dockerfile b/libraries/ostrich/backend/Dockerfile deleted file mode 100644 index 82e4da1..0000000 --- a/libraries/ostrich/backend/Dockerfile +++ /dev/null @@ -1,73 +0,0 @@ -# Dockerfile for Apache Marmotta Ostrich Triplestore - -FROM debian:stretch -MAINTAINER Sergio Fernández <[email protected]> - -ADD . /src -WORKDIR /src - -# configuration -ENV DEBIAN_FRONTEND noninteractive -ENV DB_PATH /data/ostrich -ENV DB_PORT 10000 - -# base environment -RUN apt-get update -qq \ - && apt-get install -qq -y \ - locales \ - apt-utils \ - git \ - && localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 -ENV LANG en_US.utf8 - -RUN apt-get update -qq \ - && apt-get install -y \ - build-essential \ - autoconf \ - libtool \ - cmake \ - libc++-dev \ - python-all-dev \ - python-virtualenv \ - && apt-get install -y \ - libraptor2-dev \ - librasqal3-dev \ - libgoogle-glog-dev \ - libgflags-dev \ - libleveldb-dev \ - zlib1g-dev \ - libgflags-dev \ - libgtest-dev \ - libboost-all-dev \ - libgoogle-perftools-dev - -RUN git clone https://github.com/grpc/grpc.git \ - && cd grpc \ - && git submodule update --init --recursive \ - && make \ - && cd third_party/protobuf \ - && make install \ - && cd ../.. \ - && make install \ - && cd .. - -# build -RUN mkdir build \ - && cd build \ - && cmake -D CMAKE_BUILD_TYPE=Release .. \ - && make \ - && cd .. - -# clean -RUN apt-get -y clean -qq \ - && apt-get -y autoclean -qq \ - && apt-get -y autoremove -qq \ - && rm -rf /var/lib/apt/lists/* \ - && rm -rf /src/grpc - -RUN mkdir -p ${DB_PATH} - -EXPOSE 10000 - -CMD ["/src/build/persistence/marmotta_persistence", "-db", "${DB_PATH}", "-port" "${DB_PORT}"] - http://git-wip-us.apache.org/repos/asf/marmotta/blob/02d04c42/libraries/ostrich/backend/README.md ---------------------------------------------------------------------- diff --git a/libraries/ostrich/backend/README.md b/libraries/ostrich/backend/README.md deleted file mode 100644 index 3eab294..0000000 --- a/libraries/ostrich/backend/README.md +++ /dev/null @@ -1,99 +0,0 @@ -# Apache Marmotta Ostrich Backend - -This repository implements an experimental high-performance backend for Apache Marmotta -using LevelDB as storage and gRPC as communication channel between the Java frontend -and the C++ backend. - -If it proves to be useful, the repository will eventually be merged into the main -development branch of Apache Marmotta - -## Dependencies (C++) - -To compile the C++ backend, you need to have the following dependencies installed: - - * libraptor (used for parsing/serializing in C++) - * librasqal (used for server-side SPARQL evaluation) - * libglog (logging) - * libgflags (command line arguments) - * libleveldb (database backend) - * libgrpc (gRPC runtime) - * libprotobuf (messaging, data model) - -With the exception of libgrpc and libprotobuf, all libraries are available in Linux repositories. -Debian: - - apt-get install libraptor2-dev librasqal3-dev libgoogle-glog-dev libgflags-dev libleveldb-dev - -The backend uses the new Proto 3 format and the gRPC SDK. These need to be installed separately; -please follow the instructions at [https://github.com/grpc/grpc](https://github.com/grpc/grpc/blob/master/INSTALL). - - -## Compilation (C++) - -The backend uses cmake to compile the modules. Create a new directory `build`, run cmake, and run make: - - mkdir build && cd build - cmake .. - make - -## Compilation (Java) - -The frontend is compiled with Maven and depends on many Apache Marmotta modules to work. Build it with - - cd java - mvn clean install - -## Running C++ Backend - -Start the backend from the cmake build directory as follows: - - ./persistence/marmotta_persistence -db /path/to/database -port 10000 - -The binary accepts many different options. Please see `--help` for details. - -## Running Docker - -The C++ backend can be ran in the provided Docker image. Just build it: - - docker build -t apachemarmotta/ostrich . - -Then you can run Ostrich as a container: - - docker run -t -d -p 8080:8080 -p 10000:10000 apachemarmotta/ostrich - -connecting normally to `localhost:10000`. - -## Running Sharding - -The repository contains an experimental implementation of a sharding server that proxies and -distributes requests based on a hash calculation over statements. In heavy load environments, -this is potentially much faster than running a single persistence backend. The setup requires -several persistence backends (shards) and a sharding proxy. To experiment, you can start these -on the same machine as follows: - - ./service/marmotta_persistence -db /path/to/shard1 -port 10001 - ./service/marmotta_persistence -db /path/to/shard2 -port 10002 - ./sharding/marmotta_sharding --port 10000 --backends localhost:10001,localhost:10002 - -You can then access the sharding server through Marmotta like the persistence server. Running all instances -on the same host is only useful for testing. In production environments, you would of course run all three -(or more) instances on different hosts. Note that the number and order of backends should not change once -data has been imported, because otherwise the hashing algorithm will do the wrong thing. - -## Running Apache Marmotta - -A preconfigured version of Apache Marmotta is available in `java/webapp`. It connects to -`localhost:10000` by default and can be started with: - - mvn tomcat7:run - -Afterwards, point your browser to `localhost:8080`. - -## Command Line Client - -A C++ command line client is available for very fast bulk imports and simple queries. To import -a large turtle file, run: - - ./client/marmotta_client --format=turtle import file.ttl - -The client connects by default to `localhost:10000` (change with `--host` and `--port` flags).
