Build: Add docker image for MADlib JIRA: MADLIB-920
- Add docker files that would help developers download a docker image with Postgres-9.6 and MADlib depedencies installed. A developer's local source code changes can be built on this image's container to quickly build and run install-checks. Requires docker installed on the developer's environment. - Add a bash script (jenkins_build.sh) that would be a starting point towards getting a Jenkins build for MADlib master branch. - Add help in README on how to build MADlib source code, after pulling down the base MADlib image from docker hub. Closes #103 Project: http://git-wip-us.apache.org/repos/asf/incubator-madlib/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-madlib/commit/8679cbdf Tree: http://git-wip-us.apache.org/repos/asf/incubator-madlib/tree/8679cbdf Diff: http://git-wip-us.apache.org/repos/asf/incubator-madlib/diff/8679cbdf Branch: refs/heads/latest_release Commit: 8679cbdf94600b1d4b49a0c20c82e2529ba3ff9c Parents: 01586c0 Author: Nandish Jayaram <njaya...@apache.org> Authored: Tue Mar 14 10:04:58 2017 -0700 Committer: Nandish Jayaram <njaya...@apache.org> Committed: Tue Mar 14 10:04:58 2017 -0700 ---------------------------------------------------------------------- README.md | 44 ++++++++++++ tool/docker/base/Dockerfile_gpdb_4_3_10 | 71 ++++++++++++++++++++ tool/docker/base/Dockerfile_postgres_9_6 | 57 ++++++++++++++++ .../docker/base/Dockerfile_postgres_9_6_Jenkins | 42 ++++++++++++ tool/jenkins/jenkins_build.sh | 43 ++++++++++++ 5 files changed, 257 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-madlib/blob/8679cbdf/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index 12678e0..eaff324 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,50 @@ See the project webpage [`MADlib Home`](http://madlib.incubator.apache.org/) fo latest binary and source packages. For installation and contribution guides, please see [`MADlib Wiki`](https://cwiki.apache.org/confluence/display/MADLIB/) +Development with Docker +======================= +We provide a Docker image with necessary dependencies required to compile and test MADlib on PostgreSQL 9.6. You can view the dependency Docker file at ./tool/docker/base/Dockerfile_postgres_9_6. The image is hosted on Docker Hub at madlib/postgres_9.6:latest. Later we will provide a similar Docker image for Greenplum Database. + +Some useful commands to use the docker file: +``` +## 1) Pull down the `madlib/postgres_9.6:latest` image from docker hub: +docker pull madlib/postgres_9.6:latest + +## 2) Launch a container corresponding to the MADlib image, mounting the source code folder to the container: +docker run -d -it --name madlib -v (path to incubator-madlib directory):/incubator-madlib/ madlib/postgres_9.6 +# where incubator-madlib is the directory where the MADlib source code resides. + +############################################## * WARNING * ################################################## +# Please be aware that when mounting a volume as shown above, any changes you make in the "incubator-madlib" +# folder inside the Docker container will be reflected on your local disk (and vice versa). This means that +# deleting data in the mounted volume from a Docker container will delete the data from your local disk also. +############################################################################################################# + +## 3) When the container is up, connect to it and build MADlib: +docker exec -it madlib bash +mkdir /incubator-madlib/build-docker +cd /incubator-madlib/build-docker +cmake .. +make +make doc +make install + +## 4) Install MADlib: +src/bin/madpack -p postgres -c postgres/postgres@localhost:5432/postgres install + +## 5) Several other commands, apart from the ones above can now be run, such as: +# Run install check, on all modules: +src/bin/madpack -p postgres -c postgres/postgres@localhost:5432/postgres install-check +# Run install check, on a specific module, say svm: +src/bin/madpack -p postgres -c postgres/postgres@localhost:5432/postgres install-check -t svm +# Reinstall MADlib: +src/bin/madpack -p postgres -c postgres/postgres@localhost:5432/postgres reinstall + +## 6) Kill and remove containers (after exiting the container): +docker kill madlib +docker rm madlib +``` + User and Developer Documentation ================================== The latest documentation of MADlib modules can be found at [`MADlib http://git-wip-us.apache.org/repos/asf/incubator-madlib/blob/8679cbdf/tool/docker/base/Dockerfile_gpdb_4_3_10 ---------------------------------------------------------------------- diff --git a/tool/docker/base/Dockerfile_gpdb_4_3_10 b/tool/docker/base/Dockerfile_gpdb_4_3_10 new file mode 100644 index 0000000..620b379 --- /dev/null +++ b/tool/docker/base/Dockerfile_gpdb_4_3_10 @@ -0,0 +1,71 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +### This is work in progress, does not work at the moment. +#FROM pivotaldata/gpdb4310:latest +# +#### Get postgres specific add-ons +#RUN yum -y update \ +# && yum -y groupinstall "Development tools" \ +# && yum -y install epel-release \ +# && yum -y install cmake \ +# openssl-devel \ +# openssl-libs \ +# openssh-server \ +# python-devel +# +# +#### Build MADlib +#ADD ./ /incubator-madlib +##RUN cd incubator-madlib && \ +## mkdir build && \ +# cd build && \ +# cmake .. && \ +# make && \ +# make install +# +################################################################################################### +################## PLACEHOLDER COMMANDS ################## +#### WARNING: This is under construction, for future reference#################### +### Build the image from this docker file: +## docker build -t gpdb -f tool/gpdb/Dockerfile_4_3_10 . +# +#### Steps to use the image for installing MADlib, building changed source code: +### Run the container, mounting the source code's folder to the container. For example: +## 1) docker run -d -it --name gpdb -v (path-to-incubator-madlib)/src:/incubator-madlib/src gpdb bash +# +### When the container is up, connect to it and execute (Install MADlib): +## 2) docker exec -it gpdb /incubator-madlib/build/src/bin/madpack -p greenplum -c gpadmin@127.0.0.1:5432/gpadmin install +# +### Go into the container to build and run commands like install-check for modules: +## 3) docker exec -it gpdb sh +# +### The above command gives us terminal access to the container, run commands such as: +## - cd /incubator-madlib/build +## - make (This can be run after changing code in the incubator-madlib source code) +## - src/bin/madpack -p postgres -c postgres/postgres@localhost:5432/postgres install-check -t svm +### Install or reinstall MADlib if required: +## - src/bin/madpack -p postgres -c postgres/postgres@localhost:5432/postgres install +## - src/bin/madpack -p postgres -c postgres/postgres@localhost:5432/postgres reinstall +# +# +#### Common docker commands: +### Kill and remove containers: +## - docker kill gpdb +## - docker rm gpdb +# \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-madlib/blob/8679cbdf/tool/docker/base/Dockerfile_postgres_9_6 ---------------------------------------------------------------------- diff --git a/tool/docker/base/Dockerfile_postgres_9_6 b/tool/docker/base/Dockerfile_postgres_9_6 new file mode 100644 index 0000000..4dc5a4c --- /dev/null +++ b/tool/docker/base/Dockerfile_postgres_9_6 @@ -0,0 +1,57 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +FROM postgres:9.6 + +### Get postgres specific add-ons +RUN apt-get update && apt-get install -y wget \ + build-essential \ + postgresql-server-dev-9.6 \ + postgresql-plpython-9.6 \ + openssl \ + libssl-dev \ + libboost-all-dev \ + m4 \ + wget \ + vim \ + pgxnclient \ + flex \ + bison \ + graphviz + +### Build custom CMake with SSQL support +RUN wget https://cmake.org/files/v3.6/cmake-3.6.1.tar.gz && \ + tar -zxvf cmake-3.6.1.tar.gz && \ + cd cmake-3.6.1 && \ + sed -i 's/-DCMAKE_BOOTSTRAP=1/-DCMAKE_BOOTSTRAP=1 -DCMAKE_USE_OPENSSL=ON/g' bootstrap && \ + ./configure && \ + make -j2 && \ + make install + +### Install doxygen-1.8.13: +RUN wget http://ftp.stack.nl/pub/users/dimitri/doxygen-1.8.13.src.tar.gz && \ + tar xf doxygen-1.8.13.src.tar.gz && \ + cd doxygen-1.8.13 && \ + mkdir build && \ + cd build && \ + cmake -G "Unix Makefiles" .. && \ + make && \ + make install + +## To build an image from this docker file, from incubator-madlib folder, run: +# docker build -t madlib/postgres_9.6:latest -f tool/docker/base/Dockerfile_postgres_9_6 . http://git-wip-us.apache.org/repos/asf/incubator-madlib/blob/8679cbdf/tool/docker/base/Dockerfile_postgres_9_6_Jenkins ---------------------------------------------------------------------- diff --git a/tool/docker/base/Dockerfile_postgres_9_6_Jenkins b/tool/docker/base/Dockerfile_postgres_9_6_Jenkins new file mode 100644 index 0000000..137842e --- /dev/null +++ b/tool/docker/base/Dockerfile_postgres_9_6_Jenkins @@ -0,0 +1,42 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +FROM postgres:9.6 + +### Get postgres specific add-ons +RUN apt-get update && apt-get install -y wget \ + build-essential \ + postgresql-server-dev-9.6 \ + postgresql-plpython-9.6 \ + openssl \ + libssl-dev \ + libboost-all-dev \ + m4 \ + wget + +### Build custom CMake with SSQL support +RUN wget https://cmake.org/files/v3.6/cmake-3.6.1.tar.gz && \ + tar -zxvf cmake-3.6.1.tar.gz && \ + cd cmake-3.6.1 && \ + sed -i 's/-DCMAKE_BOOTSTRAP=1/-DCMAKE_BOOTSTRAP=1 -DCMAKE_USE_OPENSSL=ON/g' bootstrap && \ + ./configure && \ + make -j2 && \ + make install + +## To build an image from this docker file, from incubator-madlib folder, run: +# docker build -t madlib/postgres_9.6:jenkins -f tool/docker/base/Dockerfile_postgres_9_6_Jenkins . http://git-wip-us.apache.org/repos/asf/incubator-madlib/blob/8679cbdf/tool/jenkins/jenkins_build.sh ---------------------------------------------------------------------- diff --git a/tool/jenkins/jenkins_build.sh b/tool/jenkins/jenkins_build.sh new file mode 100644 index 0000000..72ada55 --- /dev/null +++ b/tool/jenkins/jenkins_build.sh @@ -0,0 +1,43 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +#!/bin/sh + +##################################################################################### +### If this bash script is executed as a stand-alone file, assuming this +### is not part of the MADlib source code, then the following two commands +### may have to be used: +# git clone https://github.com/apache/incubator-madlib.git +# pushd incubator-madlib +##################################################################################### + +# Pull down the base docker images +docker pull madlib/postgres_9_6:jenkins +# Assuming git clone of incubator-madlib has been done, launch a container with the volume mounted +docker run -d --name madlib -v incubator-madlib:/incubator-madlib madlib/postgres_9.6:jenkins +## This sleep is required since it takes a couple of seconds for the docker +## container to come up, which is required by the docker exec command that follows. +sleep 5 +# cmake, make and make install MADlib +docker exec madlib bash -c 'mkdir /incubator-madlib/build ; cd /incubator-madlib/build ; cmake .. ; make ; make install' +# Install MADlib and run install check +docker exec -it madlib /incubator-madlib/build/src/bin/madpack -p postgres -c postgres/postgres@localhost:5432/postgres install +docker exec -it madlib /incubator-madlib/build/src/bin/madpack -p postgres -c postgres/postgres@localhost:5432/postgres install-check + +docker kill madlib +docker rm madlib
