Repository: reef Updated Branches: refs/heads/master 09a034673 -> 83f905db5
[REEF-1181] Add dockerfiles for Mesos 0.26 This adds dockerfiles for integration tests on Mesos 0.26. JIRA: [REEF-1181](https://issues.apache.org/jira/browse/REEF-1181) Pull Request: Closes #820 Project: http://git-wip-us.apache.org/repos/asf/reef/repo Commit: http://git-wip-us.apache.org/repos/asf/reef/commit/83f905db Tree: http://git-wip-us.apache.org/repos/asf/reef/tree/83f905db Diff: http://git-wip-us.apache.org/repos/asf/reef/diff/83f905db Branch: refs/heads/master Commit: 83f905db5c1b92306248e53a3cd80210fe0e1842 Parents: 09a0346 Author: Dongjoon Hyun <[email protected]> Authored: Wed Feb 3 10:24:39 2016 -0800 Committer: Yunseong Lee <[email protected]> Committed: Tue Feb 23 13:29:02 2016 +0900 ---------------------------------------------------------------------- dev/docker/README.md | 7 ++-- .../ubuntu12.04-jdk7-mesos0.25/Dockerfile | 2 +- .../ubuntu12.04-jdk7-mesos0.26/Dockerfile | 35 ++++++++++++++++++ .../ubuntu12.04-jdk7-mesos0.26/init-nn.sh | 39 ++++++++++++++++++++ 4 files changed, 78 insertions(+), 5 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/reef/blob/83f905db/dev/docker/README.md ---------------------------------------------------------------------- diff --git a/dev/docker/README.md b/dev/docker/README.md index 470f9b1..2e13a46 100644 --- a/dev/docker/README.md +++ b/dev/docker/README.md @@ -16,8 +16,7 @@ Docker-based Test Cluster | YARN | Ubuntu 12.04 | HDP 2.2.8 (Hadoop 2.6.0) | | hdp2.2 | | YARN | Ubuntu 12.04 | HDP 2.3.2 (Hadoop 2.7.1) | | hdp2.3 | | YARN | Ubuntu 12.04 | Apache Hadoop 2.7.2 | | apache2.7 | -| MESOS | Ubuntu 12.04 | Apache Mesos 0.24.1 | | mesos0.24 | -| MESOS | Ubuntu 12.04 | Apache Mesos 0.25.0 | | mesos0.25 | +| MESOS | Ubuntu 12.04 | Apache Mesos 0.24~0.26 | | mesos0.24 ~ 0.26 | Please note that all images use Oracle JDK 7u80. @@ -76,9 +75,9 @@ If you see 'OK', it means all tests are passed. Test Apache REEF on a docker-based Mesos cluster ----------------------------------------------- -Among reefrt/mesos0.24 and reefrt/mesos0.25, +From reefrt/mesos0.24 to reefrt/mesos0.26, choose one and run it as described in section _Run a docker-based cluster_. -If you want to test on Mesos 0.25, choose reefrt/mesos0.25. +If you want to test on Mesos 0.26, choose reefrt/mesos0.26. After running a cluster, use the following commands. ```sh http://git-wip-us.apache.org/repos/asf/reef/blob/83f905db/dev/docker/ubuntu12.04-jdk7-mesos0.25/Dockerfile ---------------------------------------------------------------------- diff --git a/dev/docker/ubuntu12.04-jdk7-mesos0.25/Dockerfile b/dev/docker/ubuntu12.04-jdk7-mesos0.25/Dockerfile index 6acfedd..fb4923c 100644 --- a/dev/docker/ubuntu12.04-jdk7-mesos0.25/Dockerfile +++ b/dev/docker/ubuntu12.04-jdk7-mesos0.25/Dockerfile @@ -25,7 +25,7 @@ RUN \ apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF && \ echo "deb http://repos.mesosphere.io/ubuntu precise main" > /etc/apt/sources.list.d/mesosphere.list && \ apt-get -y update && \ - apt-get install -y mesos && \ + apt-get install -y mesos=0.25.0-0.2.70.ubuntu1204 && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* ENV HADOOP_HOME=$HADOOP_PREFIX http://git-wip-us.apache.org/repos/asf/reef/blob/83f905db/dev/docker/ubuntu12.04-jdk7-mesos0.26/Dockerfile ---------------------------------------------------------------------- diff --git a/dev/docker/ubuntu12.04-jdk7-mesos0.26/Dockerfile b/dev/docker/ubuntu12.04-jdk7-mesos0.26/Dockerfile new file mode 100644 index 0000000..622b89d --- /dev/null +++ b/dev/docker/ubuntu12.04-jdk7-mesos0.26/Dockerfile @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +FROM reefrt/apache2.7 +MAINTAINER Apache REEF <[email protected]> + +RUN ln -s /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so /usr/bin/libjvm.so + +# Apache Mesos 0.26.0 +RUN \ + apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF && \ + echo "deb http://repos.mesosphere.io/ubuntu precise main" > /etc/apt/sources.list.d/mesosphere.list && \ + apt-get -y update && \ + apt-get install -y mesos=0.26.0-0.2.145.ubuntu1204 && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists/* +ENV HADOOP_HOME=$HADOOP_PREFIX +RUN echo 'PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/hadoop/bin"' >> /etc/environment +COPY init-nn.sh /root/ + +EXPOSE 22 5050 http://git-wip-us.apache.org/repos/asf/reef/blob/83f905db/dev/docker/ubuntu12.04-jdk7-mesos0.26/init-nn.sh ---------------------------------------------------------------------- diff --git a/dev/docker/ubuntu12.04-jdk7-mesos0.26/init-nn.sh b/dev/docker/ubuntu12.04-jdk7-mesos0.26/init-nn.sh new file mode 100755 index 0000000..d848849 --- /dev/null +++ b/dev/docker/ubuntu12.04-jdk7-mesos0.26/init-nn.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +/usr/sbin/sshd + +hostname > /usr/etc/mesos/masters +grep hdn /etc/hosts | awk '{print $1}' | sort | uniq > $HADOOP_PREFIX/etc/hadoop/slaves +cp $HADOOP_PREFIX/etc/hadoop/slaves /usr/etc/mesos/slaves +for host in `cat $HADOOP_PREFIX/etc/hadoop/slaves` +do + scp /etc/hosts $host:/etc/hosts + scp $HADOOP_PREFIX/etc/hadoop/slaves $host:$HADOOP_PREFIX/etc/hadoop/slaves + scp /usr/etc/mesos/masters $host:/usr/etc/mesos/masters + scp /usr/etc/mesos/slaves $host:/usr/etc/mesos/slaves + ssh $host mesos-daemon.sh mesos-slave --master=hnn-001-01:5050 +done + +/usr/local/hadoop/bin/hdfs namenode -format +/usr/local/hadoop/sbin/start-dfs.sh +mesos-daemon.sh mesos-master --cluster=REEF --work_dir=/var/lib/mesos --log_dir=/var/log/mesos + +cd ~ && /bin/bash
