Repository: reef Updated Branches: refs/heads/master 3d9b5c2fc -> a3ddf2171
[REEF-1213] Test Mesos 0.27.2 and 0.28.0 and add to `dev/docker` Now, REEF integration test passes on Apache Mesos 0.27.2 and 0.28.0. JIRA: [REEF-1213](https://issues.apache.org/jira/browse/REEF-1213) Pull request: This closes #858 Project: http://git-wip-us.apache.org/repos/asf/reef/repo Commit: http://git-wip-us.apache.org/repos/asf/reef/commit/a3ddf217 Tree: http://git-wip-us.apache.org/repos/asf/reef/tree/a3ddf217 Diff: http://git-wip-us.apache.org/repos/asf/reef/diff/a3ddf217 Branch: refs/heads/master Commit: a3ddf21710d3beb5f44a42e2e9c3298eeff87273 Parents: 3d9b5c2 Author: Dongjoon Hyun <[email protected]> Authored: Wed Feb 24 10:15:09 2016 -0800 Committer: John Yang <[email protected]> Committed: Sun Apr 3 10:13:47 2016 +0900 ---------------------------------------------------------------------- dev/docker/README.md | 6 +-- .../ubuntu12.04-jdk7-mesos0.27/Dockerfile | 35 ++++++++++++++++++ .../ubuntu12.04-jdk7-mesos0.27/init-nn.sh | 39 ++++++++++++++++++++ .../ubuntu12.04-jdk7-mesos0.28/Dockerfile | 35 ++++++++++++++++++ .../ubuntu12.04-jdk7-mesos0.28/init-nn.sh | 39 ++++++++++++++++++++ 5 files changed, 151 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/reef/blob/a3ddf217/dev/docker/README.md ---------------------------------------------------------------------- diff --git a/dev/docker/README.md b/dev/docker/README.md index 2e13a46..e85ff8b 100644 --- a/dev/docker/README.md +++ b/dev/docker/README.md @@ -16,7 +16,7 @@ Docker-based Test Cluster | YARN | Ubuntu 12.04 | HDP 2.2.8 (Hadoop 2.6.0) | | hdp2.2 | | YARN | Ubuntu 12.04 | HDP 2.3.2 (Hadoop 2.7.1) | | hdp2.3 | | YARN | Ubuntu 12.04 | Apache Hadoop 2.7.2 | | apache2.7 | -| MESOS | Ubuntu 12.04 | Apache Mesos 0.24~0.26 | | mesos0.24 ~ 0.26 | +| MESOS | Ubuntu 12.04 | Apache Mesos 0.24~0.28 | | mesos0.24 ~ 0.28 | Please note that all images use Oracle JDK 7u80. @@ -75,9 +75,9 @@ If you see 'OK', it means all tests are passed. Test Apache REEF on a docker-based Mesos cluster ----------------------------------------------- -From reefrt/mesos0.24 to reefrt/mesos0.26, +From reefrt/mesos0.24 to reefrt/mesos0.28, choose one and run it as described in section _Run a docker-based cluster_. -If you want to test on Mesos 0.26, choose reefrt/mesos0.26. +If you want to test on Mesos 0.28, choose reefrt/mesos0.28. After running a cluster, use the following commands. ```sh http://git-wip-us.apache.org/repos/asf/reef/blob/a3ddf217/dev/docker/ubuntu12.04-jdk7-mesos0.27/Dockerfile ---------------------------------------------------------------------- diff --git a/dev/docker/ubuntu12.04-jdk7-mesos0.27/Dockerfile b/dev/docker/ubuntu12.04-jdk7-mesos0.27/Dockerfile new file mode 100644 index 0000000..40a4683 --- /dev/null +++ b/dev/docker/ubuntu12.04-jdk7-mesos0.27/Dockerfile @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +FROM reefrt/apache2.7 +MAINTAINER Apache REEF <[email protected]> + +RUN ln -s /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so /usr/bin/libjvm.so + +# Apache Mesos 0.27.2 +RUN \ + apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF && \ + echo "deb http://repos.mesosphere.io/ubuntu precise main" > /etc/apt/sources.list.d/mesosphere.list && \ + apt-get -y update && \ + apt-get install -y mesos=0.27.2-2.0.15.ubuntu1204 && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists/* +ENV HADOOP_HOME=$HADOOP_PREFIX +RUN echo 'PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/hadoop/bin"' >> /etc/environment +COPY init-nn.sh /root/ + +EXPOSE 22 5050 http://git-wip-us.apache.org/repos/asf/reef/blob/a3ddf217/dev/docker/ubuntu12.04-jdk7-mesos0.27/init-nn.sh ---------------------------------------------------------------------- diff --git a/dev/docker/ubuntu12.04-jdk7-mesos0.27/init-nn.sh b/dev/docker/ubuntu12.04-jdk7-mesos0.27/init-nn.sh new file mode 100755 index 0000000..d848849 --- /dev/null +++ b/dev/docker/ubuntu12.04-jdk7-mesos0.27/init-nn.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +/usr/sbin/sshd + +hostname > /usr/etc/mesos/masters +grep hdn /etc/hosts | awk '{print $1}' | sort | uniq > $HADOOP_PREFIX/etc/hadoop/slaves +cp $HADOOP_PREFIX/etc/hadoop/slaves /usr/etc/mesos/slaves +for host in `cat $HADOOP_PREFIX/etc/hadoop/slaves` +do + scp /etc/hosts $host:/etc/hosts + scp $HADOOP_PREFIX/etc/hadoop/slaves $host:$HADOOP_PREFIX/etc/hadoop/slaves + scp /usr/etc/mesos/masters $host:/usr/etc/mesos/masters + scp /usr/etc/mesos/slaves $host:/usr/etc/mesos/slaves + ssh $host mesos-daemon.sh mesos-slave --master=hnn-001-01:5050 +done + +/usr/local/hadoop/bin/hdfs namenode -format +/usr/local/hadoop/sbin/start-dfs.sh +mesos-daemon.sh mesos-master --cluster=REEF --work_dir=/var/lib/mesos --log_dir=/var/log/mesos + +cd ~ && /bin/bash http://git-wip-us.apache.org/repos/asf/reef/blob/a3ddf217/dev/docker/ubuntu12.04-jdk7-mesos0.28/Dockerfile ---------------------------------------------------------------------- diff --git a/dev/docker/ubuntu12.04-jdk7-mesos0.28/Dockerfile b/dev/docker/ubuntu12.04-jdk7-mesos0.28/Dockerfile new file mode 100644 index 0000000..2a466b4 --- /dev/null +++ b/dev/docker/ubuntu12.04-jdk7-mesos0.28/Dockerfile @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +FROM reefrt/apache2.7 +MAINTAINER Apache REEF <[email protected]> + +RUN ln -s /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so /usr/bin/libjvm.so + +# Apache Mesos 0.28.0 +RUN \ + apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF && \ + echo "deb http://repos.mesosphere.io/ubuntu precise main" > /etc/apt/sources.list.d/mesosphere.list && \ + apt-get -y update && \ + apt-get install -y mesos=0.28.0-2.0.16.ubuntu1204 && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists/* +ENV HADOOP_HOME=$HADOOP_PREFIX +RUN echo 'PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/hadoop/bin"' >> /etc/environment +COPY init-nn.sh /root/ + +EXPOSE 22 5050 http://git-wip-us.apache.org/repos/asf/reef/blob/a3ddf217/dev/docker/ubuntu12.04-jdk7-mesos0.28/init-nn.sh ---------------------------------------------------------------------- diff --git a/dev/docker/ubuntu12.04-jdk7-mesos0.28/init-nn.sh b/dev/docker/ubuntu12.04-jdk7-mesos0.28/init-nn.sh new file mode 100755 index 0000000..d848849 --- /dev/null +++ b/dev/docker/ubuntu12.04-jdk7-mesos0.28/init-nn.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +/usr/sbin/sshd + +hostname > /usr/etc/mesos/masters +grep hdn /etc/hosts | awk '{print $1}' | sort | uniq > $HADOOP_PREFIX/etc/hadoop/slaves +cp $HADOOP_PREFIX/etc/hadoop/slaves /usr/etc/mesos/slaves +for host in `cat $HADOOP_PREFIX/etc/hadoop/slaves` +do + scp /etc/hosts $host:/etc/hosts + scp $HADOOP_PREFIX/etc/hadoop/slaves $host:$HADOOP_PREFIX/etc/hadoop/slaves + scp /usr/etc/mesos/masters $host:/usr/etc/mesos/masters + scp /usr/etc/mesos/slaves $host:/usr/etc/mesos/slaves + ssh $host mesos-daemon.sh mesos-slave --master=hnn-001-01:5050 +done + +/usr/local/hadoop/bin/hdfs namenode -format +/usr/local/hadoop/sbin/start-dfs.sh +mesos-daemon.sh mesos-master --cluster=REEF --work_dir=/var/lib/mesos --log_dir=/var/log/mesos + +cd ~ && /bin/bash
