Repository: incubator-rya
Updated Branches:
refs/heads/develop 64284db6f -> 193723396
closes RYA-54 Abort on errors, quieter, rerunable, wait for deploy race fix.
TODO:replace dropbox.
Conflicts:
extras/vagrantExample/src/main/vagrant/Vagrantfile
Project: http://git-wip-us.apache.org/repos/asf/incubator-rya/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-rya/commit/19372339
Tree: http://git-wip-us.apache.org/repos/asf/incubator-rya/tree/19372339
Diff: http://git-wip-us.apache.org/repos/asf/incubator-rya/diff/19372339
Branch: refs/heads/develop
Commit: 1937233962e9fabbdce036820c17731cb15de3e6
Parents: 64284db
Author: David W. Lotts <david.lo...@parsons.com>
Authored: Thu Jun 2 18:03:44 2016 -0400
Committer: Aaron Mihalik <miha...@alum.mit.edu>
Committed: Wed Aug 24 09:33:51 2016 -0400
----------------------------------------------------------------------
.../vagrantExample/src/main/vagrant/Vagrantfile | 329 ++++++++++++-------
.../vagrantExample/src/main/vagrant/readme.md | 101 +++++-
2 files changed, 293 insertions(+), 137 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-rya/blob/19372339/extras/vagrantExample/src/main/vagrant/Vagrantfile
----------------------------------------------------------------------
diff --git a/extras/vagrantExample/src/main/vagrant/Vagrantfile
b/extras/vagrantExample/src/main/vagrant/Vagrantfile
index 6fb2ae7..e3f2a73 100644
--- a/extras/vagrantExample/src/main/vagrant/Vagrantfile
+++ b/extras/vagrantExample/src/main/vagrant/Vagrantfile
@@ -18,6 +18,11 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
+#
+# Builds a single node Rya on Accumulo on Hadoop and Zookeeper.
+# Deploys Rya, and the Sesame openrdf-workbench on Tomcat accessable from the
host's browser.
+# See the accompanying readme for URL's, verification, and troubleshooting.
+#
# Note: Machine's ip is 192.168.33.10
# username : vagrant
# password : vagrant
@@ -28,11 +33,8 @@
Vagrant.configure(2) do |config|
- config.vm.box = "ubuntu/trusty64"
+ config.vm.box = "ubuntu/trusty64"
config.vm.box_url =
"https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box";
-
-
-
config.vm.provider "virtualbox" do |vb|
vb.name = "rya-example-box"
@@ -43,9 +45,12 @@ Vagrant.configure(2) do |config|
config.vm.network :private_network, ip: "192.168.33.10"
config.vm.hostname = "rya-example-box"
- config.vm.provision "shell", inline: <<-SHELL
+ config.vm.provision "shell", inline: <<-SHELL
- export ACCUMULO_VERSION=1.7.1
+ ###set -x ## turn on command echo with expanded variables
+ # List of dependency versions
+ export ACCUMULO_VERSION=1.6.5
+ ###export ACCUMULO_VERSION=1.7.1
export HADOOP_VERSION=2.7.2
export RYA_EXAMPLE_VERSION=3.2.10-SNAPSHOT
export SESAME_VERSION=2.7.6
@@ -57,33 +62,39 @@ Vagrant.configure(2) do |config|
192.168.33.10 zoo1 zoo2 zoo3
EOF
- sudo apt-get update
+ sudo -E apt-get -qq update
- echo "Installing Sun Java..."
- sudo add-apt-repository ppa:webupd8team/java
- sudo apt-get update
+ echo "Installing Java installer..."
+ sudo -E add-apt-repository ppa:webupd8team/java || exit $?
+ sudo -E apt-get -qq update || exit $?
echo debconf shared/accepted-oracle-license-v1-1 select true | \
- sudo /usr/bin/debconf-set-selections
+ sudo -E /usr/bin/debconf-set-selections
echo debconf shared/accepted-oracle-license-v1-1 seen true | \
- sudo /usr/bin/debconf-set-selections
- sudo apt-get install -y oracle-java8-installer
- sudo ln -s /usr/lib/jvm/java-8-oracle/ /usr/lib/jvm/default-java
+ sudo -E /usr/bin/debconf-set-selections
+ sudo mkdir --parents /var/cache/oracle-jdk8-installer || exit $?
+ echo verbose=off >> /var/cache/oracle-jdk8-installer/wgetrc || exit $?
+
+ echo "Installing Java..."
+ sudo -E apt-get -qq install -y oracle-java8-installer || exit $?
+ sudo ln --force -s /usr/lib/jvm/java-8-oracle/ /usr/lib/jvm/default-java
echo "Installing Tomcat..."
- sudo apt-get install -y tomcat7
+ sudo -E apt-get install -y tomcat7 || exit $?
echo "Installing Unzip..."
- apt-get install unzip
+ apt-get install unzip || exit $?
echo "Setting up environment..."
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export HADOOP_HOME=/home/vagrant/hadoop-${HADOOP_VERSION}
export ZOOKEEPER_HOME=/home/vagrant/zookeeper-${ZOOKEEPER_VERSION}
+ export ZOO_LOG_DIR=${ZOOKEEPER_HOME}/logs/
export ACCUMULO_HOME=/home/vagrant/accumulo-${ACCUMULO_VERSION}
- export PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$ACCUMULO_HOME/bin
+ export
PATHADD=$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$ACCUMULO_HOME/bin:$HADOOP_HOME/bin
+ export PATH=$PATH:$PATHADD
export HADOOP_PREFIX="$HADOOP_HOME"
- export HADOOP_CONF_DIR="$HADOOP_PREFIX/conf"
+ export HADOOP_CONF_DIR="$HADOOP_PREFIX/etc/hadoop"
export ACCUMULO_LOG_DIR=$ACCUMULO_HOME/logs
export ACCUMULO_TSERVER_OPTS="-Xmx384m -Xms384m "
export ACCUMULO_MASTER_OPTS="-Xmx128m -Xms128m"
@@ -92,67 +103,120 @@ EOF
export ACCUMULO_GENERAL_OPTS="-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -Djava.net.preferIPv4Stack=true"
export ACCUMULO_OTHER_OPTS="-Xmx128m -Xms64m"
export ACCUMULO_KILL_CMD='kill -9 %p'
+ # Shell environment includes Accumulo resources.
+ ACCUMULO_RC=/home/vagrant/.accumulo_rc.sh
+ cat > ${ACCUMULO_RC} <<EOF
- cat >> /home/vagrant/.bashrc <<EOF
- export JAVA_HOME=/usr/lib/jvm/java-8-oracle
- export HADOOP_HOME=/home/vagrant/hadoop-${HADOOP_VERSION}
- export ZOOKEEPER_HOME=/home/vagrant/zookeeper-${ZOOKEEPER_VERSION}
- export ACCUMULO_HOME=/home/vagrant/accumulo-${ACCUMULO_VERSION}
- export PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$ACCUMULO_HOME/bin
+ export JAVA_HOME=\'$JAVA_HOME\'
+ export HADOOP_HOME=\'$HADOOP_HOME\'
+ export ZOOKEEPER_HOME=\'$ZOOKEEPER_HOME\'
+ export ZOO_LOG_DIR=\'$ZOO_LOG_DIR\'
+ export ACCUMULO_HOME=\'$ACCUMULO_HOME\'
+ export PATH=\\$PATH:$PATHADD
- export HADOOP_PREFIX="$HADOOP_HOME"
- export HADOOP_CONF_DIR="$HADOOP_PREFIX/conf"
- export ACCUMULO_LOG_DIR=$ACCUMULO_HOME/logs
- export ACCUMULO_TSERVER_OPTS="-Xmx384m -Xms384m "
- export ACCUMULO_MASTER_OPTS="-Xmx128m -Xms128m"
- export ACCUMULO_MONITOR_OPTS="-Xmx64m -Xms64m"
- export ACCUMULO_GC_OPTS="-Xmx64m -Xms64m"
- export ACCUMULO_GENERAL_OPTS="-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -Djava.net.preferIPv4Stack=true"
- export ACCUMULO_OTHER_OPTS="-Xmx128m -Xms64m"
- export ACCUMULO_KILL_CMD='kill -9 %p'
+ export HADOOP_PREFIX=\'$HADOOP_PREFIX\'
+ export HADOOP_CONF_DIR=\'$HADOOP_CONF_DIR\'
+ export ACCUMULO_LOG_DIR=\'$ACCUMULO_LOG_DIR\'
+ export ACCUMULO_TSERVER_OPTS=\'$ACCUMULO_TSERVER_OPTS\'
+ export ACCUMULO_MASTER_OPTS=\'$ACCUMULO_MASTER_OPTS\'
+ export ACCUMULO_MONITOR_OPTS=\'$ACCUMULO_MONITOR_OPTS\'
+ export ACCUMULO_GC_OPTS=\'$ACCUMULO_GC_OPTS\'
+ export ACCUMULO_GENERAL_OPTS=\'$ACCUMULO_GENERAL_OPTS\'
+ export ACCUMULO_OTHER_OPTS=\'$ACCUMULO_OTHER_OPTS\'
+ export ACCUMULO_KILL_CMD=\'$ACCUMULO_KILL_CMD\'
+
+ ### command to list the 7 correct java processes: tomcat-catalina,
zookeeper, and 5 Accumulo: tracer, master, monitor, tserver, gc.
+ function ryaps() { ps -ef | grep java | tr ' ' '\\n' | egrep
'^org\\.apache|^tracer|^master|^monitor|^tserver|^gc' | sed '/\\.Main/ N ;
s/\\n/ /' ; }
EOF
-
-
- echo "Acquiring and Extracting ..."
+ source ${ACCUMULO_RC} || exit 151
+
+ # include it at the beginning of both shell configuration files.
+
+ for BASHRC in /home/vagrant/.bashrc /home/vagrant/.bash_profile ;
+ do touch ${BASHRC} ;
+ cat - ${BASHRC} > ${BASHRC}.new <<EOF && mv ${BASHRC}.new ${BASHRC}
|| exit 152
+source ${ACCUMULO_RC}
+EOF
+ done
+ echo "Acquiring and Extracting ..."
+ function echoerr() { printf "%s\n" "$*" >&2; }
+
function download {
- curl -f "$@"
+ ### curl --fail treat http status >= 400 as an error. --location follow
redirects status>=300
+ curl --silent --show-error --fail --location "$@"
if [ $? -ne 0 ]; then
- echo "--------------------------"
- echo "-"
- echo "- download failed" "$@"
- echo "-"
- echo "-" exiting ...
- echo "-"
- echo "--------------------------"
- exit 1
+ echoerr "--------------------------"
+ echoerr "-"
+ echoerr "- download failed" "$@"
+ echoerr "-"
+ echoerr "-" exiting ...
+ echoerr "-"
+ echoerr "--------------------------"
+ exit 800
fi
}
+ ### wait for a directory to exist or 60 seconds timeout
+ function waitForDeploy {
+ waitfordir="$1"
+ timeout=60
+ while [[ ! -d "$waitfordir" ]]
+ do
+ sleep 5
+ let timeout-=5
+ if [[ $timeout -le "0" ]]; then
+ echo "Timeout waiting for war to deploy, $waitfordir still
does not exist.";
+ exit 401
+ fi
+ done
+ }
echo "- Hadoop"
- download -SLO
http://apache.mirrors.tds.net/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz
- tar -zxf hadoop-${HADOOP_VERSION}.tar.gz
+
hadoopUrl=http://apache.mirrors.tds.net/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz
+ if [[ ! -d ${HADOOP_HOME} ]] ; then
+ echo "Downloading $hadoopUrl"
+ download $hadoopUrl \
+ | tar -zxC /home/vagrant || exit 101
+ fi
echo "- Zookeeper"
- download -SLO
http://archive-primary.cloudera.com/cdh4/cdh/4/zookeeper-${ZOOKEEPER_VERSION}.tar.gz
- tar -zxf zookeeper-${ZOOKEEPER_VERSION}.tar.gz
-
+
zookeeperUrl=http://archive-primary.cloudera.com/cdh4/cdh/4/zookeeper-${ZOOKEEPER_VERSION}.tar.gz
+ if [[ ! -d ${ZOOKEEPER_HOME} ]] ; then
+ echo "Downloading $zookeeperUrl"
+ download $zookeeperUrl \
+ | tar -zxC /home/vagrant || exit 102
+ fi
+
echo "- Accumulo"
- download -SLO
http://apache.mirrors.pair.com/accumulo/${ACCUMULO_VERSION}/accumulo-${ACCUMULO_VERSION}-bin.tar.gz
- tar -zxf accumulo-${ACCUMULO_VERSION}-bin.tar.gz
+
accumuloUrl=http://apache.mirrors.pair.com/accumulo/${ACCUMULO_VERSION}/accumulo-${ACCUMULO_VERSION}-bin.tar.gz
+ if [[ ! -d ${ACCUMULO_HOME} ]] ; then
+ echo "Downloading $accumuloUrl"
+ download $accumuloUrl \
+ | tar -zxC /home/vagrant || exit 103
+ fi
echo "Configuring Zookeeper..."
- sudo mkdir /var/zookeeper
+ sudo mkdir --parents /var/zookeeper
sudo chown vagrant:vagrant /var/zookeeper
+ sudo cp ${ZOOKEEPER_HOME}/conf/zoo_sample.cfg
${ZOOKEEPER_HOME}/conf/zoo.cfg
+ # Change the dataDir to /var/zookeeper -- strange that backslashes need to
be escaped only on this line.
+ sudo sed -i 's/^\\s*dataDir\\s*=.*$/dataDir=\\/var\\/zookeeper/'
${ZOOKEEPER_HOME}/conf/zoo.cfg || exit 104
+ # Conflicts with Accumulo and maybe Zookeeper
+ sudo rm --force ${HADOOP_HOME}/share/hadoop/common/lib/slf4j-api-1.7.10.jar
+ sudo rm --force
${HADOOP_HOME}/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar
+
+ # Assure logs are creatable and writeable
+ sudo mkdir --parents ${ZOO_LOG_DIR}
+ sudo touch "${ZOO_LOG_DIR}/zookeeper.out"
+ sudo chmod -R a+wX ${ZOO_LOG_DIR}
echo "Running Zookeeper..."
- cp zookeeper-${ZOOKEEPER_VERSION}/conf/zoo_sample.cfg
zookeeper-${ZOOKEEPER_VERSION}/conf/zoo.cfg
- sudo zookeeper-${ZOOKEEPER_VERSION}/bin/zkServer.sh start
+ sudo -E ${ZOOKEEPER_HOME}/bin/zkServer.sh start
echo "Configuring Accumulo..."
- cp accumulo-${ACCUMULO_VERSION}/conf/examples/1GB/standalone/*
accumulo-${ACCUMULO_VERSION}/conf/
- rm accumulo-${ACCUMULO_VERSION}/conf/accumulo-site.xml
- cat >> accumulo-${ACCUMULO_VERSION}/conf/accumulo-site.xml <<EOF
+ cp ${ACCUMULO_HOME}/conf/examples/1GB/standalone/* ${ACCUMULO_HOME}/conf/
+ rm --force ${ACCUMULO_HOME}/conf/accumulo-site.xml
+ cat >> ${ACCUMULO_HOME}/conf/accumulo-site.xml <<EOF
<configuration>
<property><name>instance.dfs.uri</name><value>file:///</value></property>
<property><name>instance.dfs.dir</name><value>/data/accumulo</value></property>
@@ -171,26 +235,26 @@ EOF
<property><name>tserver.compaction.major.delay</name><value>3</value></property>
<property><name>general.classpaths</name><value>
/data/accumulo/lib/[^.].*.jar,
- /home/vagrant/hadoop-${HADOOP_VERSION}/share/hadoop/common/.*.jar,
-
/home/vagrant/hadoop-${HADOOP_VERSION}/share/hadoop/common/lib/.*.jar,
- /home/vagrant/hadoop-${HADOOP_VERSION}/share/hadoop/hdfs/.*.jar,
-
/home/vagrant/hadoop-${HADOOP_VERSION}/share/hadoop/mapreduce/.*.jar,
- /home/vagrant/hadoop-${HADOOP_VERSION}/share/hadoop/yarn/.*.jar,
- /home/vagrant/accumulo-${ACCUMULO_VERSION}/server/target/classes/,
- /home/vagrant/accumulo-${ACCUMULO_VERSION}/lib/accumulo-server.jar,
- /home/vagrant/accumulo-${ACCUMULO_VERSION}/core/target/classes/,
- /home/vagrant/accumulo-${ACCUMULO_VERSION}/lib/accumulo-core.jar,
- /home/vagrant/accumulo-${ACCUMULO_VERSION}/start/target/classes/,
- /home/vagrant/accumulo-${ACCUMULO_VERSION}/lib/accumulo-start.jar,
- /home/vagrant/accumulo-${ACCUMULO_VERSION}/fate/target/classes/,
- /home/vagrant/accumulo-${ACCUMULO_VERSION}/lib/accumulo-fate.jar,
- /home/vagrant/accumulo-${ACCUMULO_VERSION}/proxy/target/classes/,
- /home/vagrant/accumulo-${ACCUMULO_VERSION}/lib/accumulo-proxy.jar,
- /home/vagrant/accumulo-${ACCUMULO_VERSION}/lib/[^.].*.jar,
- /home/vagrant/zookeeper-${ZOOKEEPER_VERSION}/zookeeper[^.].*.jar,
+ ${HADOOP_HOME}/share/hadoop/common/.*.jar,
+ ${HADOOP_HOME}/share/hadoop/common/lib/.*.jar,
+ ${HADOOP_HOME}/share/hadoop/hdfs/.*.jar,
+ ${HADOOP_HOME}/share/hadoop/mapreduce/.*.jar,
+ ${HADOOP_HOME}/share/hadoop/yarn/.*.jar,
+ ${ACCUMULO_HOME}/server/target/classes/,
+ ${ACCUMULO_HOME}/lib/accumulo-server.jar,
+ ${ACCUMULO_HOME}/core/target/classes/,
+ ${ACCUMULO_HOME}/lib/accumulo-core.jar,
+ ${ACCUMULO_HOME}/start/target/classes/,
+ ${ACCUMULO_HOME}/lib/accumulo-start.jar,
+ ${ACCUMULO_HOME}/fate/target/classes/,
+ ${ACCUMULO_HOME}/lib/accumulo-fate.jar,
+ ${ACCUMULO_HOME}/proxy/target/classes/,
+ ${ACCUMULO_HOME}/lib/accumulo-proxy.jar,
+ ${ACCUMULO_HOME}/lib/[^.].*.jar,
+ /home/vagrant/${ZOOKEEPER_HOME}/zookeeper[^.].*.jar,
$HADOOP_CONF_DIR,
- /home/vagrant/hadoop-${HADOOP_VERSION}/[^.].*.jar,
- /home/vagrant/hadoop-${HADOOP_VERSION}/lib/[^.].*.jar,
+ ${HADOOP_HOME}/[^.].*.jar,
+ ${HADOOP_HOME}/lib/[^.].*.jar,
</value></property>
<property><name>general.dynamic.classpaths</name><value>/data/accumulo/lib/ext/[^.].*.jar</value></property>
<property><name>trace.port.client</name><value>0</value></property>
@@ -200,74 +264,101 @@ EOF
<property><name>gc.port.client</name><value>0</value></property>
</configuration>
EOF
- cat > accumulo-${ACCUMULO_VERSION}/conf/masters <<EOF
+ cat > ${ACCUMULO_HOME}/conf/masters <<EOF
rya-example-box
EOF
- cat > accumulo-${ACCUMULO_VERSION}/conf/slaves <<EOF
+ cat > ${ACCUMULO_HOME}/conf/slaves <<EOF
rya-example-box
EOF
- sudo mkdir /data
+ sudo mkdir --parents /data
sudo chown vagrant:vagrant /data
- mkdir /data/accumulo
- mkdir /data/accumulo/lib
- mkdir /data/accumulo/lib/ext
+ mkdir --parents /data/accumulo/lib/ext
+
+ sudo chmod -R a+rwX ${ACCUMULO_HOME}/logs/
echo "Starting Accumulo..."
- accumulo-${ACCUMULO_VERSION}/bin/accumulo init --instance-name dev
--password root
- accumulo-${ACCUMULO_VERSION}/bin/start-all.sh
+ echo "Init will fail during a re-provision, but you can ignore it:
'FATAL: It appears the directories [...] were previously initialized.'"
+ echo "Also, you may see an indefinitely repeating: 'Waiting for accumulo
to be initialized' which means Accumulo won't start."
+ echo "Either issue can be resolved by removing the directory: 'sudo rm -r
/data/accumulo' then re-provision. Warning: this will erase all Rya/Accumulo
data."
+ ${ACCUMULO_HOME}/bin/accumulo init --instance-name dev --password root
+ ${ACCUMULO_HOME}/bin/start-all.sh || exit 107
+
+ sudo chmod -R a+rwX ${ACCUMULO_HOME}/logs/
echo 'Done!'
echo "Installing Sesame Server"
# creating log dir sesame-http-server-${SESAME_VERSION}
- sudo mkdir -p /usr/share/tomcat7/.aduna
+ sudo mkdir --parents /usr/share/tomcat7/.aduna
sudo chown -R tomcat7:tomcat7 /usr/share/tomcat7
- sudo ln -s /usr/share/tomcat7/.aduna/openrdf-sesame/logs
/var/log/tomcat7/openrdf-sesame
-
- download -O
http://repo1.maven.org/maven2/org/openrdf/sesame/sesame-http-server/${SESAME_VERSION}/sesame-http-server-${SESAME_VERSION}.war
- sudo mv sesame-http-server-${SESAME_VERSION}.war
/var/lib/tomcat7/webapps/openrdf-sesame.war
+ sudo ln --force -s /usr/share/tomcat7/.aduna/openrdf-sesame/logs
/var/log/tomcat7/openrdf-sesame
+ sesamewar=/var/lib/tomcat7/webapps/openrdf-sesame.war
+ if [[ ! -s $sesamewar ]] ; then
+ echo "Downloading"
+ download --output $sesamewar
http://repo1.maven.org/maven2/org/openrdf/sesame/sesame-http-server/${SESAME_VERSION}/sesame-http-server-${SESAME_VERSION}.war
|| exit 110
+ fi
echo "Sesame http server deployed at
http://rya-example-box:8080/openrdf-sesame";
echo "Installing Sesame Workbench"
- download -O
http://repo1.maven.org/maven2/org/openrdf/sesame/sesame-http-workbench/${SESAME_VERSION}/sesame-http-workbench-${SESAME_VERSION}.war
- sudo mv sesame-http-workbench-${SESAME_VERSION}.war
/var/lib/tomcat7/webapps/openrdf-workbench.war
+ workbench=/var/lib/tomcat7/webapps/openrdf-workbench.war
+ if [[ ! -s $workbench ]] ; then
+ echo "Downloading"
+ download --output $workbench
http://repo1.maven.org/maven2/org/openrdf/sesame/sesame-http-workbench/${SESAME_VERSION}/sesame-http-workbench-${SESAME_VERSION}.war
|| exit 111
+ fi
echo "Sesame workbench deployed at
http://rya-example-box:8080/openrdf-workbench";
- echo "Downloading Rya"
- # Right now it's on dropbox, but eventually it'll be on maven...
-
- download -L
https://www.dropbox.com/s/7e74yiuq4jmu0od/rya.indexing.example-${RYA_EXAMPLE_VERSION}-distribution.zip?dl=0
-o rya.indexing.example-${RYA_EXAMPLE_VERSION}-distribution.zip
- sudo mkdir rya.indexing.example-${RYA_EXAMPLE_VERSION}-distribution
- sudo unzip rya.indexing.example-${RYA_EXAMPLE_VERSION}-distribution.zip
-d rya.indexing.example-${RYA_EXAMPLE_VERSION}-distribution
+ echo "Installing Rya"
+ ryaIndexing=rya.indexing.example-${RYA_EXAMPLE_VERSION}-distribution
+ if [[ ! -s ${ryaIndexing}.zip ]] ; then
+ # Right now it's on dropbox, but eventually it'll be on maven...
+ echo "Downloading quietly, this will take some minutes with no
output..."
+ download --output ${ryaIndexing}.zip
https://dl.dropbox.com/s/7e74yiuq4jmu0od/${ryaIndexing}.zip?raw=1 || exit 112
+ fi
+ sudo mkdir --parents ${ryaIndexing}
+ sudo unzip -q -o ${ryaIndexing}.zip -d ${ryaIndexing}
+ # before continueing, wait for tomcat to deploy wars:
+ waitForDeploy /var/lib/tomcat7/webapps/openrdf-workbench/WEB-INF/lib/
+ waitForDeploy /var/lib/tomcat7/webapps/openrdf-sesame/WEB-INF/lib/
+
# soft linking the files doesn't seem to work in tomcat, so we copy
them instead :(
- sudo cp
rya.indexing.example-${RYA_EXAMPLE_VERSION}-distribution/dist/lib/*
/var/lib/tomcat7/webapps/openrdf-workbench/WEB-INF/lib/
- sudo cp
rya.indexing.example-${RYA_EXAMPLE_VERSION}-distribution/dist/lib/*
/var/lib/tomcat7/webapps/openrdf-sesame/WEB-INF/lib/
+ sudo cp ${ryaIndexing}/dist/lib/*
/var/lib/tomcat7/webapps/openrdf-workbench/WEB-INF/lib/ || exit 113
+ sudo cp ${ryaIndexing}/dist/lib/*
/var/lib/tomcat7/webapps/openrdf-sesame/WEB-INF/lib/ || exit 114
# These are older libs that breaks tomcat 7
- sudo rm
/var/lib/tomcat7/webapps/openrdf-workbench/WEB-INF/lib/servlet-api-2.5.jar
- sudo rm
/var/lib/tomcat7/webapps/openrdf-workbench/WEB-INF/lib/jsp-api-2.1.jar
- sudo rm
/var/lib/tomcat7/webapps/openrdf-sesame/WEB-INF/lib/servlet-api-2.5.jar
- sudo rm
/var/lib/tomcat7/webapps/openrdf-sesame/WEB-INF/lib/jsp-api-2.1.jar
+ sudo rm --force
/var/lib/tomcat7/webapps/openrdf-workbench/WEB-INF/lib/servlet-api-2.5.jar
+ sudo rm --force
/var/lib/tomcat7/webapps/openrdf-workbench/WEB-INF/lib/jsp-api-2.1.jar
+ sudo rm --force
/var/lib/tomcat7/webapps/openrdf-sesame/WEB-INF/lib/servlet-api-2.5.jar
+ sudo rm --force
/var/lib/tomcat7/webapps/openrdf-sesame/WEB-INF/lib/jsp-api-2.1.jar
sudo chown -R tomcat7:tomcat7
/var/lib/tomcat7/webapps/openrdf-workbench/WEB-INF/lib/
sudo chown -R tomcat7:tomcat7
/var/lib/tomcat7/webapps/openrdf-sesame/WEB-INF/lib/
- # Download and install new templates for OpenRdf WorkBench
- download -L -O
https://www.dropbox.com/s/dgw63m66nubyy4z/rya.vagrant.example-${RYA_EXAMPLE_VERSION}.jar
- sudo mkdir rya.vagrant.example-${RYA_EXAMPLE_VERSION}
- sudo unzip rya.vagrant.example-${RYA_EXAMPLE_VERSION}.jar -d
rya.vagrant.example-${RYA_EXAMPLE_VERSION}
- sudo cp rya.vagrant.example-${RYA_EXAMPLE_VERSION}/*.xsl
/var/lib/tomcat7/webapps/openrdf-workbench/transformations/
+ echo "Downloading and installing new templates for OpenRdf WorkBench"
+ ryaVagrant=rya.vagrant.example-${RYA_EXAMPLE_VERSION}
+ if [[ ! -s ${ryaVagrant}.jar ]] ; then
+ echo "Downloading"
+ download --output ${ryaVagrant}.jar
https://dl.dropbox.com/s/dgw63m66nubyy4z/${ryaVagrant}.jar?raw=1 || exit 120
+ fi
+ sudo mkdir --parents ${ryaVagrant}
+ sudo unzip -q -o ${ryaVagrant}.jar -d ${ryaVagrant}
+ sudo cp ${ryaVagrant}/*.xsl
/var/lib/tomcat7/webapps/openrdf-workbench/transformations/
+ sudo chown tomcat7:tomcat7
/var/lib/tomcat7/webapps/openrdf-workbench/transformations/*
echo "Deploying Rya Web"
- download -L
https://www.dropbox.com/s/332wr4b2f34dp6e/web.rya-${RYA_EXAMPLE_VERSION}.war?dl=0
-o web.rya.war
- sudo cp web.rya.war /var/lib/tomcat7/webapps/web.rya.war
+ ryaWar=web.rya-${RYA_EXAMPLE_VERSION}.war
+ if [[ ! -s ${ryaWar} ]] ; then
+ echo "Downloading"
+ download
https://dl.dropbox.com/s/332wr4b2f34dp6e/${ryaWar}?raw=1 --output ${ryaWar} ||
exit 121
+ fi
+ sudo cp ${ryaWar} /var/lib/tomcat7/webapps/web.rya.war
# Wait for the war to deploy
- sudo sleep 10
+ waitForDeploy /var/lib/tomcat7/webapps/web.rya/WEB-INF/classes/
+
# These are older libs that breaks tomcat 7
- sudo rm
/var/lib/tomcat7/webapps/web.rya/WEB-INF/lib/servlet-api-2.5*.jar
- sudo rm /var/lib/tomcat7/webapps/web.rya/WEB-INF/lib/jsp-api-2.1.jar
+ sudo rm --force
/var/lib/tomcat7/webapps/web.rya/WEB-INF/lib/servlet-api-2.5*.jar
+ sudo rm --force
/var/lib/tomcat7/webapps/web.rya/WEB-INF/lib/jsp-api-2.1.jar
echo "Modify Rya Web Config"
cat >
/var/lib/tomcat7/webapps/web.rya/WEB-INF/classes/environment.properties <<EOF
@@ -282,12 +373,10 @@ EOF
echo "Rya web deployed at
http://rya-example-box:8080/web.rya/sparqlQuery.jsp";
# restart tomcat
- sudo service tomcat7 restart
-
-
-
-
+ sudo -E service tomcat7 restart
+ echo "Finished and ready to use!"
+ echo "You can re-apply these settings without losing data by running the
command 'vagrant provision'"
SHELL
end
http://git-wip-us.apache.org/repos/asf/incubator-rya/blob/19372339/extras/vagrantExample/src/main/vagrant/readme.md
----------------------------------------------------------------------
diff --git a/extras/vagrantExample/src/main/vagrant/readme.md
b/extras/vagrantExample/src/main/vagrant/readme.md
index c5b89a3..235d28d 100644
--- a/extras/vagrantExample/src/main/vagrant/readme.md
+++ b/extras/vagrantExample/src/main/vagrant/readme.md
@@ -47,9 +47,9 @@ Once Vagrant is installed, starting up the Rya-Example-Box is
fairly straightfor
By default, the VM should be assigned the IP address of `192.168.33.10`. This
value is defined in the Vagrantfile and it is configurable. All of the
procedures in this in this document assume that the hostname `rya-example-box`
resolves to the VMâs IP address `192.168.33.10`. The easiest way to do this
is to add the entry `192.168.33.10 rya-example-box` to your machineâs host
file (e.g., for windows users modify `C:\Windows\System32\drivers\etc\hosts` or
for Mac/Linux users modify `/etc/hosts`)
-1. **Verify the Tomcat instance**: Open a browser to
<http://rya-example-box:8080/>. You should see a webpage that says âIt
works! If you're seeing this page via a web browser, it means you've setup
Tomcat successfully. Congratulations!â
+1. **Verify the Tomcat instance**: Open a browser to
<http://rya-example-box:8080/>. You should see a webpage that says âIt
works! If you're seeing this page via a web browser, it means you've setup
Tomcat successfully. Congratulations!
-1. **Verify the deployed OpenRDF Sesame service**: Open your browser to
<http://rya-example-box:8080/openrdf-sesame/protocol> and you should see a
â6â (this is the OpenRDF Protocol Version).
+1. **Verify the deployed OpenRDF Sesame service**: Open your browser to
<http://rya-example-box:8080/openrdf-sesame/protocol> and you should see a `6`
(this is the OpenRDF Protocol Version).
1. **Verify the deployed OpenRDF Workbench**: Open your browser to
<http://rya-example-box:8080/openrdf-workbench>
@@ -57,7 +57,7 @@ By default, the VM should be assigned the IP address of
`192.168.33.10`. This v
1. **Ssh into into the VM**: SSH from your host machine to `rya-example-box`
with user/pass of vagrant/vagrant.
-1. **Test the Accumulo shell**: After ssh'ing into the VM, run the command:
`/home/vagrant/accumulo-1.6.4/bin/accumulo shell -u root -p root`
+1. **Test the Accumulo shell**: After ssh'ing into the VM, run the command:
`/home/vagrant/accumulo-1.6.5/bin/accumulo shell -u root -p root`
### Common Errors on the VM
@@ -94,28 +94,93 @@ If these files do note exists, open the vagrant file and
look for the line `echo
Below is a list of other useful commands on the VMs
-*Restart Tomcat*
+#####*Log into the VM*
+1. Go to the folder containing "vagrantfile", run:
+ `cd extras/vagrantExample/src/main/vagrant/`
+2. Login -- no password required, uses crypto keys, run:
+ `vagrant ssh`
+Alternatively:
1. Log into the vm (run: `ssh vagrant@rya-example-box` with pass: `vagrant`)
+
+#####*Running as Root*
+
+Most start and stop scripts require running as root with variables, like
%HADOOP_PREFIX% assigned.
+
+1. Run thecommand as root preserving environment variables, run:
+ `sudo -E thecommand theParameters`
+The `-E` preserves environment variables from the current session.
+
+Alternatively:
+
2. Switch to root (run: `su` with pass: `vagrant`)
-3. Restart tomcat (run: `service tomcat7 restart`)
+2. Load the environment variables. (run: `source .accumulo_rc.sh`)
+2. Enter thecommand with parameters after the "#" prompt.
+2. Exit when done (run: `exit` )
+
+#####*Restart Tomcat*
+
+1. Login into the VM, see above.
+2. Restart tomcat, run:
+ `sudo -E service tomcat7 restart`
-*Restart Accumulo*
+#####*Restart Accumulo*
-1. Log into the vm (run: `ssh vagrant@rya-example-box` with pass: `vagrant`)
-1. Switch to root (run: `su` with pass: `vagrant`)
-1. Stop Accumulo (run: `/home/vagrant/accumulo-1.6.4/bin/stop-all.sh`)
- * If `stop-all` doesn't complete, hit `ctrl-c` once and you should see
`Initiating forced shutdown in 15 seconds`. Wait 15 seconds
-1. Start Accumulo (run: `/home/vagrant/accumulo-1.6.4/bin/start-all.sh`)
+1. Login into the VM, see above.
+2. Stop Accumulo (run: `sudo -E /home/vagrant/accumulo-1.6.5/bin/stop-all.sh`)
+ * If `stop-all` doesn't complete, hit `ctrl-c` once and you should see
`Initiating forced shutdown in 15 seconds`. Wait 15 seconds.
+1. Start Accumulo (run: `sudo -E
/home/vagrant/accumulo-1.6.5/bin/start-all.sh`)
-*Test and Restart Zookeeper*
+#####*Test and Restart Zookeeper*
-1. Log into the vm (run: `ssh vagrant@rya-example-box` with pass: `vagrant`)
-1. Switch to root (run: `su` with pass: `vagrant`)
-1. Ping Zookeeper (run: `echo ruok | nc 127.0.0.1 2181`).
- * If Zookeeper is okay, you should see the response `imok`
- * Otherwise, restart Zookeeper (run
`/home/vagrant/zookeeper-3.4.5-cdh4.5.0/bin/zkServer.sh start`)
+1. Log into the vm, see above.
+2. Ping Zookeeper, run:
+ `echo ruok | (nc 127.0.0.1 2181 ; echo)`
+ * If Zookeeper is okay, you should immediately see the response
+ `imok`
+ * Otherwise, restart Zookeeper, run:
+ `sudo -E /home/vagrant/zookeeper-3.4.5-cdh4.5.0/bin/zkServer.sh start`
+
+##### *Rya Process list*
+
+List the processes specific to rya.
+From the VM, run:
+ `ryaps`
+Output should look like this:
+```
+org.apache.zookeeper.server.quorum.QuorumPeerMain
+org.apache.accumulo.start.Main monitor
+org.apache.accumulo.start.Main tserver
+org.apache.accumulo.start.Main master
+org.apache.accumulo.start.Main gc
+org.apache.accumulo.start.Main tracer
+org.apache.catalina.startup.Bootstrap
+```
+
+##### *Re-provision, reseting the configuration*
+
+If you are having issues, or modified the vagrantfile and want to put
everything back as specified, then use the provision command. This will
overwrite most settings. It will not download nor unpack the libraries, unless
you first remove there corresponding folders from the VM's vagrant home folder.
+
+1. Go to the folder containing "vagrantfile", on the host machine, (not the
VM), run:
+ `cd extras/vagrantExample/src/main/vagrant/`
+2. Run the provision command:
+ `vagrant provision`
+To get the full factory-reset, do some of the following before you
re-provision:
+
+###### Logs
+
+From within the VM, run:
+ `sudo rm -r zookeeper-3.4.5-cdh4.5.0/logs`
+ `sudo rm -r accumulo-1.6.5/logs/`
+
+###### Delete all data in Rya and Accumulo!
+If you run into the issue where it repeatedly prints: `Waiting for accumulo to
be initialized`
+This will allow Accumulo to start up. Remove the data, then reprovision as
above.
+From within the VM, run:
+ `sudo rm -r /data`
+ `sudo rm -r /var/zookeeper/`
+
## Interacting with Rya on the VM
### Connecting to Rya via OpenRDF Workbench
@@ -144,6 +209,7 @@ Once we've created a Rya repository and uploaded data, we
can query Rya via the
1. Verify that OpenRDF Workbench is connected to Rya. The OpenRDF Workbench
screen should have `Current Selections: Repository: Rya Accumulo (
RyaAccumulo )` at the top of the page.
1. Click on `Query` on the left side of the page.
1. Use the example SPARQL query below to query for Currencies with a Short
Name of "dollar"
+
```
PREFIX money:<http://telegraphis.net/ontology/money/money#>
@@ -167,6 +233,7 @@ The Rya Web Client provides a user a web gui to query Rya.
Once data has been l
1. Open your browser to the [Rya
Web](http://rya-example-box:8080/web.rya/sparqlQuery.jsp) page.
1. Use the example SPARQL query below to query for Currencies with a Short
Name of "dollar"
+
```
PREFIX money:<http://telegraphis.net/ontology/money/money#>
