This is an automated email from the ASF dual-hosted git repository.
mmiller pushed a commit to branch 1.10
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/1.10 by this push:
new c55821213f General improvements to CI (#2715)
c55821213f is described below
commit c55821213f202f5b131f4475c3d4a755942ac8d6
Author: Mike Miller <mmil...@apache.org>
AuthorDate: Tue May 17 19:26:00 2022 +0000
General improvements to CI (#2715)
---
assemble/src/main/assemblies/component.xml | 3 ---
test/system/continuous/README.md | 21 +++++++++++++++------
...{batch_walkers.txt.example => batch_walkers.txt} | 0
...{continuous-env.sh.example => continuous-env.sh} | 17 ++++++++++-------
.../{ingesters.txt.example => ingesters.txt} | 0
.../{scanners.txt.example => scanners.txt} | 0
.../continuous/{walkers.txt.example => walkers.txt} | 0
7 files changed, 25 insertions(+), 16 deletions(-)
diff --git a/assemble/src/main/assemblies/component.xml
b/assemble/src/main/assemblies/component.xml
index 3df09db612..b1522eb467 100644
--- a/assemble/src/main/assemblies/component.xml
+++ b/assemble/src/main/assemblies/component.xml
@@ -181,7 +181,6 @@
<excludes>
<exclude>src/**</exclude>
<exclude>target/**</exclude>
- <exclude>**/continuous-env.sh</exclude>
</excludes>
</fileSet>
<fileSet>
@@ -198,8 +197,6 @@
<exclude>**/*.pl</exclude>
<exclude>**/*.pyc</exclude>
<exclude>**/*.pyo</exclude>
- <exclude>**/walkers.txt</exclude>
- <exclude>**/ingesters.txt</exclude>
<exclude>**/randomwalk/conf/logger.xml</exclude>
<exclude>**/randomwalk/conf/randomwalk.conf</exclude>
<exclude>**/randomwalk/conf/walkers</exclude>
diff --git a/test/system/continuous/README.md b/test/system/continuous/README.md
index 31ee4bdafa..412ef62c62 100644
--- a/test/system/continuous/README.md
+++ b/test/system/continuous/README.md
@@ -21,14 +21,23 @@ Continuous Query and Ingest
This directory contains a suite of scripts for placing continuous query and
ingest load on accumulo. The purpose of these script is two-fold. First,
place continuous load on accumulo to see if breaks. Second, collect
-statistics in order to understand how accumulo behaves. To run these scripts
-copy all of the `.example` files and modify them. You can put these scripts in
-the current directory or define a `CONTINUOUS_CONF_DIR` where the files will be
+statistics in order to understand how accumulo behaves. To run these scripts:
+
+1) Modify the `*.txt` config files.
+2) Enter cluster values in `continuous-env.sh`
+
+You can put these scripts in the current directory or define a
`CONTINUOUS_CONF_DIR` where the files will be
read from. These scripts rely on `pssh`. Before running any script you may need
to use `pssh` to create the log directory on each machine (if you want it
local).
-Also, create the table "ci" before running. You can run
-`org.apache.accumulo.test.continuous.GenSplits` to generate splits points for a
-continuous ingest table.
+
+First, create the table "ci" before running:
+
+> accumulo shell -e "createtable ci" -u root
+
+Optional: generate split points for CI table:
+
+> $ accumulo org.apache.accumulo.test.continuous.GenSplits 20 > /tmp/splits
+> $ accumulo shell -e "addsplits -t ci -sf /tmp/splits" -u root
The following ingest scripts insert data into accumulo that will form a random
graph.
diff --git a/test/system/continuous/batch_walkers.txt.example
b/test/system/continuous/batch_walkers.txt
similarity index 100%
rename from test/system/continuous/batch_walkers.txt.example
rename to test/system/continuous/batch_walkers.txt
diff --git a/test/system/continuous/continuous-env.sh.example
b/test/system/continuous/continuous-env.sh
similarity index 97%
rename from test/system/continuous/continuous-env.sh.example
rename to test/system/continuous/continuous-env.sh
index 10cea328bc..4527a59f95 100644
--- a/test/system/continuous/continuous-env.sh.example
+++ b/test/system/continuous/continuous-env.sh
@@ -13,7 +13,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# set this to an absolute path that exist on every machine
+# modify these values to match cluster setup
+INSTANCE_NAME=instance
+ZOO_KEEPERS=localhost:2181
+USER=user
+PASS=pass
+
# Inherit values from environment if they are already set.
HADOOP_HOME=${HADOOP_HOME:-/opt/hadoop}
HADOOP_PREFIX=${HADOOP_PREFIX:-$HADOOP_HOME}
@@ -22,14 +27,9 @@ ACCUMULO_CONF_DIR=${ACCUMULO_CONF_DIR:-$ACCUMULO_HOME/conf}
JAVA_HOME=${JAVA_HOME:-/opt/java}
ZOOKEEPER_HOME=${ZOOKEEPER_HOME:-/opt/zookeeper}
-CONTINUOUS_LOG_DIR=$ACCUMULO_HOME/test/system/continuous/logs
-INSTANCE_NAME=instance
-ZOO_KEEPERS=zhost1,zhost2
+# users
ACCUMULO_USER=$(whoami)
HDFS_USER=$(whoami)
-USER=user
-PASS=pass
-TABLE=ci
#set debug to on to enable logging of accumulo client debugging
DEBUG_INGEST=off
@@ -137,3 +137,6 @@ BULK_MAP_TASKS=10
# number of key value pairs to generate per mapper
BULK_MAP_NODES=10000
+# other static values
+TABLE=ci
+CONTINUOUS_LOG_DIR=$ACCUMULO_HOME/test/system/continuous/logs
diff --git a/test/system/continuous/ingesters.txt.example
b/test/system/continuous/ingesters.txt
similarity index 100%
rename from test/system/continuous/ingesters.txt.example
rename to test/system/continuous/ingesters.txt
diff --git a/test/system/continuous/scanners.txt.example
b/test/system/continuous/scanners.txt
similarity index 100%
rename from test/system/continuous/scanners.txt.example
rename to test/system/continuous/scanners.txt
diff --git a/test/system/continuous/walkers.txt.example
b/test/system/continuous/walkers.txt
similarity index 100%
rename from test/system/continuous/walkers.txt.example
rename to test/system/continuous/walkers.txt
