date:20210920

[accumulo] branch main updated: Added additional tests in CompactionIT (#2275)

2021-09-20 Thread dlmarion

This is an automated email from the ASF dual-hosted git repository.

dlmarion pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 279ee5f  Added additional tests in CompactionIT (#2275)
279ee5f is described below

commit 279ee5f634253581462ad14602b35e07e8570930
Author: Dave Marion 
AuthorDate: Mon Sep 20 16:43:08 2021 -0400

Added additional tests in CompactionIT (#2275)


Added tests in CompactionIT for:
  compaction cancelled on table during user compaction
  table delete during user compaction
  compaction of subset of files
  setting compression compaction iterator

Closes #2114


Co-authored-by: Jeffrey Manno 
---
 .../accumulo/test/functional/CompactionIT.java | 276 -
 1 file changed, 275 insertions(+), 1 deletion(-)

diff --git 
a/test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java 
b/test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java
index e93c6d1..a489f89 100644
--- a/test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java
+++ b/test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java
@@ -20,35 +20,53 @@ package org.apache.accumulo.test.functional;
 
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
 import static org.junit.Assert.assertTrue;
 
+import java.io.IOException;
 import java.util.ArrayList;
+import java.util.EnumSet;
 import java.util.List;
 import java.util.Map;
+import java.util.Map.Entry;
 import java.util.Objects;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
 
 import org.apache.accumulo.core.client.Accumulo;
 import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
 import org.apache.accumulo.core.client.BatchWriter;
+import org.apache.accumulo.core.client.IteratorSetting;
 import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.TableNotFoundException;
 import org.apache.accumulo.core.client.admin.CompactionConfig;
 import org.apache.accumulo.core.client.admin.NewTableConfiguration;
 import org.apache.accumulo.core.client.admin.PluginConfig;
 import org.apache.accumulo.core.client.admin.compaction.CompactableFile;
 import org.apache.accumulo.core.client.admin.compaction.CompactionSelector;
+import org.apache.accumulo.core.client.admin.compaction.CompressionConfigurer;
 import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
 import org.apache.accumulo.core.data.Mutation;
 import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.DevNull;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorEnvironment;
+import org.apache.accumulo.core.iterators.IteratorUtil.IteratorScope;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
 import org.apache.accumulo.core.metadata.MetadataTable;
 import 
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily;
 import 
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.TabletColumnFamily;
 import org.apache.accumulo.core.security.Authorizations;
 import org.apache.accumulo.harness.AccumuloClusterHarness;
 import org.apache.accumulo.miniclusterImpl.MiniAccumuloConfigImpl;
+import org.apache.accumulo.test.CompactionExecutorIT;
+import org.apache.accumulo.test.ExternalCompactionIT.FSelector;
 import org.apache.accumulo.test.VerifyIngest;
 import org.apache.accumulo.test.VerifyIngest.VerifyParams;
 import org.apache.hadoop.conf.Configuration;
@@ -56,16 +74,48 @@ import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.fs.RawLocalFileSystem;
 import org.apache.hadoop.io.Text;
+import org.bouncycastle.util.Arrays;
 import org.junit.Test;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import com.google.common.base.Preconditions;
 import com.google.common.collect.Iterators;
 import com.google.common.collect.Lists;
 
 @SuppressWarnings("removal")
 public class CompactionIT extends AccumuloClusterHarness {
 
+  public static class TestFilter extends Filter {
+
+int modulus = 1;
+
+@Override
+public void init(SortedKeyValueIterator source, 
Map options,
+IteratorEnvironment env) throws IOException {
+  super.init(source, options, env);
+
+  // if the init function is never called at all, then not setting the 
modulus option should
+  // cause the test to fail
+  if (options.containsKey("modulus")) {

[accumulo-testing] branch main updated: Migrate accumulo-testing to log4j2. (#146)

2021-09-20 Thread ctubbsii

This is an automated email from the ASF dual-hosted git repository.

ctubbsii pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-testing.git


The following commit(s) were added to refs/heads/main by this push:
 new 50e4c02  Migrate accumulo-testing to log4j2. (#146)
50e4c02 is described below

commit 50e4c021e7db4df546138d43d778f0adee4bf7a5
Author: AlbertWhitlock 
AuthorDate: Mon Sep 20 14:12:12 2021 -0400

Migrate accumulo-testing to log4j2. (#146)

Migrate accumulo-testing to use log4j2 (via slf4j)

Co-authored-by: Jeffrey Manno 
Co-authored-by: Christopher Tubbs 
---
 Dockerfile |  2 +-
 bin/cingest|  2 +-
 bin/gcs|  2 +-
 bin/monitor|  2 +-
 bin/performance| 18 +++
 bin/rwalk  |  2 +-
 conf/env.sh|  4 ++--
 conf/log4j.properties  | 27 ---
 conf/log4j2.properties | 56 +++
 pom.xml| 59 --
 10 files changed, 124 insertions(+), 50 deletions(-)

diff --git a/Dockerfile b/Dockerfile
index 36afd92..e432a95 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -28,7 +28,7 @@ ENV HADOOP_RUNTIME_JAR /opt/at/hadoop-client-runtime.jar
 ENV TEST_JAR_PATH /opt/at/accumulo-testing-shaded.jar
 ENV ACCUMULO_CLIENT_PROPS /opt/at/conf/accumulo-client.properties
 ENV TEST_PROPS /opt/at/conf/accumulo-testing.properties
-ENV TEST_LOG4J /opt/at/conf/log4j.properties
+ENV TEST_LOG4J /opt/at/conf/log4j2.properties
 
 RUN mkdir /opt/at
 RUN mkdir /opt/at/bin
diff --git a/bin/cingest b/bin/cingest
index 0753608..84452b6 100755
--- a/bin/cingest
+++ b/bin/cingest
@@ -92,5 +92,5 @@ case "$1" in
 fi
 ;;
   *)
-java $JAVA_OPTS -Dlog4j.configuration="file:$TEST_LOG4J" "$ci_main" 
"${@:2}" "$TEST_PROPS" "$ACCUMULO_CLIENT_PROPS"
+java $JAVA_OPTS -Dlog4j.configurationFile="file:$TEST_LOG4J" "$ci_main" 
"${@:2}" "$TEST_PROPS" "$ACCUMULO_CLIENT_PROPS"
 esac
diff --git a/bin/gcs b/bin/gcs
index 9959354..31c30af 100755
--- a/bin/gcs
+++ b/bin/gcs
@@ -61,4 +61,4 @@ esac
 
 export 
CLASSPATH="$TEST_JAR_PATH:$HADOOP_API_JAR:$HADOOP_RUNTIME_JAR:$CLASSPATH"
 
-java $JAVA_OPTS -Dlog4j.configuration="file:$TEST_LOG4J" "$ci_main" "${@:2}" 
"$TEST_PROPS" "$ACCUMULO_CLIENT_PROPS"
+java $JAVA_OPTS -Dlog4j.configurationFile="file:$TEST_LOG4J" "$ci_main" 
"${@:2}" "$TEST_PROPS" "$ACCUMULO_CLIENT_PROPS"
diff --git a/bin/monitor b/bin/monitor
index eb606a8..6c8498d 100755
--- a/bin/monitor
+++ b/bin/monitor
@@ -48,4 +48,4 @@ case "$1" in
 esac
 
 export 
CLASSPATH="$TEST_JAR_PATH:$HADOOP_API_JAR:$HADOOP_RUNTIME_JAR:$CLASSPATH"
-java $JAVA_OPTS -DINSTRUMENTATIONKEY="$INSTRUMENTATIONKEY" 
-Dlog4j.configuration="file:$TEST_LOG4J" "$ci_main" "${@:2}" "-c" 
"$ACCUMULO_CLIENT_PROPS"
\ No newline at end of file
+java $JAVA_OPTS -DINSTRUMENTATIONKEY="$INSTRUMENTATIONKEY" 
-Dlog4j.configurationFile="file:$TEST_LOG4J" "$ci_main" "${@:2}" "-c" 
"$ACCUMULO_CLIENT_PROPS"
\ No newline at end of file
diff --git a/bin/performance b/bin/performance
index 45e14d6..05d87d3 100755
--- a/bin/performance
+++ b/bin/performance
@@ -46,9 +46,9 @@ function build_shade_jar() {
   fi
 }
 
-log4j_config="$at_home/conf/log4j.properties"
+log4j_config="$at_home/conf/log4j2.properties"
 if [ ! -f "$log4j_config" ]; then
-  echo "Could not find logj4.properties in $at_home/conf"
+  echo "Could not find log4j2.properties in $at_home/conf"
   exit 1
 fi
 
@@ -71,7 +71,7 @@ case "$1" in
 fi
 # don't start unless we can find the class provided
 found=false
-CLASSPATH="$CP" java $JAVA_OPTS -Dlog4j.configuration="file:$log4j_config" 
${perf_pkg}.ListTests | while read -r test_class; do
+CLASSPATH="$CP" java $JAVA_OPTS 
-Dlog4j.configurationFile="file:$log4j_config" ${perf_pkg}.ListTests | while 
read -r test_class; do
   if [[ "$test_class" == "$3" ]]; then
 found=true
   fi
@@ -83,30 +83,30 @@ case "$1" in
 fi
 mkdir -p "$2"
 start_cluster
-CLASSPATH="$CP" java $JAVA_OPTS -Dlog4j.configuration="file:$log4j_config" 
${perf_pkg}.ListTests | while read -r test_class; do
+CLASSPATH="$CP" java $JAVA_OPTS 
-Dlog4j.configurationFile="file:$log4j_config" ${perf_pkg}.ListTests | while 
read -r test_class; do
   if [[ -z $3 || $test_class == *$3* ]]; then
 echo "Running test $test_class"
 pt_tmp=$(mktemp -d -t accumulo_pt_XXX)
 setup_accumulo
 get_config_file accumulo.properties "$pt_tmp"
-CLASSPATH="$CP" java $JAVA_OPTS 
-Dlog4j.configuration="file:$log4j_config"  ${perf_pkg}.MergeSiteConfig 
"$test_class" "$pt_tmp"
+CLASSPATH="$CP" java $JAVA_OPTS 
-Dlog4j.configurationFile="file:$log4j_config"  ${perf_pkg}.MergeSiteConfig 
"$test_class" "$pt_tmp"
 put_config_file "$pt_tmp/accumulo.properties"
 put_server_code "$at_home/target/accumulo-testing-$at_version.jar"
 start_accumul

[accumulo] branch main updated: Added additional tests in CompactionIT (#2275)

[accumulo-testing] branch main updated: Migrate accumulo-testing to log4j2. (#146)

2 matches

Site Navigation

Mail list logo

Footer information