[accumulo-website] branch main updated: Added me to the contributors list (#307)

2022-01-14 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new d2f51d4  Added me to the contributors list (#307)
d2f51d4 is described below

commit d2f51d4d38d7ab71d257d7eb7964a0ac81323457
Author: Filipe Rodrigues 
AuthorDate: Fri Jan 14 20:56:00 2022 +

Added me to the contributors list (#307)
---
 pages/people.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/pages/people.md b/pages/people.md
index 4b693b3..70b01c5 100644
--- a/pages/people.md
+++ b/pages/people.md
@@ -100,6 +100,7 @@ GitHub also has a [contributor list][github-contributors] 
based on commits.
 | Edward Yoon |
   |   |
 | Elina Wise  | [Arctic Slope Regional Corp.][ASRC]
   | [ET][ET]  |
 | Eugene Cheipesh |
   |   |
+| Filipe Rodrigues|
   | [GMT][GMT]
 | Gary Singh  | [Sabre Engineering][SABRE] 
   | [ET][ET]  |
 | Harjit Singh|
   | [ET][ET]  |
 | Hasan Gürcan|
   | [CEST][CEST]  |


[accumulo] branch main updated: Removed deadcode regarding unused while-loop (#2418)

2022-01-14 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 4ce35a7  Removed deadcode regarding unused while-loop (#2418)
4ce35a7 is described below

commit 4ce35a7498fe3d7aca560bb4632ebe751901821d
Author: Filipe Rodrigues 
AuthorDate: Fri Jan 14 14:43:47 2022 +

Removed deadcode regarding unused while-loop (#2418)

* This PR closes #2355.
* Removed deadcode regarding an unused while-loop which condition was 
always false. This condition was evaluated with the value of a variable called 
'reservationsBlocked', which was initialized with value 'false' and it was 
never changed.
---
 .../java/org/apache/accumulo/tserver/tablet/DatafileManager.java | 9 -
 1 file changed, 9 deletions(-)

diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java
 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java
index 1068a86..e6799b8 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java
@@ -82,7 +82,6 @@ class DatafileManager {
   private final Map> scanFileReservations = new 
HashMap<>();
   private final MapCounter fileScanReferenceCounts = new 
MapCounter<>();
   private long nextScanReservationId = 0;
-  private boolean reservationsBlocked = false;
 
   static void rename(VolumeManager fs, Path src, Path dst) throws IOException {
 if (!fs.rename(src, dst)) {
@@ -93,14 +92,6 @@ class DatafileManager {
   Pair> reserveFilesForScan() {
 synchronized (tablet) {
 
-  while (reservationsBlocked) {
-try {
-  tablet.wait(50);
-} catch (InterruptedException e) {
-  log.warn("{}", e.getMessage(), e);
-}
-  }
-
   Set absFilePaths = new 
HashSet<>(datafileSizes.keySet());
 
   long rid = nextScanReservationId++;


[accumulo-testing] branch main updated: Fix broken agitator scripts (#184)

2022-01-12 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-testing.git


The following commit(s) were added to refs/heads/main by this push:
 new 267dc84  Fix broken agitator scripts (#184)
267dc84 is described below

commit 267dc84971d6fa37af6183dc5a0f3706f9762911
Author: Mike Miller 
AuthorDate: Wed Jan 12 15:28:01 2022 -0500

Fix broken agitator scripts (#184)

* Create bash function to start agitators
* Drop broken perl scripts in favor of bash functions
* Add function to call ClusterConfigParser to read cluster.yaml
* Make agitators only log to one file
* Rename env variables from master to manager

Co-authored-by: Christopher Tubbs 
Co-authored-by: Dom G. 
---
 bin/agitator | 175 ++-
 conf/env.sh  |   8 +-
 libexec/datanode-agitator.pl | 136 -
 libexec/master-agitator.pl   |  98 
 libexec/tserver-agitator.pl  | 130 
 5 files changed, 143 insertions(+), 404 deletions(-)

diff --git a/bin/agitator b/bin/agitator
index 91e456c..3f532a7 100755
--- a/bin/agitator
+++ b/bin/agitator
@@ -30,6 +30,85 @@ Possible commands:
 EOF
 }
 
+# Starts a app specific agitator
+# usage: start_app_agitator app_name kill_sleep_time restart_sleep_time 
min_kill max_kill start_cmd kill_cmd
+# Requires that a list of hosts to be set in $APP_HOSTS
+function start_app_agitator() {
+  local app_name=$1
+  local kill_sleep_time=$2
+  local restart_sleep_time=$3
+  local min_kill=$4
+  local max_kill=$5
+  local start_cmd=$6
+  local kill_cmd=$7
+  local hosts_array; readarray -td' ' hosts_array < <(get_app_hosts 
"$app_name")
+  local num_hosts=${#hosts_array[@]}
+  local node_to_kill
+  nodes_to_kill_array=()
+  local T
+  local ENV_VARS="ACCUMULO_HOME=$ACCUMULO_HOME ZOOKEEPER_HOME=$ZOOKEEPER_HOME 
HADOOP_HOME=$HADOOP_HOME JAVA_HOME=$JAVA_HOME"
+
+  if (( num_hosts == 0 )); then
+echo "ERROR: No hosts were found in env for $app_name"
+exit 1
+  fi
+  if (( max_kill > num_hosts )); then
+echo "ERROR: Max kill $max_kill is greater than number of hosts $num_hosts"
+exit 1
+  fi
+  if (( max_kill < min_kill )); then
+echo "ERROR: Max kill $max_kill is less than min kill $min_kill"
+exit 1
+  fi
+
+  T="$(date +'%Y%m%d %H:%M:%S')"
+  echo "$T Starting $app_name agitation. Kill every $kill_sleep_time minutes, 
restart every $restart_sleep_time minutes."
+  echo "$T Will randomly kill between $min_kill and $max_kill of the 
following: ${hosts_array[*]}"
+  while true; do
+echo "$T Sleeping for $kill_sleep_time minutes"
+sleep $((kill_sleep_time * 60))
+
+T="$(date +'%Y%m%d %H:%M:%S')"
+if ((max_kill == 1)) ; then
+  node_to_kill=${hosts_array[0]}
+  echo "$T Killing $app_name at $node_to_kill"
+  ssh "$node_to_kill" "$kill_cmd"
+else
+  local num_to_kill=$((min_kill + RANDOM % max_kill))
+  # get the random nodes to kill
+  local count=0
+  while [[ $count -lt $num_to_kill ]]; do
+randomHostIndex=$((1 + RANDOM % num_hosts))
+node_to_kill=${hosts_array[randomHostIndex]}
+# only add host to the array if its not already there
+if [[ ! " ${nodes_to_kill_array[*]} " =~ $node_to_kill ]]; then
+  nodes_to_kill_array[count]=$node_to_kill
+fi
+count=${#nodes_to_kill_array[@]}
+  done
+  echo "$T Killing $count $app_name nodes"
+  for i in "${nodes_to_kill_array[@]}"; do
+ssh "$i" "$kill_cmd"
+  done
+fi
+
+T="$(date +'%Y%m%d %H:%M:%S')"
+echo "$T Sleeping for $restart_sleep_time minutes."
+sleep $((restart_sleep_time * 60))
+
+T="$(date +'%Y%m%d %H:%M:%S')"
+if ((max_kill == 1)) ; then
+  echo "$T Restarting $app_name at $node_to_kill"
+  ssh "$node_to_kill" "bash -c '${ENV_VARS} $start_cmd'"
+else
+  for i in "${nodes_to_kill_array[@]}"; do
+echo "$T Restarting $app_name node at ${i}"
+ssh "$i" "bash -c '${ENV_VARS} $start_cmd'"
+  done
+fi
+  done
+}
+
 function start_agitator() {
   ## check that pssh is installed, falling back to parallel-ssh if needed
   ## make sure to export it, so it can be seen inside the agitator perl script
@@ -40,58 +119,82 @@ function start_agitator() {
   else
 echo >&2 "The agitator requires pssh/parallel-ssh to be installed. 
Aborting."; exit 1;
   fi
+  ## read configuration into env variables
+  read_cluster_conf
 
   mkdir -p "${at_home}/logs"
   log_base="${at_home

[accumulo] branch main updated: Update protobuf to 3.19.2 (#2412)

2022-01-12 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 30c49ef  Update protobuf to 3.19.2 (#2412)
30c49ef is described below

commit 30c49ef1a193bd6712efe4c28e304e107b47a1c3
Author: Mike Miller 
AuthorDate: Wed Jan 12 07:48:34 2022 -0500

Update protobuf to 3.19.2 (#2412)
---
 .../accumulo/core/protobuf/ProtobufUtil.java   |   6 +-
 pom.xml|   2 +-
 .../server/replication/proto/Replication.java  | 682 ++---
 server/base/src/main/protobuf/replication.proto|   1 +
 server/base/src/main/scripts/generate-protobuf.sh  |   2 +-
 5 files changed, 452 insertions(+), 241 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/protobuf/ProtobufUtil.java 
b/core/src/main/java/org/apache/accumulo/core/protobuf/ProtobufUtil.java
index 000916b..57baf5a 100644
--- a/core/src/main/java/org/apache/accumulo/core/protobuf/ProtobufUtil.java
+++ b/core/src/main/java/org/apache/accumulo/core/protobuf/ProtobufUtil.java
@@ -20,7 +20,7 @@ package org.apache.accumulo.core.protobuf;
 
 import org.apache.accumulo.core.data.Value;
 
-import com.google.protobuf.GeneratedMessage;
+import com.google.protobuf.GeneratedMessageV3;
 import com.google.protobuf.TextFormat;
 
 /**
@@ -29,11 +29,11 @@ import com.google.protobuf.TextFormat;
 public class ProtobufUtil {
   private static final char LEFT_BRACKET = '[', RIGHT_BRACKET = ']';
 
-  public static Value toValue(GeneratedMessage msg) {
+  public static Value toValue(GeneratedMessageV3 msg) {
 return new Value(msg.toByteArray());
   }
 
-  public static String toString(GeneratedMessage msg) {
+  public static String toString(GeneratedMessageV3 msg) {
 // Too much typing
 return LEFT_BRACKET + TextFormat.shortDebugString(msg) + RIGHT_BRACKET;
   }
diff --git a/pom.xml b/pom.xml
index 2aaa691..12112c1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -286,7 +286,7 @@
   
 com.google.protobuf
 protobuf-java
-3.7.1
+3.19.2
   
   
 com.lmax
diff --git 
a/server/base/src/main/java/org/apache/accumulo/server/replication/proto/Replication.java
 
b/server/base/src/main/java/org/apache/accumulo/server/replication/proto/Replication.java
index 4a20d87..043ff72 100644
--- 
a/server/base/src/main/java/org/apache/accumulo/server/replication/proto/Replication.java
+++ 
b/server/base/src/main/java/org/apache/accumulo/server/replication/proto/Replication.java
@@ -21,102 +21,113 @@
 
 package org.apache.accumulo.server.replication.proto;
 
-@SuppressWarnings({"unused"})
-public final class Replication {
+@SuppressWarnings({"unused"}) public final class Replication {
   private Replication() {}
   public static void registerAllExtensions(
+  com.google.protobuf.ExtensionRegistryLite registry) {
+  }
+
+  public static void registerAllExtensions(
   com.google.protobuf.ExtensionRegistry registry) {
+registerAllExtensions(
+(com.google.protobuf.ExtensionRegistryLite) registry);
   }
-  public interface StatusOrBuilder
-  extends com.google.protobuf.MessageOrBuilder {
+  public interface StatusOrBuilder extends
+  // @@protoc_insertion_point(interface_extends:Status)
+  com.google.protobuf.MessageOrBuilder {
 
-// optional int64 begin = 1 [default = 0];
 /**
- * optional int64 begin = 1 [default = 0];
- *
  * 
  * offset where replication should start
  * 
+ *
+ * optional int64 begin = 1 [default = 0];
+ * @return Whether the begin field is set.
  */
 boolean hasBegin();
 /**
- * optional int64 begin = 1 [default = 0];
- *
  * 
  * offset where replication should start
  * 
+ *
+ * optional int64 begin = 1 [default = 0];
+ * @return The begin.
  */
 long getBegin();
 
-// optional int64 end = 2 [default = 0];
 /**
- * optional int64 end = 2 [default = 0];
- *
  * 
  * offset where data is ready for replication
  * 
+ *
+ * optional int64 end = 2 [default = 0];
+ * @return Whether the end field is set.
  */
 boolean hasEnd();
 /**
- * optional int64 end = 2 [default = 0];
- *
  * 
  * offset where data is ready for replication
  * 
+ *
+ * optional int64 end = 2 [default = 0];
+ * @return The end.
  */
 long getEnd();
 
-// optional bool infiniteEnd = 3 [default = false];
 /**
- * optional bool infiniteEnd = 3 [default = false];
- *
  * 
  * do we have a discrete 'end'
  * 
+ *
+ * optional bool infiniteEnd = 3 [default = false];
+ * @return Whether the infiniteEnd field is set.
  */
 boolean hasInfiniteEnd();
 /**
- * optional bool infiniteEnd = 3 [default = false];
- *
  *

[accumulo] branch main updated: Create RunningCompactionInfo in core for reuse (#2403)

2022-01-03 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 681139e  Create RunningCompactionInfo in core for reuse (#2403)
681139e is described below

commit 681139e786787bdc5c9778bbb2a6062962e63f51
Author: Mike Miller 
AuthorDate: Mon Jan 3 08:57:15 2022 -0500

Create RunningCompactionInfo in core for reuse (#2403)

* Create new object in core for reuse
* Make private vars of CompactorInfo final
---
 .../core/util/compaction/RunningCompaction.java|  6 +-
 .../util/compaction/RunningCompactionInfo.java | 93 +++---
 .../rest/compactions/external/CompactorInfo.java   |  8 +-
 .../rest/compactions/external/ECResource.java  |  5 +-
 .../compactions/external/RunningCompactions.java   |  6 +-
 .../external/RunningCompactorDetails.java  | 15 ++--
 .../compaction/ExternalCompactionProgressIT.java   |  8 +-
 7 files changed, 69 insertions(+), 72 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/util/compaction/RunningCompaction.java
 
b/core/src/main/java/org/apache/accumulo/core/util/compaction/RunningCompaction.java
index 5ed976d..d057da1 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/util/compaction/RunningCompaction.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/util/compaction/RunningCompaction.java
@@ -22,6 +22,7 @@ import java.util.Map;
 import java.util.TreeMap;
 
 import org.apache.accumulo.core.compaction.thrift.TCompactionStatusUpdate;
+import org.apache.accumulo.core.compaction.thrift.TExternalCompaction;
 import org.apache.accumulo.core.tabletserver.thrift.TExternalCompactionJob;
 
 public class RunningCompaction {
@@ -32,12 +33,15 @@ public class RunningCompaction {
   private final Map updates = new TreeMap<>();
 
   public RunningCompaction(TExternalCompactionJob job, String 
compactorAddress, String queueName) {
-super();
 this.job = job;
 this.compactorAddress = compactorAddress;
 this.queueName = queueName;
   }
 
+  public RunningCompaction(TExternalCompaction tEC) {
+this(tEC.getJob(), tEC.getCompactor(), tEC.getQueueName());
+  }
+
   public Map getUpdates() {
 return updates;
   }
diff --git 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/compactions/external/RunningCompactorInfo.java
 
b/core/src/main/java/org/apache/accumulo/core/util/compaction/RunningCompactionInfo.java
similarity index 67%
rename from 
server/monitor/src/main/java/org/apache/accumulo/monitor/rest/compactions/external/RunningCompactorInfo.java
rename to 
core/src/main/java/org/apache/accumulo/core/util/compaction/RunningCompactionInfo.java
index 7ef070f..4f67417 100644
--- 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/compactions/external/RunningCompactorInfo.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/util/compaction/RunningCompactionInfo.java
@@ -16,60 +16,54 @@
  * specific language governing permissions and limitations
  * under the License.
  */
-package org.apache.accumulo.monitor.rest.compactions.external;
+package org.apache.accumulo.core.util.compaction;
+
+import static java.util.Objects.requireNonNull;
 
-import java.util.Map;
 import java.util.TreeMap;
 import java.util.concurrent.TimeUnit;
 
-import jakarta.validation.constraints.NotNull;
-
 import org.apache.accumulo.core.compaction.thrift.TCompactionStatusUpdate;
 import org.apache.accumulo.core.compaction.thrift.TExternalCompaction;
 import org.apache.accumulo.core.dataImpl.KeyExtent;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-public class RunningCompactorInfo extends CompactorInfo {
-  private static final Logger log = 
LoggerFactory.getLogger(RunningCompactorInfo.class);
+public class RunningCompactionInfo {
+  private static final Logger log = 
LoggerFactory.getLogger(RunningCompactionInfo.class);
 
-  // Variable names become JSON keys
-  public String ecid;
-  public String kind;
-  public String tableId;
-  public int numFiles;
-  public float progress = 0f;
-  public long duration;
-  public String status;
-  public long lastUpdate;
+  // DO NOT CHANGE Variable names - they map to JSON keys in the Monitor
+  public final String server;
+  public final String queueName;
+  public final String ecid;
+  public final String kind;
+  public final String tableId;
+  public final int numFiles;
+  public final float progress;
+  public final long duration;
+  public final String status;
+  public final long lastUpdate;
 
-  public RunningCompactorInfo() {
-super();
-  }
+  /**
+   * Info parsed about the external running compaction. Calculate the 
progress, which is defined as
+   * the percentage of bytesRead / bytesToBeCompacted of the last update.
+   */
+  public RunningCompactionInfo(TExternalCompaction ec) {
+requireNonNull(ec, "Thrift external compact

[accumulo] branch main updated: Update log4j2 to 2.17.1 and slf4j to 1.7.32 (#2401)

2021-12-29 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 976af80  Update log4j2 to 2.17.1 and slf4j to 1.7.32 (#2401)
976af80 is described below

commit 976af80c1591a177265009aa4ed1eb03b8e21c93
Author: EdColeman 
AuthorDate: Wed Dec 29 08:32:31 2021 -0500

Update log4j2 to 2.17.1 and slf4j to 1.7.32 (#2401)
---
 pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pom.xml b/pom.xml
index 93a4113..2aaa691 100644
--- a/pom.xml
+++ b/pom.xml
@@ -144,7 +144,7 @@
 
 
2020-12-17T22:06:50Z
 true
-1.7.30
+1.7.32
 
source-release-tar
 
 false
@@ -199,7 +199,7 @@
   
 org.apache.logging.log4j
 log4j-bom
-2.17.0
+2.17.1
 pom
 import
   


[accumulo-website] branch asf-site updated (9477f3f -> eff0b0c)

2021-12-28 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git.


from 9477f3f  Automatic Site Publish by Buildbot
 add 94b1dfd  Automatic Site Publish by Buildbot
 add 2b1bc53  Automatic Site Publish by Buildbot
 add 2c5d093  Automatic Site Publish by Buildbot
 add 6273240  Automatic Site Publish by Buildbot
 add eff0b0c  Automatic Site Publish by Buildbot

No new revisions were added by this update.

Summary of changes:
 output/docs/2.x/administration/in-depth-install.html | 8 
 output/feed.xml  | 4 ++--
 output/search_data.json  | 2 +-
 3 files changed, 11 insertions(+), 3 deletions(-)


[accumulo-website] branch main updated: Add suspended state description (#305)

2021-12-28 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new e5b9206  Add suspended state description (#305)
e5b9206 is described below

commit e5b9206fb3a35c2974b5d89610e0331df547e4e0
Author: EdColeman 
AuthorDate: Tue Dec 28 09:57:29 2021 -0500

Add suspended state description (#305)

Co-authored-by: Mike Miller 
Co-authored-by: Dom G. <47725857+domgargu...@users.noreply.github.com>
---
 _docs-2/administration/in-depth-install.md | 8 
 1 file changed, 8 insertions(+)

diff --git a/_docs-2/administration/in-depth-install.md 
b/_docs-2/administration/in-depth-install.md
index a70187e..e0e3c23 100644
--- a/_docs-2/administration/in-depth-install.md
+++ b/_docs-2/administration/in-depth-install.md
@@ -442,6 +442,14 @@ TabletServer to return before reassigning that 
TabletServer's responsibilities t
 TabletServers. If the TabletServer returns to the cluster before the specified 
timeout has elapsed,
 Accumulo will assign the TabletServer its original responsibilities.
 
+Tablet Status: Normally tablets will be in a HOSTED state. When a tserver goes 
off-line, the tablets
+assigned will transition to UNASSIGNED until they are reassigned by the 
Manager process to another
+active tserver.  With the [table.suspend.duration] set to > 0, a tablet will 
go from HOSTED to
+SUSPENDED when the tserver goes offline. The tablets will stay SUSPENDED until 
the tserver comes
+back online or, if the [table.suspend.duration] has passed. If the 
table.suspend.duration has passed
+before the tserver has returned, it will then become UNASSIGNED and eligible 
for reassignment
+by the Manager. If a tablet is UNASSIGNED it will not enter the SUSPENDED 
state.
+
 It is important not to choose too large a value for [table.suspend.duration], 
as during this time,
 all scans against the data that TabletServer had hosted will block (or time 
out).
 


[accumulo-website] branch main updated: Revert "add tablet state description"

2021-12-27 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new b7ee9e9  Revert "add tablet state description"
b7ee9e9 is described below

commit b7ee9e9ae68062a012eb90196e66b2ca4cb2ed24
Author: Mike Miller 
AuthorDate: Mon Dec 27 12:05:43 2021 -0500

Revert "add tablet state description"

This reverts commit d80c2a393c7f62edb360b1b7621571e1ab0ec4ca.
---
 _docs-2/administration/in-depth-install.md | 8 
 1 file changed, 8 deletions(-)

diff --git a/_docs-2/administration/in-depth-install.md 
b/_docs-2/administration/in-depth-install.md
index 99fe80c..a70187e 100644
--- a/_docs-2/administration/in-depth-install.md
+++ b/_docs-2/administration/in-depth-install.md
@@ -442,14 +442,6 @@ TabletServer to return before reassigning that 
TabletServer's responsibilities t
 TabletServers. If the TabletServer returns to the cluster before the specified 
timeout has elapsed,
 Accumulo will assign the TabletServer its original responsibilities.
 
-Tablet Status: Normally tablets will be in a HOSTED state. When a tserver goes 
off-line, the tablets
-assigned will transition to UNASSIGNED until they are reassigned by the 
Manager process to another 
-active tserver.  With the [table.suspend.duration] set to > 0, a tablet will 
go from HOSTED to 
-SUSPENDED when the tserver goes offline. The tablets will stay SUSPENDED until 
the tserver comes
-back online or, if the [table.suspend.duration] has passed. If the 
table.suspend.duration has passed
-before the tserver has returned, it will the become UNASSIGNED and eligible 
for reassignment
-by the Manager. If a tablet is UNASSIGNED it will not enter the SUSPENDED 
state. 
-
 It is important not to choose too large a value for [table.suspend.duration], 
as during this time,
 all scans against the data that TabletServer had hosted will block (or time 
out).
 


[accumulo-website] branch main updated: Revert "Revert "Update release notes for new EC page in monitor (#303)""

2021-12-27 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new b9e37ab  Revert "Revert "Update release notes for new EC page in 
monitor (#303)""
b9e37ab is described below

commit b9e37ab9fb6727012173113edaf7cb4ec7f414ef
Author: Mike Miller 
AuthorDate: Mon Dec 27 12:04:20 2021 -0500

Revert "Revert "Update release notes for new EC page in monitor (#303)""

This reverts commit ce16c1658b748c8c8a81ef87bf80631df57c641c.
---
 _posts/release/2020-01-19-accumulo-2.1.0.md |  15 ---
 images/release/active-compactions-table.png | Bin 26834 -> 0 bytes
 images/release/ec-running-details.png   | Bin 0 -> 131341 bytes
 images/release/ec-running2.png  | Bin 0 -> 66055 bytes
 4 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/_posts/release/2020-01-19-accumulo-2.1.0.md 
b/_posts/release/2020-01-19-accumulo-2.1.0.md
index 8e966b2..3ada29b 100644
--- a/_posts/release/2020-01-19-accumulo-2.1.0.md
+++ b/_posts/release/2020-01-19-accumulo-2.1.0.md
@@ -213,6 +213,12 @@ TABLE: accumulo.metadata
 2default_tablet  0 0 0 0 HOSTED 
CURRENT:ip-10-113-12-25:9997   !0~+INF
 ```
 
+### New Utility for generating splits
+
+A new command line utility was created to generate split points from 1 or more 
rfiles. One or more HDFS directories can
+be given as well. The utility will iterate over all the files provided and 
determine the proper split points
+based on either the size or number given. It uses Apache Datasketches to get 
the split points from the data. {% ghi 2361 %} {% ghi 2368 %}
+
 ### New option for cloning offline
 
 Added option to leave cloned tables offline {% ghi 1474 %} {% ghi 1475 %}
@@ -253,15 +259,18 @@ TODO clarify what was done for this. {% ghi 1787 %} {% 
ghi 1788 %} {% ghi 1798 %
 
 Created new "jshell" convenience entry point {% ghi 1870 %} {% ghi 1910 %}
 
-### New Monitor Compactions page & optimization
+### New Monitor Pages & Improvments
 
 A page was added to the Monitor that lists the active compactions and the 
longest running
 active compaction. As an optimization, the page will only fetch data if a user 
loads the
 page and will only do so a maximum of once a minute. The same optimization was 
added for the Active Scans page,
-along with the addition of a "Fetched" column indicating when the data was 
retrieved. {% ghi 2283 %} {% ghi 2294 %}
+along with the addition of a "Fetched" column indicating when the data was 
retrieved. A new page was also created 
+for External Compactions that allows seeing the progress of compactions and 
many details about ongoing compactions.
+{% ghi 2283 %} {% ghi 2294 %} {% ghi 2358 %}
 
-![Active Compactions](/images/release/active-compactions-table.png)
+
 
+
 
 ### Other notable changes
 
diff --git a/images/release/active-compactions-table.png 
b/images/release/active-compactions-table.png
deleted file mode 100644
index d2c9925..000
Binary files a/images/release/active-compactions-table.png and /dev/null differ
diff --git a/images/release/ec-running-details.png 
b/images/release/ec-running-details.png
new file mode 100644
index 000..506fd5a
Binary files /dev/null and b/images/release/ec-running-details.png differ
diff --git a/images/release/ec-running2.png b/images/release/ec-running2.png
new file mode 100644
index 000..3311235
Binary files /dev/null and b/images/release/ec-running2.png differ


[accumulo] branch main updated: Fix typos in Admin

2021-12-22 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new b216d17  Fix typos in Admin
b216d17 is described below

commit b216d17731c924fac825c113eb80e6af4c249db9
Author: Mike Miller 
AuthorDate: Wed Dec 22 06:49:39 2021 -0500

Fix typos in Admin
---
 .../base/src/main/java/org/apache/accumulo/server/util/Admin.java  | 7 +++
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git 
a/server/base/src/main/java/org/apache/accumulo/server/util/Admin.java 
b/server/base/src/main/java/org/apache/accumulo/server/util/Admin.java
index 299b1d6..049165c 100644
--- a/server/base/src/main/java/org/apache/accumulo/server/util/Admin.java
+++ b/server/base/src/main/java/org/apache/accumulo/server/util/Admin.java
@@ -315,9 +315,8 @@ public class Admin implements KeywordExecutable {
   }
 
   /**
-   * flushing during shutdown is a performance optimization, its not required. 
The method will make
-   * an attempt to initiate flushes of all tables and give up if it takes too 
long.
-   *
+   * Flushing during shutdown is a performance optimization, it's not 
required. This method will
+   * attempt to initiate flushes of all tables and give up if it takes too 
long.
*/
   private static void flushAll(final ClientContext context) {
 
@@ -337,7 +336,7 @@ public class Admin implements KeywordExecutable {
   }
 }
   } catch (Exception e) {
-log.warn("Failed to intiate flush {}", e.getMessage());
+log.warn("Failed to initiate flush {}", e.getMessage());
   }
 };
 


[accumulo] branch main updated: Fix coordinator check in Monitor (#2392)

2021-12-21 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 9a56c45  Fix coordinator check in Monitor (#2392)
9a56c45 is described below

commit 9a56c45c8dd6283ea78f283dc49de065a1a6f6d2
Author: Mike Miller 
AuthorDate: Tue Dec 21 08:37:39 2021 -0500

Fix coordinator check in Monitor (#2392)

* Changes for the monitor external compactions page to detect
when the Coordinator can't be found or has moved. The page itself will
throw pop-up errors with a message that is not very helpful. With this
change you can still see those errors but after a minute the monitor will
detect that its gone and show the user friendly message on the page.
* Also fix log message in ECResource
---
 .../src/main/java/org/apache/accumulo/monitor/Monitor.java| 11 ---
 .../monitor/rest/compactions/external/ECResource.java |  2 +-
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java 
b/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java
index 25e6fbd..d62f780 100644
--- a/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java
+++ b/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java
@@ -171,6 +171,7 @@ public class Monitor extends AbstractServer implements 
HighlyAvailableService {
   private Exception problemException;
   private GCStatus gcStatus;
   private Optional coordinatorHost = Optional.empty();
+  private long coordinatorCheckNanos = 0L;
   private CompactionCoordinatorService.Client coordinatorClient;
   private final String coordinatorMissingMsg =
   "Error getting the compaction coordinator. Check that it is running. It 
is not "
@@ -378,10 +379,14 @@ public class Monitor extends AbstractServer implements 
HighlyAvailableService {
 this.problemException = e;
   }
 
-  if (coordinatorHost.isEmpty()) {
+  // check for compaction coordinator host and only notify its discovery
+  Optional previousHost;
+  if (System.nanoTime() - coordinatorCheckNanos > fetchTimeNanos) {
+previousHost = coordinatorHost;
 coordinatorHost = 
ExternalCompactionUtil.findCompactionCoordinator(context);
-  } else {
-log.info("External Compaction Coordinator found at {}", 
coordinatorHost.get());
+coordinatorCheckNanos = System.nanoTime();
+if (previousHost.isEmpty() && coordinatorHost.isPresent())
+  log.info("External Compaction Coordinator found at {}", 
coordinatorHost.get());
   }
 
 } finally {
diff --git 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/compactions/external/ECResource.java
 
b/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/compactions/external/ECResource.java
index d7035d6..131abc8 100644
--- 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/compactions/external/ECResource.java
+++ 
b/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/compactions/external/ECResource.java
@@ -46,7 +46,7 @@ public class ECResource {
   @GET
   public CoordinatorInfo getCoordinator() {
 var cc = monitor.getCompactorsInfo();
-log.info("Got coordinator from monitor = {}", cc);
+log.info("Got coordinator from monitor = {}", cc.getCoordinatorHost());
 return new CoordinatorInfo(cc.getCoordinatorHost(), cc);
   }
 


[accumulo] branch main updated (c029c89 -> e0853b7)

2021-12-17 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from c029c89  Fix accumulo-env.sh variables (#2391)
 add f5121ec  Make test scripts clean up duplicate classes
 add e0853b7  Merge remote-tracking branch 'upstream/1.10' into main

No new revisions were added by this update.

Summary of changes:
 start/src/test/shell/makeHelloWorldJars.sh | 2 ++
 start/src/test/shell/makeTestJars.sh   | 1 +
 2 files changed, 3 insertions(+)


[accumulo] branch 1.10 updated: Make test scripts clean up duplicate classes

2021-12-17 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch 1.10
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/1.10 by this push:
 new f5121ec  Make test scripts clean up duplicate classes
f5121ec is described below

commit f5121ec42460358eef9d25f5e5ca1914d1842a76
Author: Mike Miller 
AuthorDate: Fri Dec 17 13:23:59 2021 -0500

Make test scripts clean up duplicate classes

* The test scripts in start create multiple classes with the same name
that can cause problems for IDEs. The classes are only used to create
jars for the VFS tests so just drop the intermediate class files when done
creating the jars.
---
 start/src/test/shell/makeHelloWorldJars.sh | 2 ++
 start/src/test/shell/makeTestJars.sh   | 1 +
 2 files changed, 3 insertions(+)

diff --git a/start/src/test/shell/makeHelloWorldJars.sh 
b/start/src/test/shell/makeHelloWorldJars.sh
index 9f4a990..53d75bd 100755
--- a/start/src/test/shell/makeHelloWorldJars.sh
+++ b/start/src/test/shell/makeHelloWorldJars.sh
@@ -19,8 +19,10 @@ mkdir -p target/generated-sources/HelloWorld/test
 sed "s/%%/Hello World\!/" < src/test/java/test/HelloWorldTemplate > 
target/generated-sources/HelloWorld/test/HelloWorld.java
 $JAVA_HOME/bin/javac target/generated-sources/HelloWorld/test/HelloWorld.java 
-d target/generated-sources/HelloWorld
 $JAVA_HOME/bin/jar -cf target/test-classes/HelloWorld.jar -C 
target/generated-sources/HelloWorld test/HelloWorld.class
+rm -r target/generated-sources/HelloWorld/test
 
 mkdir -p target/generated-sources/HalloWelt/test
 sed "s/%%/Hallo Welt/" < src/test/java/test/HelloWorldTemplate > 
target/generated-sources/HalloWelt/test/HelloWorld.java
 $JAVA_HOME/bin/javac target/generated-sources/HalloWelt/test/HelloWorld.java 
-d target/generated-sources/HalloWelt
 $JAVA_HOME/bin/jar -cf target/test-classes/HelloWorld2.jar -C 
target/generated-sources/HalloWelt test/HelloWorld.class
+rm -r target/generated-sources/HalloWelt/test
\ No newline at end of file
diff --git a/start/src/test/shell/makeTestJars.sh 
b/start/src/test/shell/makeTestJars.sh
index d4dc9fc..539431e 100755
--- a/start/src/test/shell/makeTestJars.sh
+++ b/start/src/test/shell/makeTestJars.sh
@@ -21,4 +21,5 @@ do
 sed "s/testX/test$x/" < src/test/java/test/TestTemplate > 
target/generated-sources/$x/test/TestObject.java
 $JAVA_HOME/bin/javac -cp target/test-classes 
target/generated-sources/$x/test/TestObject.java -d target/generated-sources/$x
 $JAVA_HOME/bin/jar -cf target/test-classes/ClassLoaderTest$x/Test.jar -C 
target/generated-sources/$x test/TestObject.class
+rm -r target/generated-sources/$x
 done


[accumulo] branch main updated: Improvements to monitor external compactions page (#2385)

2021-12-15 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 02de510  Improvements to monitor external compactions page (#2385)
02de510 is described below

commit 02de5104c4c115d913e5547e1dd6f0b6d26390d1
Author: Mike Miller 
AuthorDate: Wed Dec 15 17:14:50 2021 -0500

Improvements to monitor external compactions page (#2385)

* Add endpoint to monitor for external compactions details and modify
page to make ajax calls when getting the details of a running compaction
* Cache up to 50 of the details JSON objects in session storage
* Use ConcurrentHashMap for storing running compactions in Monitor
---
 .../java/org/apache/accumulo/monitor/Monitor.java  | 14 ++--
 .../rest/compactions/external/CompactorInfo.java   |  8 +-
 .../rest/compactions/external/ECResource.java  | 28 ++-
 .../external/RunningCompactorDetails.java  | 51 
 .../compactions/external/RunningCompactorInfo.java | 28 +++
 .../org/apache/accumulo/monitor/resources/js/ec.js | 91 ++
 .../compaction/ExternalCompactionProgressIT.java   |  2 +-
 7 files changed, 178 insertions(+), 44 deletions(-)

diff --git 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java 
b/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java
index dcd7b37..25e6fbd 100644
--- a/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java
+++ b/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java
@@ -37,6 +37,7 @@ import java.util.Map.Entry;
 import java.util.Optional;
 import java.util.Set;
 import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicLong;
@@ -587,7 +588,7 @@ public class Monitor extends AbstractServer implements 
HighlyAvailableService {
   private final Map allCompactions = new 
HashMap<>();
   private final RecentLogs recentLogs = new RecentLogs();
   private final ExternalCompactionInfo ecInfo = new ExternalCompactionInfo();
-  private final Map ecRunningMap = new HashMap<>();
+  private final Map ecRunningMap = new 
ConcurrentHashMap<>();
   private long scansFetchedNanos = 0L;
   private long compactsFetchedNanos = 0L;
   private long ecInfoFetchedNanos = 0L;
@@ -639,7 +640,7 @@ public class Monitor extends AbstractServer implements 
HighlyAvailableService {
* user fetches since RPC calls are going to the coordinator. This allows 
for fine grain updates
* of external compaction progress.
*/
-  public synchronized Map getRunningInfo() {
+  public synchronized Map fetchRunningInfo() {
 if (coordinatorHost.isEmpty()) {
   throw new IllegalStateException(coordinatorMissingMsg);
 }
@@ -655,15 +656,16 @@ public class Monitor extends AbstractServer implements 
HighlyAvailableService {
 
 ecRunningMap.clear();
 if (running.getCompactions() != null) {
-  running.getCompactions().forEach((queue, ec) -> {
-log.trace("Found Compactions running on queue {} -> {}", queue, ec);
-ecRunningMap.put(queue, ec);
-  });
+  ecRunningMap.putAll(running.getCompactions());
 }
 
 return ecRunningMap;
   }
 
+  public Map getEcRunningMap() {
+return ecRunningMap;
+  }
+
   private CompactionCoordinatorService.Client getCoordinator(HostAndPort 
address) {
 if (coordinatorClient == null) {
   try {
diff --git 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/compactions/external/CompactorInfo.java
 
b/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/compactions/external/CompactorInfo.java
index 1363ece..a7dc56a 100644
--- 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/compactions/external/CompactorInfo.java
+++ 
b/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/compactions/external/CompactorInfo.java
@@ -21,9 +21,11 @@ package 
org.apache.accumulo.monitor.rest.compactions.external;
 public class CompactorInfo {
 
   // Variable names become JSON keys
-  public long lastContact;
-  public String server;
-  public String queueName;
+  public long lastContact = 0L;
+  public String server = "";
+  public String queueName = "";
+
+  public CompactorInfo() {}
 
   public CompactorInfo(long fetchedTimeMillis, String queue, String 
hostAndPort) {
 lastContact = System.currentTimeMillis() - fetchedTimeMillis;
diff --git 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/compactions/external/ECResource.java
 
b/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/compactions/external/ECResource.java
index 6c1a050..d7035d6 100644
--- 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/compactions/e

[accumulo-website] branch asf-site updated (e833b91 -> 9477f3f)

2021-12-13 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git.


from e833b91  Automatic Site Publish by Buildbot
 add 9477f3f  Automatic Site Publish by Buildbot

No new revisions were added by this update.

Summary of changes:
 output/feed.xml|   4 ++--
 output/images/release/active-compactions-table.png | Bin 26834 -> 0 bytes
 output/images/release/ec-running-details.png   | Bin 0 -> 131341 bytes
 output/images/release/ec-running2.png  | Bin 0 -> 66055 bytes
 output/release/accumulo-2.1.0/index.html   |  16 +---
 output/search_data.json|   2 +-
 6 files changed, 16 insertions(+), 6 deletions(-)
 delete mode 100644 output/images/release/active-compactions-table.png
 create mode 100644 output/images/release/ec-running-details.png
 create mode 100644 output/images/release/ec-running2.png


[accumulo-website] branch main updated: Update release notes for new EC page in monitor (#303)

2021-12-13 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new 4b3d095  Update release notes for new EC page in monitor (#303)
4b3d095 is described below

commit 4b3d095ff35cc2fb8d3f3d5c1cf0f8ca36f4d8e1
Author: Mike Miller 
AuthorDate: Mon Dec 13 16:39:38 2021 -0500

Update release notes for new EC page in monitor (#303)
---
 _posts/release/2020-01-19-accumulo-2.1.0.md |  15 ---
 images/release/active-compactions-table.png | Bin 26834 -> 0 bytes
 images/release/ec-running-details.png   | Bin 0 -> 131341 bytes
 images/release/ec-running2.png  | Bin 0 -> 66055 bytes
 4 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/_posts/release/2020-01-19-accumulo-2.1.0.md 
b/_posts/release/2020-01-19-accumulo-2.1.0.md
index 8e966b2..3ada29b 100644
--- a/_posts/release/2020-01-19-accumulo-2.1.0.md
+++ b/_posts/release/2020-01-19-accumulo-2.1.0.md
@@ -213,6 +213,12 @@ TABLE: accumulo.metadata
 2default_tablet  0 0 0 0 HOSTED 
CURRENT:ip-10-113-12-25:9997   !0~+INF
 ```
 
+### New Utility for generating splits
+
+A new command line utility was created to generate split points from 1 or more 
rfiles. One or more HDFS directories can
+be given as well. The utility will iterate over all the files provided and 
determine the proper split points
+based on either the size or number given. It uses Apache Datasketches to get 
the split points from the data. {% ghi 2361 %} {% ghi 2368 %}
+
 ### New option for cloning offline
 
 Added option to leave cloned tables offline {% ghi 1474 %} {% ghi 1475 %}
@@ -253,15 +259,18 @@ TODO clarify what was done for this. {% ghi 1787 %} {% 
ghi 1788 %} {% ghi 1798 %
 
 Created new "jshell" convenience entry point {% ghi 1870 %} {% ghi 1910 %}
 
-### New Monitor Compactions page & optimization
+### New Monitor Pages & Improvments
 
 A page was added to the Monitor that lists the active compactions and the 
longest running
 active compaction. As an optimization, the page will only fetch data if a user 
loads the
 page and will only do so a maximum of once a minute. The same optimization was 
added for the Active Scans page,
-along with the addition of a "Fetched" column indicating when the data was 
retrieved. {% ghi 2283 %} {% ghi 2294 %}
+along with the addition of a "Fetched" column indicating when the data was 
retrieved. A new page was also created 
+for External Compactions that allows seeing the progress of compactions and 
many details about ongoing compactions.
+{% ghi 2283 %} {% ghi 2294 %} {% ghi 2358 %}
 
-![Active Compactions](/images/release/active-compactions-table.png)
+
 
+
 
 ### Other notable changes
 
diff --git a/images/release/active-compactions-table.png 
b/images/release/active-compactions-table.png
deleted file mode 100644
index d2c9925..000
Binary files a/images/release/active-compactions-table.png and /dev/null differ
diff --git a/images/release/ec-running-details.png 
b/images/release/ec-running-details.png
new file mode 100644
index 000..506fd5a
Binary files /dev/null and b/images/release/ec-running-details.png differ
diff --git a/images/release/ec-running2.png b/images/release/ec-running2.png
new file mode 100644
index 000..3311235
Binary files /dev/null and b/images/release/ec-running2.png differ


[accumulo] branch main updated (83211c2 -> ff4e542)

2021-12-13 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from 83211c2  Refactor listSplits operation when using maxSplits (#2381)
 add ff4e542  Create GenerateSplits utility (#2368)

No new revisions were added by this update.

Summary of changes:
 assemble/pom.xml   |   5 +
 assemble/src/main/assemblies/component.xml |   1 +
 core/pom.xml   |   4 +
 .../accumulo/core/file/rfile/GenerateSplits.java   | 361 +
 .../apache/accumulo/core/file/rfile/PrintInfo.java |  27 +-
 .../core/file/rfile/GenerateSplitsTest.java| 164 ++
 pom.xml|   5 +
 .../apache/accumulo/test/start/KeywordStartIT.java |   2 +
 8 files changed, 557 insertions(+), 12 deletions(-)
 create mode 100644 
core/src/main/java/org/apache/accumulo/core/file/rfile/GenerateSplits.java
 create mode 100644 
core/src/test/java/org/apache/accumulo/core/file/rfile/GenerateSplitsTest.java


[accumulo] branch main updated: Add support for LZ4 and BZip2 Codecs (#2367)

2021-11-24 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 45a4a93  Add support for LZ4 and BZip2 Codecs (#2367)
45a4a93 is described below

commit 45a4a93d2926760d6a916ae2da1d7e668ec4dcb1
Author: Damon Brown <70720273+trietops...@users.noreply.github.com>
AuthorDate: Wed Nov 24 12:52:29 2021 -0500

Add support for LZ4 and BZip2 Codecs (#2367)

* Hadoop compression support for 'lz4' and 'bzip2' codecs.  Updates javadoc 
and normalizes comments for lz4, bzip2, and zstd.
---
 .../client/mapred/AccumuloFileOutputFormat.java|   2 +-
 .../client/mapreduce/AccumuloFileOutputFormat.java |   2 +-
 .../mapreduce/lib/FileOutputConfigurator.java  |   8 +-
 .../org/apache/accumulo/core/conf/Property.java|   2 +-
 .../core/file/rfile/bcfile/Compression.java| 153 +
 .../core/file/rfile/bcfile/CompressionTest.java|  42 ++
 .../hadoop/mapreduce/FileOutputFormatBuilder.java  |   2 +-
 .../mapreduce/lib/FileOutputConfigurator.java  |   8 +-
 .../strategies/BasicCompactionStrategy.java|   4 +-
 .../accumulo/shell/commands/CompactCommand.java|   4 +-
 10 files changed, 211 insertions(+), 16 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/client/mapred/AccumuloFileOutputFormat.java
 
b/core/src/main/java/org/apache/accumulo/core/client/mapred/AccumuloFileOutputFormat.java
index 4599340..498bc1d 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/client/mapred/AccumuloFileOutputFormat.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/client/mapred/AccumuloFileOutputFormat.java
@@ -67,7 +67,7 @@ public class AccumuloFileOutputFormat extends 
FileOutputFormat {
* @param job
*  the Hadoop job instance to be configured
* @param compressionType
-   *  one of "none", "gz", "lzo", or "snappy"
+   *  one of "none", "gz", "bzip2", "lzo", "lz4", "snappy", or "zstd"
* @since 1.5.0
*/
   public static void setCompressionType(JobConf job, String compressionType) {
diff --git 
a/core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloFileOutputFormat.java
 
b/core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloFileOutputFormat.java
index abf73ce..a0dca00 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloFileOutputFormat.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloFileOutputFormat.java
@@ -65,7 +65,7 @@ public class AccumuloFileOutputFormat extends 
FileOutputFormat {
* @param job
*  the Hadoop job instance to be configured
* @param compressionType
-   *  one of "none", "gz", "lzo", or "snappy"
+   *  one of "none", "gz", "bzip2", "lzo", "lz4", "snappy", or "zstd"
* @since 1.5.0
*/
   public static void setCompressionType(Job job, String compressionType) {
diff --git 
a/core/src/main/java/org/apache/accumulo/core/clientImpl/mapreduce/lib/FileOutputConfigurator.java
 
b/core/src/main/java/org/apache/accumulo/core/clientImpl/mapreduce/lib/FileOutputConfigurator.java
index 2557b31..d557525 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/clientImpl/mapreduce/lib/FileOutputConfigurator.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/clientImpl/mapreduce/lib/FileOutputConfigurator.java
@@ -137,15 +137,15 @@ public class FileOutputConfigurator extends 
ConfiguratorBase {
* @param conf
*  the Hadoop configuration object to configure
* @param compressionType
-   *  one of "none", "gz", "lzo", "snappy", or "zstd"
+   *  one of "none", "gz", "bzip2", "lzo", "lz4", "snappy", or "zstd"
* @since 1.6.0
*/
   public static void setCompressionType(Class implementingClass, 
Configuration conf,
   String compressionType) {
-if (compressionType == null
-|| !Arrays.asList("none", "gz", "lzo", "snappy", 
"zstd").contains(compressionType))
+if (compressionType == null || !Arrays
+.asList("none", "gz", "bzip2", "lzo", "lz4", "snappy", 
"zstd").contains(compressionType))
   throw new IllegalArgumentException(
-  "Compression type must be one of: none, gz, lzo, snappy, zstd");
+  "Compression type must be one of: none, gz, bzip2, lzo, lz4, snappy, 
zstd");
 setAccumuloProperty(implementi

[accumulo] branch main updated: Improve logging message and comments if tablet overlap occurs. (#2349)

2021-11-24 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new a3969c3  Improve logging message and comments if tablet overlap 
occurs. (#2349)
a3969c3 is described below

commit a3969c3f7e6c91e4ae047b27386146ad547f1b12
Author: AlbertWhitlock 
AuthorDate: Wed Nov 24 06:33:35 2021 -0500

Improve logging message and comments if tablet overlap occurs. (#2349)
---
 .../org/apache/accumulo/tserver/ThriftClientHandler.java| 13 +++--
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/ThriftClientHandler.java
 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/ThriftClientHandler.java
index e9b3d3d..dd97375 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/ThriftClientHandler.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/ThriftClientHandler.java
@@ -1470,10 +1470,9 @@ public class ThriftClientHandler extends 
ClientServiceHandler implements TabletC
   synchronized (server.openingTablets) {
 synchronized (server.onlineTablets) {
 
-  // checking if this exact tablet is in any of the sets
-  // below is not a strong enough check
-  // when splits and fix splits occurring
-
+  // Checking if the current tablet is in any of the sets
+  // below is not a strong enough check to catch all overlapping 
tablets
+  // when splits and fix splits are occurring
   Set unopenedOverlapping =
   KeyExtent.findOverlapping(extent, server.unopenedTablets);
   Set openingOverlapping =
@@ -1501,8 +1500,10 @@ public class ThriftClientHandler extends 
ClientServiceHandler implements TabletC
 all.remove(extent);
 
 if (!all.isEmpty()) {
-  log.error("Tablet {} overlaps previously assigned {} {} {}", 
extent,
-  unopenedOverlapping, openingOverlapping, onlineOverlapping + 
" " + all);
+  log.error(
+  "Tablet {} overlaps a previously assigned tablet, possibly 
due to a recent split. "
+  + "Overlapping tablets:  Unopened: {}, Opening: {}, 
Online: {}",
+  extent, unopenedOverlapping, openingOverlapping, 
onlineOverlapping);
 }
 return;
   }


[accumulo] branch main updated: Create page in Monitor for external compactions (#2358)

2021-11-19 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 2bce893  Create page in Monitor for external compactions (#2358)
2bce893 is described below

commit 2bce8939145b49a765f998f5dbe2b6242a64e3e6
Author: Mike Miller 
AuthorDate: Fri Nov 19 10:42:48 2021 -0500

Create page in Monitor for external compactions (#2358)

* Create multiple new classes for displaying 3 different tables of data
in the new external compaction page in the monitor
* Create 3 new ajax endpoints in ECResource
* Modify Compactor and ExternalCompactionUtil to return Optional for the
compaction coordinator instead of null
* Add check for compaction coordinator to Monitor.fetchData()
* New ExternalCompactionProgressIT for testing progress
* Use new bootstrap panel and badges for coordinator info
* Closes #2290

Co-authored-by: Dom G. <47725857+domgargu...@users.noreply.github.com>
---
 .../util/compaction/ExternalCompactionUtil.java|  10 +-
 .../org/apache/accumulo/compactor/Compactor.java   |   8 +-
 .../java/org/apache/accumulo/monitor/Monitor.java  |  89 +
 .../compactions/external/CompactionInputFile.java  |  38 
 .../rest/compactions/external/CompactorInfo.java   |  33 
 .../rest/compactions/external/Compactors.java  |  40 
 .../rest/compactions/external/CoordinatorInfo.java |  41 +
 .../rest/compactions/external/ECResource.java  |  62 +++
 .../external/ExternalCompactionInfo.java   |  60 ++
 .../compactions/external/RunningCompactions.java   |  39 
 .../compactions/external/RunningCompactorInfo.java | 133 ++
 .../org/apache/accumulo/monitor/view/WebViews.java |  25 +++
 .../org/apache/accumulo/monitor/resources/js/ec.js | 201 +
 .../org/apache/accumulo/monitor/templates/ec.ftl   |  80 
 .../apache/accumulo/monitor/templates/navbar.ftl   |   1 +
 .../compaction/ExternalCompactionProgressIT.java   | 166 +
 .../compaction/ExternalCompactionTestUtils.java|  42 +++--
 17 files changed, 1044 insertions(+), 24 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/util/compaction/ExternalCompactionUtil.java
 
b/core/src/main/java/org/apache/accumulo/core/util/compaction/ExternalCompactionUtil.java
index a5538b6..24dff44 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/util/compaction/ExternalCompactionUtil.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/util/compaction/ExternalCompactionUtil.java
@@ -24,6 +24,7 @@ import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
+import java.util.Optional;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Future;
@@ -93,19 +94,18 @@ public class ExternalCompactionUtil {
 
   /**
*
-   * @return null if Coordinator node not found, else HostAndPort
+   * @return Optional HostAndPort of Coordinator node if found
*/
-  public static HostAndPort findCompactionCoordinator(ClientContext context) {
+  public static Optional findCompactionCoordinator(ClientContext 
context) {
 final String lockPath = context.getZooKeeperRoot() + 
Constants.ZCOORDINATOR_LOCK;
 try {
   var zk = ZooSession.getAnonymousSession(context.getZooKeepers(),
   context.getZooKeepersSessionTimeOut());
   byte[] address = ServiceLock.getLockData(zk, ServiceLock.path(lockPath));
   if (null == address) {
-return null;
+return Optional.empty();
   }
-  String coordinatorAddress = new String(address);
-  return HostAndPort.fromString(coordinatorAddress);
+  return Optional.of(HostAndPort.fromString(new String(address)));
 } catch (KeeperException | InterruptedException e) {
   throw new RuntimeException(e);
 }
diff --git 
a/server/compactor/src/main/java/org/apache/accumulo/compactor/Compactor.java 
b/server/compactor/src/main/java/org/apache/accumulo/compactor/Compactor.java
index 898177e..6565280 100644
--- 
a/server/compactor/src/main/java/org/apache/accumulo/compactor/Compactor.java
+++ 
b/server/compactor/src/main/java/org/apache/accumulo/compactor/Compactor.java
@@ -474,12 +474,12 @@ public class Compactor extends AbstractServer implements 
CompactorService.Iface
*   when unable to get client
*/
   protected CompactionCoordinatorService.Client getCoordinatorClient() throws 
TTransportException {
-HostAndPort coordinatorHost = 
ExternalCompactionUtil.findCompactionCoordinator(getContext());
-if (null == coordinatorHost) {
+var coordinatorHost = 
ExternalCompactionUtil.findCompactionCoordinator(getContext());
+if (coordinatorHost.isEmpty()) {
   throw new TTransportException("Unable to get Compaction

[accumulo] branch main updated (bf4d8e4 -> 6cfb918)

2021-11-10 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from bf4d8e4  Improve error messages in PerTableVolumeChooser (#2354)
 add 6cfb918  New IteratorMincClassCastBugIT and fixes to #1411 (#2347)

No new revisions were added by this update.

Summary of changes:
 .../org/apache/accumulo/core/file/rfile/RFile.java |  2 +-
 .../system/InterruptibleIterator.java  |  9 ++
 .../system/InterruptibleMapIterator.java   | 68 ---
 .../system}/IterationInterruptedException.java |  5 +-
 .../core/iteratorsImpl/system/MapFileIterator.java |  1 -
 .../system}/SortedMapIterator.java | 36 ++--
 .../accumulo/core/conf/IterConfigUtilTest.java |  2 +-
 .../iterators/FirstEntryInRowIteratorTest.java |  3 +-
 .../core/iterators/FirstEntryInRowTest.java|  1 +
 .../core/iterators/SortedMapIteratorTest.java  |  1 +
 .../system/ColumnFamilySkippingIteratorTest.java   |  2 +-
 .../core/iterators/system/ColumnFilterTest.java|  2 +-
 .../iterators/system/DeletingIteratorTest.java |  2 +-
 .../core/iterators/system/MultiIteratorTest.java   |  2 +-
 .../system/SourceSwitchingIteratorTest.java|  7 +-
 .../iterators/system/TimeSettingIteratorTest.java  |  2 +-
 .../iterators/system/VisibilityFilterTest.java |  2 +-
 .../iterators/user/BigDecimalCombinerTest.java |  2 +-
 .../core/iterators/user/ColumnSliceFilterTest.java |  2 +-
 .../accumulo/core/iterators/user/CombinerTest.java |  2 +-
 .../accumulo/core/iterators/user/FilterTest.java   |  2 +-
 .../core/iterators/user/GrepIteratorTest.java  |  2 +-
 .../iterators/user/IntersectingIteratorTest.java   |  2 +-
 .../core/iterators/user/LargeRowFilterTest.java|  2 +-
 .../core/iterators/user/RegExFilterTest.java   |  2 +-
 .../iterators/user/RowDeletingIteratorTest.java|  2 +-
 .../iterators/user/RowEncodingIteratorTest.java|  2 +-
 .../core/iterators/user/RowFilterTest.java |  2 +-
 .../core/iterators/user/TestCfCqSlice.java |  2 +-
 .../iterators/user/TransformingIteratorTest.java   |  2 +-
 .../iterators/user/VersioningIteratorTest.java |  2 +-
 .../core/iterators/user/VisibilityFilterTest.java  |  2 +-
 .../user/WholeColumnFamilyIteratorTest.java|  2 +-
 .../core/iterators/user/WholeRowIteratorTest.java  |  2 +-
 .../accumulo/iteratortest/IteratorTestUtil.java|  2 +-
 .../iterators/MetadataBulkLoadFilterTest.java  |  2 +-
 .../org/apache/accumulo/tserver/InMemoryMap.java   |  4 +-
 .../org/apache/accumulo/tserver/NativeMap.java |  2 +-
 .../accumulo/tserver/ThriftClientHandler.java  |  2 +-
 .../apache/accumulo/tserver/scan/LookupTask.java   |  2 +-
 .../accumulo/tserver/scan/NextBatchTask.java   |  2 +-
 .../accumulo/tserver/tablet/ScanDataSource.java|  2 +-
 .../apache/accumulo/tserver/tablet/Scanner.java|  2 +-
 .../org/apache/accumulo/tserver/tablet/Tablet.java |  2 +-
 .../accumulo/tserver/CountingIteratorTest.java |  2 +-
 .../apache/accumulo/tserver/InMemoryMapTest.java   |  2 +-
 ...FileIT.java => IteratorMincClassCastBugIT.java} | 97 +++---
 .../test/performance/scan/CollectTabletStats.java  |  2 +-
 .../apache/accumulo/test/iterator/RegExTest.java   |  2 +-
 49 files changed, 139 insertions(+), 169 deletions(-)
 delete mode 100644 
core/src/main/java/org/apache/accumulo/core/iteratorsImpl/system/InterruptibleMapIterator.java
 rename core/src/main/java/org/apache/accumulo/core/{iterators => 
iteratorsImpl/system}/IterationInterruptedException.java (90%)
 rename core/src/main/java/org/apache/accumulo/core/{iterators => 
iteratorsImpl/system}/SortedMapIterator.java (73%)
 copy 
test/src/main/java/org/apache/accumulo/test/functional/{FlushNoFileIT.java => 
IteratorMincClassCastBugIT.java} (57%)


[accumulo] branch main updated: Create FlushNoFileIT and fix NPE in tablet file flush code (#2342)

2021-11-09 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 52e80b1  Create FlushNoFileIT and fix NPE in tablet file flush code 
(#2342)
52e80b1 is described below

commit 52e80b1d11f0093a07a2d886d85aa74f7998096b
Author: Mike Miller 
AuthorDate: Tue Nov 9 16:55:06 2021 -0500

Create FlushNoFileIT and fix NPE in tablet file flush code (#2342)

* Make flush methods return Optional to handle case
when no file is written
* Clean up a bunch of comments in flush code
* Create FlushNoFileIT for testing flush that doesn't write out a file
* Add checkFlushId to FunctionalTestUtils
* Update TabletLogger.flushed() for when no file is written

Co-authored-by: Keith Turner 
---
 .../apache/accumulo/core/logging/TabletLogger.java |   9 +-
 .../accumulo/server/util/ManagerMetadataUtil.java  |  21 ++-
 .../accumulo/tserver/tablet/DatafileManager.java   |  36 +++---
 .../org/apache/accumulo/tserver/tablet/Tablet.java |  10 +-
 .../accumulo/test/functional/FlushNoFileIT.java| 142 +
 .../test/functional/FunctionalTestUtils.java   |  35 +
 6 files changed, 220 insertions(+), 33 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/logging/TabletLogger.java 
b/core/src/main/java/org/apache/accumulo/core/logging/TabletLogger.java
index ab8808a..7ab5341 100644
--- a/core/src/main/java/org/apache/accumulo/core/logging/TabletLogger.java
+++ b/core/src/main/java/org/apache/accumulo/core/logging/TabletLogger.java
@@ -22,11 +22,13 @@ import static java.util.stream.Collectors.toList;
 
 import java.util.Collection;
 import java.util.List;
+import java.util.Optional;
 import java.util.concurrent.TimeUnit;
 
 import org.apache.accumulo.core.client.admin.CompactionConfig;
 import org.apache.accumulo.core.client.admin.compaction.CompactableFile;
 import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
 import org.apache.accumulo.core.metadata.TServerInstance;
 import org.apache.accumulo.core.metadata.TabletFile;
 import org.apache.accumulo.core.spi.compaction.CompactionJob;
@@ -143,8 +145,11 @@ public class TabletLogger {
 asFileNames(job.getFiles()));
   }
 
-  public static void flushed(KeyExtent extent, TabletFile newDatafile) {
-fileLog.debug("Flushed {} created {} from [memory]", extent, newDatafile);
+  public static void flushed(KeyExtent extent, Optional 
newDatafile) {
+if (newDatafile.isPresent())
+  fileLog.debug("Flushed {} created {} from [memory]", extent, 
newDatafile.get());
+else
+  fileLog.debug("Flushed {} from [memory] but no file was written.", 
extent);
   }
 
   public static void bulkImported(KeyExtent extent, TabletFile file) {
diff --git 
a/server/base/src/main/java/org/apache/accumulo/server/util/ManagerMetadataUtil.java
 
b/server/base/src/main/java/org/apache/accumulo/server/util/ManagerMetadataUtil.java
index 9c557c6..3728377 100644
--- 
a/server/base/src/main/java/org/apache/accumulo/server/util/ManagerMetadataUtil.java
+++ 
b/server/base/src/main/java/org/apache/accumulo/server/util/ManagerMetadataUtil.java
@@ -215,23 +215,22 @@ public class ManagerMetadataUtil {
   }
 
   /**
-   * new data file update function adds one data file to a tablet's list
-   *
-   * @param path
-   *  should be relative to the table directory
-   *
+   * Update tablet file data from flush. Returns a StoredTabletFile if there 
are data entries.
*/
-  public static StoredTabletFile updateTabletDataFile(ServerContext context, 
KeyExtent extent,
-  TabletFile path, DataFileValue dfv, MetadataTime time, String address, 
ServiceLock zooLock,
-  Set unusedWalLogs, TServerInstance lastLocation, long flushId) {
+  public static Optional updateTabletDataFile(ServerContext 
context,
+  KeyExtent extent, TabletFile newDatafile, DataFileValue dfv, 
MetadataTime time,
+  String address, ServiceLock zooLock, Set unusedWalLogs, 
TServerInstance lastLocation,
+  long flushId) {
 
 TabletMutator tablet = context.getAmple().mutateTablet(extent);
-StoredTabletFile newFile = null;
+// if there are no entries, the path doesn't get stored in metadata table, 
only the flush ID
+Optional newFile = Optional.empty();
 
+// if entries are present, write to path to metadata table
 if (dfv.getNumEntries() > 0) {
-  tablet.putFile(path, dfv);
+  tablet.putFile(newDatafile, dfv);
   tablet.putTime(time);
-  newFile = path.insert();
+  newFile = Optional.of(newDatafile.insert());
 
   TServerInstance self = getTServerInstance(address, zooLock);
   tablet.putLocation(self, LocationType.LAST);
diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java

[accumulo] branch main updated: Modify shutdown test in ManagerApiIT (#2333)

2021-11-01 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 82df376  Modify shutdown test in ManagerApiIT (#2333)
82df376 is described below

commit 82df376b1fe5b222c6156a64afeac252beb26441
Author: Mike Miller 
AuthorDate: Mon Nov 1 15:24:30 2021 -0400

Modify shutdown test in ManagerApiIT (#2333)

* Closes #2327
* Rework last test to create 2 connections before shutting down
---
 .../accumulo/test/functional/ManagerApiIT.java | 25 --
 1 file changed, 19 insertions(+), 6 deletions(-)

diff --git 
a/test/src/main/java/org/apache/accumulo/test/functional/ManagerApiIT.java 
b/test/src/main/java/org/apache/accumulo/test/functional/ManagerApiIT.java
index 199cfab..1b39b4c 100644
--- a/test/src/main/java/org/apache/accumulo/test/functional/ManagerApiIT.java
+++ b/test/src/main/java/org/apache/accumulo/test/functional/ManagerApiIT.java
@@ -210,12 +210,19 @@ public class ManagerApiIT extends SharedMiniClusterBase {
   // see the junit annotation to control test ordering at the top of this class
   @Test
   public void z99_testPermissions_shutdown() throws Exception {
-// To shutdown, user needs SystemPermission.SYSTEM
-op = user -> client -> client.shutdown(null, user, false);
-expectPermissionDenied(op, regularUser);
-// We should be able to do both of the following RPC calls before it 
actually shuts down
-expectPermissionSuccess(op, rootUser);
-expectPermissionSuccess(op, privilegedUser);
+// grab connections before shutting down
+var rootUserBuilder = 
Accumulo.newClient().from(getClientProps()).as(rootUser.getPrincipal(),
+rootUser.getToken());
+var privUserBuilder = Accumulo.newClient().from(getClientProps())
+.as(privilegedUser.getPrincipal(), privilegedUser.getToken());
+try (var rootClient = rootUserBuilder.build(); var privClient = 
privUserBuilder.build()) {
+  // To shutdown, user needs SystemPermission.SYSTEM
+  op = user -> client -> client.shutdown(null, user, false);
+  expectPermissionDenied(op, regularUser);
+  // We should be able to do both of the following RPC calls before it 
actually shuts down
+  expectPermissionSuccess(op, (ClientContext) rootClient);
+  expectPermissionSuccess(op, (ClientContext) privClient);
+}
   }
 
   private static void expectPermissionSuccess(
@@ -228,6 +235,12 @@ public class ManagerApiIT extends SharedMiniClusterBase {
 }
   }
 
+  private static void expectPermissionSuccess(
+  Function> op, 
ClientContext context)
+  throws Exception {
+ManagerClient.executeVoid(context, op.apply(context.rpcCreds()));
+  }
+
   private static void expectPermissionDenied(
   Function> op, 
Credentials user)
   throws Exception {


[accumulo-website] branch asf-site updated (3babbc8 -> e833b91)

2021-10-28 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git.


from 3babbc8  Automatic Site Publish by Buildbot
 add 4b36924  Automatic Site Publish by Buildbot
 add e833b91  Automatic Site Publish by Buildbot

No new revisions were added by this update.

Summary of changes:
 output/feed.xml  |  4 ++--
 output/release/accumulo-2.1.0/index.html | 14 +++---
 output/search_data.json  |  2 +-
 3 files changed, 10 insertions(+), 10 deletions(-)


[accumulo-website] branch main updated: Drop LTM note in release notes

2021-10-28 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new 3b7447c  Drop LTM note in release notes
3b7447c is described below

commit 3b7447c6606202123324e954ae6d1508ecdc47bd
Author: Mike Miller 
AuthorDate: Thu Oct 28 13:24:43 2021 -0400

Drop LTM note in release notes
---
 _posts/release/2020-01-19-accumulo-2.1.0.md | 6 +-
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/_posts/release/2020-01-19-accumulo-2.1.0.md 
b/_posts/release/2020-01-19-accumulo-2.1.0.md
index bbb9478..8e966b2 100644
--- a/_posts/release/2020-01-19-accumulo-2.1.0.md
+++ b/_posts/release/2020-01-19-accumulo-2.1.0.md
@@ -13,10 +13,6 @@ TODO high level description of the release
 
 TODO summarize contributions
 
-According to the [Long Term Maintenance (LTM)][LTM] strategy, the intent is to
-maintain the X.X release line with critical bug and security fixes until one
-year after the next LTM version is released.
-
 TODO Version Upgrade guidance
 
 ## Minimum Requirements
@@ -314,4 +310,4 @@ View the [Upgrading Accumulo documentation][upgrade] for 
guidance.
 [ext_comp_project]: https://github.com/apache/accumulo/projects/20
 [vfs_system_classloader]: 
https://github.com/apache/accumulo-classloaders/tree/main/modules/vfs-class-loader
 [opentelemetry_home]: https://opentelemetry.io/
-[micrometer_home]: https://micrometer.io/
\ No newline at end of file
+[micrometer_home]: https://micrometer.io/


[accumulo-website] branch main updated: Expand note about CompactionStrategy (#297)

2021-10-28 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new 3b537e1  Expand note about CompactionStrategy (#297)
3b537e1 is described below

commit 3b537e184ead755fbdc1f82d929c0d23038d73cc
Author: Mike Miller 
AuthorDate: Thu Oct 28 13:21:45 2021 -0400

Expand note about CompactionStrategy (#297)

* Update _posts/release/2020-01-19-accumulo-2.1.0.md

Co-authored-by: Keith Turner 
---
 _posts/release/2020-01-19-accumulo-2.1.0.md | 10 +++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/_posts/release/2020-01-19-accumulo-2.1.0.md 
b/_posts/release/2020-01-19-accumulo-2.1.0.md
index 53a9867..bbb9478 100644
--- a/_posts/release/2020-01-19-accumulo-2.1.0.md
+++ b/_posts/release/2020-01-19-accumulo-2.1.0.md
@@ -62,9 +62,13 @@ Significant changes were made to how Accumulo compacts files 
in this release.  S
The new compaction changes in this release can satisfy this use case while
doing a logarithmic amount of work.
 
-CompactionStrategy was deprecated in favor of new public APIs.
-See its [javadoc]({% jurl 
org.apache.accumulo.tserver.compaction.CompactionStrategy %}) 
-for more information. Github tickets related to these changes: {% ghi 564 %} 
{% ghi 1605 %} {% ghi 1609 %} {% ghi 1649 %} {% ghi %}
+CompactionStrategy was deprecated in favor of new public APIs. 
CompactionStrategy was never public API as it
+used internal types and one of these types `FileRef` was removed in 2.1. Users 
who have written a CompactionStrategy
+can replace `FileRef` with its replacement internal type `StoredTabletFile` 
but this is not recommended. Since it is
+very likely that CompactionStrategy will be removed in a future release, any 
work put into rewriting a CompactionStrategy
+will be lost. It is recommended that users implement CompactionSelector, 
CompactionConfigurer, and CompactionPlanner instead.  The new compaction 
changes in 2.1 introduce new algorithms for optimally scheduling compactions 
across multiple thread pools, configuring a deprecated compaction strategy may 
result is missing out on the benefits of these new algorithms. See
+[javadoc]({% jurl org.apache.accumulo.tserver.compaction.CompactionStrategy %})
+for more information. Github tickets related to these changes: {% ghi 564 %} 
{% ghi 1605 %} {% ghi 1609 %} {% ghi 1649 %}
 
 ### External Compactions (experimental)
 


[accumulo] branch main updated: Make CleanUp shutdownNow close Connector (#2332)

2021-10-28 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new bc921ba  Make CleanUp shutdownNow close Connector (#2332)
bc921ba is described below

commit bc921baf1f96adf1ccb5aec7a8f8a5b6ae66b3bd
Author: Mike Miller 
AuthorDate: Thu Oct 28 11:28:34 2021 -0400

Make CleanUp shutdownNow close Connector (#2332)

* Recent changes in #2303 made static ThriftTransportPool a part of
ClientContext. In order to allow legacy Connector clean up the threads
as it did previously, make the shutdownNow method take a connector and
call close on it. Made null an option in case the Connector is
unavailable to pass as a parameter.
* Fixes failure in CleanUpIT. Partial fix of #2327
---
 core/src/main/java/org/apache/accumulo/core/util/CleanUp.java | 11 ++-
 .../java/org/apache/accumulo/test/functional/CleanUpIT.java   |  2 +-
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/core/src/main/java/org/apache/accumulo/core/util/CleanUp.java 
b/core/src/main/java/org/apache/accumulo/core/util/CleanUp.java
index af83fa9..0b31365 100644
--- a/core/src/main/java/org/apache/accumulo/core/util/CleanUp.java
+++ b/core/src/main/java/org/apache/accumulo/core/util/CleanUp.java
@@ -21,6 +21,8 @@ package org.apache.accumulo.core.util;
 import java.util.Set;
 
 import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.accumulo.core.clientImpl.ConnectorImpl;
 import org.apache.accumulo.core.singletons.SingletonManager;
 import org.apache.accumulo.core.singletons.SingletonManager.Mode;
 import org.slf4j.Logger;
@@ -47,10 +49,17 @@ public class CleanUp {
   /**
* kills all threads created by internal Accumulo singleton resources. After 
this method is
* called, no Connector will work in the current classloader.
+   *
+   * @param conn
+   *  If available, Connector object to close resources on. Will 
accept null otherwise.
*/
-  public static void shutdownNow() {
+  public static void shutdownNow(Connector conn) {
 SingletonManager.setMode(Mode.CLIENT);
 waitForZooKeeperClientThreads();
+if (conn != null) {
+  ConnectorImpl connImpl = (ConnectorImpl) conn;
+  connImpl.getAccumuloClient().close();
+}
   }
 
   /**
diff --git 
a/test/src/main/java/org/apache/accumulo/test/functional/CleanUpIT.java 
b/test/src/main/java/org/apache/accumulo/test/functional/CleanUpIT.java
index 34309cd..4e3c153 100644
--- a/test/src/main/java/org/apache/accumulo/test/functional/CleanUpIT.java
+++ b/test/src/main/java/org/apache/accumulo/test/functional/CleanUpIT.java
@@ -110,7 +110,7 @@ public class CleanUpIT extends SharedMiniClusterBase {
 fail("Not seeing expected threads. Saw " + threadCount);
   }
 
-  org.apache.accumulo.core.util.CleanUp.shutdownNow();
+  org.apache.accumulo.core.util.CleanUp.shutdownNow(conn);
 
   Mutation m2 = new Mutation("r2");
   m2.put("cf1", "cq1", 1, "6");


[accumulo] branch main updated: Fix javadoc typo

2021-10-25 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 4738d5a  Fix javadoc typo
4738d5a is described below

commit 4738d5ad55047a93a5e6869d0defea9b5b4956da
Author: Mike Miller 
AuthorDate: Mon Oct 25 09:01:38 2021 -0400

Fix javadoc typo
---
 .../java/org/apache/accumulo/core/client/lexicoder/AbstractEncoder.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/client/lexicoder/AbstractEncoder.java
 
b/core/src/main/java/org/apache/accumulo/core/client/lexicoder/AbstractEncoder.java
index 255a478..84d1257 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/client/lexicoder/AbstractEncoder.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/client/lexicoder/AbstractEncoder.java
@@ -23,7 +23,7 @@ import static java.util.Objects.requireNonNull;
 
 /**
  * AbstractEncoder is an {@link Encoder} that implements all of Encoder's 
methods validating the
- * input, but has those methods defer logic to to a new method,
+ * input, but has those methods defer logic to a new method,
  * {@link #decodeUnchecked(byte[], int, int)}.
  *
  * @since 1.7.0


[accumulo-website] branch next-release updated (c064970 -> 4ba7465)

2021-10-22 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch next-release
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git.


from c064970  Merge branch 'main' into next-release
 add 4ba7465  Drop sections that no longer apply (#296)

No new revisions were added by this update.

Summary of changes:
 _docs-2/getting-started/features.md| 10 --
 _docs-2/security/on-disk-encryption.md |  6 --
 2 files changed, 16 deletions(-)


[accumulo] branch main updated: Make ThriftTransportPool a part of ClientContext (#2303)

2021-10-22 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new c9e3877  Make ThriftTransportPool a part of ClientContext (#2303)
c9e3877 is described below

commit c9e3877373303997d42b465b2cdb62fd3fffd5d8
Author: Mike Miller 
AuthorDate: Fri Oct 22 10:51:43 2021 -0400

Make ThriftTransportPool a part of ClientContext (#2303)

* Create getTransportPool() in ClientContext that will create a
ThriftTransportPool once and call startCheckerThread()
* Make close method in ClientContext shutdown the ThriftTransportPool
* Pass ClientContext to static methods where needed to return 
ThriftTransportPool
* Drop unused transport object from TabletServerBatchReaderIterator
* Internal ThriftTransportPool refactoring included: Dropping singleton 
code,
making private objects final, changing access modifiers of methods and 
replacing
getConnectionPool() with direct references to object
---
 .../accumulo/core/clientImpl/ClientContext.java| 13 +++
 .../core/clientImpl/ConditionalWriterImpl.java |  6 +-
 .../core/clientImpl/InstanceOperationsImpl.java|  4 +-
 .../accumulo/core/clientImpl/ManagerClient.java|  8 +-
 .../core/clientImpl/ReplicationClient.java |  8 +-
 .../accumulo/core/clientImpl/ServerClient.java | 10 +--
 .../core/clientImpl/TableOperationsImpl.java   | 16 ++--
 .../TabletServerBatchReaderIterator.java   |  6 +-
 .../core/clientImpl/TabletServerBatchWriter.java   |  2 +-
 .../accumulo/core/clientImpl/ThriftScanner.java|  6 +-
 .../core/clientImpl/ThriftTransportPool.java   | 99 --
 .../apache/accumulo/core/clientImpl/Writer.java|  2 +-
 .../org/apache/accumulo/core/rpc/ThriftUtil.java   | 10 +--
 .../org/apache/accumulo/core/summary/Gatherer.java |  2 +-
 .../util/compaction/ExternalCompactionUtil.java|  6 +-
 .../miniclusterImpl/MiniAccumuloClusterImpl.java   | 34 
 .../accumulo/server/client/BulkImporter.java   |  4 +-
 .../accumulo/server/manager/LiveTServerSet.java| 18 ++--
 .../manager/balancer/BalancerEnvironmentImpl.java  |  2 +-
 .../server/master/balancer/TabletBalancer.java |  2 +-
 .../server/util/VerifyTabletAssignments.java   |  2 +-
 .../coordinator/CompactionCoordinator.java |  9 +-
 .../accumulo/coordinator/CompactionFinalizer.java  |  2 +-
 .../org/apache/accumulo/compactor/Compactor.java   |  8 +-
 .../java/org/apache/accumulo/manager/Manager.java  |  4 +-
 .../manager/tableOps/bulkVer1/LoadFiles.java   |  2 +-
 .../manager/tableOps/bulkVer2/LoadFiles.java   |  2 +-
 .../java/org/apache/accumulo/monitor/Monitor.java  |  8 +-
 .../rest/tservers/TabletServerResource.java|  2 +-
 .../org/apache/accumulo/tserver/TabletServer.java  |  2 +-
 .../accumulo/shell/commands/ListBulkCommand.java   |  2 +-
 .../accumulo/test/DetectDeadTabletServersIT.java   |  2 +-
 .../org/apache/accumulo/test/GetManagerStats.java  |  2 +-
 .../apache/accumulo/test/TransportCachingIT.java   |  2 +-
 .../functional/BalanceAfterCommsFailureIT.java |  2 +-
 .../BalanceInPresenceOfOfflineTableIT.java |  2 +-
 .../accumulo/test/functional/BulkFailureIT.java|  4 +-
 .../test/functional/MetadataMaxFilesIT.java|  2 +-
 .../test/functional/SimpleBalancerFairnessIT.java  |  2 +-
 39 files changed, 130 insertions(+), 189 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientContext.java 
b/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientContext.java
index 70425f9..9e3fdce 100644
--- a/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientContext.java
+++ b/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientContext.java
@@ -113,6 +113,7 @@ public class ClientContext implements AccumuloClient {
   private final Supplier saslSupplier;
   private final Supplier sslSupplier;
   private TCredentials rpcCreds;
+  private ThriftTransportPool thriftTransportPool;
 
   private volatile boolean closed = false;
 
@@ -711,6 +712,9 @@ public class ClientContext implements AccumuloClient {
   @Override
   public void close() {
 closed = true;
+if (thriftTransportPool != null) {
+  thriftTransportPool.shutdown();
+}
 singletonReservation.close();
   }
 
@@ -915,4 +919,13 @@ public class ClientContext implements AccumuloClient {
   setProperty(property, Integer.toString(value));
 }
   }
+
+  public synchronized ThriftTransportPool getTransportPool() {
+ensureOpen();
+if (thriftTransportPool == null) {
+  thriftTransportPool = new ThriftTransportPool();
+  thriftTransportPool.startCheckerThread();
+}
+return thriftTransportPool;
+  }
 }
diff --git 
a/core/src/main/java/org/apache/accumulo/core/clientImpl/ConditionalWriterImpl.java
 
b/core/src/main/java/org/apache

[accumulo-website] branch asf-site updated (05e57d2 -> 79e3954)

2021-10-22 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git.


from 05e57d2  Automatic Site Publish by Buildbot
 add 79e3954  Automatic Site Publish by Buildbot

No new revisions were added by this update.

Summary of changes:
 output/docs/2.x/getting-started/features.html | 2 +-
 output/feed.xml   | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)


[accumulo-website] branch main updated: Fix closing tag in features.md

2021-10-22 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new 528fbd0  Fix closing tag in features.md
528fbd0 is described below

commit 528fbd035279216a7374963f5110bc1a3489ef23
Author: Mike Miller 
AuthorDate: Fri Oct 22 10:14:10 2021 -0400

Fix closing tag in features.md
---
 _docs-2/getting-started/features.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/_docs-2/getting-started/features.md 
b/_docs-2/getting-started/features.md
index bddbac4..c7257c9 100644
--- a/_docs-2/getting-started/features.md
+++ b/_docs-2/getting-started/features.md
@@ -261,7 +261,7 @@ the Javadoc for [ConditionalMutation] and 
[ConditionalWriter].
 
 ### Lexicoders
 
-[Lexicoders]({% durl getting-started/table_design#lexicoders) (since 1.6.0) 
help encode data (i.e numbers, dates)
+[Lexicoders]({% durl getting-started/table_design#lexicoders %}) (since 1.6.0) 
help encode data (i.e numbers, dates)
 into Accumulo keys in a way that their natural sort order is preserved.
 
 ## Extensible Behaviors


[accumulo-website] branch next-release updated (b701adf -> c064970)

2021-10-22 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch next-release
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git.


from b701adf  Documentation for external compactions (#282)
 add 9de1a36  Update website for 2021 ApacheCon events image/link
 add c85ac0c  Add comment in css
 add 66be02b  Fix ApacheCon event image generation
 add 795fcc1  Update gems
 add ed46e04  Tweak the download page button layout
 add 7bdfd93  Remove unused gem and tighten version constraints
 add 94038c0  Add Harjit Singh to contributors (#267)
 add 2d336e9  Add prominent link to public API docs (#268)
 add 876b998  Improve 2.x quickstart doc (#271)
 add 81e15de  Improve description of loc in metadata
 add b456d36  corrected text per review comments.
 add 9561908  Update _docs-2/troubleshooting/system-metadata-tables.md
 add b576e44  Update _docs-2/troubleshooting/system-metadata-tables.md
 add 7ace637  Update _docs-2/troubleshooting/system-metadata-tables.md
 add baf509e  Merge pull request #272 from 
EdColeman/update_metadata_loc_description
 add aeb13a2  Remove self from Reviewers lines on blog posts
 add e732cd2  Fix downloads script for mirror selection
 add db33093  Update people.md
 add 1114a92  Update people.md
 add 729d361  Merge pull request #275 from Umang228/patch-1
 add b670b9a  Add build command to pages/how-to-contribute.md (#276)
 add ce52f6c  Added Ali Mustafa as a contributor in people.md (#277)
 add 2921523  Update some gems (#278)
 add 8fcd22c  Update people.md
 add 4319d88  Merge pull request #279 from hguercan/patch-1
 add 67386aa  Update contributors list to add Amisha Sahu (#280)
 add cc77c89  Added 'Akshit Mangotra' in people.md (#281)
 add 6e18ff4  Add JShell Accumulo Feature Blog (#274)
 add 8f0de12  Update people.md (#283)
 add 0900900  Add Luke Foster to people.md (#284)
 add c386fc7  Update tools doc for CheckForMetadataProblems (#285)
 add 2ca6f78  Update tools.md for new CheckForMetadataProblems output (#287)
 add 1a76ea0  External Compaction Blog post (#286)
 add 0f0a071  Update gems
 add c3fbc84  Update people.md (#288)
 add 17c0196  Moved name tp PMC list (#290)
 add 085cb1b  contributor list updated (#291)
 add 67a66cb  Update Contributor list (#293)
 add 4ee208a  Add a bunch to release notes for 2.1.0
 add e8c86a8  Updated relnotes (#294)
 add 66477b5  Release notes updates (#295)
 add f7d672e  Add notable change and correct wording in release notes
 add 7aac9f3  Add new property to release notes
 add c064970  Merge branch 'main' into next-release

No new revisions were added by this update.

Summary of changes:
 Gemfile|   5 +-
 Gemfile.lock   |  38 +-
 _docs-2/getting-started/quickstart.md  | 240 ++
 _docs-2/troubleshooting/system-metadata-tables.md  |  11 +-
 _docs-2/troubleshooting/tools.md   |  10 +-
 _includes/nav.html |   1 +
 _layouts/default.html  |   1 +
 _posts/blog/2016-10-28-durability-performance.md   |   2 +-
 _posts/blog/2019-08-12-why-upgrade.md  |   2 +-
 _posts/blog/2021-04-21-jshell-accumulo-feature.md  | 157 +++
 _posts/blog/2021-07-08-external-compactions.md | 491 +
 _posts/release/2020-01-19-accumulo-2.1.0.md| 180 +++-
 .../202107_ecomp/accumulo-compactor-muchos.yaml| 180 
 images/blog/202107_ecomp/ci-entries.png| Bin 0 -> 31996 bytes
 images/blog/202107_ecomp/ci-files-per-tablet.png   | Bin 0 -> 3 bytes
 images/blog/202107_ecomp/ci-ingest-rate.png| Bin 0 -> 43035 bytes
 images/blog/202107_ecomp/ci-online-tablets.png | Bin 0 -> 29720 bytes
 images/blog/202107_ecomp/ci-pods-running.png   | Bin 0 -> 35805 bytes
 images/blog/202107_ecomp/ci-queued.png | Bin 0 -> 45185 bytes
 images/blog/202107_ecomp/ci-running.png| Bin 0 -> 58471 bytes
 images/blog/202107_ecomp/clusters-layout.png   | Bin 0 -> 107997 bytes
 images/blog/202107_ecomp/files_over_time.html  |  23 +
 .../202107_ecomp/full-table-compaction-queued.png  | Bin 0 -> 35444 bytes
 .../202107_ecomp/full-table-compaction-running.png | Bin 0 -> 35926 bytes
 .../tablet-2_1fa2e8ba2e8ba328-files.gif| Bin 0 -> 6501483 bytes
 .../tablet-2_385d1745d1745d88-files.gif| Bin 0 -> 6535064 bytes
 .../tablet-2_6a8ba2e8ba2e8c78-files.gif| Bin 0 -> 6467307 bytes
 .../blog/202107_ecomp/tablet-2_default-files.gif   | Bin 0 -> 5817065 bytes
 images/release/active-compactions-table.png| Bin 0 -> 26834 bytes
 index.html |   2 +-
 pages/downloads.md

[accumulo-website] branch asf-site updated (65c8b61 -> 05e57d2)

2021-10-21 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git.


from 65c8b61  Automatic Site Publish by Buildbot
 add c29b047  Automatic Site Publish by Buildbot
 add ffc4ed9  Automatic Site Publish by Buildbot
 add e3950b3  Automatic Site Publish by Buildbot
 add 05e57d2  Automatic Site Publish by Buildbot

No new revisions were added by this update.

Summary of changes:
 output/feed.xml|   4 +-
 output/images/release/active-compactions-table.png | Bin 0 -> 26834 bytes
 output/release/accumulo-2.1.0/index.html   | 111 ++---
 output/release/index.html  |   2 +-
 output/search_data.json|   2 +-
 5 files changed, 101 insertions(+), 18 deletions(-)
 create mode 100644 output/images/release/active-compactions-table.png


[accumulo-website] branch main updated: Add new property to release notes

2021-10-21 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new 7aac9f3  Add new property to release notes
7aac9f3 is described below

commit 7aac9f30453351a5ae4f500259ebee3220dd88bc
Author: Mike Miller 
AuthorDate: Thu Oct 21 15:16:27 2021 -0400

Add new property to release notes
---
 _posts/release/2020-01-19-accumulo-2.1.0.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/_posts/release/2020-01-19-accumulo-2.1.0.md 
b/_posts/release/2020-01-19-accumulo-2.1.0.md
index 5db345b..53a9867 100644
--- a/_posts/release/2020-01-19-accumulo-2.1.0.md
+++ b/_posts/release/2020-01-19-accumulo-2.1.0.md
@@ -292,7 +292,7 @@ along with the addition of a "Fetched" column indicating 
when the data was retri
 * {% ghi 1848 %} Update column name in the monitor UI
 * {% ghi 2039 %} {% ghi 2045 %} Add bulk import option to ignore empty dirs
 * {% ghi 2064 %} Add option to disable encryption to AESCryptoService
-* {% ghi 2117 %} Make sorted recovery write to RFiles
+* {% ghi 2117 %} {% ghi 2236 %} Make sorted recovery write to RFiles. New 
`tserver.wal.sort.file.` property to configure
 * {% ghi 2273 %} Upgrade Thrift to 0.15.0
 
 ## Upgrading


[accumulo-website] branch main updated: Add notable change and correct wording in release notes

2021-10-21 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new f7d672e  Add notable change and correct wording in release notes
f7d672e is described below

commit f7d672ee078e848c442f4f5c91c12a24178cd6f1
Author: Mike Miller 
AuthorDate: Thu Oct 21 14:55:53 2021 -0400

Add notable change and correct wording in release notes
---
 _posts/release/2020-01-19-accumulo-2.1.0.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/_posts/release/2020-01-19-accumulo-2.1.0.md 
b/_posts/release/2020-01-19-accumulo-2.1.0.md
index 62ece86..5db345b 100644
--- a/_posts/release/2020-01-19-accumulo-2.1.0.md
+++ b/_posts/release/2020-01-19-accumulo-2.1.0.md
@@ -253,7 +253,7 @@ TODO clarify what was done for this. {% ghi 1787 %} {% ghi 
1788 %} {% ghi 1798 %
 
 Created new "jshell" convenience entry point {% ghi 1870 %} {% ghi 1910 %}
 
-### New Monitor Compactions page & performance improvements
+### New Monitor Compactions page & optimization
 
 A page was added to the Monitor that lists the active compactions and the 
longest running
 active compaction. As an optimization, the page will only fetch data if a user 
loads the
@@ -293,6 +293,7 @@ along with the addition of a "Fetched" column indicating 
when the data was retri
 * {% ghi 2039 %} {% ghi 2045 %} Add bulk import option to ignore empty dirs
 * {% ghi 2064 %} Add option to disable encryption to AESCryptoService
 * {% ghi 2117 %} Make sorted recovery write to RFiles
+* {% ghi 2273 %} Upgrade Thrift to 0.15.0
 
 ## Upgrading
 


[accumulo-website] branch main updated: Release notes updates (#295)

2021-10-21 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new 66477b5  Release notes updates (#295)
66477b5 is described below

commit 66477b5a135be31ebc2e3e41198cc9e48e0dd72b
Author: Mike Miller 
AuthorDate: Thu Oct 21 14:51:36 2021 -0400

Release notes updates (#295)

* Expand some sections with more detail
* Add LTM to header
* Add informative sections at the beginning with TODOs
* Add TODOs for sections that need more info
* Add bullet to notable changes
---
 _posts/release/2020-01-19-accumulo-2.1.0.md |  73 
 images/release/active-compactions-table.png | Bin 0 -> 26834 bytes
 2 files changed, 63 insertions(+), 10 deletions(-)

diff --git a/_posts/release/2020-01-19-accumulo-2.1.0.md 
b/_posts/release/2020-01-19-accumulo-2.1.0.md
index 6517d99..62ece86 100644
--- a/_posts/release/2020-01-19-accumulo-2.1.0.md
+++ b/_posts/release/2020-01-19-accumulo-2.1.0.md
@@ -1,11 +1,28 @@
 ---
 title: Apache Accumulo 2.1.0
 sortableversion: '02.01.00'
+LTM: true
 draft: true
 ---
 
 ** DRAFT RELEASE NOTES **
 
+## About
+
+TODO high level description of the release
+
+TODO summarize contributions
+
+According to the [Long Term Maintenance (LTM)][LTM] strategy, the intent is to
+maintain the X.X release line with critical bug and security fixes until one
+year after the next LTM version is released.
+
+TODO Version Upgrade guidance
+
+## Minimum Requirements
+
+TODO describe requirements
+
 ## Binary Incompatibility
 
 This release is known to be incompatible with prior versions of the client
@@ -14,9 +31,9 @@ be able to communicate with a 2.1.0 or later installation of 
Accumulo, nor will
 the 2.1.0 or later version of the client libraries communicate with a 2.0.1 or
 earlier installation.
 
-## Notable Changes
+## Major Changes
 
-### Compaction Changes
+### Overhaul of Table Compactions
 
 Significant changes were made to how Accumulo compacts files in this release.  
See 
 {% dlink administration/compaction %} for details, below are some highlights.
@@ -169,11 +186,32 @@ metrics to supported Micrometer sinks. {% ghi 1134 %}
 
 ### New SPI Package
 
-New SPI Package {% ghi 1900 %} {% ghi 1905 %} {% ghi 1880 %} {% ghi 1891 %} {% 
ghi 1426 %}
+TODO describe New SPI Package {% ghi 1900 %} {% ghi 1905 %} {% ghi 1880 %} {% 
ghi 1891 %} {% ghi 1426 %}
 
 ### New listtablets shell command.
 
-New command for debugging tablets called listtablets {% ghi 1317 %} {% ghi 
1821 %}
+A new command was created for debugging called listtablets, that shows 
detailed tablet information 
+on a single line. This command aggregates data about a tablet such as status, 
location, size, 
+number of entries and HDFS directory name. It even shows the start and end 
rows of tablets, 
+displaying them in the same sorted order they are stored in the metadata. See 
example command output below.
+{% ghi 1317 %} {% ghi 1821 %}
+
+```
+root@uno> listtablets -t test_ingest -h
+2021-01-04T15:12:47,663 [Shell.audit] INFO : root@uno> listtablets -t 
test_ingest -h
+NUM  TABLET_DIR  FILES WALS  ENTRIES   SIZE  STATUS LOCATION   
IDSTART (Exclusive)END 
+TABLE: test_ingest
+1t-007   1 060   552 HOSTED 
CURRENT:ip-10-113-12-25:9997   2 -INF row_05  
+2t-006   1 0   500 2.71K HOSTED 
CURRENT:ip-10-113-12-25:9997   2 row_05   row_55  
+3t-008   1 0 5.00K24.74K HOSTED 
CURRENT:ip-10-113-12-25:9997   2 row_55   row_000555  
+4default_tablet  1 0 4.44K22.01K HOSTED 
CURRENT:ip-10-113-12-25:9997   2 row_000555   +INF  
   
+root@uno> listtablets -t accumulo.metadata
+2021-01-04T15:13:21,750 [Shell.audit] INFO : root@uno> listtablets -t 
accumulo.metadata
+NUM  TABLET_DIR  FILES WALS  ENTRIES   SIZE  STATUS LOCATION   
IDSTART (Exclusive)END 
+TABLE: accumulo.metadata
+1table_info  2 0 7 524   HOSTED 
CURRENT:ip-10-113-12-25:9997   !0-INF ~   
+2default_tablet  0 0 0 0 HOSTED 
CURRENT:ip-10-113-12-25:9997   !0~+INF
+```
 
 ### New option for cloning offline
 
@@ -181,7 +219,10 @@ Added option to leave cloned tables offline {% ghi 1474 %} 
{% ghi 1475 %}
 
 ### New max tablets option in bulk import
 
-Created max tablets property in new bulk import {% ghi 1614 %}
+The property `table.bulk.max.tablets` was created in new bulk import 
technique. This property acts as a cluster performance 
+failsafe to prevent a single ingested 

[accumulo-website] branch asf-site updated (d8a979e -> 65c8b61)

2021-10-20 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git.


from d8a979e  Automatic Site Publish by Buildbot
 add 65c8b61  Automatic Site Publish by Buildbot

No new revisions were added by this update.

Summary of changes:
 output/feed.xml  |  4 +-
 output/release/accumulo-2.1.0/index.html | 82 +++-
 output/search_data.json  |  2 +-
 3 files changed, 83 insertions(+), 5 deletions(-)


[accumulo-website] branch main updated: Add a bunch to release notes for 2.1.0

2021-10-20 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new 4ee208a  Add a bunch to release notes for 2.1.0
4ee208a is described below

commit 4ee208a58ffbfc69a846f8fb0cbba506480d71cc
Author: Mike Miller 
AuthorDate: Wed Oct 20 12:46:54 2021 -0400

Add a bunch to release notes for 2.1.0
---
 _posts/release/2020-01-19-accumulo-2.1.0.md | 84 +++--
 1 file changed, 81 insertions(+), 3 deletions(-)

diff --git a/_posts/release/2020-01-19-accumulo-2.1.0.md 
b/_posts/release/2020-01-19-accumulo-2.1.0.md
index 7dee888..d7a022e 100644
--- a/_posts/release/2020-01-19-accumulo-2.1.0.md
+++ b/_posts/release/2020-01-19-accumulo-2.1.0.md
@@ -47,14 +47,14 @@ Significant changes were made to how Accumulo compacts 
files in this release.  S
 
 CompactionStrategy was deprecated in favor of new public APIs.
 See its [javadoc]({% jurl 
org.apache.accumulo.tserver.compaction.CompactionStrategy %}) 
-for more information.
+for more information. Github tickets related to these changes: {% ghi 564 %} 
{% ghi 1605 %} {% ghi 1609 %} {% ghi 1649 %} {% ghi %}
 
 ### Fixed GC Metadata hotspots
 
 Prior to this release, Accumulo stored GC file candidates in the metadata table
 using rows of the form `~del`. This row schema lead to uneven load on
 the metadata table and metadata tablets that were eventually never used. In {%
-ghi 1043 %} the row format was changed to `~del` resulting in
+ghi 1043 %} {% ghi 1344 %} the row format was changed to 
`~del` resulting in
 even load on the metadata table and even data spread in the tablets. After
 upgrading, there may still be splits in the metadata table using the old row
 format. These splits can be merged away as shown in the example below which
@@ -142,4 +142,82 @@ Some other specific examples of these changes include:
   `org.apache.accumulo.manager.state.SetGoalState` instead.
 * `masterMemory` in `minicluster.properties` has been deprecated and
   `managerMemory` should be used instead in any `minicluster.properties` files
-  you have configured.
\ No newline at end of file
+  you have configured.
+* See also {% ghi 1640 %} {% ghi 1642 %} {% ghi 1703 %} {% ghi 1704 %} {% ghi 
1873 %} {% ghi 1907 %}
+
+### New SPI Package
+
+New SPI Package {% ghi 1900 %} {% ghi 1905 %} {% ghi 1880 %} {% ghi 1891 %} {% 
ghi 1426 %}
+
+### New listtablets shell command.
+
+New command for debugging tablets called listtablets {% ghi 1317 %} {% ghi 
1821 %}
+
+### New option for cloning offline
+
+Added option to leave cloned tables offline {% ghi 1474 %} {% ghi 1475 %}
+
+### New max tablets option in bulk import
+
+Created max tablets property in new bulk import {% ghi 1614 %}
+
+### New ability for user to define context classloaders
+
+Deprecated VFS ClassLoader and added ability for user to define context 
classloaders {% ghi 1747 %} {% ghi 1715 %}
+
+### Updated hash algorithm
+
+TODO clarify what was done for this. {% ghi 1787 %} {% ghi 1788 %} {% ghi 1798 
%} {% ghi 1810 %}
+
+### Various Performance improvements when deleting tables
+
+Make delete table operations cancel user compactions {% ghi 2030 %}.
+Added check to not flush when table is being deleted {% ghi 1887 %}.
+Make the UnloadTabletHandler log before waiting for deletes to finish {% ghi 
1881 %}.
+Stop user flush if table is being deleted {% ghi 1931 %}
+
+### New jshell entry point
+
+Created new "jshell" convenience entry point {% ghi 1870 %} {% ghi 1910 %}
+
+### Other notable changes
+
+* {% ghi 1309 %} Prevent cloning metadata table
+* {% ghi 1294 %} {% ghi 1299 %} Add optional -t tablename to importdirectory 
shell command.
+* {% ghi 1332 %} Disable FileSystemMonitor checks by default
+* {% ghi 1345 %} {% ghi 1352 %} Allow gc initiated compaction as option.
+* {% ghi 1090 %} {% ghi 1330 %} Consolidate/Simplify ServerOpts
+* {% ghi 1389 %} Always choose volume for new tablet files.
+* {% ghi 1390 %} {% ghi 1400 %} {% ghi 1411 %} Declare core.iterators package 
public API
+* {% ghi 1441 %} {% ghi 1405 %} Improvements to Upgrades
+* {% ghi 1397 %} {% ghi 1461 %} Replace relative paths on upgrade.
+* {% ghi 1456 %} {% ghi 1457 %} Prevent catastrophic tserver shutdown by rate 
limiting the shutdown
+* {% ghi 1528 %} {% ghi 1514 %} {% ghi 1515 %} {% ghi 1516 %} Upgraded to 
Log4j2
+* {% ghi 1513 %} {% ghi 1530 %} {% ghi 1531 %} {% ghi 1877 %} Upgraded to 
Zookeeper 3.5.9
+* {% ghi 1053 %} {% ghi 1060 %} {% ghi 1576 %} Support multiple volumes in 
import table
+* {% ghi 1464 %} Leverage SimpleThreadPool for improving importtable 
performance
+* {% ghi 1568 %} Support multiple tservers / node in accumulo-service
+* {% ghi 1644 %} {% ghi 1645 %} Fix issue with minor compaction not retrying
+* {% ghi 1660 %} Dropped MemoryManager property
+* {% ghi 1543 %} {% ghi 1650 %} Use batchin

[accumulo] branch main updated (98cf4b5 -> b563904)

2021-10-18 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from 98cf4b5  Fix minor checkstyle issues (#2319)
 add b563904  Fix error message typo in Compactor

No new revisions were added by this update.

Summary of changes:
 .../src/main/java/org/apache/accumulo/compactor/Compactor.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[accumulo] branch main updated: Add check & tests for DefaultCompactionPlanner (#2316)

2021-10-14 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 1cb6973  Add check & tests for DefaultCompactionPlanner (#2316)
1cb6973 is described below

commit 1cb697324418a823de26c7485e87454845d9813d
Author: Mike Miller 
AuthorDate: Thu Oct 14 09:45:54 2021 -0400

Add check & tests for DefaultCompactionPlanner (#2316)

* Add check to DefaultCompactionPlanner to throw error if there are
duplicate maxSizes configured
* Add tests for error conditions to DefaultCompactionPlannerTest
---
 .../spi/compaction/DefaultCompactionPlanner.java   |   9 ++
 .../compaction/DefaultCompactionPlannerTest.java   | 148 +
 2 files changed, 157 insertions(+)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlanner.java
 
b/core/src/main/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlanner.java
index 2cd1b96..edb9b37 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlanner.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlanner.java
@@ -198,6 +198,15 @@ public class DefaultCompactionPlanner implements 
CompactionPlanner {
   "Can only have one executor w/o a maxSize. " + 
params.getOptions().get("executors"));
 }
 
+// use the add method on the Set interface to check for duplicate maxSizes
+Set maxSizes = new HashSet<>();
+executors.forEach(e -> {
+  if (!maxSizes.add(e.getMaxSize())) {
+throw new IllegalArgumentException(
+"Duplicate maxSize set in executors. " + 
params.getOptions().get("executors"));
+  }
+});
+
 determineMaxFilesToCompact(params);
   }
 
diff --git 
a/core/src/test/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlannerTest.java
 
b/core/src/test/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlannerTest.java
index 34a7d02..95e3335 100644
--- 
a/core/src/test/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlannerTest.java
+++ 
b/core/src/test/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlannerTest.java
@@ -19,6 +19,7 @@
 package org.apache.accumulo.core.spi.compaction;
 
 import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThrows;
 import static org.junit.Assert.assertTrue;
 import static org.junit.Assert.fail;
 
@@ -234,6 +235,153 @@ public class DefaultCompactionPlannerTest {
 assertEquals(CompactionExecutorIdImpl.externalId("large"), 
job.getExecutor());
   }
 
+  /**
+   * Tests internal type executor with no numThreads set throws error
+   */
+  @Test
+  public void testErrorInternalTypeNoNumThreads() {
+DefaultCompactionPlanner planner = new DefaultCompactionPlanner();
+Configuration conf = EasyMock.createMock(Configuration.class);
+
EasyMock.expect(conf.isSet(EasyMock.anyString())).andReturn(false).anyTimes();
+
+ServiceEnvironment senv = EasyMock.createMock(ServiceEnvironment.class);
+EasyMock.expect(senv.getConfiguration()).andReturn(conf).anyTimes();
+EasyMock.replay(conf, senv);
+
+String executors = getExecutors("'type': 'internal','maxSize':'32M'",
+"'type': 'internal','maxSize':'128M','numThreads':2",
+"'type': 'internal','maxSize':'512M','numThreads':3");
+var e = assertThrows("Failed to throw error", NullPointerException.class,
+() -> planner.init(getInitParams(senv, executors)));
+assertTrue("Error message didn't contain numThreads", 
e.getMessage().contains("numThreads"));
+  }
+
+  /**
+   * Test external type executor with numThreads set throws error.
+   */
+  @Test
+  public void testErrorExternalTypeNumThreads() {
+DefaultCompactionPlanner planner = new DefaultCompactionPlanner();
+Configuration conf = EasyMock.createMock(Configuration.class);
+
EasyMock.expect(conf.isSet(EasyMock.anyString())).andReturn(false).anyTimes();
+
+ServiceEnvironment senv = EasyMock.createMock(ServiceEnvironment.class);
+EasyMock.expect(senv.getConfiguration()).andReturn(conf).anyTimes();
+EasyMock.replay(conf, senv);
+
+String executors = getExecutors("'type': 
'internal','maxSize':'32M','numThreads':1",
+"'type': 'internal','maxSize':'128M','numThreads':2",
+"'type': 'external','maxSize':'512M','numThreads':3");
+var e = assertThrows("Failed to throw error", 
IllegalArgumentException.class,
+() -> planner.init(getInitParams(senv, executors)));
+assertTrue("Error message didn't contain numThreads", 
e.getMessage().contains("numThreads"));
+  }
+
+  /**
+   * Tests external typ

[accumulo] branch main updated: Improve error message in CompactionService. Closes #2311 (#2315)

2021-10-13 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new eaa8268  Improve error message in CompactionService. Closes #2311 
(#2315)
eaa8268 is described below

commit eaa8268f3835733283f8a1ffcabdf5101135a4f3
Author: Mike Miller 
AuthorDate: Wed Oct 13 13:45:52 2021 -0400

Improve error message in CompactionService. Closes #2311 (#2315)
---
 .../org/apache/accumulo/tserver/compactions/CompactionService.java | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java
 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java
index 43f1363..c5ea507 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java
@@ -132,7 +132,8 @@ public class CompactionService {
 @Override
 public CompactionExecutorId getExternalExecutor(String name) {
   var ceid = CompactionExecutorIdImpl.externalId(name);
-  
Preconditions.checkArgument(!requestedExternalExecutors.contains(ceid));
+  
Preconditions.checkArgument(!requestedExternalExecutors.contains(ceid),
+  "Duplicate external executor for queue " + name);
   requestedExternalExecutors.add(ceid);
   return ceid;
 }


[accumulo] branch main updated: Rename config check classes (#2314)

2021-10-13 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 0357c9b  Rename config check classes (#2314)
0357c9b is described below

commit 0357c9b66e0fcd49cc8778c3e51d91a60b494e38
Author: Mike Miller 
AuthorDate: Wed Oct 13 08:26:57 2021 -0400

Rename config check classes (#2314)

* Rename classes that check configuration to more specific names
* Rename ConfigSanityCheck to CheckServerConfig to match keyword
---
 ...ConfigSanityCheck.java => ConfigCheckUtil.java} | 14 ++--
 .../accumulo/core/conf/SiteConfiguration.java  |  2 +-
 ...nityCheckTest.java => ConfigCheckUtilTest.java} | 26 +++---
 .../core/conf/DefaultConfigurationTest.java|  2 +-
 ...nfigSanityCheck.java => CheckServerConfig.java} |  2 +-
 .../server/conf/ServerConfigurationFactory.java|  8 +++
 .../conf/ServerConfigurationFactoryTest.java   |  2 +-
 .../apache/accumulo/test/start/KeywordStartIT.java |  4 ++--
 8 files changed, 30 insertions(+), 30 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/conf/ConfigSanityCheck.java 
b/core/src/main/java/org/apache/accumulo/core/conf/ConfigCheckUtil.java
similarity index 94%
rename from 
core/src/main/java/org/apache/accumulo/core/conf/ConfigSanityCheck.java
rename to core/src/main/java/org/apache/accumulo/core/conf/ConfigCheckUtil.java
index fada2ba..b0bc099 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/ConfigSanityCheck.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/ConfigCheckUtil.java
@@ -31,9 +31,9 @@ import com.google.common.base.Preconditions;
 /**
  * A utility class for validating {@link AccumuloConfiguration} instances.
  */
-public class ConfigSanityCheck {
+public class ConfigCheckUtil {
 
-  private static final Logger log = 
LoggerFactory.getLogger(ConfigSanityCheck.class);
+  private static final Logger log = 
LoggerFactory.getLogger(ConfigCheckUtil.class);
   private static final String PREFIX = "BAD CONFIG ";
 
   /**
@@ -44,7 +44,7 @@ public class ConfigSanityCheck {
*
* @param entries
*  iterable through configuration keys and values
-   * @throws SanityCheckException
+   * @throws ConfigCheckException
*   if a fatal configuration error is found
*/
   public static void validate(Iterable> entries) {
@@ -126,15 +126,15 @@ public class ConfigSanityCheck {
   }
 
   /**
-   * The exception thrown when {@link ConfigSanityCheck#validate(Iterable)} 
fails.
+   * The exception thrown when {@link ConfigCheckUtil#validate(Iterable)} 
fails.
*/
-  public static class SanityCheckException extends RuntimeException {
+  public static class ConfigCheckException extends RuntimeException {
 private static final long serialVersionUID = 1L;
 
 /**
  * Creates a new exception with the given message.
  */
-public SanityCheckException(String msg) {
+public ConfigCheckException(String msg) {
   super(msg);
 }
   }
@@ -143,7 +143,7 @@ public class ConfigSanityCheck {
 // ACCUMULO-3651 Level changed from fatal to error and FATAL added to 
message for slf4j
 // compatibility
 log.error("FATAL: {}", msg);
-throw new SanityCheckException(msg);
+throw new ConfigCheckException(msg);
   }
 
   /**
diff --git 
a/core/src/main/java/org/apache/accumulo/core/conf/SiteConfiguration.java 
b/core/src/main/java/org/apache/accumulo/core/conf/SiteConfiguration.java
index 35bda16..63c5c28 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/SiteConfiguration.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/SiteConfiguration.java
@@ -206,7 +206,7 @@ public class SiteConfiguration extends 
AccumuloConfiguration {
   private final Map config;
 
   private SiteConfiguration(Map config) {
-ConfigSanityCheck.validate(config.entrySet());
+ConfigCheckUtil.validate(config.entrySet());
 this.config = config;
   }
 
diff --git 
a/core/src/test/java/org/apache/accumulo/core/conf/ConfigSanityCheckTest.java 
b/core/src/test/java/org/apache/accumulo/core/conf/ConfigCheckUtilTest.java
similarity index 78%
rename from 
core/src/test/java/org/apache/accumulo/core/conf/ConfigSanityCheckTest.java
rename to 
core/src/test/java/org/apache/accumulo/core/conf/ConfigCheckUtilTest.java
index 30f99dd..9574e15 100644
--- 
a/core/src/test/java/org/apache/accumulo/core/conf/ConfigSanityCheckTest.java
+++ b/core/src/test/java/org/apache/accumulo/core/conf/ConfigCheckUtilTest.java
@@ -20,11 +20,11 @@ package org.apache.accumulo.core.conf;
 
 import java.util.Map;
 
-import org.apache.accumulo.core.conf.ConfigSanityCheck.SanityCheckException;
+import org.apache.accumulo.core.conf.ConfigCheckUtil.ConfigCheckException;
 import org.junit.Before;
 import org.junit.Test;
 
-public class Confi

[accumulo] branch main updated (bbc18e3 -> 0f0bcff)

2021-10-12 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from bbc18e3  Fix External Compaction status percentage in logs (#2310)
 add 0f0bcff  Log exception in CompactionManager (#2312)

No new revisions were added by this update.

Summary of changes:
 .../java/org/apache/accumulo/tserver/compactions/CompactionManager.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[accumulo] branch main updated (eba5059 -> 2c81f2f)

2021-10-07 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from eba5059  ConcurrentDeleteTableIT bug fix and improvements (#2304)
 add 2c81f2f  Fix javadoc

No new revisions were added by this update.

Summary of changes:
 .../apache/accumulo/monitor/rest/compactions/CompactionsResource.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[accumulo] branch main updated: Rework GC rest code in Monitor (#2297)

2021-10-05 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 8fb5d43  Rework GC rest code in Monitor (#2297)
8fb5d43 is described below

commit 8fb5d43ae19112497f9809e206b0b636ec2fde69
Author: Mike Miller 
AuthorDate: Tue Oct 5 11:01:59 2021 -0400

Rework GC rest code in Monitor (#2297)

* Make the GC rest code in Monitor return one list
* Create GarbageCollectorStats object to replace GarbageCollectorCycle
* Drop unused rest endpoints
* Make GC page use datatables to display data
---
 .../monitor/rest/gc/GarbageCollection.java |  67 ---
 .../monitor/rest/gc/GarbageCollectorResource.java  |  91 ---
 ...lectorCycle.java => GarbageCollectorStats.java} |  36 +++---
 .../monitor/rest/gc/GarbageCollectorStatus.java|  20 ++--
 .../accumulo/monitor/resources/js/functions.js |  10 --
 .../org/apache/accumulo/monitor/resources/js/gc.js | 130 +
 .../org/apache/accumulo/monitor/templates/gc.ftl   |  12 +-
 7 files changed, 86 insertions(+), 280 deletions(-)

diff --git 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/gc/GarbageCollection.java
 
b/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/gc/GarbageCollection.java
deleted file mode 100644
index 47dcd65..000
--- 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/gc/GarbageCollection.java
+++ /dev/null
@@ -1,67 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *   http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.accumulo.monitor.rest.gc;
-
-import org.apache.accumulo.core.gc.thrift.GcCycleStats;
-
-/**
- * GarbageCollection is responsible for creating the gc JSON object
- *
- * @since 2.0.0
- */
-public class GarbageCollection {
-
-  private static final GarbageCollection EMPTY = new GarbageCollection();
-
-  // Variable names become JSON key
-  public GarbageCollectorCycle lastCycle = new GarbageCollectorCycle();
-  public GarbageCollectorCycle currentCycle = new GarbageCollectorCycle();
-
-  public GarbageCollection() {}
-
-  /**
-   * Creates a new Garbage Collector JSON object
-   *
-   * @param last
-   *  last GC cycle
-   * @param current
-   *  current GC cycle
-   */
-  public GarbageCollection(GcCycleStats last, GcCycleStats current) {
-this.lastCycle = new GarbageCollectorCycle(last);
-this.currentCycle = new GarbageCollectorCycle(current);
-  }
-
-  /**
-   * Creates a new Garbage Collector JSON object
-   *
-   * @param last
-   *  last GC cycle
-   * @param current
-   *  current GC cycle
-   */
-  public GarbageCollection(GarbageCollectorCycle last, GarbageCollectorCycle 
current) {
-this.lastCycle = last;
-this.currentCycle = current;
-  }
-
-  public static GarbageCollection getEmpty() {
-return EMPTY;
-  }
-}
diff --git 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/gc/GarbageCollectorResource.java
 
b/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/gc/GarbageCollectorResource.java
index 50e0ec2..3309c6f 100644
--- 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/gc/GarbageCollectorResource.java
+++ 
b/server/monitor/src/main/java/org/apache/accumulo/monitor/rest/gc/GarbageCollectorResource.java
@@ -24,7 +24,6 @@ import jakarta.ws.rs.Path;
 import jakarta.ws.rs.Produces;
 import jakarta.ws.rs.core.MediaType;
 
-import org.apache.accumulo.core.gc.thrift.GCStatus;
 import org.apache.accumulo.monitor.Monitor;
 
 /**
@@ -48,94 +47,4 @@ public class GarbageCollectorResource {
   public GarbageCollectorStatus getStatus() {
 return new GarbageCollectorStatus(monitor.getGcStatus());
   }
-
-  /**
-   * Generates current and last file gc object
-   *
-   * @return file gc object
-   */
-  @Path("files")
-  @GET
-  public GarbageCollection getFileStatus() {
-GCStatus gcStatus = monitor.getGcStatus();
-if (gcStatus == null) {
-  return GarbageCollection.getEmpty();
-}
-return new GarbageCollection(gcStatus.last, gcStatus.current);
-  }
-
-  /**
- 

[accumulo] branch main updated: Improve fetching scans and compactions in Monitor (#2294)

2021-10-01 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new db22fdd  Improve fetching scans and compactions in Monitor (#2294)
db22fdd is described below

commit db22fdd03db44a44ae87c6ae8add23edc63c21e8
Author: Mike Miller 
AuthorDate: Fri Oct 1 11:17:21 2021 -0400

Improve fetching scans and compactions in Monitor (#2294)

* Drop fetchScans and fetchCompactions threads and just let the page
refresh get the data as needed. Created constant to prevent fetching
more than once a minute. Created time constant for age off
* Replace active scans table with datatables and add fetched column
* Add fetched column to Active compactions table
* Refactored fetch methods to catch thrift error and log.
* Made entry point methods getCompactions and getScans synchronized to
allow removal of synchronized code blocks.
---
 .../java/org/apache/accumulo/monitor/Monitor.java  | 78 ++
 .../monitor/rest/compactions/CompactionInfo.java   | 16 ++---
 .../rest/compactions/CompactionsResource.java  |  2 +-
 .../monitor/rest/scans/ScanInformation.java| 16 ++---
 .../accumulo/monitor/rest/scans/ScansResource.java |  4 +-
 .../accumulo/monitor/resources/js/compactions.js   | 11 ++-
 .../apache/accumulo/monitor/resources/js/scans.js  | 77 ++---
 .../accumulo/monitor/templates/compactions.ftl |  1 +
 .../apache/accumulo/monitor/templates/scans.ftl| 13 ++--
 9 files changed, 108 insertions(+), 110 deletions(-)

diff --git 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java 
b/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java
index 6217347..9e48e9d 100644
--- a/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java
+++ b/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java
@@ -474,28 +474,6 @@ public class Monitor extends AbstractServer implements 
HighlyAvailableService {
   }
 }).start();
 
-Threads.createThread("Scan scanner", () -> {
-  while (true) {
-try {
-  fetchScans();
-} catch (Exception e) {
-  log.warn("{}", e.getMessage(), e);
-}
-sleepUninterruptibly(5, TimeUnit.SECONDS);
-  }
-}).start();
-
-Threads.createThread("Compaction fetcher", () -> {
-  while (true) {
-try {
-  fetchCompactions();
-} catch (Exception e) {
-  log.warn("{}", e.getMessage(), e);
-}
-sleepUninterruptibly(5, TimeUnit.SECONDS);
-  }
-}).start();
-
 monitorInitialized.set(true);
   }
 
@@ -561,6 +539,7 @@ public class Monitor extends AbstractServer implements 
HighlyAvailableService {
 oldest = Math.max(oldest, scan.age);
   }
   this.oldestScan = oldest < 0 ? null : oldest;
+  // use clock time for date friendly display
   this.fetched = System.currentTimeMillis();
 }
   }
@@ -577,6 +556,7 @@ public class Monitor extends AbstractServer implements 
HighlyAvailableService {
 oldest = Math.max(oldest, a.age);
   }
   this.oldest = oldest < 0 ? null : oldest;
+  // use clock time for date friendly display
   this.fetched = System.currentTimeMillis();
 }
   }
@@ -584,56 +564,71 @@ public class Monitor extends AbstractServer implements 
HighlyAvailableService {
   private final Map allScans = new HashMap<>();
   private final Map allCompactions = new 
HashMap<>();
   private final RecentLogs recentLogs = new RecentLogs();
+  private long scansFetchedNanos = 0L;
+  private long compactsFetchedNanos = 0L;
+  private final long fetchTimeNanos = TimeUnit.MINUTES.toNanos(1);
+  private final long ageOffEntriesMillis = TimeUnit.MINUTES.toMillis(15);
 
-  public Map getScans() {
-synchronized (allScans) {
-  return new HashMap<>(allScans);
+  /**
+   * Fetch the active scans but only if fetchTimeNanos has elapsed.
+   */
+  public synchronized Map getScans() {
+if (System.nanoTime() - scansFetchedNanos > fetchTimeNanos) {
+  log.info("User initiated fetch of Active Scans");
+  fetchScans();
 }
+return Map.copyOf(allScans);
   }
 
-  public Map getCompactions() {
-synchronized (allCompactions) {
-  return new HashMap<>(allCompactions);
+  /**
+   * Fetch the active compactions but only if fetchTimeNanos has elapsed.
+   */
+  public synchronized Map getCompactions() {
+if (System.nanoTime() - compactsFetchedNanos > fetchTimeNanos) {
+  log.info("User initiated fetch of Active Compactions");
+  fetchCompactions();
 }
+return Map.copyOf(allCompactions);
   }
 
-  private void fetchScans() throws Exception {
+  private void fetchScans() {
 ServerContext context = g

[accumulo] branch main updated: Upgrade Thrift to 0.15.0 (#2273)

2021-09-29 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 04bf24f  Upgrade Thrift to 0.15.0 (#2273)
04bf24f is described below

commit 04bf24f777a3422d968240c66adf9167ffc4b2e0
Author: Mike Miller 
AuthorDate: Wed Sep 29 07:27:37 2021 -0400

Upgrade Thrift to 0.15.0 (#2273)
---
 .../core/clientImpl/ThriftTransportPool.java   | 26 +++
 .../iteratorsImpl/system/SystemIteratorUtil.java   |  5 ++-
 .../apache/accumulo/core/rpc/FilterTransport.java  | 16 +
 .../apache/accumulo/core/rpc/TBufferedSocket.java  |  3 +-
 .../accumulo/core/rpc/TTimeoutTransport.java   | 38 --
 .../org/apache/accumulo/core/rpc/ThriftUtil.java   | 12 +++
 .../core/rpc/UGIAssumingTransportFactory.java  |  9 -
 .../accumulo/core/util/ThriftMessageUtil.java  | 21 ++--
 core/src/main/scripts/generate-thrift.sh   |  2 +-
 .../accumulo/core/rpc/TTimeoutTransportTest.java   |  9 ++---
 .../accumulo/core/util/ThriftMessageUtilTest.java  |  2 +-
 pom.xml|  2 +-
 .../server/rpc/CustomNonBlockingServer.java|  6 ++--
 .../org/apache/accumulo/tracer/TraceFormatter.java |  8 ++---
 .../org/apache/accumulo/tracer/TraceServer.java| 16 +
 .../java/org/apache/accumulo/test/rpc/Mocket.java  | 24 +++---
 16 files changed, 143 insertions(+), 56 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/clientImpl/ThriftTransportPool.java
 
b/core/src/main/java/org/apache/accumulo/core/clientImpl/ThriftTransportPool.java
index df43f51..4be3e67 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/clientImpl/ThriftTransportPool.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/clientImpl/ThriftTransportPool.java
@@ -42,6 +42,7 @@ import org.apache.accumulo.core.singletons.SingletonService;
 import org.apache.accumulo.core.util.HostAndPort;
 import org.apache.accumulo.core.util.Pair;
 import org.apache.accumulo.core.util.threads.Threads;
+import org.apache.thrift.TConfiguration;
 import org.apache.thrift.transport.TTransport;
 import org.apache.thrift.transport.TTransportException;
 import org.slf4j.Logger;
@@ -654,6 +655,31 @@ public class ThriftTransportPool {
   }
 }
 
+@Override
+public TConfiguration getConfiguration() {
+  return wrappedTransport.getConfiguration();
+}
+
+@Override
+public void updateKnownMessageSize(long size) throws TTransportException {
+  try {
+ioCount++;
+wrappedTransport.updateKnownMessageSize(size);
+  } finally {
+ioCount++;
+  }
+}
+
+@Override
+public void checkReadBytesAvailable(long numBytes) throws 
TTransportException {
+  try {
+ioCount++;
+wrappedTransport.checkReadBytesAvailable(numBytes);
+  } finally {
+ioCount++;
+  }
+}
+
 public ThriftTransportKey getCacheKey() {
   return cacheKey;
 }
diff --git 
a/core/src/main/java/org/apache/accumulo/core/iteratorsImpl/system/SystemIteratorUtil.java
 
b/core/src/main/java/org/apache/accumulo/core/iteratorsImpl/system/SystemIteratorUtil.java
index 9e241c2..41a7acc 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/iteratorsImpl/system/SystemIteratorUtil.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/iteratorsImpl/system/SystemIteratorUtil.java
@@ -72,9 +72,8 @@ public class SystemIteratorUtil {
   }
 
   public static byte[] encodeIteratorSettings(IteratorConfig iterators) {
-TSerializer tser = new TSerializer(new TBinaryProtocol.Factory());
-
 try {
+  TSerializer tser = new TSerializer(new TBinaryProtocol.Factory());
   return tser.serialize(iterators);
 } catch (TException e) {
   throw new RuntimeException(e);
@@ -86,9 +85,9 @@ public class SystemIteratorUtil {
   }
 
   public static List decodeIteratorSettings(byte[] enc) {
-TDeserializer tdser = new TDeserializer(new TBinaryProtocol.Factory());
 IteratorConfig ic = new IteratorConfig();
 try {
+  TDeserializer tdser = new TDeserializer(new TBinaryProtocol.Factory());
   tdser.deserialize(ic, enc);
 } catch (TException e) {
   throw new RuntimeException(e);
diff --git 
a/core/src/main/java/org/apache/accumulo/core/rpc/FilterTransport.java 
b/core/src/main/java/org/apache/accumulo/core/rpc/FilterTransport.java
index f80dc94..3aaba4a 100644
--- a/core/src/main/java/org/apache/accumulo/core/rpc/FilterTransport.java
+++ b/core/src/main/java/org/apache/accumulo/core/rpc/FilterTransport.java
@@ -20,6 +20,7 @@ package org.apache.accumulo.core.rpc;
 
 import static java.util.Objects.requireNonNull;
 
+import org.apache.thrift.TConfiguration;
 import org.apache.thrift.transport.TTransport;
 import org.apache.thrift.transport.TTransportException;
 
@@ -103,4 +104,19

[accumulo-website] branch asf-site updated (60f43f3 -> d8a979e)

2021-09-27 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git.


from 60f43f3  Automatic Site Publish by Buildbot
 add d8a979e  Automatic Site Publish by Buildbot

No new revisions were added by this update.

Summary of changes:
 output/feed.xml  | 4 ++--
 output/people/index.html | 5 +
 2 files changed, 7 insertions(+), 2 deletions(-)


[accumulo-website] branch main updated: Update Contributor list (#293)

2021-09-27 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new 67a66cb  Update Contributor list (#293)
67a66cb is described below

commit 67a66cba3fc8f7def6d851e581d01b90b2f8ae00
Author: wx930910 
AuthorDate: Mon Sep 27 09:25:19 2021 -0400

Update Contributor list (#293)
---
 pages/people.md | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/pages/people.md b/pages/people.md
index 5b13eab..4b693b3 100644
--- a/pages/people.md
+++ b/pages/people.md
@@ -183,6 +183,7 @@ GitHub also has a [contributor list][github-contributors] 
based on commits.
 | Vincent Russell |
   |   |
 | Volth   |
   |   |
 | Wil Selwood | [Satellite Applications Catapult][SACAT]   
   | [GMT][GMT]/[BST][BST] |
+| Xiao Wang   | [Stevens Institute of Technology][SIT] 
   | [ET][ET]  |
 
 [github-contributors]: https://github.com/apache/accumulo/graphs/contributors
 [ANAVATION]: https://www.anavationllc.com
@@ -221,6 +222,7 @@ GitHub also has a [contributor list][github-contributors] 
based on commits.
 [PRAXIS]: https://www.praxiseng.com
 [SABRE]: https://www.sabre-eng.com
 [SACAT]: https://sa.catapult.org.uk
+[SIT]: https://www.stevens.edu
 [SPLYT]: https://www.splyt.com
 [SQRRL]: http://sqrrl.com
 [SRA]: https://sra.com


[accumulo] branch main updated: Add Active Compactions page to Monitor (#2283)

2021-09-23 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 9451dd0  Add Active Compactions page to Monitor (#2283)
9451dd0 is described below

commit 9451dd0dda6b4d9ebb5203e8314782233b28b75f
Author: Mike Miller 
AuthorDate: Thu Sep 23 10:38:43 2021 -0400

Add Active Compactions page to Monitor (#2283)
---
 .../java/org/apache/accumulo/monitor/Monitor.java  | 62 +++
 .../monitor/rest/compactions/CompactionInfo.java   | 53 +
 .../monitor/rest/compactions/Compactions.java  | 43 ++
 .../CompactionsResource.java}  | 28 -
 .../accumulo/monitor/rest/scans/ScansResource.java |  2 +-
 .../org/apache/accumulo/monitor/view/WebViews.java | 20 ++-
 .../accumulo/monitor/resources/js/compactions.js   | 69 ++
 .../accumulo/monitor/templates/compactions.ftl | 39 
 .../apache/accumulo/monitor/templates/navbar.ftl   |  1 +
 9 files changed, 300 insertions(+), 17 deletions(-)

diff --git 
a/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java 
b/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java
index 378923f..6217347 100644
--- a/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java
+++ b/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java
@@ -53,6 +53,7 @@ import 
org.apache.accumulo.core.manager.thrift.ManagerMonitorInfo;
 import org.apache.accumulo.core.master.thrift.TableInfo;
 import org.apache.accumulo.core.master.thrift.TabletServerStatus;
 import org.apache.accumulo.core.rpc.ThriftUtil;
+import org.apache.accumulo.core.tabletserver.thrift.ActiveCompaction;
 import org.apache.accumulo.core.tabletserver.thrift.ActiveScan;
 import org.apache.accumulo.core.tabletserver.thrift.TabletClientService.Client;
 import org.apache.accumulo.core.trace.TraceUtil;
@@ -484,6 +485,17 @@ public class Monitor extends AbstractServer implements 
HighlyAvailableService {
   }
 }).start();
 
+Threads.createThread("Compaction fetcher", () -> {
+  while (true) {
+try {
+  fetchCompactions();
+} catch (Exception e) {
+  log.warn("{}", e.getMessage(), e);
+}
+sleepUninterruptibly(5, TimeUnit.SECONDS);
+  }
+}).start();
+
 monitorInitialized.set(true);
   }
 
@@ -553,7 +565,24 @@ public class Monitor extends AbstractServer implements 
HighlyAvailableService {
 }
   }
 
+  public static class CompactionStats {
+public final long count;
+public final Long oldest;
+public final long fetched;
+
+CompactionStats(List active) {
+  this.count = active.size();
+  long oldest = -1;
+  for (ActiveCompaction a : active) {
+oldest = Math.max(oldest, a.age);
+  }
+  this.oldest = oldest < 0 ? null : oldest;
+  this.fetched = System.currentTimeMillis();
+}
+  }
+
   private final Map allScans = new HashMap<>();
+  private final Map allCompactions = new 
HashMap<>();
   private final RecentLogs recentLogs = new RecentLogs();
 
   public Map getScans() {
@@ -562,6 +591,12 @@ public class Monitor extends AbstractServer implements 
HighlyAvailableService {
 }
   }
 
+  public Map getCompactions() {
+synchronized (allCompactions) {
+  return new HashMap<>(allCompactions);
+}
+  }
+
   private void fetchScans() throws Exception {
 ServerContext context = getContext();
 for (String server : context.instanceOperations().getTabletServers()) {
@@ -589,6 +624,33 @@ public class Monitor extends AbstractServer implements 
HighlyAvailableService {
 }
   }
 
+  private void fetchCompactions() throws Exception {
+ServerContext context = getContext();
+for (String server : context.instanceOperations().getTabletServers()) {
+  final HostAndPort parsedServer = HostAndPort.fromString(server);
+  Client tserver = ThriftUtil.getTServerClient(parsedServer, context);
+  try {
+var compacts = tserver.getActiveCompactions(null, context.rpcCreds());
+synchronized (allCompactions) {
+  allCompactions.put(parsedServer, new CompactionStats(compacts));
+}
+  } catch (Exception ex) {
+log.debug("Failed to get active compactions from {}", server, ex);
+  } finally {
+ThriftUtil.returnClient(tserver);
+  }
+}
+// Age off old compaction information
+var entryIter = allCompactions.entrySet().iterator();
+long now = System.currentTimeMillis();
+while (entryIter.hasNext()) {
+  var entry = entryIter.next();
+  if (now - entry.getValue().fetched > 5 * 60 * 1000) {
+entryIter.remove();
+  }
+}
+  }
+
   /**
* Get the monitor lock in ZooKeeper
*/
diff --git 
a/server/monitor/s

[accumulo] branch main updated: Rename CompactionTask to InternalJob (#2270)

2021-09-16 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new f73af60  Rename CompactionTask to InternalJob (#2270)
f73af60 is described below

commit f73af60b86e452ef9cd429b56ccc53e93c602eda
Author: Mike Miller 
AuthorDate: Thu Sep 16 11:50:25 2021 -0400

Rename CompactionTask to InternalJob (#2270)

* Rename CompactionTask to InternalJob to be consistent with ExternalJob
* Rename variables to match InternalJob type
* Fix some wording in comments
* Add javadoc comments
---
 .../tserver/compactions/CompactionExecutor.java|  6 +++
 .../compactions/ExternalCompactionExecutor.java| 35 ++--
 .../compactions/InternalCompactionExecutor.java| 63 +++---
 .../accumulo/tserver/compactions/SubmittedJob.java |  3 ++
 4 files changed, 61 insertions(+), 46 deletions(-)

diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionExecutor.java
 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionExecutor.java
index 85581a8..0e6e823 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionExecutor.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionExecutor.java
@@ -24,6 +24,12 @@ import org.apache.accumulo.core.dataImpl.KeyExtent;
 import org.apache.accumulo.core.spi.compaction.CompactionJob;
 import org.apache.accumulo.core.spi.compaction.CompactionServiceId;
 
+/**
+ * A non-pluggable component that executes compactions using multiple threads 
and has a priority
+ * queue. There are two types: Internal and External. The {@link 
InternalCompactionExecutor} runs
+ * within the Accumulo tserver process. The {@link ExternalCompactionExecutor} 
runs compactions
+ * outside the tserver.
+ */
 public interface CompactionExecutor {
 
   SubmittedJob submit(CompactionServiceId csid, CompactionJob job, Compactable 
compactable,
diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/ExternalCompactionExecutor.java
 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/ExternalCompactionExecutor.java
index 7d2dcd5..7acf20f 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/ExternalCompactionExecutor.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/ExternalCompactionExecutor.java
@@ -41,13 +41,16 @@ import 
org.apache.accumulo.tserver.compactions.SubmittedJob.Status;
 
 import com.google.common.base.Preconditions;
 
+/**
+ * Runs compactions outside the tserver, typically by a process external to 
Accumulo.
+ */
 public class ExternalCompactionExecutor implements CompactionExecutor {
 
-  // This exist to provide an accurate count of queued compactions for 
metrics. The PriorityQueue is
+  // This set provides an accurate count of queued compactions for metrics. 
The PriorityQueue is
   // not used because its size may be off due to it containing cancelled 
compactions. The collection
   // below should not contain cancelled compactions. A concurrent set was not 
used because those do
   // not have constant time size operations.
-  private Set queuedTask = Collections.synchronizedSet(new 
HashSet<>());
+  private final Set queuedJob = Collections.synchronizedSet(new 
HashSet<>());
 
   private class ExternalJob extends SubmittedJob {
 private final AtomicReference status = new 
AtomicReference<>(Status.QUEUED);
@@ -61,7 +64,7 @@ public class ExternalCompactionExecutor implements 
CompactionExecutor {
   super(job);
   this.compactable = compactable;
   this.csid = csid;
-  queuedTask.add(this);
+  queuedJob.add(this);
   this.timeCreated = System.currentTimeMillis();
 }
 
@@ -83,11 +86,11 @@ public class ExternalCompactionExecutor implements 
CompactionExecutor {
   if (expectedStatus == Status.QUEUED) {
 canceled = status.compareAndSet(expectedStatus, Status.CANCELED);
 if (canceled) {
-  queuedTask.remove(this);
+  queuedJob.remove(this);
 }
 
 if (canceled && cancelCount.incrementAndGet() % 1024 == 0) {
-  // Occasionally clean the queue of canceled tasks that have hung 
around because of their
+  // Occasionally clean the queue of canceled jobs that have hung 
around because of their
   // low priority. This runs periodically, instead of every time 
something is canceled, to
   // avoid hurting performance.
   queue.removeIf(ej -> ej.getStatus() == Status.CANCELED);
@@ -106,8 +109,8 @@ public class ExternalCompactionExecutor implements 
CompactionExecutor {
 }
   }
 
-  private PriorityBlockingQueue queue;
-  private CompactionExecutorId ceid;
+  private final PriorityBlocking

[accumulo] branch main updated: Upgrade Thrift to 0.13.0 (#1780)

2021-09-16 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 9e101e1  Upgrade Thrift to 0.13.0 (#1780)
9e101e1 is described below

commit 9e101e189e20943af930144878d6b65722f42a8c
Author: BukrosSzabolcs 
AuthorDate: Thu Sep 16 15:54:22 2021 +0200

Upgrade Thrift to 0.13.0 (#1780)
---
 .../accumulo/core/util/ThriftMessageUtil.java  | 28 --
 pom.xml|  2 +-
 .../apache/accumulo/server/rpc/TimedProcessor.java |  4 ++--
 .../accumulo/server/rpc/UGIAssumingProcessor.java  |  9 +++
 4 files changed, 23 insertions(+), 20 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/util/ThriftMessageUtil.java 
b/core/src/main/java/org/apache/accumulo/core/util/ThriftMessageUtil.java
index f7372cf..e993695 100644
--- a/core/src/main/java/org/apache/accumulo/core/util/ThriftMessageUtil.java
+++ b/core/src/main/java/org/apache/accumulo/core/util/ThriftMessageUtil.java
@@ -27,7 +27,7 @@ import org.apache.thrift.TBase;
 import org.apache.thrift.TException;
 import org.apache.thrift.protocol.TCompactProtocol;
 import org.apache.thrift.protocol.TProtocol;
-import org.apache.thrift.transport.AutoExpandingBufferWriteTransport;
+import org.apache.thrift.transport.TMemoryBuffer;
 import org.apache.thrift.transport.TMemoryInputTransport;
 
 /**
@@ -36,18 +36,21 @@ import org.apache.thrift.transport.TMemoryInputTransport;
  */
 public class ThriftMessageUtil {
 
-  private final AutoExpandingBufferWriteTransport transport;
-  private final TProtocol protocol;
+  private final int initialCapacity;
+
+  private final TMemoryInputTransport inputTransport;
+  private final TCompactProtocol inputProtocol;
 
   public ThriftMessageUtil() {
-this(64, 1.5);
+this(64);
   }
 
-  public ThriftMessageUtil(int initialCapacity, double growthCoefficient) {
+  public ThriftMessageUtil(int initialCapacity) {
 // TODO does this make sense? better to push this down to the serialize 
method (accept the
 // transport as an argument)?
-this.transport = new AutoExpandingBufferWriteTransport(initialCapacity, 
growthCoefficient);
-this.protocol = new TCompactProtocol(transport);
+this.initialCapacity = initialCapacity;
+this.inputTransport = new TMemoryInputTransport();
+this.inputProtocol = new TCompactProtocol(inputTransport);
   }
 
   /**
@@ -61,14 +64,14 @@ public class ThriftMessageUtil {
*/
   public ByteBuffer serialize(TBase msg) throws IOException {
 requireNonNull(msg);
-transport.reset();
+TMemoryBuffer transport = new TMemoryBuffer(initialCapacity);
+TProtocol protocol = new TCompactProtocol(transport);
 try {
   msg.write(protocol);
-  // We should flush(), but we know its a noop
 } catch (TException e) {
   throw new IOException(e);
 }
-return ByteBuffer.wrap(transport.getBuf().array(), 0, transport.getPos());
+return ByteBuffer.wrap(transport.getArray(), 0, transport.length());
   }
 
   /**
@@ -94,10 +97,9 @@ public class ThriftMessageUtil {
   public > T deserialize(byte[] serialized, int offset, 
int length, T instance)
   throws IOException {
 requireNonNull(instance);
-TCompactProtocol proto =
-new TCompactProtocol(new TMemoryInputTransport(serialized, offset, 
length));
+inputTransport.reset(serialized, offset, length);
 try {
-  instance.read(proto);
+  instance.read(inputProtocol);
 } catch (TException e) {
   throw new IOException(e);
 }
diff --git a/pom.xml b/pom.xml
index 4cec469..d602b26 100644
--- a/pom.xml
+++ b/pom.xml
@@ -154,7 +154,7 @@
 
 3.0.0-M4
 
-0.12.0
+0.13.0
 -Xmx1G
 
 3.5.9
diff --git 
a/server/base/src/main/java/org/apache/accumulo/server/rpc/TimedProcessor.java 
b/server/base/src/main/java/org/apache/accumulo/server/rpc/TimedProcessor.java
index eee13fd..453d9b8 100644
--- 
a/server/base/src/main/java/org/apache/accumulo/server/rpc/TimedProcessor.java
+++ 
b/server/base/src/main/java/org/apache/accumulo/server/rpc/TimedProcessor.java
@@ -55,12 +55,12 @@ public class TimedProcessor implements TProcessor {
   }
 
   @Override
-  public boolean process(TProtocol in, TProtocol out) throws TException {
+  public void process(TProtocol in, TProtocol out) throws TException {
 long now = 0;
 now = System.currentTimeMillis();
 thriftMetrics.addIdle(now - idleStart);
 try {
-  return other.process(in, out);
+  other.process(in, out);
 } finally {
   idleStart = System.currentTimeMillis();
   thriftMetrics.addExecute(idleStart - now);
diff --git 
a/server/base/src/main/java/org/apache/accumulo/server/rpc/UGIAssumingProcessor.java
 
b/server/base/src/main/java/org/apache/accumulo/server/rpc/UGIAssumingProcessor.java
index 2f46082..fa84f59 100

[accumulo] branch main updated: Clarify compact method (#2268)

2021-09-14 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 0d24673  Clarify compact method (#2268)
0d24673 is described below

commit 0d246732c5fe5af67aa5a2be129f659106370685
Author: Mike Miller 
AuthorDate: Tue Sep 14 07:52:58 2021 -0400

Clarify compact method (#2268)

* Make compact method in CompactableUtils just call compact and return
the majc stats, removing the need to mutate the object across method calls.
* Pass the CompactionInfo object as a param to reduce the number of params
to the compact method.
* Pull logic from compact method and put into new bringOnline method, 
further
reducing the number of params to the compact method. The bringOnline method
returns the completed StoredTabletFile
* Replace anonymous CompactionEnv classes with 2 new classes: MinCEnv & 
MajCEnv
* Rename CompactionEnvironment to ExtCEnv to differentiate from the
other types of CompactionEnv implementations
* Add javadoc comments describing methods
---
 .../org/apache/accumulo/compactor/Compactor.java   |   2 +-
 .../{CompactionEnvironment.java => ExtCEnv.java}   |   4 +-
 .../accumulo/tserver/tablet/CompactableImpl.java   |  32 ---
 .../accumulo/tserver/tablet/CompactableUtils.java  | 105 +
 .../apache/accumulo/tserver/tablet/MajCEnv.java}   |  68 +
 .../apache/accumulo/tserver/tablet/MinCEnv.java}   |  68 +
 .../accumulo/tserver/tablet/MinorCompactor.java|  57 +--
 .../apache/accumulo/test/ExternalCompactionIT.java |   2 +-
 8 files changed, 95 insertions(+), 243 deletions(-)

diff --git 
a/server/compactor/src/main/java/org/apache/accumulo/compactor/Compactor.java 
b/server/compactor/src/main/java/org/apache/accumulo/compactor/Compactor.java
index c72e286..305de7f 100644
--- 
a/server/compactor/src/main/java/org/apache/accumulo/compactor/Compactor.java
+++ 
b/server/compactor/src/main/java/org/apache/accumulo/compactor/Compactor.java
@@ -542,7 +542,7 @@ public class Compactor extends AbstractServer implements 
CompactorService.Iface
   job.getIteratorSettings().getIterators()
   .forEach(tis -> 
iters.add(SystemIteratorUtil.toIteratorSetting(tis)));
 
-  CompactionEnvironment cenv = new CompactionEnvironment(JOB_HOLDER, 
queueName);
+  ExtCEnv cenv = new ExtCEnv(JOB_HOLDER, queueName);
   FileCompactor compactor = new FileCompactor(getContext(), extent, 
files, outputFile,
   job.isPropagateDeletes(), cenv, iters, tConfig);
 
diff --git 
a/server/compactor/src/main/java/org/apache/accumulo/compactor/CompactionEnvironment.java
 b/server/compactor/src/main/java/org/apache/accumulo/compactor/ExtCEnv.java
similarity index 96%
copy from 
server/compactor/src/main/java/org/apache/accumulo/compactor/CompactionEnvironment.java
copy to 
server/compactor/src/main/java/org/apache/accumulo/compactor/ExtCEnv.java
index 6f99bb0..5c6062e 100644
--- 
a/server/compactor/src/main/java/org/apache/accumulo/compactor/CompactionEnvironment.java
+++ b/server/compactor/src/main/java/org/apache/accumulo/compactor/ExtCEnv.java
@@ -36,7 +36,7 @@ import 
org.apache.accumulo.server.iterators.TabletIteratorEnvironment;
 
 import com.google.common.annotations.VisibleForTesting;
 
-public class CompactionEnvironment implements CompactionEnv {
+public class ExtCEnv implements CompactionEnv {
 
   private final CompactionJobHolder jobHolder;
   private TExternalCompactionJob job;
@@ -58,7 +58,7 @@ public class CompactionEnvironment implements CompactionEnv {
 }
   }
 
-  CompactionEnvironment(CompactionJobHolder jobHolder, String queueName) {
+  ExtCEnv(CompactionJobHolder jobHolder, String queueName) {
 this.jobHolder = jobHolder;
 this.job = jobHolder.getJob();
 this.queueName = queueName;
diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java
 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java
index e64d53a..4345999 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java
@@ -1031,7 +1031,7 @@ public class CompactableImpl implements Compactable {
 }
   }
 
-  private static class CompactionInfo {
+  static class CompactionInfo {
 Set jobFiles;
 Long checkCompactionId = null;
 boolean propagateDeletes = true;
@@ -1147,31 +1147,37 @@ public class CompactableImpl implements Compactable {
   return;
 
 var cInfo = ocInfo.get();
-StoredTabletFile metaFile = null;
+StoredTabletFile newFile = null;
 long startTime = System.currentTimeMillis();
-// create an empty stats object to be populated by 
CompactableUti

[accumulo] branch main updated: Split compact method into two more specific methods (#2263)

2021-09-10 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new ff67c09  Split compact method into two more specific methods (#2263)
ff67c09 is described below

commit ff67c0948507db573a313da5a71f690a6a92fff9
Author: Mike Miller 
AuthorDate: Fri Sep 10 06:42:24 2021 -0400

Split compact method into two more specific methods (#2263)

* Split compact method in CompactionService into to 2 methods:
getCompactionPlan() and submitCompactionJob()
* Rename compact() to submitCompaction()
* Add javadoc comments to various methods
* Add final to various private members
* Drop redundant enum check and boolean assignment

Co-authored-by: Keith Turner 
---
 .../client/admin/compaction/CompactableFile.java   |  2 ++
 .../core/spi/compaction/CompactionPlan.java|  3 ++
 .../core/util/compaction/CompactionPlanImpl.java   |  8 ++---
 .../tserver/compactions/CompactionManager.java | 11 ---
 .../tserver/compactions/CompactionService.java | 36 ++
 .../accumulo/tserver/tablet/CompactableImpl.java   |  7 ++---
 6 files changed, 42 insertions(+), 25 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/CompactableFile.java
 
b/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/CompactableFile.java
index 3e0a21e..8be4985 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/CompactableFile.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/CompactableFile.java
@@ -23,6 +23,8 @@ import java.net.URI;
 import org.apache.accumulo.core.metadata.CompactableFileImpl;
 
 /**
+ * A single file ready to compact, that will come in a set of possible 
candidates.
+ *
  * @since 2.1.0
  */
 public interface CompactableFile {
diff --git 
a/core/src/main/java/org/apache/accumulo/core/spi/compaction/CompactionPlan.java
 
b/core/src/main/java/org/apache/accumulo/core/spi/compaction/CompactionPlan.java
index 592f5b4..410dccd 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/spi/compaction/CompactionPlan.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/spi/compaction/CompactionPlan.java
@@ -55,5 +55,8 @@ public interface CompactionPlan {
 CompactionPlan build();
   }
 
+  /**
+   * Return the set of jobs this plan will submit for compaction.
+   */
   Collection getJobs();
 }
diff --git 
a/core/src/main/java/org/apache/accumulo/core/util/compaction/CompactionPlanImpl.java
 
b/core/src/main/java/org/apache/accumulo/core/util/compaction/CompactionPlanImpl.java
index 111c386..e204cb8 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/util/compaction/CompactionPlanImpl.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/util/compaction/CompactionPlanImpl.java
@@ -54,11 +54,11 @@ public class CompactionPlanImpl implements CompactionPlan {
 
   public static class BuilderImpl implements CompactionPlan.Builder {
 
-private CompactionKind kind;
+private final CompactionKind kind;
 private ArrayList jobs = new ArrayList<>();
-private Set allFiles;
-private Set seenFiles = new HashSet<>();
-private Set candidates;
+private final Set allFiles;
+private final Set seenFiles = new HashSet<>();
+private final Set candidates;
 
 public BuilderImpl(CompactionKind kind, Set allFiles,
 Set candidates) {
diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionManager.java
 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionManager.java
index 5e65f7a..f7c5b2a 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionManager.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionManager.java
@@ -254,7 +254,7 @@ public class CompactionManager {
   new HashSet<>(runningExternalCompactions.keySet());
   for (Compactable compactable : compactables) {
 last = compactable;
-compact(compactable);
+submitCompaction(compactable);
 // remove anything from snapshot that tablets know are running
 compactable.getExternalCompactionIds(runningEcids::remove);
   }
@@ -267,7 +267,7 @@ public class CompactionManager {
   compactablesToCheck.poll(maxTimeBetweenChecks - passed, 
TimeUnit.MILLISECONDS);
   if (compactable != null) {
 last = compactable;
-compact(compactable);
+submitCompaction(compactable);
   }
 }
 
@@ -290,7 +290,10 @@ public class CompactionManager {
 }
   }
 
-  private void compact(Compactable compactable) {
+  /**
+   * Get each configured service for the compactable tablet and submit for 
comp

[accumulo] branch main updated (e6ec4cd -> e08cf60)

2021-09-09 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from e6ec4cd  Fix check in verifyExternalCompactions (#2267)
 add e08cf60  Fix spelling in CompactableImpl

No new revisions were added by this update.

Summary of changes:
 .../main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


[accumulo] branch main updated (7715766 -> e6ec4cd)

2021-09-09 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from 7715766  Replacing MockReateLimiter with mocking object to improve 
test design (#2254)
 add e6ec4cd  Fix check in verifyExternalCompactions (#2267)

No new revisions were added by this update.

Summary of changes:
 .../apache/accumulo/tserver/tablet/CompactableImpl.java   | 15 ---
 1 file changed, 8 insertions(+), 7 deletions(-)


[accumulo] branch main updated: Replacing MockReateLimiter with mocking object to improve test design (#2254)

2021-09-08 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 7715766  Replacing MockReateLimiter with mocking object to improve 
test design (#2254)
7715766 is described below

commit 77157661af4e6de790e4b0cd34debd02f69bdce2
Author: wx930910 
AuthorDate: Wed Sep 8 10:40:58 2021 -0400

Replacing MockReateLimiter with mocking object to improve test design 
(#2254)
---
 .../core/file/streams/MockRateLimiter.java | 41 --
 .../file/streams/RateLimitedInputStreamTest.java   | 29 +--
 .../file/streams/RateLimitedOutputStreamTest.java  | 17 +++--
 3 files changed, 33 insertions(+), 54 deletions(-)

diff --git 
a/core/src/test/java/org/apache/accumulo/core/file/streams/MockRateLimiter.java 
b/core/src/test/java/org/apache/accumulo/core/file/streams/MockRateLimiter.java
deleted file mode 100644
index 905224b..000
--- 
a/core/src/test/java/org/apache/accumulo/core/file/streams/MockRateLimiter.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *   http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.accumulo.core.file.streams;
-
-import java.util.concurrent.atomic.AtomicLong;
-
-import org.apache.accumulo.core.util.ratelimit.RateLimiter;
-
-public class MockRateLimiter implements RateLimiter {
-  private final AtomicLong permitsAcquired = new AtomicLong();
-
-  @Override
-  public long getRate() {
-return 0;
-  }
-
-  @Override
-  public void acquire(long permits) {
-permitsAcquired.addAndGet(permits);
-  }
-
-  public long getPermitsAcquired() {
-return permitsAcquired.get();
-  }
-}
diff --git 
a/core/src/test/java/org/apache/accumulo/core/file/streams/RateLimitedInputStreamTest.java
 
b/core/src/test/java/org/apache/accumulo/core/file/streams/RateLimitedInputStreamTest.java
index d23ee31..18bf228 100644
--- 
a/core/src/test/java/org/apache/accumulo/core/file/streams/RateLimitedInputStreamTest.java
+++ 
b/core/src/test/java/org/apache/accumulo/core/file/streams/RateLimitedInputStreamTest.java
@@ -23,8 +23,11 @@ import static org.junit.Assert.assertEquals;
 import java.io.InputStream;
 import java.security.SecureRandom;
 import java.util.Random;
+import java.util.concurrent.atomic.AtomicLong;
 
+import org.apache.accumulo.core.util.ratelimit.RateLimiter;
 import org.apache.hadoop.fs.Seekable;
+import org.easymock.EasyMock;
 import org.junit.Test;
 
 public class RateLimitedInputStreamTest {
@@ -32,7 +35,17 @@ public class RateLimitedInputStreamTest {
   @Test
   public void permitsAreProperlyAcquired() throws Exception {
 Random randGen = new SecureRandom();
-MockRateLimiter rateLimiter = new MockRateLimiter();
+// Create variables for tracking behaviors of mock object
+AtomicLong rateLimiterPermitsAcquired = new AtomicLong();
+// Construct mock object
+RateLimiter rateLimiter = EasyMock.niceMock(RateLimiter.class);
+// Stub Mock Method
+rateLimiter.acquire(EasyMock.anyLong());
+EasyMock.expectLastCall()
+.andAnswer(() -> 
rateLimiterPermitsAcquired.addAndGet(EasyMock.getCurrentArgument(0)))
+.anyTimes();
+EasyMock.replay(rateLimiter);
+
 long bytesRetrieved = 0;
 try (InputStream is = new RateLimitedInputStream(new RandomInputStream(), 
rateLimiter)) {
   for (int i = 0; i < 100; ++i) {
@@ -42,7 +55,7 @@ public class RateLimitedInputStreamTest {
 bytesRetrieved += count;
   }
 }
-assertEquals(bytesRetrieved, rateLimiter.getPermitsAcquired());
+assertEquals(bytesRetrieved, rateLimiterPermitsAcquired.get());
   }
 
   private static class RandomInputStream extends InputStream implements 
Seekable {
@@ -55,23 +68,17 @@ public class RateLimitedInputStreamTest {
 
 @Override
 public void seek(long pos) {
-  throw new UnsupportedOperationException("Not supported yet."); // To 
change body of generated
- // 
methods, choose Tools |
-

[accumulo] branch main updated: Add check for deleted Table to Tablet compactAll (#2262)

2021-09-07 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 68ca8c1  Add check for deleted Table to Tablet compactAll (#2262)
68ca8c1 is described below

commit 68ca8c1c584d7987602a8c1dd901c45d0f9cd7ad
Author: Mike Miller 
AuthorDate: Tue Sep 7 09:47:07 2021 -0400

Add check for deleted Table to Tablet compactAll (#2262)

* Also drop unnecessary boolean and add comment
---
 .../main/java/org/apache/accumulo/tserver/tablet/Tablet.java | 12 +++-
 1 file changed, 3 insertions(+), 9 deletions(-)

diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
index 360ac77..709ed43 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
@@ -2096,8 +2096,6 @@ public class Tablet {
 
   public void compactAll(long compactionId, CompactionConfig compactionConfig) 
{
 
-boolean shouldInitiate = false;
-
 synchronized (this) {
   if (lastCompactID >= compactionId) {
 return;
@@ -2116,17 +2114,13 @@ public class Tablet {
 }
   }
 
-  if (isClosing() || isClosed()) {
+  if (isClosing() || isClosed() || isBeingDeleted()) {
 return;
   }
-
-  shouldInitiate = true;
-
 }
 
-if (shouldInitiate) {
-  compactable.initiateUserCompaction(compactionId, compactionConfig);
-}
+// passed all verification checks so initiate compaction
+compactable.initiateUserCompaction(compactionId, compactionConfig);
   }
 
   public TableConfiguration getTableConfiguration() {


[accumulo] branch main updated (647b03a -> 506f3c5)

2021-09-03 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from 647b03a  Reword some comments to clarify code (#2260)
 add 506f3c5  Add 3 ITs to Sunny Tests. Closes #2147 (#2261)

No new revisions were added by this update.

Summary of changes:
 test/src/main/java/org/apache/accumulo/test/functional/BulkIT.java | 3 +++
 .../main/java/org/apache/accumulo/test/functional/ScanIteratorIT.java  | 3 +++
 .../main/java/org/apache/accumulo/test/functional/WALSunnyDayIT.java   | 3 +++
 3 files changed, 9 insertions(+)


[accumulo] branch main updated: Reword some comments to clarify code (#2260)

2021-09-02 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 647b03a  Reword some comments to clarify code (#2260)
647b03a is described below

commit 647b03a231c43d84e1a949668bb3c3cc8fa1b917
Author: Mike Miller 
AuthorDate: Thu Sep 2 14:58:58 2021 -0400

Reword some comments to clarify code (#2260)

* Also rename some methods in CompactableImpl
* Drop confusing sentence in comments
* Small improvements to comments

Co-authored-by: Dom G <47725857+domgargu...@users.noreply.github.com>
---
 .../accumulo/tserver/tablet/CompactableImpl.java   | 32 ++
 1 file changed, 15 insertions(+), 17 deletions(-)

diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java
 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java
index 47b1861..1e60d1c 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java
@@ -171,7 +171,7 @@ public class CompactableImpl implements Compactable {
*
* 
* This class does no synchronization of its own and relies on 
CompactableImpl to do all needed
-   * synchronization. CompactableImpl must makes changes to files and other 
state like running jobs
+   * synchronization. CompactableImpl must make changes to files and other 
state like running jobs
* in a mutually exclusive manner, so synchronization at this level is 
unnecessary.
*
*/
@@ -181,10 +181,9 @@ public class CompactableImpl implements Compactable {
 private CompactionKind selectKind = null;
 
 // Tracks if when a set of files was selected, if at that time the set was 
all of the tablets
-// files. Because a set of selected files can be compacted over one or 
more compactions, its
+// files. Because a set of selected files can be compacted over one or 
more compactions, it's
 // important to track this in order to know if the last compaction is a 
full compaction and
-// should
-// not propagate deletes.
+// should not propagate deletes.
 private boolean initiallySelectedAll = false;
 private Set selectedFiles = new HashSet<>();
 
@@ -362,7 +361,7 @@ public class CompactableImpl implements Compactable {
 log.trace("Ignoring because compacting not a subset {}", getExtent());
 
 // A compaction finished, so things are out of date. This can happen 
because CompactableImpl
-// and Tablet have separate locks, its ok.
+// and Tablet have separate locks, it's ok.
 return Set.of();
   }
 
@@ -561,9 +560,9 @@ public class CompactableImpl implements Compactable {
 
 Map extCompactionsToRemove = new HashMap<>();
 
-var extSelInfo = initializeSelection(extCompactions, tablet, 
extCompactionsToRemove);
+var extSelInfo = initExternalSelection(extCompactions, tablet, 
extCompactionsToRemove);
 
-sanityCheckExternalCompactions(extCompactions, dataFileSizes.keySet(), 
extCompactionsToRemove);
+verifyExternalCompactions(extCompactions, dataFileSizes.keySet(), 
extCompactionsToRemove);
 
 extCompactionsToRemove.forEach((ecid, reason) -> {
   log.warn("Removing external compaction {} for {} because {} meta: {}", 
ecid,
@@ -615,7 +614,7 @@ public class CompactableImpl implements Compactable {
 this.fileMgr = new FileManager(tablet.getExtent(), extCompactingFiles, 
extSelInfo);
   }
 
-  private void sanityCheckExternalCompactions(
+  private void verifyExternalCompactions(
   Map extCompactions,
   Set tabletFiles, Map 
extCompactionsToRemove) {
 
@@ -635,7 +634,6 @@ public class CompactableImpl implements Compactable {
 extCompactionsToRemove.putIfAbsent(ecid, "Some external compaction 
files overlap");
   });
 }
-
   }
 
   private synchronized boolean addJob(CompactionJob job) {
@@ -760,14 +758,14 @@ public class CompactableImpl implements Compactable {
   }
 
   /**
-   * For user compactions a set of files is selected. Then those files 
compacted by one or more
+   * For user compactions a set of files is selected. Those files then get 
compacted by one or more
* compactions until the set is empty. This method attempts to reconstruct 
the selected set of
-   * files when a tablet is loaded that has an external user compaction. Doing 
this avoid repeating
-   * work and also when a user compaction completes it does checks against the 
selected set. So the
-   * selected set must be initialized. Since the data is coming from persisted 
storage, lots of
-   * sanity checks are done in this method rather than assuming the persisted 
data is just correct.
+   * files when a tablet is loaded with an 

[accumulo] branch main updated: Create inner class in CompactionService (#2251)

2021-08-31 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 705dfe0  Create inner class in CompactionService (#2251)
705dfe0 is described below

commit 705dfe07f3af0bfd0dc6bb0f62a01fb9a0164f37
Author: Mike Miller 
AuthorDate: Tue Aug 31 14:35:30 2021 -0400

Create inner class in CompactionService (#2251)

* Improve readability of CompactionService by creating inner class
for the implementation of PlanningParameters
---
 .../tserver/compactions/CompactionService.java | 108 +++--
 1 file changed, 59 insertions(+), 49 deletions(-)

diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java
 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java
index ca30011..ef19cbf 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java
@@ -243,68 +243,78 @@ public class CompactionService {
 }
   }
 
-  private void planCompaction(CompactionKind kind, Compactable compactable,
-  Consumer completionCallback) {
-var files = compactable.getFiles(myId, kind);
+  private class CpPlanParams implements PlanningParameters {
+private final CompactionKind kind;
+private final Compactable comp;
+private final Compactable.Files files;
+
+public CpPlanParams(CompactionKind kind, Compactable comp, 
Compactable.Files files) {
+  this.kind = kind;
+  this.comp = comp;
+  this.files = files;
+}
 
-if (files.isEmpty() || files.get().candidates.isEmpty()) {
-  log.trace("Compactable returned no files {} {} {}", 
compactable.getExtent(), kind, files);
-  return;
+private final ServiceEnvironment senv = new 
ServiceEnvironmentImpl(context);
+
+@Override
+public TableId getTableId() {
+  return comp.getTableId();
 }
 
-PlanningParameters params = new PlanningParameters() {
+@Override
+public ServiceEnvironment getServiceEnvironment() {
+  return senv;
+}
 
-  private final ServiceEnvironment senv = new 
ServiceEnvironmentImpl(context);
+@Override
+public double getRatio() {
+  return comp.getCompactionRatio();
+}
 
-  @Override
-  public TableId getTableId() {
-return compactable.getTableId();
-  }
+@Override
+public CompactionKind getKind() {
+  return kind;
+}
 
-  @Override
-  public ServiceEnvironment getServiceEnvironment() {
-return senv;
-  }
+@Override
+public Collection getRunningCompactions() {
+  return files.compacting;
+}
 
-  @Override
-  public double getRatio() {
-return compactable.getCompactionRatio();
-  }
+@Override
+public Collection getCandidates() {
+  return files.candidates;
+}
 
-  @Override
-  public CompactionKind getKind() {
-return kind;
-  }
+@Override
+public Collection getAll() {
+  return files.allFiles;
+}
 
-  @Override
-  public Collection getRunningCompactions() {
-return files.get().compacting;
-  }
+@Override
+public Map getExecutionHints() {
+  if (kind == CompactionKind.USER)
+return files.executionHints;
+  else
+return Map.of();
+}
 
-  @Override
-  public Collection getCandidates() {
-return files.get().candidates;
-  }
+@Override
+public CompactionPlan.Builder createPlanBuilder() {
+  return new CompactionPlanImpl.BuilderImpl(kind, files.allFiles, 
files.candidates);
+}
+  }
 
-  @Override
-  public Collection getAll() {
-return files.get().allFiles;
-  }
+  private void planCompaction(CompactionKind kind, Compactable compactable,
+  Consumer completionCallback) {
+var files = compactable.getFiles(myId, kind);
 
-  @Override
-  public Map getExecutionHints() {
-if (kind == CompactionKind.USER)
-  return files.get().executionHints;
-else
-  return Map.of();
-  }
+if (files.isEmpty() || files.get().candidates.isEmpty()) {
+  log.trace("Compactable returned no files {} {} {}", 
compactable.getExtent(), kind, files);
+  return;
+}
 
-  @Override
-  public CompactionPlan.Builder createPlanBuilder() {
-return new CompactionPlanImpl.BuilderImpl(kind, files.get().allFiles,
-files.get().candidates);
-  }
-};
+PlanningParameters params = new CpPlanParams(kind, compactable, 
files.get());
 
 log.trace("Planning compactions {} {} {} {}", planner.getClass().getName(),
 compactable.getExtent(), kind, files);


[accumulo] branch main updated: Fix javadoc comment

2021-08-31 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 4b21ebe  Fix javadoc comment
4b21ebe is described below

commit 4b21ebe3d1cfe02cb60161a96b2acf238487aace
Author: Mike Miller 
AuthorDate: Tue Aug 31 12:02:19 2021 -0400

Fix javadoc comment
---
 .../java/org/apache/accumulo/core/spi/compaction/CompactionPlanner.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/spi/compaction/CompactionPlanner.java
 
b/core/src/main/java/org/apache/accumulo/core/spi/compaction/CompactionPlanner.java
index 523c65a..64d300ed 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/spi/compaction/CompactionPlanner.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/spi/compaction/CompactionPlanner.java
@@ -34,7 +34,7 @@ import org.apache.accumulo.core.spi.common.ServiceEnvironment;
  */
 public interface CompactionPlanner {
 
-  /*
+  /**
* This interface exists so the API can evolve and additional parameters can 
be passed to the
* method in the future.
*


[accumulo-examples] branch main updated: Update docs/rgbalancer.md

2021-08-31 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-examples.git


The following commit(s) were added to refs/heads/main by this push:
 new d1cc653  Update docs/rgbalancer.md
d1cc653 is described below

commit d1cc653e97dc2186f261c6e27ff5c1c8af7fc9d5
Author: Mike Miller 
AuthorDate: Tue Aug 31 10:47:53 2021 -0400

Update docs/rgbalancer.md
---
 docs/rgbalancer.md | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/docs/rgbalancer.md b/docs/rgbalancer.md
index 57d87eb..19bb404 100644
--- a/docs/rgbalancer.md
+++ b/docs/rgbalancer.md
@@ -28,6 +28,9 @@ different tservers.  This gives us four groups of tablets: 
01, 02, 03, and 04.
 root@accumulo> createnamespace examples
 root@accumulo> createtable examples.testRGB
 root@accumulo examples.testRGB> addsplits -t examples.testRGB 01b 01m 01r 
01z 02b 02m 02r 02z 03b 03m 03r 03z 04a 04b 04c 04d 04e 04f 04g 04h 04i 04j 04k 
04l 04m 04n 04o 04p
+
+Run the tables command with the "-l" option to find the table ID.
+
 root@accumulo examples.testRGB> tables -l
 accumulo.metadata=>!0
 accumulo.replication =>  +rep
@@ -35,7 +38,7 @@ different tservers.  This gives us four groups of tablets: 
01, 02, 03, and 04.
 testRGB  => 2
 trace=> 1
 
-After adding the splits we look at the locations in the metadata table.
+Using the table ID for part of the begin and end row scan options, look at the 
locations in the metadata table.
 
 root@accumulo examples.testRGB> scan -t accumulo.metadata -b 2; -e 2< -c 
loc
 2;01b loc:34a5f6e086b000c []ip-10-1-2-25:9997
@@ -98,7 +101,7 @@ default tablet are configured to be in group 04.
 
 root@accumulo examples.testRGB> config -t examples.testRGB -s 
table.custom.balancer.group.regex.pattern=(\d\d).*
 root@accumulo examples.testRGB> config -t examples.testRGB -s 
table.custom.balancer.group.regex.default=04
-root@accumulo examples.testRGB> config -t examples.testRGB -s 
table.balancer=org.apache.accumulo.server.master.balancer.RegexGroupBalancer
+root@accumulo examples.testRGB> config -t examples.testRGB -s 
table.balancer=org.apache.accumulo.core.spi.balancer.RegexGroupBalancer
 
 After waiting a bit, look at the tablet locations again and all is good.
 


[accumulo-examples] branch main updated: Update wording in docs/helloworld.md

2021-08-31 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-examples.git


The following commit(s) were added to refs/heads/main by this push:
 new 14858bd  Update wording in docs/helloworld.md
14858bd is described below

commit 14858bd591b185bb68f64e387e1c761520cc04e9
Author: Mike Miller 
AuthorDate: Tue Aug 31 10:19:07 2021 -0400

Update wording in docs/helloworld.md
---
 docs/helloworld.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/helloworld.md b/docs/helloworld.md
index b0d4381..88e8679 100644
--- a/docs/helloworld.md
+++ b/docs/helloworld.md
@@ -25,9 +25,9 @@ Inserts data with a BatchWriter:
 
 $ ./bin/runex helloworld.Insert
 
-On the accumulo status page at the URL below (where 'master' is replaced with 
the name or IP of your accumulo master), you should see 50K entries
+On the accumulo status page at the URL below (you may need to replace 
'localhost' with the name or IP of your server), you should see 50K entries
 
-http://master:9995/
+http://localhost:9995/
 
 To view the entries, use the shell (run `accumulo shell -u username -p 
password` to access it) to scan the table:
 


[accumulo] branch main updated: Clarify some code in CompactionService (#2250)

2021-08-31 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new ae7d18f  Clarify some code in CompactionService (#2250)
ae7d18f is described below

commit ae7d18f9779e346e010be7ce54b24ec16f429270
Author: Mike Miller 
AuthorDate: Tue Aug 31 08:04:00 2021 -0400

Clarify some code in CompactionService (#2250)

* Split up some single lines of code to make it easier to read
* Add a comment
---
 .../accumulo/tserver/compactions/CompactionService.java  | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java
 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java
index 145c47d..ca30011 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java
@@ -319,9 +319,9 @@ public class CompactionService {
 }
 
 plan = convertPlan(plan, kind, files.get().allFiles, 
files.get().candidates);
-
-if (compactable.getExtent().isMeta() && plan.getJobs().stream().map(cj -> 
cj.getExecutor())
-.anyMatch(ceid -> ((CompactionExecutorIdImpl) ceid).isExternalId())) {
+// log error if tablet is metadata and compaction is external
+var execIds = plan.getJobs().stream().map(cj -> (CompactionExecutorIdImpl) 
cj.getExecutor());
+if (compactable.getExtent().isMeta() && execIds.anyMatch(ceid -> 
ceid.isExternalId())) {
   log.error(
   "Compacting metadata tablets on external compactors is not 
supported, please change "
   + "config for compaction service ({}) and/or table ASAP.  {} is 
not compacting, "
@@ -344,11 +344,11 @@ public class CompactionService {
 
 if (reconcile(jobs, submitted)) {
   for (CompactionJob job : jobs) {
-var sjob =
-executors.get(job.getExecutor()).submit(myId, job, compactable, 
completionCallback);
+CompactionExecutor executor = executors.get(job.getExecutor());
+var submittedJob = executor.submit(myId, job, compactable, 
completionCallback);
 // its important that the collection created in computeIfAbsent 
supports concurrency
 submittedJobs.computeIfAbsent(compactable.getExtent(), k -> new 
ConcurrentLinkedQueue<>())
-.add(sjob);
+.add(submittedJob);
   }
 
   if (!jobs.isEmpty()) {


[accumulo] branch main updated (7c9c827 -> dcca6ce)

2021-08-24 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from 7c9c827  Rename all ServerContext variables to context (#2241)
 add dcca6ce  Rename new Class loader variable to contextName (#2242)

No new revisions were added by this update.

Summary of changes:
 .../core/spi/common/ContextClassLoaderFactory.java | 18 +-
 .../start/classloader/vfs/AccumuloVFSClassLoader.java  |  8 
 2 files changed, 13 insertions(+), 13 deletions(-)


[accumulo] branch main updated (2987f37 -> 7c9c827)

2021-08-24 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from 2987f37  Make variables final in Server and Client Info
 add 7c9c827  Rename all ServerContext variables to context (#2241)

No new revisions were added by this update.

Summary of changes:
 .../accumulo/server/ServiceEnvironmentImpl.java|  16 +--
 .../apache/accumulo/server/init/Initialize.java|  24 ++--
 .../manager/balancer/BalancerEnvironmentImpl.java  |   4 +-
 .../server/metadata/RootTabletMutatorImpl.java |   8 +-
 .../accumulo/server/metadata/ServerAmpleImpl.java  |   6 +-
 .../server/metadata/TabletMutatorBase.java |   4 +-
 .../apache/accumulo/server/rpc/TServerUtils.java   |  14 +--
 .../apache/accumulo/server/MockServerContext.java  |  11 +-
 .../server/ServiceEnvironmentImplTest.java |  12 +-
 .../accumulo/server/util/TServerUtilsTest.java |  41 +++---
 .../coordinator/CompactionCoordinatorTest.java |  50 
 .../apache/accumulo/compactor/CompactorTest.java   |  40 +++---
 .../manager/tableOps/create/PopulateMetadata.java  |   4 +-
 .../manager/upgrade/RenameMasterDirInZK.java   |   8 +-
 .../apache/accumulo/manager/upgrade/Upgrader.java  |  12 +-
 .../accumulo/manager/upgrade/Upgrader8to9.java |   6 +-
 .../accumulo/manager/upgrade/Upgrader9to10.java| 138 +++--
 .../accumulo/tserver/ConditionCheckerContext.java  |   6 +-
 .../org/apache/accumulo/tserver/InMemoryMap.java   |   4 +-
 .../tserver/compactions/CompactionManager.java |  17 +--
 .../tserver/compactions/CompactionService.java |  15 +--
 .../test/functional/ManyWriteAheadLogsIT.java  |  10 +-
 .../replication/ReplicationOperationsImplIT.java   |   6 +-
 23 files changed, 231 insertions(+), 225 deletions(-)


[accumulo] branch main updated: Make variables final in Server and Client Info

2021-08-23 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 2987f37  Make variables final in Server and Client Info
2987f37 is described below

commit 2987f3762afbba81501ad52b5a188aa25bdd7ab8
Author: Mike Miller 
AuthorDate: Mon Aug 23 15:02:07 2021 -0400

Make variables final in Server and Client Info
---
 .../apache/accumulo/core/clientImpl/ClientInfoImpl.java  |  4 ++--
 .../main/java/org/apache/accumulo/server/ServerInfo.java | 16 
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientInfoImpl.java 
b/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientInfoImpl.java
index 816604c..5493b01 100644
--- a/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientInfoImpl.java
+++ b/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientInfoImpl.java
@@ -35,9 +35,9 @@ import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
 
 public class ClientInfoImpl implements ClientInfo {
 
-  private Properties properties;
+  private final Properties properties;
   private AuthenticationToken token;
-  private Configuration hadoopConf;
+  private final Configuration hadoopConf;
 
   public ClientInfoImpl(Path propertiesFile) {
 this(ClientInfoImpl.toProperties(propertiesFile));
diff --git 
a/server/base/src/main/java/org/apache/accumulo/server/ServerInfo.java 
b/server/base/src/main/java/org/apache/accumulo/server/ServerInfo.java
index c37e65b..4427c99 100644
--- a/server/base/src/main/java/org/apache/accumulo/server/ServerInfo.java
+++ b/server/base/src/main/java/org/apache/accumulo/server/ServerInfo.java
@@ -45,14 +45,14 @@ import org.apache.hadoop.fs.Path;
 
 public class ServerInfo implements ClientInfo {
 
-  private SiteConfiguration siteConfig;
-  private Configuration hadoopConf;
-  private String instanceID;
-  private String instanceName;
-  private String zooKeepers;
-  private int zooKeepersSessionTimeOut;
-  private VolumeManager volumeManager;
-  private ZooCache zooCache;
+  private final SiteConfiguration siteConfig;
+  private final Configuration hadoopConf;
+  private final String instanceID;
+  private final String instanceName;
+  private final String zooKeepers;
+  private final int zooKeepersSessionTimeOut;
+  private final VolumeManager volumeManager;
+  private final ZooCache zooCache;
   private final ServerDirs serverDirs;
 
   ServerInfo(SiteConfiguration siteConfig, String instanceName, String 
zooKeepers,


[accumulo] branch main updated: Rework sorted WAL Upgrade code (#2231)

2021-08-23 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 87d4aa5  Rework sorted WAL Upgrade code (#2231)
87d4aa5 is described below

commit 87d4aa5d51c8dc9ac1bc6d714998228b21270411
Author: Mike Miller 
AuthorDate: Mon Aug 23 12:42:16 2021 -0400

Rework sorted WAL Upgrade code (#2231)

* Follow on work for #2185
* Improve upgrade code in Upgrader9to10.dropSortedMapWALFiles()
---
 .../accumulo/manager/upgrade/Upgrader9to10.java| 55 +-
 .../manager/upgrade/Upgrader9to10Test.java | 66 +++---
 2 files changed, 74 insertions(+), 47 deletions(-)

diff --git 
a/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
 
b/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
index 597f88c..9334164 100644
--- 
a/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
+++ 
b/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
@@ -108,7 +108,7 @@ import com.google.common.base.Preconditions;
  *
  * Sorted recovery was updated to use RFiles instead of map files. So to 
prevent issues during
  * tablet recovery, remove the old temporary map files and resort using 
RFiles. This is done in
- * {@link #dropSortedMapWALFiles(VolumeManager)}. For more information see the 
following issues:
+ * {@link #dropSortedMapWALFiles(ServerContext)}. For more information see the 
following issues:
  * https://github.com/apache/accumulo/issues/2117;>#2117 and
  * https://github.com/apache/accumulo/issues/2179;>#2179
  */
@@ -134,6 +134,8 @@ public class Upgrader9to10 implements Upgrader {
 upgradeRootTabletMetadata(ctx);
 renameOldMasterPropsinZK(ctx);
 createExternalCompactionNodes(ctx);
+// special case where old files need to be deleted
+dropSortedMapWALFiles(ctx);
   }
 
   @Override
@@ -148,8 +150,6 @@ public class Upgrader9to10 implements Upgrader {
 upgradeRelativePaths(ctx, Ample.DataLevel.USER);
 upgradeDirColumns(ctx, Ample.DataLevel.USER);
 upgradeFileDeletes(ctx, Ample.DataLevel.USER);
-// special case where old files need to be deleted
-dropSortedMapWALFiles(ctx.getVolumeManager());
   }
 
   private void setMetaTableProps(ServerContext ctx) {
@@ -738,33 +738,36 @@ public class Upgrader9to10 implements Upgrader {
   /**
* Remove old temporary map files to prevent problems during recovery.
*/
-  static void dropSortedMapWALFiles(VolumeManager vm) {
-Path recoveryDir = new Path("/accumulo/recovery");
-try {
-  if (!vm.exists(recoveryDir)) {
-log.info("There are no recovery files in /accumulo/recovery");
-return;
-  }
-  List directoriesToDrop = new ArrayList<>();
-  for (FileStatus walDir : vm.listStatus(recoveryDir)) {
-// map files will be in a directory starting with "part"
-Path walDirPath = walDir.getPath();
-for (FileStatus dirOrFile : vm.listStatus(walDirPath)) {
-  if (dirOrFile.isDirectory()) {
-directoriesToDrop.add(walDirPath);
-break;
+  static void dropSortedMapWALFiles(ServerContext context) {
+VolumeManager vm = context.getVolumeManager();
+for (String recoveryDir : context.getRecoveryDirs()) {
+  Path recoveryDirPath = new Path(recoveryDir);
+  try {
+if (!vm.exists(recoveryDirPath)) {
+  log.info("There are no recovery files in {}", recoveryDir);
+  continue;
+}
+List directoriesToDrop = new ArrayList<>();
+for (FileStatus walDir : vm.listStatus(recoveryDirPath)) {
+  // map files will be in a directory starting with "part"
+  Path walDirPath = walDir.getPath();
+  for (FileStatus dirOrFile : vm.listStatus(walDirPath)) {
+if (dirOrFile.isDirectory()) {
+  directoriesToDrop.add(walDirPath);
+  break;
+}
   }
 }
-  }
-  if (!directoriesToDrop.isEmpty()) {
-log.info("Found {} old sorted map directories to delete.", 
directoriesToDrop.size());
-for (Path dir : directoriesToDrop) {
-  log.info("Deleting everything in old sorted map directory: {}", dir);
-  vm.deleteRecursively(dir);
+if (!directoriesToDrop.isEmpty()) {
+  log.info("Found {} old sorted map directories to delete.", 
directoriesToDrop.size());
+  for (Path dir : directoriesToDrop) {
+log.info("Deleting everything in old sorted map directory: {}", 
dir);
+vm.deleteRecursively(dir);
+  }
 }
+  } catch (IOException ioe) {
+throw new UncheckedIOException(ioe);
   }
-   

[accumulo] branch main updated: Rename LogSorter properties to be consistent (#2237)

2021-08-23 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 38e4cce  Rename LogSorter properties to be consistent (#2237)
38e4cce is described below

commit 38e4cced3de382e737950e9cfd46c2f4826fc571
Author: Mike Miller 
AuthorDate: Mon Aug 23 08:44:01 2021 -0400

Rename LogSorter properties to be consistent (#2237)

* Create tserver.wal.sort.buffer.size to replace tserver.sort.buffer.size
* Create tserver.wal.sort.concurrent.max to replace 
tserver.recovery.concurrent.max
* All LogSorter properties now start with tserver.wal.sort
---
 core/src/main/java/org/apache/accumulo/core/conf/Property.java | 10 +-
 .../main/java/org/apache/accumulo/tserver/log/LogSorter.java   |  9 +++--
 2 files changed, 16 insertions(+), 3 deletions(-)

diff --git a/core/src/main/java/org/apache/accumulo/core/conf/Property.java 
b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
index e41f898..77fd8ea 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
@@ -671,8 +671,16 @@ public enum Property {
   "The replication to use when writing the Write-Ahead log to HDFS. If"
   + " zero, it will use the HDFS default replication setting.",
   "1.5.0"),
+  TSERV_WAL_SORT_MAX_CONCURRENT("tserver.wal.sort.concurrent.max", "2", 
PropertyType.COUNT,
+  "The maximum number of threads to use to sort logs during recovery", 
"2.1.0"),
+  @Deprecated(since = "2.1.0")
+  @ReplacedBy(property = Property.TSERV_WAL_SORT_MAX_CONCURRENT)
   TSERV_RECOVERY_MAX_CONCURRENT("tserver.recovery.concurrent.max", "2", 
PropertyType.COUNT,
-  "The maximum number of threads to use to sort logs during" + " 
recovery", "1.5.0"),
+  "The maximum number of threads to use to sort logs during recovery", 
"1.5.0"),
+  TSERV_WAL_SORT_BUFFER_SIZE("tserver.wal.sort.buffer.size", "10%", 
PropertyType.MEMORY,
+  "The amount of memory to use when sorting logs during recovery.", 
"2.1.0"),
+  @Deprecated(since = "2.1.0")
+  @ReplacedBy(property = Property.TSERV_WAL_SORT_BUFFER_SIZE)
   TSERV_SORT_BUFFER_SIZE("tserver.sort.buffer.size", "10%", 
PropertyType.MEMORY,
   "The amount of memory to use when sorting logs during recovery.", 
"1.5.0"),
   TSERV_WAL_SORT_FILE_PREFIX("tserver.wal.sort.file.", null, 
PropertyType.PREFIX,
diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java
index 3d593dc..ed8a601 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java
@@ -151,7 +151,10 @@ public class LogSorter {
 return;
   }
 
-  final long bufferSize = 
sortedLogConf.getAsBytes(Property.TSERV_SORT_BUFFER_SIZE);
+  @SuppressWarnings("deprecation")
+  Property prop = 
sortedLogConf.resolve(Property.TSERV_WAL_SORT_BUFFER_SIZE,
+  Property.TSERV_SORT_BUFFER_SIZE);
+  final long bufferSize = sortedLogConf.getAsBytes(prop);
   Thread.currentThread().setName("Sorting " + name + " for recovery");
   while (true) {
 final ArrayList> buffer = new 
ArrayList<>();
@@ -208,7 +211,9 @@ public class LogSorter {
   public LogSorter(ServerContext context, AccumuloConfiguration conf) {
 this.context = context;
 this.sortedLogConf = extractSortedLogConfig(conf);
-int threadPoolSize = conf.getCount(Property.TSERV_RECOVERY_MAX_CONCURRENT);
+@SuppressWarnings("deprecation")
+int threadPoolSize = 
conf.getCount(conf.resolve(Property.TSERV_WAL_SORT_MAX_CONCURRENT,
+Property.TSERV_RECOVERY_MAX_CONCURRENT));
 this.threadPool =
 ThreadPools.createFixedThreadPool(threadPoolSize, 
this.getClass().getName(), false);
 this.walBlockSize = DfsLogger.getWalBlockSize(conf);


[accumulo] branch main updated: Add new property to configure rfile sorted recovery (#2236)

2021-08-20 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 3ea979d  Add new property to configure rfile sorted recovery (#2236)
3ea979d is described below

commit 3ea979d5971016b9c2512a96cc5b6bd5d1969b2e
Author: Mike Miller 
AuthorDate: Fri Aug 20 10:44:08 2021 -0400

Add new property to configure rfile sorted recovery (#2236)

* Create new property prefix "tserver.wal.sort.file." to configure the
rfiles written during sorted recovery
* Add method to LogSorter to convert the sort file properties to table
files properties
* Create new tests in SortedLogRecoveryTest
* Make method public in Compression to use in test
* Add property to MultiTableRecoveryIT to allow testing in an IT
---
 .../org/apache/accumulo/core/conf/Property.java|   5 +
 .../core/file/rfile/bcfile/Compression.java|   2 +-
 .../org/apache/accumulo/tserver/log/LogSorter.java |  33 --
 .../tserver/log/SortedLogRecoveryTest.java | 112 +
 .../apache/accumulo/test/MultiTableRecoveryIT.java |   2 +
 5 files changed, 147 insertions(+), 7 deletions(-)

diff --git a/core/src/main/java/org/apache/accumulo/core/conf/Property.java 
b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
index f97e802..e41f898 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
@@ -675,6 +675,11 @@ public enum Property {
   "The maximum number of threads to use to sort logs during" + " 
recovery", "1.5.0"),
   TSERV_SORT_BUFFER_SIZE("tserver.sort.buffer.size", "10%", 
PropertyType.MEMORY,
   "The amount of memory to use when sorting logs during recovery.", 
"1.5.0"),
+  TSERV_WAL_SORT_FILE_PREFIX("tserver.wal.sort.file.", null, 
PropertyType.PREFIX,
+  "The rfile properties to use when sorting logs during recovery. Most of 
the properties"
+  + " that begin with 'table.file' can be used here. For example, to 
set the compression"
+  + " of the sorted recovery files to snappy use 
'tserver.wal.sort.file.compress.type=snappy'",
+  "2.1.0"),
   TSERV_WORKQ_THREADS("tserver.workq.threads", "2", PropertyType.COUNT,
   "The number of threads for the distributed work queue. These threads are"
   + " used for copying failed bulk import RFiles.",
diff --git 
a/core/src/main/java/org/apache/accumulo/core/file/rfile/bcfile/Compression.java
 
b/core/src/main/java/org/apache/accumulo/core/file/rfile/bcfile/Compression.java
index dcd41a6..d4089df 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/file/rfile/bcfile/Compression.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/file/rfile/bcfile/Compression.java
@@ -696,7 +696,7 @@ public final class Compression {
 return supportedAlgorithms.toArray(new String[0]);
   }
 
-  static Algorithm getCompressionAlgorithmByName(final String name) {
+  public static Algorithm getCompressionAlgorithmByName(final String name) {
 Algorithm[] algorithms = Algorithm.class.getEnumConstants();
 for (Algorithm algorithm : algorithms) {
   if (algorithm.getName().equals(name)) {
diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java
index bd299af..3d593dc 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java
@@ -32,6 +32,7 @@ import java.util.concurrent.ThreadPoolExecutor;
 
 import org.apache.accumulo.core.Constants;
 import org.apache.accumulo.core.conf.AccumuloConfiguration;
+import org.apache.accumulo.core.conf.ConfigurationCopy;
 import org.apache.accumulo.core.conf.Property;
 import org.apache.accumulo.core.data.Key;
 import org.apache.accumulo.core.data.Mutation;
@@ -59,7 +60,7 @@ import com.google.common.annotations.VisibleForTesting;
 public class LogSorter {
 
   private static final Logger log = LoggerFactory.getLogger(LogSorter.class);
-  AccumuloConfiguration conf;
+  AccumuloConfiguration sortedLogConf;
 
   private final Map currentWork = 
Collections.synchronizedMap(new HashMap<>());
 
@@ -139,7 +140,7 @@ public class LogSorter {
 
   input = fs.open(srcPath);
   try {
-decryptingInput = DfsLogger.getDecryptingStream(input, conf);
+decryptingInput = DfsLogger.getDecryptingStream(input, sortedLogConf);
   } catch (LogHeaderIncompleteException e) {
 log.warn("Could not read header from write-ahead log {}. Not 
sorting.", srcPath);
 // Creating a 

[accumulo] branch main updated: Create AccumuloDataVersion (#2234)

2021-08-18 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new bd46219  Create AccumuloDataVersion (#2234)
bd46219 is described below

commit bd46219580375f36861961c092fba59fa96041a3
Author: Mike Miller 
AuthorDate: Wed Aug 18 09:06:21 2021 -0400

Create AccumuloDataVersion (#2234)

* Create new class for data version called AccumuloDataVersion
* Move data version from ServerConstants to AccumuloDataVersion and add
javadoc description of the different version numers
* Create AccumuloDataVersion.get() method for getting version
* Move other constants from ServerConstants to Constants
* Rename ServerConstants to ServerDirs
---
 .../java/org/apache/accumulo/core/Constants.java   | 10 ++-
 .../apache/accumulo/core/file/rfile/RFileTest.java |  8 ---
 .../miniclusterImpl/MiniAccumuloClusterImpl.java   |  6 +-
 .../accumulo/server/AccumuloDataVersion.java   | 81 ++
 .../org/apache/accumulo/server/ServerContext.java  | 20 +++---
 .../{ServerConstants.java => ServerDirs.java}  | 63 -
 .../org/apache/accumulo/server/ServerInfo.java | 14 ++--
 .../apache/accumulo/server/fs/VolumeManager.java   |  6 +-
 .../apache/accumulo/server/gc/GcVolumeUtil.java|  6 +-
 .../apache/accumulo/server/init/Initialize.java| 42 +--
 .../apache/accumulo/server/util/ChangeSecret.java  | 18 ++---
 ...erverConstantsTest.java => ServerDirsTest.java} | 35 +-
 .../server/security/SystemCredentialsTest.java | 15 ++--
 .../accumulo/gc/GarbageCollectionAlgorithm.java|  5 +-
 .../tableOps/tableExport/WriteExportFiles.java |  4 +-
 .../manager/tableOps/tableImport/ImportTable.java  |  4 +-
 .../manager/upgrade/UpgradeCoordinator.java| 32 -
 .../accumulo/manager/upgrade/Upgrader8to9.java |  4 +-
 .../accumulo/manager/upgrade/AccumuloTest.java | 27 
 .../org/apache/accumulo/tserver/TabletServer.java  |  3 +-
 .../org/apache/accumulo/tserver/log/DfsLogger.java |  6 +-
 .../test/MissingWalHeaderCompletesRecoveryIT.java  |  6 +-
 .../java/org/apache/accumulo/test/VolumeIT.java|  4 +-
 .../UnusedWalDoesntCloseReplicationStatusIT.java   |  4 +-
 24 files changed, 232 insertions(+), 191 deletions(-)

diff --git a/core/src/main/java/org/apache/accumulo/core/Constants.java 
b/core/src/main/java/org/apache/accumulo/core/Constants.java
index 8559656..a46803f 100644
--- a/core/src/main/java/org/apache/accumulo/core/Constants.java
+++ b/core/src/main/java/org/apache/accumulo/core/Constants.java
@@ -21,8 +21,15 @@ package org.apache.accumulo.core;
 import static java.nio.charset.StandardCharsets.UTF_8;
 
 public class Constants {
-
+  // defines Accumulo data version constants
   public static final String VERSION = FilteredConstants.VERSION;
+  public static final String VERSION_DIR = "version";
+
+  // important directories
+  public static final String INSTANCE_ID_DIR = "instance_id";
+  public static final String TABLE_DIR = "tables";
+  public static final String RECOVERY_DIR = "recovery";
+  public static final String WAL_DIR = "wal";
 
   // Zookeeper locations
   public static final String ZROOT = "/accumulo";
@@ -116,5 +123,4 @@ public class Constants {
   public static final String HDFS_TABLES_DIR = "/tables";
 
   public static final int DEFAULT_VISIBILITY_CACHE_SIZE = 1000;
-
 }
diff --git 
a/core/src/test/java/org/apache/accumulo/core/file/rfile/RFileTest.java 
b/core/src/test/java/org/apache/accumulo/core/file/rfile/RFileTest.java
index ed62565..94270ad 100644
--- a/core/src/test/java/org/apache/accumulo/core/file/rfile/RFileTest.java
+++ b/core/src/test/java/org/apache/accumulo/core/file/rfile/RFileTest.java
@@ -2324,14 +2324,6 @@ public class RFileTest {
 // This tests that the normal set of operations used to populate a root 
tablet
 conf = getAccumuloConfig(CryptoTest.CRYPTO_ON_CONF);
 
-// populate the root tablet with info about the default tablet
-// the root tablet contains the key extent and locations of all the
-// metadata tablets
-// String initRootTabFile = ServerConstants.getMetadataTableDir() + 
"/root_tablet/0_0."
-// + 
FileOperations.getNewFileExtension(AccumuloConfiguration.getDefaultConfiguration());
-// FileSKVWriter mfw = 
FileOperations.getInstance().openWriter(initRootTabFile, fs, conf,
-// AccumuloConfiguration.getDefaultConfiguration());
-
 TestRFile testRfile = new TestRFile(conf);
 testRfile.openWriter();
 
diff --git 
a/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterImpl.java
 
b/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterImpl.java
index 679f725..a5a85e7 100644
--- 
a/minicluster

[accumulo] branch main updated: Remove unused constant from ServerConstants

2021-08-17 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 1359960  Remove unused constant from ServerConstants
1359960 is described below

commit 13599604dbb507d0ee9a28046528b5e0519deb31
Author: Mike Miller 
AuthorDate: Tue Aug 17 10:03:32 2021 -0400

Remove unused constant from ServerConstants
---
 .../base/src/main/java/org/apache/accumulo/server/ServerConstants.java | 3 ---
 1 file changed, 3 deletions(-)

diff --git 
a/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java 
b/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java
index 0195e8c..48e0172 100644
--- a/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java
+++ b/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java
@@ -42,8 +42,6 @@ import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
 
-import com.google.common.collect.Sets;
-
 public class ServerConstants {
 
   public static final String VERSION_DIR = "version";
@@ -84,7 +82,6 @@ public class ServerConstants {
 
   public static final Set CAN_RUN =
   Set.of(SHORTEN_RFILE_KEYS, CRYPTO_CHANGES, DATA_VERSION);
-  public static final Set NEEDS_UPGRADE = Sets.difference(CAN_RUN, 
Set.of(DATA_VERSION));
   public static final String TABLE_DIR = "tables";
   public static final String RECOVERY_DIR = "recovery";
   public static final String WAL_DIR = "wal";


[accumulo] branch main updated: Drop ServerUtil and static state (#2233)

2021-08-17 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new efbaf2d  Drop ServerUtil and static state (#2233)
efbaf2d is described below

commit efbaf2d52d10110d8dbbec0a11daf5c596f2d278
Author: Mike Miller 
AuthorDate: Tue Aug 17 09:51:23 2021 -0400

Drop ServerUtil and static state (#2233)

* Move static state from ServerUtil in to ServerConstants object methods
* Move creation of ServerConstants object into ServerInfo
* Move upgrade code from ServerUtil into UpgradeCoordinator
* Move methods dealing with server stuff to ServerContext
* Create getter for ServerConstants in ServerContext
* Move AccumuloTest to manager to avoid adding a dependency
* Drop ServerUtilTest and move testing of method to ServerContextTest
* Make InterruptedException handled the same as other KeeperException
in ServerContext waitForZookeeperAndHdfs()

Co-authored-by: EdColeman 
---
 .../miniclusterImpl/MiniAccumuloClusterImpl.java   |   5 +-
 .../org/apache/accumulo/server/AbstractServer.java |   2 +-
 .../apache/accumulo/server/ServerConstants.java|  37 ++-
 .../org/apache/accumulo/server/ServerContext.java  | 156 ++-
 .../org/apache/accumulo/server/ServerInfo.java |  10 +-
 .../org/apache/accumulo/server/ServerUtil.java | 288 -
 .../apache/accumulo/server/init/Initialize.java|  10 +-
 .../apache/accumulo/server/util/ChangeSecret.java  |  20 +-
 .../apache/accumulo/server/ServerContextTest.java  |   6 +
 .../org/apache/accumulo/server/ServerUtilTest.java |  29 ---
 .../accumulo/manager/state/SetGoalState.java   |   3 +-
 .../manager/upgrade/UpgradeCoordinator.java|  79 +-
 .../accumulo/manager/upgrade}/AccumuloTest.java|  26 +-
 .../org/apache/accumulo/tracer/TraceServer.java|   3 +-
 14 files changed, 318 insertions(+), 356 deletions(-)

diff --git 
a/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterImpl.java
 
b/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterImpl.java
index 7b315e8..679f725 100644
--- 
a/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterImpl.java
+++ 
b/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterImpl.java
@@ -78,8 +78,8 @@ import org.apache.accumulo.fate.zookeeper.ZooUtil;
 import org.apache.accumulo.manager.state.SetGoalState;
 import org.apache.accumulo.minicluster.MiniAccumuloCluster;
 import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.server.ServerConstants;
 import org.apache.accumulo.server.ServerContext;
-import org.apache.accumulo.server.ServerUtil;
 import org.apache.accumulo.server.fs.VolumeManager;
 import org.apache.accumulo.server.fs.VolumeManagerImpl;
 import org.apache.accumulo.server.init.Initialize;
@@ -470,11 +470,12 @@ public class MiniAccumuloClusterImpl implements 
AccumuloCluster {
 if (config.useExistingInstance()) {
   AccumuloConfiguration acuConf = config.getAccumuloConfiguration();
   Configuration hadoopConf = config.getHadoopConfiguration();
+  ServerConstants serverConstants = new ServerConstants(acuConf, 
hadoopConf);
 
   ConfigurationCopy cc = new ConfigurationCopy(acuConf);
   Path instanceIdPath;
   try (var fs = VolumeManagerImpl.get(cc, hadoopConf)) {
-instanceIdPath = ServerUtil.getAccumuloInstanceIdPath(fs);
+instanceIdPath = serverConstants.getInstanceIdLocation(fs.getFirst());
   } catch (IOException e) {
 throw new RuntimeException(e);
   }
diff --git 
a/server/base/src/main/java/org/apache/accumulo/server/AbstractServer.java 
b/server/base/src/main/java/org/apache/accumulo/server/AbstractServer.java
index f2ebe63..49e03b0 100644
--- a/server/base/src/main/java/org/apache/accumulo/server/AbstractServer.java
+++ b/server/base/src/main/java/org/apache/accumulo/server/AbstractServer.java
@@ -49,7 +49,7 @@ public abstract class AbstractServer implements 
AutoCloseable, Runnable {
 context = new ServerContext(siteConfig);
 log.info("Version " + Constants.VERSION);
 log.info("Instance " + context.getInstanceID());
-ServerUtil.init(context, appName);
+context.init(appName);
 ClassLoaderUtil.initContextFactory(context.getConfiguration());
 this.metricsSystem = Metrics.initSystem(getClass().getSimpleName());
 TraceUtil.enableServerTraces(hostname, appName, 
context.getConfiguration());
diff --git 
a/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java 
b/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java
index 46d39f1..0195e8c 100644
--- a/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java
+++ b/server/base/src/main/java/org/ap

[accumulo] branch main updated: Replace static state in ServerConstants (#2232)

2021-08-16 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new f2adfed  Replace static state in ServerConstants (#2232)
f2adfed is described below

commit f2adfedda8d4efee48c5138b1373b2d4e2cd7598
Author: Mike Miller 
AuthorDate: Mon Aug 16 10:56:23 2021 -0400

Replace static state in ServerConstants (#2232)

* Replace static state from ServerConstants with object methods and remove 
duplicate methods
* Add final object to ServerContext for getting baseUris and 
volumeReplacements
* Move static methods that use Volume to ServerUtil
---
 .../apache/accumulo/server/ServerConstants.java| 156 ++---
 .../org/apache/accumulo/server/ServerContext.java  |  22 +++
 .../org/apache/accumulo/server/ServerUtil.java |  16 ++-
 .../org/apache/accumulo/server/fs/VolumeUtil.java  |   4 +-
 .../apache/accumulo/server/init/Initialize.java|  27 ++--
 .../apache/accumulo/server/util/ChangeSecret.java  |   6 +-
 .../accumulo/server/ServerConstantsTest.java   |  32 +++--
 .../accumulo/gc/GarbageCollectWriteAheadLogs.java  |   3 +-
 .../apache/accumulo/gc/SimpleGarbageCollector.java |   6 +-
 .../accumulo/manager/recovery/RecoveryManager.java |   6 +-
 .../manager/tableOps/bulkVer1/BulkImport.java  |   4 +-
 .../manager/tableOps/bulkVer2/PrepBulkImport.java  |   4 +-
 .../accumulo/manager/tableOps/delete/CleanUp.java  |   3 +-
 .../tableOps/tableImport/CreateImportDir.java  |   3 +-
 .../org/apache/accumulo/tserver/TabletServer.java  |   2 +-
 .../org/apache/accumulo/tserver/log/DfsLogger.java |   2 +-
 .../accumulo/tserver/tablet/DatafileManager.java   |   3 +-
 .../org/apache/accumulo/tserver/tablet/Tablet.java |   7 +-
 .../accumulo/test/functional/SplitRecoveryIT.java  |   5 +-
 19 files changed, 169 insertions(+), 142 deletions(-)

diff --git 
a/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java 
b/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java
index bb70aa8..46d39f1 100644
--- a/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java
+++ b/server/base/src/main/java/org/apache/accumulo/server/ServerConstants.java
@@ -25,13 +25,13 @@ import java.util.Collections;
 import java.util.HashSet;
 import java.util.LinkedHashSet;
 import java.util.List;
+import java.util.Objects;
 import java.util.Set;
 import java.util.stream.Collectors;
 
 import org.apache.accumulo.core.conf.AccumuloConfiguration;
 import org.apache.accumulo.core.conf.Property;
 import org.apache.accumulo.core.util.Pair;
-import org.apache.accumulo.core.volume.Volume;
 import org.apache.accumulo.core.volume.VolumeConfiguration;
 import org.apache.accumulo.server.fs.VolumeManager;
 import org.apache.accumulo.server.fs.VolumeUtil;
@@ -81,27 +81,33 @@ public class ServerConstants {
   public static final Set CAN_RUN =
   Set.of(SHORTEN_RFILE_KEYS, CRYPTO_CHANGES, DATA_VERSION);
   public static final Set NEEDS_UPGRADE = Sets.difference(CAN_RUN, 
Set.of(DATA_VERSION));
+  public static final String TABLE_DIR = "tables";
+  public static final String RECOVERY_DIR = "recovery";
+  public static final String WAL_DIR = "wal";
 
-  private static Set baseUris = null;
+  private Set baseUris;
+  private Set tablesDirs;
+  private Set recoveryDirs;
 
-  private static List> replacementsList = null;
+  private final List> replacementsList;
+  private final AccumuloConfiguration conf;
+  private final Configuration hadoopConf;
 
-  public static Set getBaseUris(ServerContext context) {
-return getBaseUris(context.getConfiguration(), context.getHadoopConf());
+  public ServerConstants(AccumuloConfiguration conf, Configuration hadoopConf) 
{
+this.conf = Objects.requireNonNull(conf, "Configuration cannot be null");
+this.hadoopConf = Objects.requireNonNull(hadoopConf, "Hadoop configuration 
cannot be null");
+this.replacementsList = loadVolumeReplacements();
   }
 
-  // these are functions to delay loading the Accumulo configuration unless we 
must
-  public static synchronized Set getBaseUris(AccumuloConfiguration 
conf,
-  Configuration hadoopConf) {
+  public Set getBaseUris() {
 if (baseUris == null) {
   baseUris = Collections.unmodifiableSet(
   checkBaseUris(hadoopConf, VolumeConfiguration.getVolumeUris(conf), 
false));
 }
-
 return baseUris;
   }
 
-  public static Set checkBaseUris(Configuration hadoopConf, 
Set configuredBaseDirs,
+  public Set checkBaseUris(Configuration hadoopConf, Set 
configuredBaseDirs,
   boolean ignore) {
 // all base dirs must have same instance id and data version, any dirs 
that have neither should
 // be ignored
@@ -152,100 +158,94 @@ public class ServerConstants {
 return baseDirsList;
   }
 
-  public static f

[accumulo] branch main updated (7db0561 -> baf7a6a)

2021-08-13 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from 7db0561  Remove use of JLine outside of accumulo-shell (#2228)
 add baf7a6a  Drop unused param in ServerConstants

No new revisions were added by this update.

Summary of changes:
 .../java/org/apache/accumulo/server/ServerConstants.java |  6 +++---
 .../java/org/apache/accumulo/server/init/Initialize.java |  3 +--
 .../java/org/apache/accumulo/server/ServerConstantsTest.java | 12 ++--
 3 files changed, 10 insertions(+), 11 deletions(-)


[accumulo] branch main updated (93bafa8 -> 5497d4f)

2021-08-02 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from 93bafa8  Update trace code in GarbageCollectionAlgorithm (#2216)
 add 5497d4f  Fix typos in Property descriptions

No new revisions were added by this update.

Summary of changes:
 core/src/main/java/org/apache/accumulo/core/conf/Property.java | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)


[accumulo] branch main updated: Handle old sorted map files in Upgrade (#2185)

2021-07-30 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 09e3d77  Handle old sorted map files in Upgrade (#2185)
09e3d77 is described below

commit 09e3d771ef11fcc2c2a8eb1d1b29b024d7da900f
Author: Mike Miller 
AuthorDate: Fri Jul 30 12:43:30 2021 -0400

Handle old sorted map files in Upgrade (#2185)

* Add dropSortedMapWALFiles to Upgrader9to10 for resolving sorted
map files that may still be around during upgrade. Follow up for #2117
* Closes #2179
---
 .../accumulo/manager/upgrade/Upgrader9to10.java| 41 +
 .../manager/upgrade/Upgrader9to10Test.java | 42 ++
 2 files changed, 83 insertions(+)

diff --git 
a/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
 
b/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
index 5e874c5..597f88c 100644
--- 
a/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
+++ 
b/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/Upgrader9to10.java
@@ -105,6 +105,12 @@ import com.google.common.base.Preconditions;
  * https://github.com/apache/accumulo/issues/1642;>#1642, and
  * https://github.com/apache/accumulo/issues/1643;>#1643 as 
well.
  * 
+ *
+ * Sorted recovery was updated to use RFiles instead of map files. So to 
prevent issues during
+ * tablet recovery, remove the old temporary map files and resort using 
RFiles. This is done in
+ * {@link #dropSortedMapWALFiles(VolumeManager)}. For more information see the 
following issues:
+ * https://github.com/apache/accumulo/issues/2117;>#2117 and
+ * https://github.com/apache/accumulo/issues/2179;>#2179
  */
 public class Upgrader9to10 implements Upgrader {
 
@@ -142,6 +148,8 @@ public class Upgrader9to10 implements Upgrader {
 upgradeRelativePaths(ctx, Ample.DataLevel.USER);
 upgradeDirColumns(ctx, Ample.DataLevel.USER);
 upgradeFileDeletes(ctx, Ample.DataLevel.USER);
+// special case where old files need to be deleted
+dropSortedMapWALFiles(ctx.getVolumeManager());
   }
 
   private void setMetaTableProps(ServerContext ctx) {
@@ -726,4 +734,37 @@ public class Upgrader9to10 implements Upgrader {
 }
 return new Path(upgradeProperty, 
VolumeManager.FileType.TABLE.getDirectory() + oldDelete);
   }
+
+  /**
+   * Remove old temporary map files to prevent problems during recovery.
+   */
+  static void dropSortedMapWALFiles(VolumeManager vm) {
+Path recoveryDir = new Path("/accumulo/recovery");
+try {
+  if (!vm.exists(recoveryDir)) {
+log.info("There are no recovery files in /accumulo/recovery");
+return;
+  }
+  List directoriesToDrop = new ArrayList<>();
+  for (FileStatus walDir : vm.listStatus(recoveryDir)) {
+// map files will be in a directory starting with "part"
+Path walDirPath = walDir.getPath();
+for (FileStatus dirOrFile : vm.listStatus(walDirPath)) {
+  if (dirOrFile.isDirectory()) {
+directoriesToDrop.add(walDirPath);
+break;
+  }
+}
+  }
+  if (!directoriesToDrop.isEmpty()) {
+log.info("Found {} old sorted map directories to delete.", 
directoriesToDrop.size());
+for (Path dir : directoriesToDrop) {
+  log.info("Deleting everything in old sorted map directory: {}", dir);
+  vm.deleteRecursively(dir);
+}
+  }
+} catch (IOException ioe) {
+  throw new UncheckedIOException(ioe);
+}
+  }
 }
diff --git 
a/server/manager/src/test/java/org/apache/accumulo/manager/upgrade/Upgrader9to10Test.java
 
b/server/manager/src/test/java/org/apache/accumulo/manager/upgrade/Upgrader9to10Test.java
index a5752b9..25800da 100644
--- 
a/server/manager/src/test/java/org/apache/accumulo/manager/upgrade/Upgrader9to10Test.java
+++ 
b/server/manager/src/test/java/org/apache/accumulo/manager/upgrade/Upgrader9to10Test.java
@@ -24,11 +24,13 @@ import static org.easymock.EasyMock.createMock;
 import static org.easymock.EasyMock.expect;
 import static org.easymock.EasyMock.expectLastCall;
 import static org.easymock.EasyMock.replay;
+import static org.easymock.EasyMock.reset;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertTrue;
 import static org.junit.Assert.fail;
 
+import java.io.IOException;
 import java.util.ArrayList;
 import java.util.Iterator;
 import java.util.List;
@@ -49,6 +51,7 @@ import 
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.Da
 import org.apache.accumulo.core.security.Authorizations;
 import org.apache.accumulo.server.fs.VolumeManager;
 import org.apache.accumulo.serv

[accumulo] branch main updated: Fix typo 'propogate' (#2212)

2021-07-23 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 146b1d3  Fix typo 'propogate' (#2212)
146b1d3 is described below

commit 146b1d3d0dc2ade6f1888c9905d83e37d98aedbe
Author: Dom G <47725857+domgargu...@users.noreply.github.com>
AuthorDate: Fri Jul 23 14:48:42 2021 -0400

Fix typo 'propogate' (#2212)
---
 .../accumulo/core/metadata/schema/ExternalCompactionMetadata.java   | 2 +-
 .../java/org/apache/accumulo/server/compaction/CompactionInfo.java  | 2 +-
 .../java/org/apache/accumulo/server/compaction/FileCompactor.java   | 2 +-
 .../java/org/apache/accumulo/tserver/tablet/CompactableImpl.java| 6 +++---
 4 files changed, 6 insertions(+), 6 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/metadata/schema/ExternalCompactionMetadata.java
 
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/ExternalCompactionMetadata.java
index eada2c7..02d9797 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/metadata/schema/ExternalCompactionMetadata.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/ExternalCompactionMetadata.java
@@ -94,7 +94,7 @@ public class ExternalCompactionMetadata {
 return ceid;
   }
 
-  public boolean getPropogateDeletes() {
+  public boolean getPropagateDeletes() {
 return propagateDeletes;
   }
 
diff --git 
a/server/base/src/main/java/org/apache/accumulo/server/compaction/CompactionInfo.java
 
b/server/base/src/main/java/org/apache/accumulo/server/compaction/CompactionInfo.java
index f7c0354..3615e10 100644
--- 
a/server/base/src/main/java/org/apache/accumulo/server/compaction/CompactionInfo.java
+++ 
b/server/base/src/main/java/org/apache/accumulo/server/compaction/CompactionInfo.java
@@ -81,7 +81,7 @@ public class CompactionInfo {
 type = TCompactionType.MERGE;
   else
 type = TCompactionType.MINOR;
-else if (!compactor.willPropogateDeletes())
+else if (!compactor.willPropagateDeletes())
   type = TCompactionType.FULL;
 else
   type = TCompactionType.MAJOR;
diff --git 
a/server/base/src/main/java/org/apache/accumulo/server/compaction/FileCompactor.java
 
b/server/base/src/main/java/org/apache/accumulo/server/compaction/FileCompactor.java
index 9d77443..a093f24 100644
--- 
a/server/base/src/main/java/org/apache/accumulo/server/compaction/FileCompactor.java
+++ 
b/server/base/src/main/java/org/apache/accumulo/server/compaction/FileCompactor.java
@@ -418,7 +418,7 @@ public class FileCompactor implements 
Callable {
 return env.getIteratorScope() == IteratorScope.minc;
   }
 
-  boolean willPropogateDeletes() {
+  boolean willPropagateDeletes() {
 return propagateDeletes;
   }
 
diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java
 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java
index ece9f44..d4562cc 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java
@@ -453,8 +453,8 @@ public class CompactableImpl implements Compactable {
   }
 
   if (propDel == null) {
-propDel = ecMeta.getPropogateDeletes();
-  } else if (propDel != ecMeta.getPropogateDeletes()) {
+propDel = ecMeta.getPropagateDeletes();
+  } else if (propDel != ecMeta.getPropagateDeletes()) {
 unexpectedExternal = true;
 reasons.add("Disagreement on propagateDeletes");
 break;
@@ -464,7 +464,7 @@ public class CompactableImpl implements Compactable {
 
 if (propDel != null && !propDel && count > 1) {
   unexpectedExternal = true;
-  reasons.add("Concurrent compactions not propogatingDeletes");
+  reasons.add("Concurrent compactions not propagatingDeletes");
 }
 
 Pair idAndCfg = null;


[accumulo] branch main updated: Make CachableBlockFile builder require nonNull for cacheId (#2160)

2021-07-23 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 7d176b6  Make CachableBlockFile builder require nonNull for cacheId  
(#2160)
7d176b6 is described below

commit 7d176b60fc6a2ce28fcc3cfdc9a3b68684c76f87
Author: Mike Miller 
AuthorDate: Fri Jul 23 11:27:55 2021 -0400

Make CachableBlockFile builder require nonNull for cacheId  (#2160)
---
 .../org/apache/accumulo/core/client/rfile/RFileScanner.java |  6 +++---
 .../accumulo/core/client/rfile/RFileSummariesRetriever.java |  6 +++---
 .../core/file/blockfile/impl/CachableBlockFile.java | 10 +++---
 .../org/apache/accumulo/core/summary/SummaryReader.java |  8 
 .../accumulo/core/file/rfile/MultiLevelIndexTest.java   |  8 
 .../java/org/apache/accumulo/core/file/rfile/RFileTest.java | 13 +++--
 6 files changed, 24 insertions(+), 27 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/client/rfile/RFileScanner.java 
b/core/src/main/java/org/apache/accumulo/core/client/rfile/RFileScanner.java
index 3413aec..bb4356d 100644
--- a/core/src/main/java/org/apache/accumulo/core/client/rfile/RFileScanner.java
+++ b/core/src/main/java/org/apache/accumulo/core/client/rfile/RFileScanner.java
@@ -347,9 +347,9 @@ class RFileScanner extends ScannerOptions implements 
Scanner {
   for (int i = 0; i < sources.length; i++) {
 // TODO may have been a bug with multiple files and caching in older 
version...
 FSDataInputStream inputStream = (FSDataInputStream) 
sources[i].getInputStream();
-CachableBuilder cb = new CachableBuilder().cacheId("source-" + 
i).input(inputStream)
-
.length(sources[i].getLength()).conf(opts.in.getConf()).cacheProvider(cacheProvider)
-.cryptoService(cryptoService);
+CachableBuilder cb =
+new CachableBuilder().input(inputStream, "source-" + 
i).length(sources[i].getLength())
+
.conf(opts.in.getConf()).cacheProvider(cacheProvider).cryptoService(cryptoService);
 readers.add(new RFile.Reader(cb));
   }
 
diff --git 
a/core/src/main/java/org/apache/accumulo/core/client/rfile/RFileSummariesRetriever.java
 
b/core/src/main/java/org/apache/accumulo/core/client/rfile/RFileSummariesRetriever.java
index 8dbb7a1..9341f50 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/client/rfile/RFileSummariesRetriever.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/client/rfile/RFileSummariesRetriever.java
@@ -93,9 +93,9 @@ class RFileSummariesRetriever implements 
SummaryInputArguments, SummaryFSOptions
 try {
   SummaryCollection all = new SummaryCollection();
   CryptoService cservice = CryptoServiceFactory.newInstance(acuconf, 
ClassloaderType.JAVA);
-  for (RFileSource source : sources) {
-SummaryReader fileSummary = 
SummaryReader.load(in.getFileSystem().getConf(),
-source.getInputStream(), source.getLength(), summarySelector, 
factory, cservice);
+  for (int i = 0; i < sources.length; i++) {
+SummaryReader fileSummary = 
SummaryReader.load(in.getFileSystem().getConf(), sources[i],
+"source-" + i, summarySelector, factory, cservice);
 SummaryCollection sc = fileSummary
 .getSummaries(Collections.singletonList(new 
Gatherer.RowRange(startRow, endRow)));
 all.merge(sc, factory);
diff --git 
a/core/src/main/java/org/apache/accumulo/core/file/blockfile/impl/CachableBlockFile.java
 
b/core/src/main/java/org/apache/accumulo/core/file/blockfile/impl/CachableBlockFile.java
index 2b15fe3..b4c24e2 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/file/blockfile/impl/CachableBlockFile.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/file/blockfile/impl/CachableBlockFile.java
@@ -77,11 +77,6 @@ public class CachableBlockFile {
 Configuration hadoopConf = null;
 CryptoService cryptoService = null;
 
-public CachableBuilder cacheId(String id) {
-  this.cacheId = id;
-  return this;
-}
-
 public CachableBuilder conf(Configuration hadoopConf) {
   this.hadoopConf = hadoopConf;
   return this;
@@ -94,7 +89,8 @@ public class CachableBlockFile {
   return this;
 }
 
-public CachableBuilder input(InputStream is) {
+public CachableBuilder input(InputStream is, String cacheId) {
+  this.cacheId = cacheId;
   this.inputSupplier = () -> is;
   return this;
 }
@@ -359,7 +355,7 @@ public class CachableBlockFile {
 }
 
 public Reader(CachableBuilder b) {
-  this.cacheId = b.cacheId;
+  this.cacheId = Objects.requireNonNull(b.cacheId);
   this.inputSupplier = b.inputSupplier;
   this.lengthSupplier = b.lengthSupplier;
   this.fileLenCache = b.fileLenCache;
diff --git 
a/core/src/mai

[accumulo] branch main updated: Refactor LogSorter to use config (#2191)

2021-07-07 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 908105b  Refactor LogSorter to use config (#2191)
908105b is described below

commit 908105bc8fddd64aa2d5907c22ccf2a232209306
Author: Mike Miller 
AuthorDate: Wed Jul 7 07:29:44 2021 -0400

Refactor LogSorter to use config (#2191)

* Refactor LogSorter writeBuffer to allow use of system configuration
when writing out sorted rfiles. This will use configured settings on the
sorted files instead of only the defaults.
---
 .../java/org/apache/accumulo/tserver/log/LogSorter.java | 13 ++---
 .../apache/accumulo/tserver/log/SortedLogRecoveryTest.java  |  6 --
 2 files changed, 10 insertions(+), 9 deletions(-)

diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java
index 293ff82..bd299af 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java
@@ -32,7 +32,6 @@ import java.util.concurrent.ThreadPoolExecutor;
 
 import org.apache.accumulo.core.Constants;
 import org.apache.accumulo.core.conf.AccumuloConfiguration;
-import org.apache.accumulo.core.conf.DefaultConfiguration;
 import org.apache.accumulo.core.conf.Property;
 import org.apache.accumulo.core.data.Key;
 import org.apache.accumulo.core.data.Mutation;
@@ -146,7 +145,7 @@ public class LogSorter {
 // Creating a 'finished' marker will cause recovery to proceed 
normally and the
 // empty file will be correctly ignored downstream.
 fs.mkdirs(new Path(destPath));
-writeBuffer(context, destPath, Collections.emptyList(), part++);
+writeBuffer(destPath, Collections.emptyList(), part++);
 fs.create(SortedLogState.getFinishedMarkerPath(destPath)).close();
 return;
   }
@@ -164,10 +163,10 @@ public class LogSorter {
 value.readFields(decryptingInput);
 buffer.add(new Pair<>(key, value));
   }
-  writeBuffer(context, destPath, buffer, part++);
+  writeBuffer(destPath, buffer, part++);
   buffer.clear();
 } catch (EOFException ex) {
-  writeBuffer(context, destPath, buffer, part++);
+  writeBuffer(destPath, buffer, part++);
   break;
 }
   }
@@ -215,8 +214,8 @@ public class LogSorter {
   }
 
   @VisibleForTesting
-  public static void writeBuffer(ServerContext context, String destPath,
-  List> buffer, int part) throws IOException 
{
+  void writeBuffer(String destPath, List> 
buffer, int part)
+  throws IOException {
 String filename = String.format("part-r-%05d.rf", part);
 Path path = new Path(destPath, filename);
 FileSystem fs = context.getVolumeManager().getFileSystemByPath(path);
@@ -238,7 +237,7 @@ public class LogSorter {
 
 try (var writer = FileOperations.getInstance().newWriterBuilder()
 .forFile(fullPath.toString(), fs, fs.getConf(), 
context.getCryptoService())
-.withTableConfiguration(DefaultConfiguration.getInstance()).build()) {
+.withTableConfiguration(conf).build()) {
   writer.startDefaultLocalityGroup();
   for (var entry : keyListMap.entrySet()) {
 LogFileValue val = new LogFileValue();
diff --git 
a/server/tserver/src/test/java/org/apache/accumulo/tserver/log/SortedLogRecoveryTest.java
 
b/server/tserver/src/test/java/org/apache/accumulo/tserver/log/SortedLogRecoveryTest.java
index 2121725..582b9bc 100644
--- 
a/server/tserver/src/test/java/org/apache/accumulo/tserver/log/SortedLogRecoveryTest.java
+++ 
b/server/tserver/src/test/java/org/apache/accumulo/tserver/log/SortedLogRecoveryTest.java
@@ -79,6 +79,7 @@ public class SortedLogRecoveryTest {
   static final Text cq = new Text("cq");
   static final Value value = new Value("value");
   static ServerContext context;
+  static LogSorter logSorter;
 
   @Rule
   public TemporaryFolder tempFolder =
@@ -87,6 +88,7 @@ public class SortedLogRecoveryTest {
   @Before
   public void setup() {
 context = EasyMock.createMock(ServerContext.class);
+logSorter = new LogSorter(context, DefaultConfiguration.getInstance());
   }
 
   static class KeyValue implements Comparable {
@@ -179,11 +181,11 @@ public class SortedLogRecoveryTest {
 for (KeyValue pair : entry.getValue()) {
   buffer.add(new Pair<>(pair.key, pair.value));
   if (buffer.size() >= bufferSize) {
-LogSorter.writeBuffer(context, destPath, buffer, parts++);
+logSorter.writeBuffer(destPath, buffer, parts++);
 buffer.clear();
   }
 }
-LogSorter.writeBuffer(context, destPath, buff

[accumulo] branch main updated: Catch NoNodeException in CompactRange undo (#2178)

2021-06-24 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 04a4a7b  Catch NoNodeException in CompactRange undo (#2178)
04a4a7b is described below

commit 04a4a7bcb5f6e7d38829c270b46c1484e4b42650
Author: Mike Miller 
AuthorDate: Thu Jun 24 06:36:22 2021 -0400

Catch NoNodeException in CompactRange undo (#2178)

* Fixes #1919
* It is possible for a compaction to run after a table is deleted so
catch the exception and print to debug, avoiding the FATE warning
---
 .../manager/tableOps/compact/CompactRange.java | 33 --
 1 file changed, 18 insertions(+), 15 deletions(-)

diff --git 
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/compact/CompactRange.java
 
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/compact/CompactRange.java
index dfa52c4..6d94c52 100644
--- 
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/compact/CompactRange.java
+++ 
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/compact/CompactRange.java
@@ -149,24 +149,27 @@ public class CompactRange extends ManagerRepo {
 
 ZooReaderWriter zoo = environment.getContext().getZooReaderWriter();
 
-zoo.mutateExisting(zTablePath, currentValue -> {
-  String cvs = new String(currentValue, UTF_8);
-  String[] tokens = cvs.split(",");
-  long flushID = Long.parseLong(tokens[0]);
-
-  String txidString = String.format("%016x", txid);
+try {
+  zoo.mutateExisting(zTablePath, currentValue -> {
+String cvs = new String(currentValue, UTF_8);
+String[] tokens = cvs.split(",");
+long flushID = Long.parseLong(tokens[0]);
 
-  StringBuilder encodedIterators = new StringBuilder();
-  for (int i = 1; i < tokens.length; i++) {
-if (tokens[i].startsWith(txidString))
-  continue;
-encodedIterators.append(",");
-encodedIterators.append(tokens[i]);
-  }
+String txidString = String.format("%016x", txid);
 
-  return (Long.toString(flushID) + encodedIterators).getBytes(UTF_8);
-});
+StringBuilder encodedIterators = new StringBuilder();
+for (int i = 1; i < tokens.length; i++) {
+  if (tokens[i].startsWith(txidString))
+continue;
+  encodedIterators.append(",");
+  encodedIterators.append(tokens[i]);
+}
 
+return (Long.toString(flushID) + encodedIterators).getBytes(UTF_8);
+  });
+} catch (NoNodeException ke) {
+  log.debug("Node for {} no longer exists.", tableId, ke);
+}
   }
 
   @Override


[accumulo] branch main updated: Adds tests to LogFileKeyTest (#2177)

2021-06-23 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 89b411c  Adds tests to LogFileKeyTest (#2177)
89b411c is described below

commit 89b411c391675cb3f1faed83f2dd4451b2ac6d12
Author: Mike Miller 
AuthorDate: Wed Jun 23 09:57:31 2021 -0400

Adds tests to LogFileKeyTest (#2177)
---
 .../apache/accumulo/tserver/logger/LogFileKey.java | 15 ++-
 .../accumulo/tserver/log/LogFileKeyTest.java   | 49 ++
 2 files changed, 62 insertions(+), 2 deletions(-)

diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/logger/LogFileKey.java
 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/logger/LogFileKey.java
index cf5dc21..b860be5 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/logger/LogFileKey.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/logger/LogFileKey.java
@@ -28,6 +28,7 @@ import java.io.DataInput;
 import java.io.DataOutput;
 import java.io.IOException;
 import java.io.UncheckedIOException;
+import java.util.Arrays;
 
 import org.apache.accumulo.core.data.Key;
 import org.apache.accumulo.core.data.Range;
@@ -36,10 +37,13 @@ import org.apache.hadoop.io.DataInputBuffer;
 import org.apache.hadoop.io.DataOutputBuffer;
 import org.apache.hadoop.io.Text;
 import org.apache.hadoop.io.WritableComparable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import com.google.common.base.Preconditions;
 
 public class LogFileKey implements WritableComparable {
+  private static final Logger log = LoggerFactory.getLogger(LogFileKey.class);
 
   public LogEvents event;
   public String filename = null;
@@ -258,7 +262,9 @@ public class LogFileKey implements 
WritableComparable {
 
   /**
* Format the row using 13 bytes encoded to allow proper sorting of the 
RFile Key. The highest
-   * byte is for the event number, 4 bytes for the tabletId and 8 bytes for 
the sequence long.
+   * byte is for the event number, 4 bytes for the tabletId and 8 bytes for 
the sequence long. The
+   * conversions of integer to byte[] and long to byte[] is similar to what 
DataOutputStream does
+   * for writeInt() and writeLong()
*/
   private byte[] formatRow(int tabletId, long seq) {
 byte eventNum = getEventByte();
@@ -283,11 +289,14 @@ public class LogFileKey implements 
WritableComparable {
 row[10] = (byte) ((seq >>> 16) & mask);
 row[11] = (byte) ((seq >>> 8) & mask);
 row[12] = (byte) (seq & mask);
+
+log.trace("Convert {} {} {} to row {}", event, tabletId, seq, 
Arrays.toString(row));
 return row;
   }
 
   /**
-   * Extract the tabletId integer from the byte encoded Row.
+   * Extract the tabletId integer from the byte encoded Row. Similar to what 
DataInputStream does
+   * for readInt()
*/
   private static int getTabletId(byte[] row) {
 int mask = 0xff; // use a mask of int type to convert byte to int without 
sign extension
@@ -335,6 +344,8 @@ public class LogFileKey implements 
WritableComparable {
 if (eventType(logFileKey.event) != rowParts[0]) {
   throw new AssertionError("Event in row differs from column family. Key: 
" + key);
 }
+log.trace("From row {} get {} {} {}", Arrays.toString(rowParts), 
logFileKey.event,
+logFileKey.tabletId, logFileKey.seq);
 
 // handle special cases of what is stored in the qualifier
 switch (logFileKey.event) {
diff --git 
a/server/tserver/src/test/java/org/apache/accumulo/tserver/log/LogFileKeyTest.java
 
b/server/tserver/src/test/java/org/apache/accumulo/tserver/log/LogFileKeyTest.java
index 72cc339..0f04aca 100644
--- 
a/server/tserver/src/test/java/org/apache/accumulo/tserver/log/LogFileKeyTest.java
+++ 
b/server/tserver/src/test/java/org/apache/accumulo/tserver/log/LogFileKeyTest.java
@@ -20,12 +20,17 @@ package org.apache.accumulo.tserver.log;
 
 import static org.junit.Assert.assertEquals;
 
+import java.io.IOException;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
 
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.dataImpl.KeyExtent;
 import org.apache.accumulo.tserver.logger.LogEvents;
 import org.apache.accumulo.tserver.logger.LogFileKey;
+import org.apache.hadoop.io.Text;
 import org.junit.Test;
 
 public class LogFileKeyTest {
@@ -98,4 +103,48 @@ public class LogFileKeyTest {
 }
 
   }
+
+  private void testToFromKey(int tabletId, long seq) throws IOException {
+var logFileKey = nk(LogEvents.DEFINE_TABLET, tabletId, seq);
+logFileKey.tablet = new KeyExtent(TableId.of("5"), new Text("b"), null);
+Key k = logFileKey.toKey();
+
+var converted = LogFileKey.fromKey(k);
+assertEquals("Failed 

[accumulo] branch main updated: Fix row encoding/decoding in LogFileKey (#2176)

2021-06-23 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new b8262a4  Fix row encoding/decoding in LogFileKey (#2176)
b8262a4 is described below

commit b8262a424db92729c89d6870ba5973386b9e3ea1
Author: Christopher Tubbs 
AuthorDate: Wed Jun 23 08:46:15 2021 -0400

Fix row encoding/decoding in LogFileKey (#2176)

Fix bitwise operations to encode and decode the tabletId and sequence
number to/from a row

This fixes #2173
---
 .../apache/accumulo/tserver/logger/LogFileKey.java | 54 +-
 1 file changed, 31 insertions(+), 23 deletions(-)

diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/logger/LogFileKey.java
 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/logger/LogFileKey.java
index bf08ca4..cf5dc21 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/logger/LogFileKey.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/logger/LogFileKey.java
@@ -245,8 +245,8 @@ public class LogFileKey implements 
WritableComparable {
* byte in the row.
*/
   private byte getEventByte() {
-int evenTypeInteger = eventType(event);
-return (byte) (evenTypeInteger & 0xff);
+int eventTypeInteger = eventType(event);
+return (byte) (eventTypeInteger & 0xff);
   }
 
   /**
@@ -269,19 +269,20 @@ public class LogFileKey implements 
WritableComparable {
 // encode the signed integer so negatives will sort properly for tabletId
 int encodedTabletId = tabletId ^ 0x8000;
 
+int mask = 0xff; // use a mask of int type to truncate the selected 8 bits
 row[0] = eventNum;
-row[1] = (byte) ((encodedTabletId >>> 24) & 0xff);
-row[2] = (byte) ((encodedTabletId >>> 16) & 0xff);
-row[3] = (byte) ((encodedTabletId >>> 8) & 0xff);
-row[4] = (byte) (encodedTabletId & 0xff);
-row[5] = (byte) (seq >>> 56);
-row[6] = (byte) (seq >>> 48);
-row[7] = (byte) (seq >>> 40);
-row[8] = (byte) (seq >>> 32);
-row[9] = (byte) (seq >>> 24);
-row[10] = (byte) (seq >>> 16);
-row[11] = (byte) (seq >>> 8);
-row[12] = (byte) (seq); // >>> 0
+row[1] = (byte) ((encodedTabletId >>> 24) & mask);
+row[2] = (byte) ((encodedTabletId >>> 16) & mask);
+row[3] = (byte) ((encodedTabletId >>> 8) & mask);
+row[4] = (byte) (encodedTabletId & mask);
+row[5] = (byte) ((seq >>> 56) & mask);
+row[6] = (byte) ((seq >>> 48) & mask);
+row[7] = (byte) ((seq >>> 40) & mask);
+row[8] = (byte) ((seq >>> 32) & mask);
+row[9] = (byte) ((seq >>> 24) & mask);
+row[10] = (byte) ((seq >>> 16) & mask);
+row[11] = (byte) ((seq >>> 8) & mask);
+row[12] = (byte) (seq & mask);
 return row;
   }
 
@@ -289,7 +290,13 @@ public class LogFileKey implements 
WritableComparable {
* Extract the tabletId integer from the byte encoded Row.
*/
   private static int getTabletId(byte[] row) {
-int encoded = ((row[1] << 24) + (row[2] << 16) + (row[3] << 8) + row[4]);
+int mask = 0xff; // use a mask of int type to convert byte to int without 
sign extension
+// @formatter:off
+int encoded = (row[1] & mask) << 24 |
+  (row[2] & mask) << 16 |
+  (row[3] & mask) <<  8 |
+  (row[4] & mask);
+// @formatter:on
 return encoded ^ 0x8000;
   }
 
@@ -297,15 +304,16 @@ public class LogFileKey implements 
WritableComparable {
* Extract the sequence long from the byte encoded Row.
*/
   private static long getSequence(byte[] row) {
+long mask = 0xff; // use a mask of long type to convert byte to long 
without sign extension
 // @formatter:off
-return (((long) row[5] << 56) +
-((long) (row[6] & 255) << 48) +
-((long) (row[7] & 255) << 40) +
-((long) (row[8] & 255) << 32) +
-((long) (row[9] & 255) << 24) +
-((row[10] & 255) << 16) +
-((row[11] & 255) << 8) +
-((row[12] & 255)));
+return (row[5]  & mask) << 56 |
+   (row[6]  & mask) << 48 |
+   (row[7]  & mask) << 40 |
+   (row[8]  & mask) << 32 |
+   (row[9]  & mask) << 24 |
+   (row[10] & mask) << 16 |
+   (row[11] & mask) <<  8 |
+   (row[12] & mask);
 // @formatter:on
   }
 


[accumulo] branch main updated: Make sorted recovery write to RFiles (#2117)

2021-06-21 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 3e765b1  Make sorted recovery write to RFiles (#2117)
3e765b1 is described below

commit 3e765b1f3b487092206cee036097820c37b7907e
Author: Mike Miller 
AuthorDate: Mon Jun 21 09:53:33 2021 -0400

Make sorted recovery write to RFiles (#2117)

* Change LogSorter and SortedLogRecovery to write/read RFile instead of 
MapFile
* Rewrite LogSorter writeBuffer method and make static to test
* Create toKey() and fromKey() methods in LogFileKey for converting WAL keys
* Create toValue() and FromValue() methods in LogFileValue for converting 
WAL values
* Byte encode the rows of the Key to sort WAL events properly
* Make RecoveryLogsIterator convert Key Value pairs on next()

Co-authored-by: Keith Turner 
Co-authored-by: Christopher Tubbs 
---
 .../org/apache/accumulo/tserver/TabletServer.java  |   6 +-
 .../org/apache/accumulo/tserver/log/LogSorter.java |  64 +---
 .../accumulo/tserver/log/RecoveryLogReader.java|   2 +-
 .../accumulo/tserver/log/RecoveryLogsIterator.java | 124 +++
 .../accumulo/tserver/log/SortedLogRecovery.java|  63 
 .../accumulo/tserver/log/TabletServerLogger.java   |   9 +-
 .../apache/accumulo/tserver/logger/LogFileKey.java | 167 +
 .../accumulo/tserver/logger/LogFileValue.java  |  28 
 .../tserver/log/SortedLogRecoveryTest.java |  79 +++---
 .../tserver/log/TestUpgradePathForWALogs.java  |   3 +
 10 files changed, 436 insertions(+), 109 deletions(-)

diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
index 6887f4a..2ab89fa 100644
--- a/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
+++ b/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
@@ -1133,7 +1133,7 @@ public class TabletServer extends AbstractServer {
 
   public void recover(VolumeManager fs, KeyExtent extent, List 
logEntries,
   Set tabletFiles, MutationReceiver mutationReceiver) throws 
IOException {
-List recoveryLogs = new ArrayList<>();
+List recoveryDirs = new ArrayList<>();
 List sorted = new ArrayList<>(logEntries);
 sorted.sort((e1, e2) -> (int) (e1.timestamp - e2.timestamp));
 for (LogEntry entry : sorted) {
@@ -1148,9 +1148,9 @@ public class TabletServer extends AbstractServer {
 throw new IOException(
 "Unable to find recovery files for extent " + extent + " logEntry: 
" + entry);
   }
-  recoveryLogs.add(recovery);
+  recoveryDirs.add(recovery);
 }
-logger.recover(fs, extent, recoveryLogs, tabletFiles, mutationReceiver);
+logger.recover(getContext(), extent, recoveryDirs, tabletFiles, 
mutationReceiver);
   }
 
   public int createLogId() {
diff --git 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java
index 98fe935..293ff82 100644
--- 
a/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java
+++ 
b/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java
@@ -23,16 +23,20 @@ import java.io.EOFException;
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.Collections;
-import java.util.Comparator;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Map.Entry;
+import java.util.TreeMap;
 import java.util.concurrent.ThreadPoolExecutor;
 
 import org.apache.accumulo.core.Constants;
 import org.apache.accumulo.core.conf.AccumuloConfiguration;
+import org.apache.accumulo.core.conf.DefaultConfiguration;
 import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.file.FileOperations;
 import org.apache.accumulo.core.master.thrift.RecoveryStatus;
 import org.apache.accumulo.core.util.Pair;
 import org.apache.accumulo.core.util.threads.ThreadPools;
@@ -47,11 +51,12 @@ import org.apache.accumulo.tserver.logger.LogFileValue;
 import org.apache.hadoop.fs.FSDataInputStream;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.io.MapFile;
 import org.apache.zookeeper.KeeperException;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import com.google.common.annotations.VisibleForTesting;
+
 public class LogSorter {
 
   private static final Logger log = LoggerFactory.getLogger(LogSorter.class);
@@ -141,7 +146,7 @@ public class LogSorter {
 // Creating a 'finished' marker will cause recovery to proceed

[accumulo] branch main updated: Add test and fix for bug in KeyBuilder (#2170)

2021-06-17 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 1fecdd3  Add test and fix for bug in KeyBuilder (#2170)
1fecdd3 is described below

commit 1fecdd378f2bc15955baa371ac9bb706bcc94dfc
Author: Mike Miller 
AuthorDate: Thu Jun 17 09:40:49 2021 -0400

Add test and fix for bug in KeyBuilder (#2170)

* Replace error prone CharsetEncoder with toString() and getBytes()
---
 .../main/java/org/apache/accumulo/core/data/KeyBuilder.java | 13 +
 .../java/org/apache/accumulo/core/data/KeyBuilderTest.java  | 10 ++
 2 files changed, 11 insertions(+), 12 deletions(-)

diff --git a/core/src/main/java/org/apache/accumulo/core/data/KeyBuilder.java 
b/core/src/main/java/org/apache/accumulo/core/data/KeyBuilder.java
index 100a8cc..1d625b8 100644
--- a/core/src/main/java/org/apache/accumulo/core/data/KeyBuilder.java
+++ b/core/src/main/java/org/apache/accumulo/core/data/KeyBuilder.java
@@ -20,11 +20,6 @@ package org.apache.accumulo.core.data;
 
 import static java.nio.charset.StandardCharsets.UTF_8;
 
-import java.nio.CharBuffer;
-import java.nio.charset.CharacterCodingException;
-import java.nio.charset.CharsetEncoder;
-import java.nio.charset.CodingErrorAction;
-
 import org.apache.accumulo.core.security.ColumnVisibility;
 import org.apache.hadoop.io.Text;
 
@@ -326,13 +321,7 @@ public class KeyBuilder {
 }
 
 private byte[] encodeCharSequence(CharSequence chars) {
-  CharsetEncoder encoder = 
UTF_8.newEncoder().onMalformedInput(CodingErrorAction.REPORT)
-  .onUnmappableCharacter(CodingErrorAction.REPORT);
-  try {
-return encoder.encode(CharBuffer.wrap(chars)).array();
-  } catch (CharacterCodingException ex) {
-throw new RuntimeException("KeyBuilder supports only CharSequences 
encoded in UTF-8", ex);
-  }
+  return chars.toString().getBytes(UTF_8);
 }
 
 @Override
diff --git 
a/core/src/test/java/org/apache/accumulo/core/data/KeyBuilderTest.java 
b/core/src/test/java/org/apache/accumulo/core/data/KeyBuilderTest.java
index 8f795c5..70f02e5 100644
--- a/core/src/test/java/org/apache/accumulo/core/data/KeyBuilderTest.java
+++ b/core/src/test/java/org/apache/accumulo/core/data/KeyBuilderTest.java
@@ -305,4 +305,14 @@ public class KeyBuilderTest {
 keyExpected.setDeleted(false);
 assertEquals(keyExpected, keyBuilt);
   }
+
+  /**
+   * Tests bug where a String of 10 chars or longer was being encoded 
incorrectly.
+   */
+  @Test
+  public void test10CharactersBug() {
+Key keyBuilt1 = Key.builder().row(rowText).family("1234567890").build();
+Key keyBuilt2 = Key.builder().row(rowText).family(new 
Text("1234567890")).build();
+assertEquals(keyBuilt1, keyBuilt2);
+  }
 }


[accumulo] branch main updated: Fix crypto in SplitLarge utility (#2145)

2021-06-04 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 512325e  Fix crypto in SplitLarge utility (#2145)
512325e is described below

commit 512325e1ab14351df7722ca1956b14c42fff90d7
Author: Mike Miller 
AuthorDate: Fri Jun 4 13:31:01 2021 -0400

Fix crypto in SplitLarge utility (#2145)

* The option that was added in 2.0 for the encryption only included the 
name of the service. You also need a way to pass in other options for it to 
work. This change just simplifies that by removing the option altogether and 
reading the config from classpath, similar to other utilities.
---
 .../org/apache/accumulo/core/file/rfile/SplitLarge.java| 14 +-
 1 file changed, 5 insertions(+), 9 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/file/rfile/SplitLarge.java 
b/core/src/main/java/org/apache/accumulo/core/file/rfile/SplitLarge.java
index f0ca7da..580c68d 100644
--- a/core/src/main/java/org/apache/accumulo/core/file/rfile/SplitLarge.java
+++ b/core/src/main/java/org/apache/accumulo/core/file/rfile/SplitLarge.java
@@ -21,10 +21,8 @@ package org.apache.accumulo.core.file.rfile;
 import java.util.ArrayList;
 import java.util.List;
 
-import org.apache.accumulo.core.cli.Help;
+import org.apache.accumulo.core.cli.ConfigOpts;
 import org.apache.accumulo.core.conf.AccumuloConfiguration;
-import org.apache.accumulo.core.conf.ConfigurationTypeHelper;
-import org.apache.accumulo.core.conf.DefaultConfiguration;
 import org.apache.accumulo.core.conf.Property;
 import org.apache.accumulo.core.crypto.CryptoServiceFactory;
 import org.apache.accumulo.core.data.Key;
@@ -46,12 +44,10 @@ import com.beust.jcommander.Parameter;
  */
 public class SplitLarge {
 
-  static class Opts extends Help {
+  static class Opts extends ConfigOpts {
 @Parameter(names = "-m",
 description = "the maximum size of the key/value pair to shunt to the 
small file")
 long maxSize = 10 * 1024 * 1024;
-@Parameter(names = "-crypto", description = "the class to perform 
encryption/decryption")
-String cryptoClass = Property.INSTANCE_CRYPTO_SERVICE.getDefaultValue();
 @Parameter(description = " {  ... }")
 List files = new ArrayList<>();
   }
@@ -63,9 +59,9 @@ public class SplitLarge {
 opts.parseArgs(SplitLarge.class.getName(), args);
 
 for (String file : opts.files) {
-  AccumuloConfiguration aconf = DefaultConfiguration.getInstance();
-  CryptoService cryptoService = 
ConfigurationTypeHelper.getClassInstance(null, opts.cryptoClass,
-  CryptoService.class, CryptoServiceFactory.newDefaultInstance());
+  AccumuloConfiguration aconf = opts.getSiteConfiguration();
+  CryptoService cryptoService =
+  CryptoServiceFactory.newInstance(aconf, 
CryptoServiceFactory.ClassloaderType.JAVA);
   Path path = new Path(file);
   CachableBuilder cb =
   new CachableBuilder().fsPath(fs, 
path).conf(conf).cryptoService(cryptoService);


[accumulo] branch main updated (af2d2d6 -> b15fbf3)

2021-05-21 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from af2d2d6  Improve LocalityGroupUtil.java (#2112)
 add b15fbf3  Move logic in LogSorter outside sort method (#2113)

No new revisions were added by this update.

Summary of changes:
 .../org/apache/accumulo/tserver/log/LogSorter.java | 119 ++---
 .../tserver/log/TestUpgradePathForWALogs.java  |  97 +
 .../tserver/src/test/resources/walog-from-20.walog | Bin 0 -> 982 bytes
 3 files changed, 108 insertions(+), 108 deletions(-)
 create mode 100755 server/tserver/src/test/resources/walog-from-20.walog


[accumulo-website] branch main updated: Update people.md (#283)

2021-05-20 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new 8f0de12  Update people.md (#283)
8f0de12 is described below

commit 8f0de12ec2f3190c0fbedcb0ff64aa41884f559a
Author: nicolasalarconrapela 
AuthorDate: Thu May 20 17:20:57 2021 +0300

Update people.md (#283)
---
 pages/people.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/pages/people.md b/pages/people.md
index 7c0cb80..80e142c 100644
--- a/pages/people.md
+++ b/pages/people.md
@@ -147,6 +147,7 @@ GitHub also has a [contributor list][github-contributors] 
based on commits.
 | Mike Fagan  | [Arcus Research][ARCUS]
   | [MT][MT]  |
 | Morgan Haskel   |
   |   |
 | Nguessan Kouame |
   |   |
+| Nicolás Alarcón R.  |
   | [CEST][CEST]  |
 | Oren Falkowitz  | [sqrrl][SQRRL] 
   | [ET][ET]  |
 | Phil Eberhardt  | [sqrrl][SQRRL] 
   | [ET][ET]  |
 | Philip Young|
   |   |


[accumulo] branch main updated: Fix property JSON strings (#2109)

2021-05-14 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 763cf4f  Fix property JSON strings (#2109)
763cf4f is described below

commit 763cf4fae5d93aff96a06458f98dfe7296b5b882
Author: Mike Miller 
AuthorDate: Fri May 14 14:10:01 2021 -0400

Fix property JSON strings (#2109)

* Apply replaceAll for quotes on all parts of the default JSON values in
Property.java. Closes #2105
---
 .../main/java/org/apache/accumulo/core/conf/Property.java   | 13 ++---
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/core/src/main/java/org/apache/accumulo/core/conf/Property.java 
b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
index 30b9ce1..1b14a8c 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
@@ -442,8 +442,8 @@ public enum Property {
   "The maximum number of files a compaction will open"),
   TSERV_COMPACTION_SERVICE_ROOT_EXECUTORS(
   "tserver.compaction.major.service.root.planner.opts.executors",
-  "[{'name':'small','maxSize':'32M','numThreads':1},"
-  + "{'name':'huge','numThreads':1}]".replaceAll("'", "\""),
+  
("[{'name':'small','maxSize':'32M','numThreads':1},{'name':'huge','numThreads':1}]")
+  .replaceAll("'", "\""),
   PropertyType.STRING,
   "See {% jlink -f 
org.apache.accumulo.core.spi.compaction.DefaultCompactionPlanner %} "),
   
TSERV_COMPACTION_SERVICE_META_PLANNER("tserver.compaction.major.service.meta.planner",
@@ -458,8 +458,8 @@ public enum Property {
   "The maximum number of files a compaction will open"),
   TSERV_COMPACTION_SERVICE_META_EXECUTORS(
   "tserver.compaction.major.service.meta.planner.opts.executors",
-  "[{'name':'small','maxSize':'32M','numThreads':2},"
-  + "{'name':'huge','numThreads':2}]".replaceAll("'", "\""),
+  
("[{'name':'small','maxSize':'32M','numThreads':2},{'name':'huge','numThreads':2}]")
+  .replaceAll("'", "\""),
   PropertyType.STRING,
   "See {% jlink -f 
org.apache.accumulo.core.spi.compaction.DefaultCompactionPlanner %} "),
   
TSERV_COMPACTION_SERVICE_DEFAULT_PLANNER("tserver.compaction.major.service.default.planner",
@@ -474,9 +474,8 @@ public enum Property {
   "The maximum number of files a compaction will open"),
   TSERV_COMPACTION_SERVICE_DEFAULT_EXECUTORS(
   "tserver.compaction.major.service.default.planner.opts.executors",
-  "[{'name':'small','maxSize':'32M','numThreads':2},"
-  + "{'name':'medium','maxSize':'128M','numThreads':2},"
-  + "{'name':'large','numThreads':2}]".replaceAll("'", "\""),
+  
("[{'name':'small','maxSize':'32M','numThreads':2},{'name':'medium','maxSize':'128M','numThreads':2},{'name':'large','numThreads':2}]")
+  .replaceAll("'", "\""),
   PropertyType.STRING,
   "See {% jlink -f 
org.apache.accumulo.core.spi.compaction.DefaultCompactionPlanner %} "),
   @Deprecated(since = "2.1.0", forRemoval = true)


[accumulo] branch main updated (a28b27a -> 3c039c7)

2021-05-12 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git.


from a28b27a  Add extent to metadata update failure logs (#2086)
 add 3c039c7  Add option to disable encryption to AESCryptoService (#2064)

No new revisions were added by this update.

Summary of changes:
 .../accumulo/core/spi/crypto/AESCryptoService.java | 25 ++---
 .../apache/accumulo/core/crypto/CryptoTest.java| 41 ++
 .../apache/accumulo/core/file/rfile/RFileTest.java |  5 +++
 3 files changed, 66 insertions(+), 5 deletions(-)


[accumulo] branch main updated: Fix #2070 address ShellServerIT test failures (#2087)

2021-05-10 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new a0af230  Fix #2070 address ShellServerIT test failures (#2087)
a0af230 is described below

commit a0af230ab65039ba4eda5e2d6ee415f89f08914c
Author: EdColeman 
AuthorDate: Mon May 10 12:46:29 2021 -0400

Fix #2070 address ShellServerIT test failures (#2087)

- address simple checkstyle errors
- use unique table and namespace names
- Add test for constraints jar built with 2_1
---
 .../org/apache/accumulo/test/ShellServerIT.java| 465 +++--
 test/src/main/resources/FooConstraint_2_1.jar  | Bin 0 -> 2018 bytes
 2 files changed, 252 insertions(+), 213 deletions(-)

diff --git a/test/src/main/java/org/apache/accumulo/test/ShellServerIT.java 
b/test/src/main/java/org/apache/accumulo/test/ShellServerIT.java
index bd166af..081e984 100644
--- a/test/src/main/java/org/apache/accumulo/test/ShellServerIT.java
+++ b/test/src/main/java/org/apache/accumulo/test/ShellServerIT.java
@@ -375,7 +375,8 @@ public class ShellServerIT extends SharedMiniClusterBase {
   @SuppressFBWarnings(value = "PATH_TRAVERSAL_IN", justification = "path 
provided by test")
   @Test
   public void exporttableImporttable() throws Exception {
-final String table = name.getMethodName(), table2 = table + "2";
+final String table = getUniqueNames(1)[0];
+final String table2 = table + "2";
 
 // exporttable / importtable
 ts.exec("createtable " + table + " -evc", true);
@@ -452,7 +453,7 @@ public class ShellServerIT extends SharedMiniClusterBase {
 
   @Test
   public void setscaniterDeletescaniter() throws Exception {
-final String table = name.getMethodName();
+final String table = getUniqueNames(1)[0];
 
 // setscaniter, deletescaniter
 ts.exec("createtable " + table);
@@ -482,7 +483,7 @@ public class ShellServerIT extends SharedMiniClusterBase {
 
   @Test
   public void egrep() throws Exception {
-final String table = name.getMethodName();
+final String table = getUniqueNames(1)[0];
 
 // egrep
 ts.exec("createtable " + table);
@@ -494,7 +495,7 @@ public class ShellServerIT extends SharedMiniClusterBase {
 
   @Test
   public void du() throws Exception {
-final String table = name.getMethodName();
+final String table = getUniqueNames(1)[0];
 
 // create and delete a table so we get out of a table context in the shell
 ts.exec("notable", true);
@@ -513,7 +514,7 @@ public class ShellServerIT extends SharedMiniClusterBase {
 String o = ts.output.get();
 // for some reason, there's a bit of fluctuation
 assertTrue("Output did not match regex: '" + o + "'",
-o.matches(".*[1-9][0-9][0-9]\\s\\[" + table + "\\]\\n"));
+o.matches(".*[1-9][0-9][0-9]\\s\\[" + table + "]\\n"));
 ts.exec("deletetable -f " + table);
   }
 
@@ -535,7 +536,7 @@ public class ShellServerIT extends SharedMiniClusterBase {
 
   @Test
   public void user() throws Exception {
-final String table = name.getMethodName();
+final String table = getUniqueNames(1)[0];
 final boolean kerberosEnabled = getToken() instanceof KerberosToken;
 
 // createuser, deleteuser, user, users, droptable, grant, revoke
@@ -577,7 +578,7 @@ public class ShellServerIT extends SharedMiniClusterBase {
 
   @Test
   public void durability() throws Exception {
-final String table = name.getMethodName();
+final String table = getUniqueNames(1)[0];
 ts.exec("createtable " + table);
 ts.exec("insert -d none a cf cq randomGunkaASDFWEAQRd");
 ts.exec("insert -d foo a cf cq2 2", false, "foo", true);
@@ -587,7 +588,7 @@ public class ShellServerIT extends SharedMiniClusterBase {
 
   @Test
   public void iter() throws Exception {
-final String table = name.getMethodName();
+final String table = getUniqueNames(1)[0];
 
 // setshelliter, listshelliter, deleteshelliter
 ts.exec("createtable " + table);
@@ -637,9 +638,10 @@ public class ShellServerIT extends SharedMiniClusterBase {
   @Test
   public void setIterOptionPrompt() throws Exception {
 try (AccumuloClient client = 
Accumulo.newClient().from(getClientProps()).build()) {
-  String tableName = name.getMethodName();
+  final String[] tableNames = getUniqueNames(4);
+  final String tableName0 = tableNames[0];
 
-  ts.exec("createtable " + tableName);
+  ts.exec("createtable " + tableName0);
   ts.input.set("\n\n");
   // Setting a non-optiondescriber with no name should fail
   ts.exec("setiter -scan -class " + COLUMN_FAMILY_COUNTER_ITERATOR + "

[accumulo] branch main updated: Create MockServerContext for use in unit tests (#2083)

2021-05-06 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 6a0a5ee  Create MockServerContext for use in unit tests (#2083)
6a0a5ee is described below

commit 6a0a5ee30788f9f606166b8391b22c23d65071dc
Author: Mike Miller 
AuthorDate: Thu May 6 15:13:54 2021 -0400

Create MockServerContext for use in unit tests (#2083)
---
 .../apache/accumulo/server/MockServerContext.java  | 55 ++
 .../accumulo/server/client/BulkImporterTest.java   | 11 +
 .../server/conf/NamespaceConfigurationTest.java| 10 +---
 .../conf/ServerConfigurationFactoryTest.java   | 16 +--
 .../server/conf/TableConfigurationTest.java| 10 +---
 .../server/conf/ZooConfigurationFactoryTest.java   |  7 +--
 .../master/balancer/TableLoadBalancerTest.java | 10 +---
 .../server/problems/ProblemReportTest.java |  4 +-
 .../security/handler/ZKAuthenticatorTest.java  |  7 ++-
 9 files changed, 72 insertions(+), 58 deletions(-)

diff --git 
a/server/base/src/test/java/org/apache/accumulo/server/MockServerContext.java 
b/server/base/src/test/java/org/apache/accumulo/server/MockServerContext.java
new file mode 100644
index 000..64694e8
--- /dev/null
+++ 
b/server/base/src/test/java/org/apache/accumulo/server/MockServerContext.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server;
+
+import static org.easymock.EasyMock.expect;
+
+import java.util.Properties;
+
+import org.apache.accumulo.core.conf.ConfigurationCopy;
+import org.apache.accumulo.core.conf.DefaultConfiguration;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.crypto.CryptoServiceFactory;
+import org.easymock.EasyMock;
+
+/**
+ * Get a generic mocked ServerContext with junk for testing.
+ */
+public class MockServerContext {
+
+  public static ServerContext get() {
+ServerContext sc = EasyMock.createMock(ServerContext.class);
+ConfigurationCopy conf = new 
ConfigurationCopy(DefaultConfiguration.getInstance());
+conf.set(Property.INSTANCE_VOLUMES, "file:///");
+expect(sc.getConfiguration()).andReturn(conf).anyTimes();
+
expect(sc.getCryptoService()).andReturn(CryptoServiceFactory.newDefaultInstance()).anyTimes();
+expect(sc.getProperties()).andReturn(new Properties()).anyTimes();
+return sc;
+  }
+
+  public static ServerContext getWithZK(String instanceID, String zk, int 
zkTimeout) {
+var sc = get();
+expect(sc.getZooKeeperRoot()).andReturn("/accumulo/" + 
instanceID).anyTimes();
+expect(sc.getInstanceID()).andReturn(instanceID).anyTimes();
+expect(sc.getZooKeepers()).andReturn(zk).anyTimes();
+expect(sc.getZooKeepersSessionTimeOut()).andReturn(zkTimeout).anyTimes();
+return sc;
+  }
+
+}
diff --git 
a/server/base/src/test/java/org/apache/accumulo/server/client/BulkImporterTest.java
 
b/server/base/src/test/java/org/apache/accumulo/server/client/BulkImporterTest.java
index 566596d..b0d17be 100644
--- 
a/server/base/src/test/java/org/apache/accumulo/server/client/BulkImporterTest.java
+++ 
b/server/base/src/test/java/org/apache/accumulo/server/client/BulkImporterTest.java
@@ -31,9 +31,6 @@ import java.util.TreeSet;
 import org.apache.accumulo.core.clientImpl.ClientContext;
 import org.apache.accumulo.core.clientImpl.TabletLocator;
 import org.apache.accumulo.core.clientImpl.TabletLocator.TabletLocation;
-import org.apache.accumulo.core.conf.ConfigurationCopy;
-import org.apache.accumulo.core.conf.DefaultConfiguration;
-import org.apache.accumulo.core.conf.Property;
 import org.apache.accumulo.core.crypto.CryptoServiceFactory;
 import org.apache.accumulo.core.data.Key;
 import org.apache.accumulo.core.data.Mutation;
@@ -43,6 +40,7 @@ import org.apache.accumulo.core.data.Value;
 import org.apache.accumulo.core.dataImpl.KeyExtent;
 import org.apache.accumulo.core.file.FileOperations;
 import org.apache.accumulo.core.file.FileSKVWriter;
+import org.apache.accumulo.server.MockServerContext;
 import org.apach

[accumulo] branch main updated: Fix for Constraint API (#2074)

2021-05-03 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 6094a53  Fix for Constraint API (#2074)
6094a53 is described below

commit 6094a537421f7b2d328c8d1d016a2bf830743d4c
Author: Mike Miller 
AuthorDate: Mon May 3 17:13:04 2021 -0400

Fix for Constraint API (#2074)

* Add method to deprecated API to prevent errors
* Add test to DeprecatedConstraintExtendTest
---
 .../accumulo/core/constraints/Constraint.java  | 11 +++
 .../DeprecatedConstraintExtendTest.java| 38 --
 2 files changed, 47 insertions(+), 2 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/constraints/Constraint.java 
b/core/src/main/java/org/apache/accumulo/core/constraints/Constraint.java
index 3414970..b24d613 100644
--- a/core/src/main/java/org/apache/accumulo/core/constraints/Constraint.java
+++ b/core/src/main/java/org/apache/accumulo/core/constraints/Constraint.java
@@ -84,4 +84,15 @@ public interface Constraint extends 
org.apache.accumulo.core.data.constraints.Co
* @return list of violation codes, or null if none
*/
   List check(Environment env, Mutation mutation);
+
+  /**
+   * Implemented for backwards compatibility.
+   *
+   * @since 2.1.0
+   */
+  @Override
+  default List 
check(org.apache.accumulo.core.data.constraints.Constraint.Environment env,
+  Mutation mutation) {
+return check((Environment) env, mutation);
+  }
 }
diff --git 
a/core/src/test/java/org/apache/accumulo/core/constraints/DeprecatedConstraintExtendTest.java
 
b/core/src/test/java/org/apache/accumulo/core/constraints/DeprecatedConstraintExtendTest.java
index aa7ef99..dac943d 100644
--- 
a/core/src/test/java/org/apache/accumulo/core/constraints/DeprecatedConstraintExtendTest.java
+++ 
b/core/src/test/java/org/apache/accumulo/core/constraints/DeprecatedConstraintExtendTest.java
@@ -40,13 +40,12 @@ import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
 @SuppressWarnings("deprecation")
 public class DeprecatedConstraintExtendTest {
 
-  Constraint constraint = new MinKeySizeConstraint();
-
   byte[] min = new byte[1024];
   byte[] oversized = new byte[1048577];
 
   @Test
   public void testMinKeySizeConstraint() {
+Constraint constraint = new MinKeySizeConstraint();
 
 // pass constraints
 Mutation m = new Mutation(min);
@@ -67,6 +66,20 @@ public class DeprecatedConstraintExtendTest {
 constraint.check(null, m));
   }
 
+  @Test
+  public void testFoo() {
+FooConstraint fc = new FooConstraint();
+// pass constraints
+Mutation m = new Mutation("blah");
+m.put("colf", "colq", new Value(new byte[] {}));
+assertEquals(null, fc.check(null, m));
+
+// test fail constraint
+m = new Mutation("foo");
+m.put("colf", "colq", new Value(new byte[] {}));
+assertEquals(Collections.singletonList(Short.valueOf("1")), fc.check(null, 
m));
+  }
+
   /**
* Limit the size of 1mb but also a minimum of 1KB
*/
@@ -94,4 +107,25 @@ public class DeprecatedConstraintExtendTest {
   return violations;
 }
   }
+
+  /**
+   * Test previously defined constraint.
+   */
+  public class FooConstraint implements Constraint {
+public String getViolationDescription(short violationCode) {
+  switch (violationCode) {
+case 1:
+  return "Contains foo";
+  }
+  throw new IllegalArgumentException();
+}
+
+public List check(Constraint.Environment env, Mutation mutation) {
+  if (new String(mutation.getRow()).contains("foo")) {
+return Collections.singletonList(Short.valueOf("1"));
+  }
+  return null;
+}
+  }
+
 }


[accumulo] branch main updated: Fix redundant TableId creation in TableLoadBalancer (#2060)

2021-04-30 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new 4213d76  Fix redundant TableId creation in TableLoadBalancer (#2060)
4213d76 is described below

commit 4213d76c2b5fcbeeb4163db04c1fb8b7226cfe22
Author: Mike Miller 
AuthorDate: Fri Apr 30 09:34:51 2021 -0400

Fix redundant TableId creation in TableLoadBalancer (#2060)
---
 .../org/apache/accumulo/core/spi/balancer/TableLoadBalancer.java| 6 ++
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git 
a/core/src/main/java/org/apache/accumulo/core/spi/balancer/TableLoadBalancer.java
 
b/core/src/main/java/org/apache/accumulo/core/spi/balancer/TableLoadBalancer.java
index f046984..02aef99 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/spi/balancer/TableLoadBalancer.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/spi/balancer/TableLoadBalancer.java
@@ -108,10 +108,8 @@ public class TableLoadBalancer implements TabletBalancer {
   public void getAssignments(AssignmentParameters params) {
 // separate the unassigned into tables
 Map> groupedUnassigned = new 
HashMap<>();
-params.unassignedTablets()
-.forEach((tid, lastTserver) -> groupedUnassigned
-.computeIfAbsent(TableId.of(tid.getTable().canonical()), k -> new 
HashMap<>())
-.put(tid, lastTserver));
+params.unassignedTablets().forEach((tid, lastTserver) -> groupedUnassigned
+.computeIfAbsent(tid.getTable(), k -> new HashMap<>()).put(tid, 
lastTserver));
 for (Entry> e : 
groupedUnassigned.entrySet()) {
   Map newAssignments = new HashMap<>();
   getBalancerForTable(e.getKey()).getAssignments(


[accumulo-website] branch asf-site updated (c611f61 -> a0db284)

2021-04-29 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git.


from c611f61  Automatic Site Publish by Buildbot
 add 9c2604e  Automatic Site Publish by Buildbot
 add a0db284  Automatic Site Publish by Buildbot

No new revisions were added by this update.

Summary of changes:
 output/feed.xml | 4 ++--
 output/how-to-contribute/index.html | 2 +-
 output/people/index.html| 5 +
 3 files changed, 8 insertions(+), 3 deletions(-)


[accumulo-website] branch main updated: Add build command to pages/how-to-contribute.md (#276)

2021-04-29 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git


The following commit(s) were added to refs/heads/main by this push:
 new b670b9a  Add build command to pages/how-to-contribute.md (#276)
b670b9a is described below

commit b670b9af0b054629db930c340fa747814e5829aa
Author: Mike Miller 
AuthorDate: Thu Apr 29 09:13:33 2021 -0400

Add build command to pages/how-to-contribute.md (#276)
---
 pages/how-to-contribute.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pages/how-to-contribute.md b/pages/how-to-contribute.md
index 7537bae..7857232 100644
--- a/pages/how-to-contribute.md
+++ b/pages/how-to-contribute.md
@@ -40,7 +40,7 @@ Any questions/ideas don't hesitate to [contact us][contact].
 git checkout -b accumulo-4321
 ```
 1. Do work and commit to your branch. You can reference [this link][messages] 
for a guide on how to write good commit log messages.
-1. Ensure you works satisfies the guidelines laid out in the `CONTRIBUTING.md` 
file.
+1. Ensure your branch builds cleanly using the command: ```mvn clean verify 
-DskipITs```
 1. If needed, squash to the minimum number of commits. For help on squashing 
commits, see this [tutorial][squash-tutorial] or [StackOverflow 
answer][squash-stack].
 1. [Push] your branch to your fork.
 ```


[accumulo] branch main updated: Migrate Constraints to API (#1985)

2021-04-28 Thread mmiller
This is an automated email from the ASF dual-hosted git repository.

mmiller pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git


The following commit(s) were added to refs/heads/main by this push:
 new d3b41fc  Migrate Constraints to API (#1985)
d3b41fc is described below

commit d3b41fc0ac5da70aab86e6fb95852353b9a167aa
Author: Mike Miller 
AuthorDate: Wed Apr 28 14:25:18 2021 -0400

Migrate Constraints to API (#1985)

* Create new Constraint API and equivalent classes under the
o.a.a.core.data.constraints package to become public API
* Deprecated Constraint and classes under o.a.a.core.constraints package
and make the old classes extend the new API
* Migrate tests to go along with API changes
* Create DeprecatedConstraintExtendTest to test user extending old API
* Add DefaultKeySizeConstraintTest to reuseForks exceptions
---
 core/pom.xml   |  2 +
 .../core/clientImpl/NamespaceOperationsImpl.java   |  2 +-
 .../core/clientImpl/TableOperationsImpl.java   |  2 +-
 .../apache/accumulo/core/conf/IterConfigUtil.java  |  2 +-
 .../org/apache/accumulo/core/conf/Property.java|  2 +-
 .../accumulo/core/constraints/Constraint.java  | 26 ++
 .../core/constraints/DefaultKeySizeConstraint.java | 13 ++-
 .../core/constraints/NoDeleteConstraint.java   | 12 ++-
 .../core/constraints/VisibilityConstraint.java | 13 ++-
 .../core/{ => data}/constraints/Constraint.java| 19 +++--
 .../constraints/DefaultKeySizeConstraint.java  |  4 +-
 .../{ => data}/constraints/NoDeleteConstraint.java |  4 +-
 .../constraints/VisibilityConstraint.java  |  4 +-
 .../constraints/DefaultKeySizeConstraintTest.java  |  1 +
 .../DeprecatedConstraintExtendTest.java| 97 ++
 .../core/constraints/NoDeleteConstraintTest.java   |  1 +
 .../VisibilityConstraintTest.java  | 11 +--
 .../constraints/DefaultKeySizeConstraintTest.java  |  7 +-
 .../constraints/NoDeleteConstraintTest.java|  2 +-
 .../constraints}/VisibilityConstraintTest.java |  7 +-
 .../server/constraints/MetadataConstraints.java|  2 +-
 .../server/constraints/SystemEnvironment.java  |  2 +-
 .../server/metadata/RootTabletMutatorImpl.java | 15 +++-
 .../accumulo/tserver/TservConstraintEnv.java   | 21 +++--
 .../tserver/constraints/ConstraintChecker.java | 20 -
 .../tserver/constraints/SystemConstraint.java  |  2 +-
 .../constraints/UnsatisfiableConstraint.java   |  2 +-
 .../tserver/constraints/ConstraintCheckerTest.java | 34 
 .../accumulo/shell/commands/ConstraintCommand.java |  2 +-
 .../shell/commands/CreateTableCommand.java |  2 +-
 .../apache/accumulo/test/TableOperationsIT.java|  2 +-
 .../test/constraints/AlphaNumKeyConstraint.java|  2 +-
 .../accumulo/test/constraints/MaxMutationSize.java |  2 +-
 .../test/constraints/NumericValueConstraint.java   |  2 +-
 .../accumulo/test/functional/SlowConstraint.java   |  2 +-
 35 files changed, 252 insertions(+), 91 deletions(-)

diff --git a/core/pom.xml b/core/pom.xml
index 36d9dc6..e546886 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -206,6 +206,7 @@
   **/IdleRatioScanPrioritizerTest.java
   **/TestCfCqSlice.java
   **/TestCfCqSliceSeekingFilter.java
+  
**/core/constraints/DefaultKeySizeConstraintTest.java
 
   
   
@@ -223,6 +224,7 @@
   **/IdleRatioScanPrioritizerTest.java
   **/TestCfCqSlice.java
   **/TestCfCqSliceSeekingFilter.java
+  
**/core/constraints/DefaultKeySizeConstraintTest.java
 
   
 
diff --git 
a/core/src/main/java/org/apache/accumulo/core/clientImpl/NamespaceOperationsImpl.java
 
b/core/src/main/java/org/apache/accumulo/core/clientImpl/NamespaceOperationsImpl.java
index 126b977..979d891 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/clientImpl/NamespaceOperationsImpl.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/clientImpl/NamespaceOperationsImpl.java
@@ -49,8 +49,8 @@ import org.apache.accumulo.core.client.admin.TableOperations;
 import org.apache.accumulo.core.clientImpl.thrift.SecurityErrorCode;
 import org.apache.accumulo.core.clientImpl.thrift.ThriftSecurityException;
 import 
org.apache.accumulo.core.clientImpl.thrift.ThriftTableOperationException;
-import org.apache.accumulo.core.constraints.Constraint;
 import org.apache.accumulo.core.data.NamespaceId;
+import org.apache.accumulo.core.data.constraints.Constraint;
 import org.apache.accumulo.core.iterators.IteratorUtil.IteratorScope;
 import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
 import org.apache.accumulo.core.manager.thrift.FateOperation;
diff --git 
a/core/src/main/java/org/apache/accumulo/core/clientImpl/TableOperationsImpl.java
 
b/core/src/main/java/org/

  1   2   3   4   5   6   7   8   9   10   >