(accumulo) branch main updated: Stronger type checking and other WAL improvements (#4082)
This is an automated email from the ASF dual-hosted git repository.
ctubbsii pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/main by this push:
new f9897862dd Stronger type checking and other WAL improvements (#4082)
f9897862dd is described below
commit f9897862dd4e6ff4892239ff5ebeb8ed6e34bc68
Author: Christopher Tubbs
AuthorDate: Tue Dec 19 01:24:23 2023 -0500
Stronger type checking and other WAL improvements (#4082)
* Use LogEntry type in more places where just the logEntry's path as
a String was being used
* Keep the UUID and HostAndPort in LogEntry that are created for
validation, rather than their String equivalents
* Remove no longer needed deleteWal(String) and create a
deleteFromMutation method on LogEntry to keep the schema
serialization/deserialization in LogEntry (may want to move that to
LogColumnFamily class in the MetadataSchema later, but for now, at
least it's in one place)
* Use private constructor in LogEntry, do validation in a fromPath
static method before calling the constructor with the strongly typed
results
* Update javadocs to clarify the path format
* Fix the validation of the host+port in the path (it's a plus sign
not a colon, because colons weren't valid characters in HDFS, so we
used a plus instead)
* Simplify VolumeUtil switchVolumes logic that was previously
complicated because an old LogEntry format could contain multiple
paths per entry and that's no longer the case
* Clean up DfsLogger a little bit to remove some old stuff that wasn't
needed: remove ServerResources used for testing when a mock
ServerContext would suffice, pass LogEntry instead of Strings for the
path and its metadata format (the meta reference being passed
wasn't actually what was in the table anyway, it was coming from
LogEntry, which was computing it, and Ample wasn't using this meta
for deletions/cleanup anyway - may be an existing bug; if so, that
will be a follow-on), stop using toString() to compute serialization
and remove the redundant use of the logger name itself to try to
compute the serialization in that toString() method
* Simplify names to use "path" instead of "filePath" or "filename"
* Remove unneeded toString calls
* Avoid use of Java NIO Path type in LogEntryTest, which is confusing
since the paths actually come from HDFS Path
* Add check to verify `+` exists for port delimiter
---
.../apache/accumulo/core/logging/TabletLogger.java | 3 +-
.../accumulo/core/metadata/schema/Ample.java | 4 +-
.../accumulo/core/tabletserver/log/LogEntry.java | 138 +---
.../core/metadata/schema/TabletMetadataTest.java | 7 +-
.../core/tabletserver/log/LogEntryTest.java| 139 ++---
.../org/apache/accumulo/server/fs/VolumeUtil.java | 21 +---
.../server/manager/state/MetaDataStateStore.java | 2 +-
.../server/manager/state/MetaDataTableScanner.java | 2 +-
.../server/manager/state/ZooTabletStateStore.java | 6 +-
.../server/metadata/TabletMutatorBase.java | 10 +-
.../accumulo/server/util/ListVolumesUsed.java | 2 +-
.../accumulo/server/util/ManagerMetadataUtil.java | 3 +-
.../apache/accumulo/server/fs/VolumeUtilTest.java | 12 +-
.../accumulo/tserver/TabletClientHandler.java | 8 +-
.../org/apache/accumulo/tserver/TabletServer.java | 17 +--
.../org/apache/accumulo/tserver/log/DfsLogger.java | 91 +-
.../accumulo/tserver/log/TabletServerLogger.java | 33 +++--
.../accumulo/tserver/tablet/DatafileManager.java | 3 +-
.../org/apache/accumulo/tserver/tablet/Tablet.java | 35 +++---
.../accumulo/tserver/WalRemovalOrderTest.java | 43 ---
.../test/MissingWalHeaderCompletesRecoveryIT.java | 4 +-
.../apache/accumulo/test/manager/MergeStateIT.java | 5 +-
22 files changed, 281 insertions(+), 307 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/logging/TabletLogger.java
b/core/src/main/java/org/apache/accumulo/core/logging/TabletLogger.java
index e632971cf5..44832db7f1 100644
--- a/core/src/main/java/org/apache/accumulo/core/logging/TabletLogger.java
+++ b/core/src/main/java/org/apache/accumulo/core/logging/TabletLogger.java
@@ -23,6 +23,7 @@ import static java.util.stream.Collectors.toList;
import java.util.Collection;
import java.util.List;
import java.util.Optional;
+import java.util.UUID;
import java.util.concurrent.TimeUnit;
import org.apache.accumulo.core.client.admin.CompactionConfig;
@@ -161,7 +162,7 @@ public class TabletLogger {
public static void recovering(KeyExtent extent, List logEntries) {
if (recoveryLog.isDebugEnabled()) {
- List logIds =
logEntries.stream().map(LogEntry::getUniqueID).collect(toList());
+ List logIds =
(accumulo) branch elasticity updated: Enables CompactionExecutorIT tests (#4080)
This is an automated email from the ASF dual-hosted git repository.
ddanielr pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 5ed277 Enables CompactionExecutorIT tests (#4080)
5ed277 is described below
commit 5ed277a3ad2773551860d7f38b3470bbefb1
Author: Daniel Roberts
AuthorDate: Mon Dec 18 22:26:42 2023 -0500
Enables CompactionExecutorIT tests (#4080)
* Adds additional method to stop compactors by resource group names.
* Remove failing compaction selector test
-
Co-authored-by: Keith Turner
---
.../MiniAccumuloClusterControl.java| 20 +++
.../test/compaction/CompactionExecutorIT.java | 165 ++---
2 files changed, 134 insertions(+), 51 deletions(-)
diff --git
a/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterControl.java
b/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterControl.java
index 6d3b36026e..b3977dab00 100644
---
a/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterControl.java
+++
b/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterControl.java
@@ -254,6 +254,26 @@ public class MiniAccumuloClusterControl implements
ClusterControl {
stop(server, null);
}
+ public void stopCompactorGroup(String compactorResourceGroup) {
+synchronized (compactorProcesses) {
+ var group = compactorProcesses.get(compactorResourceGroup);
+ if (group == null) {
+return;
+ }
+ group.forEach(process -> {
+try {
+ cluster.stopProcessWithTimeout(process, 30, TimeUnit.SECONDS);
+} catch (ExecutionException | TimeoutException e) {
+ log.warn("Compactor did not fully stop after 30 seconds", e);
+ throw new RuntimeException(e);
+} catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+}
+ });
+ compactorProcesses.remove(compactorResourceGroup);
+}
+ }
+
@Override
public synchronized void stop(ServerType server, String hostname) throws
IOException {
switch (server) {
diff --git
a/test/src/main/java/org/apache/accumulo/test/compaction/CompactionExecutorIT.java
b/test/src/main/java/org/apache/accumulo/test/compaction/CompactionExecutorIT.java
index a7309cb693..19f6d0bb30 100644
---
a/test/src/main/java/org/apache/accumulo/test/compaction/CompactionExecutorIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/compaction/CompactionExecutorIT.java
@@ -18,6 +18,7 @@
*/
package org.apache.accumulo.test.compaction;
+import static org.apache.accumulo.core.util.LazySingletons.GSON;
import static org.apache.accumulo.core.util.LazySingletons.RANDOM;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
@@ -29,7 +30,7 @@ import java.io.UncheckedIOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.EnumSet;
-import java.util.HashMap;
+import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
@@ -38,6 +39,7 @@ import java.util.SortedSet;
import java.util.TreeMap;
import java.util.TreeSet;
import java.util.stream.Collectors;
+import java.util.stream.Stream;
import org.apache.accumulo.core.client.Accumulo;
import org.apache.accumulo.core.client.AccumuloClient;
@@ -67,36 +69,50 @@ import
org.apache.accumulo.core.spi.compaction.CompactionExecutorId;
import org.apache.accumulo.core.spi.compaction.CompactionKind;
import org.apache.accumulo.core.spi.compaction.CompactionPlan;
import org.apache.accumulo.core.spi.compaction.CompactionPlanner;
+import org.apache.accumulo.harness.MiniClusterConfigurationCallback;
import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.miniclusterImpl.MiniAccumuloConfigImpl;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.RawLocalFileSystem;
import org.apache.hadoop.io.Text;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
-import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
-@Disabled // ELASTICITY_TODO
public class CompactionExecutorIT extends SharedMiniClusterBase {
+ public static final List compactionGroups = new LinkedList<>();
+ public static final Logger log =
LoggerFactory.getLogger(CompactionExecutorIT.class);
public static class TestPlanner implements CompactionPlanner {
+private static class ExecutorConfig {
+ String group;
+
(accumulo) branch elasticity updated: Move code from Manager class to compaction coordinator (#4011)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 4977f1277c Move code from Manager class to compaction coordinator
(#4011)
4977f1277c is described below
commit 4977f1277c1a8ecefefaa804ba40cabea3d90daf
Author: Keith Turner
AuthorDate: Mon Dec 18 19:35:57 2023 -0500
Move code from Manager class to compaction coordinator (#4011)
Moves code related to compaction coordination out of the manager class and
into the compaction coordinator class.
Co-authored-by: Daniel Roberts
---
.../java/org/apache/accumulo/manager/Manager.java | 42 +++---
.../accumulo/manager/TabletGroupWatcher.java | 2 +-
.../coordinator/CompactionCoordinator.java | 66 --
.../coordinator}/QueueMetrics.java | 13 +++--
.../accumulo/manager/metrics/ManagerMetrics.java | 3 -
5 files changed, 66 insertions(+), 60 deletions(-)
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/Manager.java
b/server/manager/src/main/java/org/apache/accumulo/manager/Manager.java
index b23e0a6bdb..2418dba1fe 100644
--- a/server/manager/src/main/java/org/apache/accumulo/manager/Manager.java
+++ b/server/manager/src/main/java/org/apache/accumulo/manager/Manager.java
@@ -113,7 +113,6 @@ import org.apache.accumulo.core.util.Retry;
import org.apache.accumulo.core.util.threads.ThreadPools;
import org.apache.accumulo.core.util.threads.Threads;
import
org.apache.accumulo.manager.compaction.coordinator.CompactionCoordinator;
-import org.apache.accumulo.manager.compaction.queue.CompactionJobQueues;
import org.apache.accumulo.manager.metrics.ManagerMetrics;
import org.apache.accumulo.manager.recovery.RecoveryManager;
import org.apache.accumulo.manager.split.Splitter;
@@ -162,7 +161,6 @@ import com.google.common.util.concurrent.RateLimiter;
import com.google.common.util.concurrent.Uninterruptibles;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
-import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.context.Scope;
@@ -538,16 +536,14 @@ public class Manager extends AbstractServer
return splitter;
}
- private CompactionJobQueues compactionJobQueues;
-
- public CompactionJobQueues getCompactionQueues() {
-return compactionJobQueues;
- }
-
public UpgradeCoordinator.UpgradeStatus getUpgradeStatus() {
return upgradeCoordinator.getStatus();
}
+ public CompactionCoordinator getCompactionCoordinator() {
+return compactionCoordinator;
+ }
+
private class MigrationCleanupThread implements Runnable {
@Override
@@ -932,17 +928,13 @@ public class Manager extends AbstractServer
final ServerContext context = getContext();
final String zroot = getZooKeeperRoot();
-this.compactionJobQueues = new CompactionJobQueues(
-
getConfiguration().getCount(Property.MANAGER_COMPACTION_SERVICE_PRIORITY_QUEUE_SIZE));
-
// ACCUMULO-4424 Put up the Thrift servers before getting the lock as a
sign of process health
// when a hot-standby
//
// Start the Manager's Fate Service
fateServiceHandler = new FateServiceHandler(this);
managerClientHandler = new ManagerClientServiceHandler(this);
-compactionCoordinator =
-new CompactionCoordinator(context, tserverSet, security,
compactionJobQueues, nextEvent);
+compactionCoordinator = new CompactionCoordinator(context, tserverSet,
security, nextEvent);
// Start the Manager's Client service
// Ensure that calls before the manager gets the lock fail
ManagerClientService.Iface haProxy =
@@ -950,7 +942,7 @@ public class Manager extends AbstractServer
ServerAddress sa;
var processor =
ThriftProcessorTypes.getManagerTProcessor(fateServiceHandler,
-compactionCoordinator, haProxy, getContext());
+compactionCoordinator.getThriftService(), haProxy, getContext());
try {
sa = TServerUtils.startServer(context, getHostname(),
Property.MANAGER_CLIENTPORT, processor,
@@ -1008,10 +1000,8 @@ public class Manager extends AbstractServer
Thread.currentThread().interrupt();
}
-// Don't call run on the CompactionCoordinator until we have tservers.
-Thread compactionCoordinatorThread =
-Threads.createThread("CompactionCoordinator Thread",
compactionCoordinator);
-compactionCoordinatorThread.start();
+// Don't call start the CompactionCoordinator until we have tservers.
+compactionCoordinator.start();
ZooReaderWriter zReaderWriter = context.getZooReaderWriter();
@@ -1163,11 +1153,6 @@ public class Manager extends AbstractServer
tableInformationStatusPool.shutdownNow();
compactionCoordinator.shutdown();
-
(accumulo) branch 2.1 updated: Update property doc generation for website
This is an automated email from the ASF dual-hosted git repository.
ctubbsii pushed a commit to branch 2.1
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/2.1 by this push:
new 8dfb4eb713 Update property doc generation for website
8dfb4eb713 is described below
commit 8dfb4eb71351b856a398db7d0f75b8ddda85ce3a
Author: Christopher Tubbs
AuthorDate: Mon Dec 18 18:54:55 2023 -0500
Update property doc generation for website
* Update generated docs code to produce files appropriate for current
website documentation for 2.x and 3.x
* Use "(2.x)" or "(3.x)" and ensure the order is correct, and 3.x docs
have the "3" on the end of the filename
---
.../main/java/org/apache/accumulo/core/conf/ClientConfigGenerate.java | 2 +-
.../main/java/org/apache/accumulo/core/conf/ConfigurationDocGen.java | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/conf/ClientConfigGenerate.java
b/core/src/main/java/org/apache/accumulo/core/conf/ClientConfigGenerate.java
index d4d01d0786..0cd21758b1 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/ClientConfigGenerate.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/ClientConfigGenerate.java
@@ -86,7 +86,7 @@ public class ClientConfigGenerate {
@Override
void pageHeader() {
doc.println("---");
- doc.println("title: Client Properties");
+ doc.println("title: Client Properties (2.x)");
doc.println("category: configuration");
doc.println("order: 3");
doc.println("---\n");
diff --git
a/core/src/main/java/org/apache/accumulo/core/conf/ConfigurationDocGen.java
b/core/src/main/java/org/apache/accumulo/core/conf/ConfigurationDocGen.java
index 74e1bc58dd..51a6e24732 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/ConfigurationDocGen.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/ConfigurationDocGen.java
@@ -62,9 +62,9 @@ public class ConfigurationDocGen {
void pageHeader() {
doc.println("---");
-doc.println("title: Server Properties");
+doc.println("title: Server Properties (2.x)");
doc.println("category: configuration");
-doc.println("order: 4");
+doc.println("order: 5");
doc.println("---\n");
doc.println("\n");
(accumulo) branch main updated (3061ff028d -> 436a6835de)
This is an automated email from the ASF dual-hosted git repository. ctubbsii pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git from 3061ff028d Remove redundant LogEntry value (#4071) new 8dfb4eb713 Update property doc generation for website new 436a6835de Merge branch '2.1' The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: core/pom.xml | 4 ++-- .../java/org/apache/accumulo/core/conf/ClientConfigGenerate.java | 8 .../java/org/apache/accumulo/core/conf/ConfigurationDocGen.java | 4 ++-- 3 files changed, 8 insertions(+), 8 deletions(-)
(accumulo) 01/01: Merge branch '2.1'
This is an automated email from the ASF dual-hosted git repository.
ctubbsii pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit 436a6835de165715c930f9d6cebe1cad76d9d578
Merge: 3061ff028d 8dfb4eb713
Author: Christopher Tubbs
AuthorDate: Mon Dec 18 19:02:37 2023 -0500
Merge branch '2.1'
core/pom.xml | 4 ++--
.../java/org/apache/accumulo/core/conf/ClientConfigGenerate.java | 8
.../java/org/apache/accumulo/core/conf/ConfigurationDocGen.java | 4 ++--
3 files changed, 8 insertions(+), 8 deletions(-)
diff --cc core/pom.xml
index e1a83b0228,7869dddb38..2ba4af4861
--- a/core/pom.xml
+++ b/core/pom.xml
@@@ -355,7 -355,7 +355,7 @@@
test
--generate-markdown
--
${project.build.directory}/generated-docs/server-properties.md
++
${project.build.directory}/generated-docs/server-properties3.md
@@@ -370,7 -370,7 +370,7 @@@
test
--generate-markdown
--
${project.build.directory}/generated-docs/client-properties.md
++
${project.build.directory}/generated-docs/client-properties3.md
diff --cc
core/src/main/java/org/apache/accumulo/core/conf/ClientConfigGenerate.java
index d4d01d0786,0cd21758b1..f95688a4ed
--- a/core/src/main/java/org/apache/accumulo/core/conf/ClientConfigGenerate.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/ClientConfigGenerate.java
@@@ -29,8 -29,8 +29,8 @@@ import java.util.TreeMap
import com.google.common.collect.Sets;
/**
-- * Generates client-properties.md for documentation on Accumulo website and
-- * accumulo-client.properties for Accumulo distribution tarball
++ * Generates client properties documentation on Accumulo website and
accumulo-client.properties for
++ * Accumulo distribution tarball
*/
public class ClientConfigGenerate {
@@@ -86,9 -86,9 +86,9 @@@
@Override
void pageHeader() {
doc.println("---");
- doc.println("title: Client Properties");
- doc.println("title: Client Properties (2.x)");
++ doc.println("title: Client Properties (3.x)");
doc.println("category: configuration");
-- doc.println("order: 3");
++ doc.println("order: 4");
doc.println("---\n");
doc.println("");
diff --cc
core/src/main/java/org/apache/accumulo/core/conf/ConfigurationDocGen.java
index 163b150eab,51a6e24732..dc5320e93e
--- a/core/src/main/java/org/apache/accumulo/core/conf/ConfigurationDocGen.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/ConfigurationDocGen.java
@@@ -62,9 -62,9 +62,9 @@@ public class ConfigurationDocGen
void pageHeader() {
doc.println("---");
- doc.println("title: Server Properties");
-doc.println("title: Server Properties (2.x)");
++doc.println("title: Server Properties (3.x)");
doc.println("category: configuration");
- doc.println("order: 4");
-doc.println("order: 5");
++doc.println("order: 6");
doc.println("---\n");
doc.println("\n");
(accumulo-website) branch asf-site updated: Automatic Site Publish by Buildbot
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/accumulo-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 0fb5300c Automatic Site Publish by Buildbot 0fb5300c is described below commit 0fb5300c68498e4106103fd11657ca94b92edee7 Author: buildbot AuthorDate: Mon Dec 18 17:39:16 2023 + Automatic Site Publish by Buildbot --- output/feed.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/output/feed.xml b/output/feed.xml index cd1e9eff..23be9a16 100644 --- a/output/feed.xml +++ b/output/feed.xml @@ -6,8 +6,8 @@ https://accumulo.apache.org/ https://accumulo.apache.org/feed.xml; rel="self" type="application/rss+xml"/> -Wed, 29 Nov 2023 07:26:29 + -Wed, 29 Nov 2023 07:26:29 + +Mon, 18 Dec 2023 17:39:10 + +Mon, 18 Dec 2023 17:39:10 + Jekyll v4.3.2
(accumulo-website) branch main updated: Bump ruby version in github actions script (#410)
This is an automated email from the ASF dual-hosted git repository. domgarguilo pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/accumulo-website.git The following commit(s) were added to refs/heads/main by this push: new 32bc2fd4 Bump ruby version in github actions script (#410) 32bc2fd4 is described below commit 32bc2fd4d61db7ef391f80e8b949aff6cf0946a1 Author: Dom G AuthorDate: Mon Dec 18 12:38:53 2023 -0500 Bump ruby version in github actions script (#410) --- .github/workflows/jekyll.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/jekyll.yaml b/.github/workflows/jekyll.yaml index 7dbea9dd..f77b7cb8 100644 --- a/.github/workflows/jekyll.yaml +++ b/.github/workflows/jekyll.yaml @@ -34,7 +34,7 @@ jobs: - name: Set up Ruby uses: ruby/setup-ruby@v1 with: -ruby-version: 2.6 +ruby-version: 3.2.2 - name: Cache gems uses: actions/cache@v3 with:
