(accumulo) branch elasticity updated: fixes deleting tablet suspension (#4575)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new e98f15b5fd fixes deleting tablet suspension (#4575)
e98f15b5fd is described below
commit e98f15b5fd282ee60d2d695531ee16b89014d459
Author: Keith Turner
AuthorDate: Mon May 20 16:17:45 2024 -0400
fixes deleting tablet suspension (#4575)
Tablet suspension was failing to delete becasue the value set on the
conditional mutation did not match the format of what was stored in the
metadata table. Fixed this issue and added some unit test.
---
.../org/apache/accumulo/core/data/Condition.java | 15 ++
.../accumulo/core/metadata/SuspendingTServer.java | 4 ++
.../core/metadata/SuspendingTServerTest.java | 63 ++
.../metadata/ConditionalTabletMutatorImpl.java | 3 +-
4 files changed, 83 insertions(+), 2 deletions(-)
diff --git a/core/src/main/java/org/apache/accumulo/core/data/Condition.java
b/core/src/main/java/org/apache/accumulo/core/data/Condition.java
index deadc0e069..dab5ee24e5 100644
--- a/core/src/main/java/org/apache/accumulo/core/data/Condition.java
+++ b/core/src/main/java/org/apache/accumulo/core/data/Condition.java
@@ -209,6 +209,21 @@ public class Condition {
return this;
}
+ /**
+ * This method sets the expected value of a column. In order for the
condition to pass the column
+ * must exist and have this value. If a value is not set, then the column
must be absent for the
+ * condition to pass. See {@link #setValue(byte[])}.
+ *
+ * @param value value
+ * @return this condition
+ * @throws IllegalArgumentException if value is null
+ * @since 4.0.0
+ */
+ public Condition setValue(Value value) {
+checkArgument(value != null, "value is null");
+return setValue(value.get());
+ }
+
/**
* Gets the value of this condition.
*
diff --git
a/core/src/main/java/org/apache/accumulo/core/metadata/SuspendingTServer.java
b/core/src/main/java/org/apache/accumulo/core/metadata/SuspendingTServer.java
index e481369a21..e45635fc80 100644
---
a/core/src/main/java/org/apache/accumulo/core/metadata/SuspendingTServer.java
+++
b/core/src/main/java/org/apache/accumulo/core/metadata/SuspendingTServer.java
@@ -49,6 +49,10 @@ public class SuspendingTServer {
return new Value(tServer.getHostPort() + "|" + suspensionTime.getMillis());
}
+ public Value toValue() {
+return new Value(server + "|" + suspensionTime.getMillis());
+ }
+
@Override
public boolean equals(Object rhsObject) {
if (!(rhsObject instanceof SuspendingTServer)) {
diff --git
a/core/src/test/java/org/apache/accumulo/core/metadata/SuspendingTServerTest.java
b/core/src/test/java/org/apache/accumulo/core/metadata/SuspendingTServerTest.java
new file mode 100644
index 00..7826915ed5
--- /dev/null
+++
b/core/src/test/java/org/apache/accumulo/core/metadata/SuspendingTServerTest.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.accumulo.core.util.time.SteadyTime;
+import org.junit.jupiter.api.Test;
+
+import com.google.common.net.HostAndPort;
+
+public class SuspendingTServerTest {
+ @Test
+ public void testToFromValue() {
+SteadyTime suspensionTime = SteadyTime.from(System.currentTimeMillis(),
TimeUnit.MILLISECONDS);
+TServerInstance ser1 = new
TServerInstance(HostAndPort.fromParts("server1", 8555), "s001");
+
+var val1 = SuspendingTServer.toValue(ser1, suspensionTime);
+var st1 = SuspendingTServer.fromValue(val1);
+assertEquals(HostAndPort.fromParts("server1", 8555), st1.server);
+assertEquals(suspensionTime, st1.suspensionTime);
+assertEquals(val1, st1.toValue());
+var st2 = new SuspendingTServer(HostAndPort.fromParts("server1", 8555),
suspensionTime);
+assertEquals(st1, st2);
+
(accumulo) 02/03: Merge branch 'main' into elasticity
This is an automated email from the ASF dual-hosted git repository. dlmarion pushed a commit to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git commit 94f37aaddb4ed01d5f0a66f709409cc2af0051a9 Merge: 96fa64fc2a 91a2ca2349 Author: Dave Marion AuthorDate: Mon May 20 17:03:29 2024 + Merge branch 'main' into elasticity .../core/clientImpl/ThriftTransportPool.java | 28 ++- .../scan/ConfigurableScanServerHostSelector.java | 157 + .../spi/scan/ConfigurableScanServerSelector.java | 49 +++--- .../ConfigurableScanServerHostSelectorTest.java| 191 + .../scan/ConfigurableScanServerSelectorTest.java | 22 +-- 5 files changed, 407 insertions(+), 40 deletions(-)
(accumulo) 01/03: Fixed ScanServerGroupConfigurationIT from error after merge
This is an automated email from the ASF dual-hosted git repository.
dlmarion pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit 96fa64fc2aa18d37b882c4c040e658bf90700f1d
Author: Dave Marion
AuthorDate: Mon May 20 17:02:26 2024 +
Fixed ScanServerGroupConfigurationIT from error after merge
---
.../apache/accumulo/test/ScanServerGroupConfigurationIT.java | 10 --
1 file changed, 4 insertions(+), 6 deletions(-)
diff --git
a/test/src/main/java/org/apache/accumulo/test/ScanServerGroupConfigurationIT.java
b/test/src/main/java/org/apache/accumulo/test/ScanServerGroupConfigurationIT.java
index 09f1cb921e..89582eeea7 100644
---
a/test/src/main/java/org/apache/accumulo/test/ScanServerGroupConfigurationIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/ScanServerGroupConfigurationIT.java
@@ -38,7 +38,6 @@ import org.apache.accumulo.harness.SharedMiniClusterBase;
import org.apache.accumulo.minicluster.ServerType;
import org.apache.accumulo.miniclusterImpl.MiniAccumuloConfigImpl;
import org.apache.accumulo.test.util.Wait;
-import org.apache.accumulo.tserver.ScanServer;
import org.apache.hadoop.conf.Configuration;
import org.apache.zookeeper.ZooKeeper;
import org.junit.jupiter.api.AfterAll;
@@ -163,11 +162,10 @@ public class ScanServerGroupConfigurationIT extends
SharedMiniClusterBase {
ScanServerIT.ingest(client, tableName, 10, 10, 10, "colf", true);
assertEquals(100, additionalIngest1);
-// Bump the number of scan serves that can run to start the GROUP1
scan server
-
getCluster().getConfig().getClusterServerConfiguration().setNumDefaultScanServers(2);
-
-getCluster()._exec(ScanServer.class, ServerType.SCAN_SERVER, Map.of(),
-new String[] {"-g", "GROUP1"});
+// A a scan server for resource group GROUP1
+getCluster().getConfig().getClusterServerConfiguration()
+.addScanServerResourceGroup("GROUP1", 1);
+getCluster().getClusterControl().start(ServerType.SCAN_SERVER);
Wait.waitFor(() -> zk.getChildren(scanServerRoot, false).size() == 2);
Wait.waitFor(() -> ((ClientContext)
client).getScanServers().values().stream().anyMatch(
(p) ->
p.getSecond().equals(ScanServerSelector.DEFAULT_SCAN_SERVER_GROUP_NAME))
(accumulo) 03/03: Fixed post-merge issue with ConfigurableScanServerSelectorTest
This is an automated email from the ASF dual-hosted git repository.
dlmarion pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit 393c7a6c2f9dd365531bf315eab926d7d528bda6
Author: Dave Marion
AuthorDate: Mon May 20 17:11:19 2024 +
Fixed post-merge issue with ConfigurableScanServerSelectorTest
---
.../accumulo/core/spi/scan/ConfigurableScanServerSelectorTest.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/core/src/test/java/org/apache/accumulo/core/spi/scan/ConfigurableScanServerSelectorTest.java
b/core/src/test/java/org/apache/accumulo/core/spi/scan/ConfigurableScanServerSelectorTest.java
index 041f5e6eb3..64fe25bb43 100644
---
a/core/src/test/java/org/apache/accumulo/core/spi/scan/ConfigurableScanServerSelectorTest.java
+++
b/core/src/test/java/org/apache/accumulo/core/spi/scan/ConfigurableScanServerSelectorTest.java
@@ -505,7 +505,7 @@ public class ConfigurableScanServerSelectorTest {
var dg = ScanServerSelector.DEFAULT_SCAN_SERVER_GROUP_NAME;
-var params = new DaParams(tabletId, Map.of(), Map.of()) {
+var params = new SelectorParams(tabletId, Map.of(), Map.of()) {
@Override
public Optional waitUntil(Supplier> condition,
Duration maxWaitTime,
String description) {
(accumulo) branch elasticity updated (6c0f610ed2 -> 393c7a6c2f)
This is an automated email from the ASF dual-hosted git repository. dlmarion pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from 6c0f610ed2 Merge remote-tracking branch 'upstream/main' into elasticity new 96fa64fc2a Fixed ScanServerGroupConfigurationIT from error after merge add 176ba9ea0b Improve ThriftTransportPool shutdown speed (#4561) add 73b97b8d1d Merge branch '2.1' add dd61442925 Created ScanServerSelector that tries to use servers on the same host (#4536) add 91a2ca2349 Merge branch '2.1' new 94f37aaddb Merge branch 'main' into elasticity new 393c7a6c2f Fixed post-merge issue with ConfigurableScanServerSelectorTest The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../core/clientImpl/ThriftTransportPool.java | 28 ++- .../scan/ConfigurableScanServerHostSelector.java | 157 + .../spi/scan/ConfigurableScanServerSelector.java | 49 +++--- .../ConfigurableScanServerHostSelectorTest.java| 191 + .../scan/ConfigurableScanServerSelectorTest.java | 24 +-- .../test/ScanServerGroupConfigurationIT.java | 10 +- 6 files changed, 412 insertions(+), 47 deletions(-) create mode 100644 core/src/main/java/org/apache/accumulo/core/spi/scan/ConfigurableScanServerHostSelector.java create mode 100644 core/src/test/java/org/apache/accumulo/core/spi/scan/ConfigurableScanServerHostSelectorTest.java
(accumulo) 01/01: Merge branch '2.1'
This is an automated email from the ASF dual-hosted git repository.
dlmarion pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit 91a2ca234903a098def9bcb5eb52a5f98a2aba2b
Merge: 73b97b8d1d dd61442925
Author: Dave Marion
AuthorDate: Mon May 20 16:41:27 2024 +
Merge branch '2.1'
.../scan/ConfigurableScanServerHostSelector.java | 157 +
.../spi/scan/ConfigurableScanServerSelector.java | 49 +++---
.../ConfigurableScanServerHostSelectorTest.java| 191 +
.../scan/ConfigurableScanServerSelectorTest.java | 22 +--
4 files changed, 386 insertions(+), 33 deletions(-)
diff --cc
core/src/main/java/org/apache/accumulo/core/spi/scan/ConfigurableScanServerHostSelector.java
index 00,d21a8799b9..f43f21e8c2
mode 00,100644..100644
---
a/core/src/main/java/org/apache/accumulo/core/spi/scan/ConfigurableScanServerHostSelector.java
+++
b/core/src/main/java/org/apache/accumulo/core/spi/scan/ConfigurableScanServerHostSelector.java
@@@ -1,0 -1,155 +1,157 @@@
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+ package org.apache.accumulo.core.spi.scan;
+
++import static org.apache.accumulo.core.util.LazySingletons.RANDOM;
++
+ import java.util.ArrayList;
+ import java.util.Collections;
+ import java.util.Comparator;
+ import java.util.HashMap;
+ import java.util.HashSet;
+ import java.util.List;
+ import java.util.Map;
+ import java.util.Set;
+
+ import org.apache.accumulo.core.data.TabletId;
+
+ import com.google.common.hash.HashCode;
+ import com.google.common.net.HostAndPort;
+
+ /**
+ * Extension of the {@code ConfigurableScanServerSelector} that can be used
when there are multiple
+ * ScanServers running on the same host and for some reason, like using a
shared off-heap cache,
+ * sending scans for the same tablet to the same host may provide a better
experience.
+ *
+ * This implementation will initially hash a Tablet to a ScanServer. If the
ScanServer is unable to
+ * execute the scan, this implementation will try to send the scan to a
ScanServer on the same host.
+ * If there are no more ScanServers to try on that host, then it will fall
back to trying a
+ * different host and the process repeats.
+ *
+ */
+ public class ConfigurableScanServerHostSelector extends
ConfigurableScanServerSelector {
+
+ private static final class PriorHostServersComparator implements
Comparator {
+
+ @Override
+ public int compare(PriorHostServers o1, PriorHostServers o2) {
+ return Integer.compare(o1.getPriorServers().size(),
o2.getPriorServers().size());
+ }
+
+ }
+
+ private static final class PriorHostServers {
+ private final String priorHost;
+ private final List priorServers = new ArrayList<>();
+
+ public PriorHostServers(String priorHost) {
+ this.priorHost = priorHost;
+ }
+
+ public String getPriorHost() {
+ return priorHost;
+ }
+
+ public List getPriorServers() {
+ return priorServers;
+ }
+ }
+
+ @Override
+ protected int selectServers(SelectorParameters params, Profile profile,
+ List orderedScanServers, Map serversToUse) {
+
+ // orderedScanServers is the set of ScanServers addresses (host:port)
+ // for the resource group designated for the profile being used for
+ // this scan. We want to group these scan servers by hostname and
+ // hash the tablet to the hostname, then randomly pick one of the
+ // scan servers in that group.
+
+ final Map> scanServerHosts = new HashMap<>();
+ for (final String address : orderedScanServers) {
+ final HostAndPort hp = HostAndPort.fromString(address);
+ scanServerHosts.computeIfAbsent(hp.getHost(), (k) -> {
+ return new ArrayList();
+ }).add(address);
+ }
+ final List hostIndex = new ArrayList<>(scanServerHosts.keySet());
+
+ final int numberOfPreviousAttempts = params.getTablets().stream()
+ .mapToInt(tablet ->
params.getAttempts(tablet).size()).max().orElse(0);
+
+ final int numServersToUseInAttemptPlan =
+ profile.getNumServers(numberOfPreviousAttempts,
orderedScanServers.size());
+
(accumulo) branch main updated (73b97b8d1d -> 91a2ca2349)
This is an automated email from the ASF dual-hosted git repository. dlmarion pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git from 73b97b8d1d Merge branch '2.1' add dd61442925 Created ScanServerSelector that tries to use servers on the same host (#4536) new 91a2ca2349 Merge branch '2.1' The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../scan/ConfigurableScanServerHostSelector.java | 157 + .../spi/scan/ConfigurableScanServerSelector.java | 49 +++--- .../ConfigurableScanServerHostSelectorTest.java| 191 + .../scan/ConfigurableScanServerSelectorTest.java | 22 +-- 4 files changed, 386 insertions(+), 33 deletions(-) create mode 100644 core/src/main/java/org/apache/accumulo/core/spi/scan/ConfigurableScanServerHostSelector.java create mode 100644 core/src/test/java/org/apache/accumulo/core/spi/scan/ConfigurableScanServerHostSelectorTest.java
(accumulo) branch 2.1 updated (176ba9ea0b -> dd61442925)
This is an automated email from the ASF dual-hosted git repository. dlmarion pushed a change to branch 2.1 in repository https://gitbox.apache.org/repos/asf/accumulo.git from 176ba9ea0b Improve ThriftTransportPool shutdown speed (#4561) add dd61442925 Created ScanServerSelector that tries to use servers on the same host (#4536) No new revisions were added by this update. Summary of changes: .../scan/ConfigurableScanServerHostSelector.java | 155 + .../spi/scan/ConfigurableScanServerSelector.java | 51 +++--- .../ConfigurableScanServerHostSelectorTest.java| 191 + .../scan/ConfigurableScanServerSelectorTest.java | 22 +-- 4 files changed, 385 insertions(+), 34 deletions(-) create mode 100644 core/src/main/java/org/apache/accumulo/core/spi/scan/ConfigurableScanServerHostSelector.java create mode 100644 core/src/test/java/org/apache/accumulo/core/spi/scan/ConfigurableScanServerHostSelectorTest.java
(accumulo) branch elasticity updated (6afdc88f8e -> 6c0f610ed2)
This is an automated email from the ASF dual-hosted git repository. edcoleman pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from 6afdc88f8e Modified ScanServer to not allow scans on Fate table as well add 5cd9cdfc70 Update MetricsIT (#4576) add c53bebc3e9 Added Scan Server Group Configuration IT (#4506) add 52be928da0 Merge branch '2.1' add 6c0f610ed2 Merge remote-tracking branch 'upstream/main' into elasticity No new revisions were added by this update. Summary of changes: .../test/ScanServerGroupConfigurationIT.java | 198 + .../org/apache/accumulo/test/ScanServerIT.java | 4 +- .../apache/accumulo/test/metrics/MetricsIT.java| 6 +- 3 files changed, 202 insertions(+), 6 deletions(-) create mode 100644 test/src/main/java/org/apache/accumulo/test/ScanServerGroupConfigurationIT.java
(accumulo) branch main updated (52be928da0 -> 73b97b8d1d)
This is an automated email from the ASF dual-hosted git repository. dlmarion pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git from 52be928da0 Merge branch '2.1' add 176ba9ea0b Improve ThriftTransportPool shutdown speed (#4561) new 73b97b8d1d Merge branch '2.1' The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../core/clientImpl/ThriftTransportPool.java | 28 -- 1 file changed, 21 insertions(+), 7 deletions(-)
(accumulo) 01/01: Merge branch '2.1'
This is an automated email from the ASF dual-hosted git repository. dlmarion pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git commit 73b97b8d1d84717793e9531d9d1ca65d664b7cc2 Merge: 52be928da0 176ba9ea0b Author: Dave Marion AuthorDate: Mon May 20 15:45:44 2024 + Merge branch '2.1' .../core/clientImpl/ThriftTransportPool.java | 28 -- 1 file changed, 21 insertions(+), 7 deletions(-)
(accumulo) branch 2.1 updated (c53bebc3e9 -> 176ba9ea0b)
This is an automated email from the ASF dual-hosted git repository. dlmarion pushed a change to branch 2.1 in repository https://gitbox.apache.org/repos/asf/accumulo.git from c53bebc3e9 Added Scan Server Group Configuration IT (#4506) add 176ba9ea0b Improve ThriftTransportPool shutdown speed (#4561) No new revisions were added by this update. Summary of changes: .../core/clientImpl/ThriftTransportPool.java | 28 -- 1 file changed, 21 insertions(+), 7 deletions(-)
(accumulo) branch main updated (5cd9cdfc70 -> 52be928da0)
This is an automated email from the ASF dual-hosted git repository. dlmarion pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git from 5cd9cdfc70 Update MetricsIT (#4576) add c53bebc3e9 Added Scan Server Group Configuration IT (#4506) new 52be928da0 Merge branch '2.1' The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../test/ScanServerGroupConfigurationIT.java | 197 + .../org/apache/accumulo/test/ScanServerIT.java | 4 +- 2 files changed, 199 insertions(+), 2 deletions(-) create mode 100644 test/src/main/java/org/apache/accumulo/test/ScanServerGroupConfigurationIT.java
(accumulo) 01/01: Merge branch '2.1'
This is an automated email from the ASF dual-hosted git repository. dlmarion pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git commit 52be928da034ce1fe9dd18971171fc60feb55ed3 Merge: 5cd9cdfc70 c53bebc3e9 Author: Dave Marion AuthorDate: Mon May 20 14:53:57 2024 + Merge branch '2.1' .../test/ScanServerGroupConfigurationIT.java | 197 + .../org/apache/accumulo/test/ScanServerIT.java | 4 +- 2 files changed, 199 insertions(+), 2 deletions(-)
(accumulo) branch 2.1 updated: Added Scan Server Group Configuration IT (#4506)
This is an automated email from the ASF dual-hosted git repository.
dlmarion pushed a commit to branch 2.1
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/2.1 by this push:
new c53bebc3e9 Added Scan Server Group Configuration IT (#4506)
c53bebc3e9 is described below
commit c53bebc3e9accf63de0c41dac3e2da5ce9597605
Author: Dave Marion
AuthorDate: Mon May 20 10:43:28 2024 -0400
Added Scan Server Group Configuration IT (#4506)
Closes #4504
---
.../test/ScanServerGroupConfigurationIT.java | 197 +
.../org/apache/accumulo/test/ScanServerIT.java | 4 +-
2 files changed, 199 insertions(+), 2 deletions(-)
diff --git
a/test/src/main/java/org/apache/accumulo/test/ScanServerGroupConfigurationIT.java
b/test/src/main/java/org/apache/accumulo/test/ScanServerGroupConfigurationIT.java
new file mode 100644
index 00..c18e6e1aff
--- /dev/null
+++
b/test/src/main/java/org/apache/accumulo/test/ScanServerGroupConfigurationIT.java
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+import java.util.Map;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.ScannerBase.ConsistencyLevel;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.ClientProperty;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.core.spi.scan.ScanServerSelector;
+import org.apache.accumulo.harness.MiniClusterConfigurationCallback;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.miniclusterImpl.MiniAccumuloConfigImpl;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.accumulo.tserver.ScanServer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.zookeeper.ZooKeeper;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import com.google.common.collect.Iterables;
+
+public class ScanServerGroupConfigurationIT extends SharedMiniClusterBase {
+
+ // @formatter:off
+ private static final String clientConfiguration =
+ "["+
+ " {"+
+ " \"isDefault\": true,"+
+ " \"maxBusyTimeout\": \"5m\","+
+ " \"busyTimeoutMultiplier\": 8,"+
+ " \"scanTypeActivations\": [],"+
+ " \"attemptPlans\": ["+
+ " {"+
+ " \"servers\": \"3\","+
+ " \"busyTimeout\": \"33ms\","+
+ " \"salt\": \"one\""+
+ " },"+
+ " {"+
+ " \"servers\": \"13\","+
+ " \"busyTimeout\": \"33ms\","+
+ " \"salt\": \"two\""+
+ " },"+
+ " {"+
+ " \"servers\": \"100%\","+
+ " \"busyTimeout\": \"33ms\""+
+ " }"+
+ " ]"+
+ " },"+
+ " {"+
+ " \"isDefault\": false,"+
+ " \"maxBusyTimeout\": \"5m\","+
+ " \"busyTimeoutMultiplier\": 8,"+
+ " \"group\": \"GROUP1\","+
+ " \"scanTypeActivations\": [\"use_group1\"],"+
+ " \"attemptPlans\": ["+
+ " {"+
+ " \"servers\": \"3\","+
+ " \"busyTimeout\": \"33ms\","+
+ " \"salt\": \"one\""+
+ " },"+
+ " {"+
+ " \"servers\": \"13\","+
+ " \"busyTimeout\": \"33ms\","+
+ " \"salt\": \"two\""+
+ " },"+
+ " {"+
+ " \"servers\": \"100%\","+
+ " \"busyTimeout\": \"33ms\""+
+ " }"+
+ " ]"+
+ " }"+
+ "]";
+ // @formatter:on
+
+ private static class Config implements MiniClusterConfigurationCallback {
+@Override
+public void configureMiniCluster(MiniAccumuloConfigImpl cfg, Configuration
coreSite) {
+ cfg.setNumScanServers(0); // start with no scan servers
+ cfg.setProperty(
(accumulo) 01/02: Merge branch 'main' into elasticity
This is an automated email from the ASF dual-hosted git repository. dlmarion pushed a commit to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git commit bd6ce7cc1cab36413fa07262111c056453a1feab Merge: f7e6183271 64de0f0aa7 Author: Dave Marion AuthorDate: Mon May 20 13:29:05 2024 + Merge branch 'main' into elasticity .../org/apache/accumulo/tserver/ScanServer.java| 17 +++ .../apache/accumulo/tserver/ScanServerTest.java| 132 ++--- 2 files changed, 134 insertions(+), 15 deletions(-)
(accumulo) branch elasticity updated (f7e6183271 -> 6afdc88f8e)
This is an automated email from the ASF dual-hosted git repository. dlmarion pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from f7e6183271 speeds up tablet mgmt iterator (#4568) add a8d3a101d7 Only allow system user to perform eventual scans on root and meta (#4531) add 64de0f0aa7 Merge branch '2.1' new bd6ce7cc1c Merge branch 'main' into elasticity new 6afdc88f8e Modified ScanServer to not allow scans on Fate table as well The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/accumulo/core/dataImpl/KeyExtent.java | 4 + .../org/apache/accumulo/tserver/ScanServer.java| 17 +++ .../apache/accumulo/tserver/ScanServerTest.java| 132 ++--- 3 files changed, 138 insertions(+), 15 deletions(-)
(accumulo) 02/02: Modified ScanServer to not allow scans on Fate table as well
This is an automated email from the ASF dual-hosted git repository.
dlmarion pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit 6afdc88f8e6ebfdd2a5b14c2837f1ff03c062a21
Author: Dave Marion
AuthorDate: Mon May 20 14:18:55 2024 +
Modified ScanServer to not allow scans on Fate table as well
---
.../main/java/org/apache/accumulo/core/dataImpl/KeyExtent.java | 4
.../src/main/java/org/apache/accumulo/tserver/ScanServer.java | 4 ++--
.../test/java/org/apache/accumulo/tserver/ScanServerTest.java | 10 +-
3 files changed, 11 insertions(+), 7 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/dataImpl/KeyExtent.java
b/core/src/main/java/org/apache/accumulo/core/dataImpl/KeyExtent.java
index e69068478f..133818ed5d 100644
--- a/core/src/main/java/org/apache/accumulo/core/dataImpl/KeyExtent.java
+++ b/core/src/main/java/org/apache/accumulo/core/dataImpl/KeyExtent.java
@@ -474,6 +474,10 @@ public class KeyExtent implements Comparable {
return prevExtent.endRow().equals(prevEndRow());
}
+ public boolean isSystemTable() {
+return AccumuloTable.allTableIds().contains(tableId());
+ }
+
public boolean isMeta() {
return tableId().equals(AccumuloTable.METADATA.tableId()) ||
isRootTablet();
}
diff --git
a/server/tserver/src/main/java/org/apache/accumulo/tserver/ScanServer.java
b/server/tserver/src/main/java/org/apache/accumulo/tserver/ScanServer.java
index c2c71d04fb..f9aed7e2cf 100644
--- a/server/tserver/src/main/java/org/apache/accumulo/tserver/ScanServer.java
+++ b/server/tserver/src/main/java/org/apache/accumulo/tserver/ScanServer.java
@@ -933,7 +933,7 @@ public class ScanServer extends AbstractServer
KeyExtent extent = getKeyExtent(textent);
-if (extent.isMeta() && !isSystemUser(credentials)) {
+if (extent.isSystemTable() && !isSystemUser(credentials)) {
throw new TException(
"Only the system user can perform eventual consistency scans on the
root and metadata tables");
}
@@ -1000,7 +1000,7 @@ public class ScanServer extends AbstractServer
for (Entry> entry : tbatch.entrySet()) {
KeyExtent extent = getKeyExtent(entry.getKey());
- if (extent.isMeta() &&
!context.getSecurityOperation().isSystemUser(credentials)) {
+ if (extent.isSystemTable() && !isSystemUser(credentials)) {
throw new TException(
"Only the system user can perform eventual consistency scans on
the root and metadata tables");
}
diff --git
a/server/tserver/src/test/java/org/apache/accumulo/tserver/ScanServerTest.java
b/server/tserver/src/test/java/org/apache/accumulo/tserver/ScanServerTest.java
index f93f095a87..bbe8ffb3fc 100644
---
a/server/tserver/src/test/java/org/apache/accumulo/tserver/ScanServerTest.java
+++
b/server/tserver/src/test/java/org/apache/accumulo/tserver/ScanServerTest.java
@@ -190,7 +190,7 @@ public class ScanServerTest {
Map execHints = new HashMap<>();
ScanReservation reservation = createMock(ScanReservation.class);
-expect(extent.isMeta()).andReturn(false).anyTimes();
+expect(extent.isSystemTable()).andReturn(false).anyTimes();
expect(extent.toThrift()).andReturn(textent).anyTimes();
expect(reservation.getFailures()).andReturn(Map.of(textent, ranges));
reservation.close();
@@ -242,7 +242,7 @@ public class ScanServerTest {
};
TestScanServer ss = partialMockBuilder(TestScanServer.class).createMock();
-expect(extent.isMeta()).andReturn(false).anyTimes();
+expect(extent.isSystemTable()).andReturn(false).anyTimes();
expect(reservation.newTablet(ss, extent)).andReturn(tablet);
expect(reservation.getTabletMetadataExtents()).andReturn(Set.of(extent));
expect(reservation.getFailures()).andReturn(Map.of());
@@ -305,7 +305,7 @@ public class ScanServerTest {
};
TestScanServer ss = partialMockBuilder(TestScanServer.class).createMock();
-expect(extent.isMeta()).andReturn(false).anyTimes();
+expect(extent.isSystemTable()).andReturn(false).anyTimes();
expect(reservation.newTablet(ss, extent)).andReturn(tablet).anyTimes();
expect(reservation.getTabletMetadataExtents()).andReturn(Set.of());
expect(reservation.getFailures()).andReturn(Map.of(textent,
ranges)).anyTimes();
@@ -395,7 +395,7 @@ public class ScanServerTest {
TabletResolver resolver = createMock(TabletResolver.class);
TestScanServer ss = partialMockBuilder(TestScanServer.class).createMock();
-expect(sextent.isMeta()).andReturn(true).anyTimes();
+expect(sextent.isSystemTable()).andReturn(true).anyTimes();
expect(reservation.newTablet(ss, sextent)).andReturn(tablet);
expect(reservation.getFailures()).andReturn(Map.of()).anyTimes();
reservation.close();
@@ -444,7 +444,7 @@ public class ScanServerTest {
TabletResolver resolver = createMock(TabletResolver.class);
TestScanServer ss = partialMockBuilder(TestScanServe
(accumulo) branch main updated: Update MetricsIT (#4576)
This is an automated email from the ASF dual-hosted git repository.
edcoleman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/main by this push:
new 5cd9cdfc70 Update MetricsIT (#4576)
5cd9cdfc70 is described below
commit 5cd9cdfc70e8da152d31d23f547eb9f005beba82
Author: EdColeman
AuthorDate: Mon May 20 09:32:52 2024 -0400
Update MetricsIT (#4576)
Changes to counters are now publishing metrics that may have not seen
before:
- METRICS_MAJC_PAUSED
- METRICS_MINC_PAUSED
- METRICS_SCAN_PAUSED_FOR_MEM
- METRICS_SCAN_RETURN_FOR_MEM
- METRICS_UPDATE_ERRORS
Also sorted metrics excludes list to make things easier to find.
---
.../main/java/org/apache/accumulo/test/metrics/MetricsIT.java | 11 ---
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/test/src/main/java/org/apache/accumulo/test/metrics/MetricsIT.java
b/test/src/main/java/org/apache/accumulo/test/metrics/MetricsIT.java
index f2be6e71a5..35efb186ef 100644
--- a/test/src/main/java/org/apache/accumulo/test/metrics/MetricsIT.java
+++ b/test/src/main/java/org/apache/accumulo/test/metrics/MetricsIT.java
@@ -99,13 +99,10 @@ public class MetricsIT extends ConfigurableMacBase
implements MetricsProducer {
doWorkToGenerateMetrics();
cluster.stop();
-Set unexpectedMetrics =
-Set.of(METRICS_SCAN_YIELDS, METRICS_UPDATE_ERRORS,
METRICS_COMPACTOR_MAJC_STUCK,
-METRICS_SCAN_BUSY_TIMEOUT_COUNTER, METRICS_SCAN_PAUSED_FOR_MEM,
-METRICS_SCAN_RETURN_FOR_MEM, METRICS_MINC_PAUSED,
METRICS_MAJC_PAUSED);
-Set flakyMetrics = Set.of(METRICS_GC_WAL_ERRORS,
METRICS_FATE_TYPE_IN_PROGRESS,
-METRICS_SCAN_BUSY_TIMEOUT_COUNTER, METRICS_SCAN_RESERVATION_TIMER,
-METRICS_SCAN_TABLET_METADATA_CACHE);
+Set unexpectedMetrics = Set.of(METRICS_COMPACTOR_MAJC_STUCK,
METRICS_SCAN_YIELDS);
+Set flakyMetrics =
+Set.of(METRICS_FATE_TYPE_IN_PROGRESS, METRICS_GC_WAL_ERRORS,
METRICS_SCAN_RESERVATION_TIMER,
+METRICS_SCAN_BUSY_TIMEOUT_COUNTER,
METRICS_SCAN_TABLET_METADATA_CACHE);
Map expectedMetricNames = this.getMetricFields();
flakyMetrics.forEach(expectedMetricNames::remove); // might not see these
(accumulo) 01/01: Merge branch '2.1'
This is an automated email from the ASF dual-hosted git repository.
dlmarion pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit 64de0f0aa768c040533107f29b715d636631e41e
Merge: 8fe933a671 a8d3a101d7
Author: Dave Marion
AuthorDate: Mon May 20 13:21:02 2024 +
Merge branch '2.1'
.../org/apache/accumulo/tserver/ScanServer.java| 17 +++
.../apache/accumulo/tserver/ScanServerTest.java| 132 ++---
2 files changed, 134 insertions(+), 15 deletions(-)
diff --cc
server/tserver/src/test/java/org/apache/accumulo/tserver/ScanServerTest.java
index cad9d15aff,73a3e0d03b..f93f095a87
---
a/server/tserver/src/test/java/org/apache/accumulo/tserver/ScanServerTest.java
+++
b/server/tserver/src/test/java/org/apache/accumulo/tserver/ScanServerTest.java
@@@ -66,8 -64,9 +66,9 @@@ public class ScanServerTest
private KeyExtent extent;
private TabletResolver resolver;
private ScanReservation reservation;
+ private boolean systemUser;
-protected TestScanServer(ScanServerOpts opts, String[] args) {
+protected TestScanServer(ConfigOpts opts, String[] args) {
super(opts, args);
}
(accumulo) branch main updated (8fe933a671 -> 64de0f0aa7)
This is an automated email from the ASF dual-hosted git repository. dlmarion pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git from 8fe933a671 Add a constraint check for suspend column (#4546) add a8d3a101d7 Only allow system user to perform eventual scans on root and meta (#4531) new 64de0f0aa7 Merge branch '2.1' The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/accumulo/tserver/ScanServer.java| 17 +++ .../apache/accumulo/tserver/ScanServerTest.java| 132 ++--- 2 files changed, 134 insertions(+), 15 deletions(-)
(accumulo) branch 2.1 updated (4b5234bd87 -> a8d3a101d7)
This is an automated email from the ASF dual-hosted git repository. dlmarion pushed a change to branch 2.1 in repository https://gitbox.apache.org/repos/asf/accumulo.git from 4b5234bd87 Added ZK cleanup thread to Manager for Scan Server nodes (#4562) add a8d3a101d7 Only allow system user to perform eventual scans on root and meta (#4531) No new revisions were added by this update. Summary of changes: .../org/apache/accumulo/tserver/ScanServer.java| 17 +++ .../apache/accumulo/tserver/ScanServerTest.java| 132 ++--- 2 files changed, 134 insertions(+), 15 deletions(-)
