(accumulo) branch 2.1 updated (4839d3bf22 -> 714fdb7e1a)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch 2.1 in repository https://gitbox.apache.org/repos/asf/accumulo.git from 4839d3bf22 logs when closing tablets are waiting on compactions (#4472) add 714fdb7e1a logs when tablet decides there is nothing to do for user compaction (#4473) No new revisions were added by this update. Summary of changes: .../java/org/apache/accumulo/tserver/tablet/CompactableImpl.java | 9 + 1 file changed, 9 insertions(+)
(accumulo) branch 2.1 updated (8521cb3178 -> 4839d3bf22)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch 2.1 in repository https://gitbox.apache.org/repos/asf/accumulo.git from 8521cb3178 improves efficiency of migration set clean up (#4474) add 4839d3bf22 logs when closing tablets are waiting on compactions (#4472) No new revisions were added by this update. Summary of changes: .../org/apache/accumulo/tserver/tablet/CompactableImpl.java | 13 +++-- 1 file changed, 11 insertions(+), 2 deletions(-)
(accumulo) branch 2.1 updated (7c6572caf3 -> 8521cb3178)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch 2.1 in repository https://gitbox.apache.org/repos/asf/accumulo.git from 7c6572caf3 fixes bug with starting multiple group of scan servers (#4445) add 8521cb3178 improves efficiency of migration set clean up (#4474) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/accumulo/manager/Manager.java | 14 +- 1 file changed, 9 insertions(+), 5 deletions(-)
(accumulo) branch elasticity updated: batches reading tablet metadata for refresh (#4439)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new cb25f26ab3 batches reading tablet metadata for refresh (#4439)
cb25f26ab3 is described below
commit cb25f26ab3be77dbfce54e9d1e7234e5974510ec
Author: Keith Turner
AuthorDate: Wed Apr 17 17:38:37 2024 -0400
batches reading tablet metadata for refresh (#4439)
Changes tablet server code that refreshes tablet metadata to read tablet
metadata for multiple tablets in a batch. Also noticed the refresh code
was not handling some race condition correctly and could result in
spurious error messages. It was also not handling closed tablets.
Refactored the code to return false when refresh is not possble for
acceptable reasons instead of throw an error. This will cause the the
RPC to retry for those tablets later. Also remove some code that was
attempting to handle complex race conditions that would be hard to test,
realized if these conditions do happen that it would be best to retry
rather than try to handle them.
Co-authored-by: Dave Marion
---
.../accumulo/tserver/TabletClientHandler.java | 86 -
.../org/apache/accumulo/tserver/tablet/Tablet.java | 137 +
2 files changed, 138 insertions(+), 85 deletions(-)
diff --git
a/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletClientHandler.java
b/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletClientHandler.java
index f8532de0b8..a1770b90cb 100644
---
a/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletClientHandler.java
+++
b/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletClientHandler.java
@@ -31,6 +31,7 @@ import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
+import java.util.Optional;
import java.util.Set;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
@@ -1096,80 +1097,47 @@ public class TabletClientHandler implements
TabletServerClientService.Iface,
// handle that more expensive case if needed.
var tabletsSnapshot = server.getOnlineTablets();
-Set notFound = new HashSet<>();
+Map refreshSessions = new HashMap<>();
+
+// Created this as synchronized list because it's passed to a lambda that
could possibly run in
+// another thread.
+List unableToRefresh = Collections.synchronizedList(new
ArrayList<>());
for (var tkextent : refreshes) {
var extent = KeyExtent.fromThrift(tkextent);
-
var tablet = tabletsSnapshot.get(extent);
if (tablet != null) {
-// ELASTICITY_TODO use a batch reader to read all tablets metadata at
once instead of one by
-// one. This may be a bit tricky from a synchronization perspective
(with multiple tablets
-// and multiple concurrent refresh request), so defer doing this until
after removing
-// functionality from the tablet. No need to make the change now and
have to change it
-// later.
-tablet.refreshMetadata(RefreshPurpose.REFRESH_RPC);
+refreshSessions.put(extent, tablet.startRefresh());
} else {
-notFound.add(extent);
+unableToRefresh.add(extent.toThrift());
}
}
-if (!notFound.isEmpty()) {
- // Some tablets were not found, lets see if they are loading or moved to
online while doing
- // the refreshes above.
- List unableToRefresh = new ArrayList<>();
- List foundTablets = new ArrayList<>();
-
- synchronized (server.unopenedTablets) {
-synchronized (server.openingTablets) {
- synchronized (server.onlineTablets) {
-// Get the snapshot again, however this time nothing will be
changing while we iterate
-// over the snapshot because all three locks are held.
-tabletsSnapshot = server.getOnlineTablets();
-for (var extent : notFound) {
- // TODO investigate if its safe to ignore tablets in the
unopened set because they
- // have not yet read any metadata
- if (server.unopenedTablets.contains(extent)
- || server.openingTablets.contains(extent)) {
-// Can not refresh these tablets that are in the process of
loading, but they may
-// still need refreshing because we don't know when they read
their metadata
-// relative to the refresh event.
-unableToRefresh.add(extent.toThrift());
- } else {
-var tablet = tabletsSnapshot.get(extent);
-if (tablet != null) {
- // Intentionally not calling refresh on the tablet while
holding these locks.
-
(accumulo) branch elasticity updated (016dda7ffe -> d3264f1ffa)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from 016dda7ffe removes selector compactions (#4397) add 5bea0f6b26 Deprecates per table selector compactions (#4398) new e3742b0d77 Merge branch 'main' into elasticity new d3264f1ffa removes uneeded deprecation suppression The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes:
(accumulo) 01/02: Merge branch 'main' into elasticity
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit e3742b0d774d0214dbcd276fa4ffcffe7db4ed4b
Merge: 016dda7ffe 5bea0f6b26
Author: Keith Turner
AuthorDate: Wed Apr 17 16:18:58 2024 -0400
Merge branch 'main' into elasticity
.../java/org/apache/accumulo/test/compaction/CompactionExecutorIT.java | 1 +
1 file changed, 1 insertion(+)
diff --cc
test/src/main/java/org/apache/accumulo/test/compaction/CompactionExecutorIT.java
index 67c01286a9,ec9b22f25c..f8d7c31467
---
a/test/src/main/java/org/apache/accumulo/test/compaction/CompactionExecutorIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/compaction/CompactionExecutorIT.java
@@@ -391,10 -323,17 +391,11 @@@ public class CompactionExecutorIT exten
}
@Test
+ @SuppressWarnings("deprecation")
public void testTooManyDeletes() throws Exception {
try (AccumuloClient client =
Accumulo.newClient().from(getClientProps()).build()) {
- Map props = Map.of(Property.TABLE_COMPACTION_SELECTOR.getKey(),
- TooManyDeletesSelector.class.getName(),
- Property.TABLE_COMPACTION_SELECTOR_OPTS.getKey() + "threshold",
".4");
var deleteSummarizerCfg =
SummarizerConfiguration.builder(DeletesSummarizer.class.getName()).build();
- client.tableOperations().create("tmd_selector", new
NewTableConfiguration()
- .setProperties(props).enableSummarization(deleteSummarizerCfg));
client.tableOperations().create("tmd_control1",
new
NewTableConfiguration().enableSummarization(deleteSummarizerCfg));
client.tableOperations().create("tmd_control2",
(accumulo) 02/02: removes uneeded deprecation suppression
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit d3264f1ffa2b6bedaf45f6c5af5b9f7eac439139
Author: Keith Turner
AuthorDate: Wed Apr 17 16:27:06 2024 -0400
removes uneeded deprecation suppression
---
.../java/org/apache/accumulo/test/compaction/CompactionExecutorIT.java | 1 -
1 file changed, 1 deletion(-)
diff --git
a/test/src/main/java/org/apache/accumulo/test/compaction/CompactionExecutorIT.java
b/test/src/main/java/org/apache/accumulo/test/compaction/CompactionExecutorIT.java
index f8d7c31467..67c01286a9 100644
---
a/test/src/main/java/org/apache/accumulo/test/compaction/CompactionExecutorIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/compaction/CompactionExecutorIT.java
@@ -391,7 +391,6 @@ public class CompactionExecutorIT extends
SharedMiniClusterBase {
}
@Test
- @SuppressWarnings("deprecation")
public void testTooManyDeletes() throws Exception {
try (AccumuloClient client =
Accumulo.newClient().from(getClientProps()).build()) {
var deleteSummarizerCfg =
(accumulo) branch elasticity updated: removes selector compactions (#4397)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 016dda7ffe removes selector compactions (#4397)
016dda7ffe is described below
commit 016dda7ffe6f67a096af38a9e6f23133747c7c2c
Author: Keith Turner
AuthorDate: Wed Apr 17 16:02:00 2024 -0400
removes selector compactions (#4397)
Co-authored-by: Daniel Roberts
---
.../core/client/admin/compaction/CompactionSelector.java | 3 +--
core/src/main/java/org/apache/accumulo/core/conf/Property.java | 9 +
.../apache/accumulo/core/spi/compaction/CompactionKind.java| 6 --
.../core/spi/compaction/RatioBasedCompactionPlanner.java | 10 --
.../accumulo/core/tabletserver/thrift/TCompactionKind.java | 3 ---
core/src/main/thrift/tabletserver.thrift | 6 +++---
.../core/util/compaction/CompactionPrioritizerTest.java| 4 ++--
.../src/main/java/org/apache/accumulo/compactor/ExtCEnv.java | 1 -
.../manager/compaction/coordinator/CompactionCoordinator.java | 2 --
.../compaction/coordinator/commit/CommitCompaction.java| 4 ++--
10 files changed, 13 insertions(+), 35 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/CompactionSelector.java
b/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/CompactionSelector.java
index 281372d43d..311ffc730e 100644
---
a/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/CompactionSelector.java
+++
b/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/CompactionSelector.java
@@ -35,8 +35,7 @@ import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
/**
- * This class selects which files a user compaction will compact. It can also
be configured per
- * table to periodically select files to compact.
+ * This class selects which files a user compaction will compact.
*
* @since 2.1.0
*/
diff --git a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
index e33eed2748..b05e0aba4e 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
@@ -855,12 +855,6 @@ public enum Property {
+ "specified time. If a system compaction cancels a hold and runs,
then the user compaction"
+ " can reselect and hold files after the system compaction runs.",
"2.1.0"),
- TABLE_COMPACTION_SELECTOR("table.compaction.selector", "",
PropertyType.CLASSNAME,
- "A configurable selector for a table that can periodically select file
for mandatory "
- + "compaction, even if the files do not meet the compaction ratio.",
- "2.1.0"),
- TABLE_COMPACTION_SELECTOR_OPTS("table.compaction.selector.opts.", null,
PropertyType.PREFIX,
- "Options for the table compaction dispatcher.", "2.1.0"),
TABLE_COMPACTION_CONFIGURER("table.compaction.configurer", "",
PropertyType.CLASSNAME,
"A plugin that can dynamically configure compaction output files based
on input files.",
"2.1.0"),
@@ -1414,8 +1408,7 @@ public enum Property {
|| key.startsWith(TABLE_SUMMARIZER_PREFIX.getKey())
|| key.startsWith(TABLE_SCAN_DISPATCHER_OPTS.getKey())
|| key.startsWith(TABLE_COMPACTION_DISPATCHER_OPTS.getKey())
-|| key.startsWith(TABLE_COMPACTION_CONFIGURER_OPTS.getKey())
-|| key.startsWith(TABLE_COMPACTION_SELECTOR_OPTS.getKey()))
+|| key.startsWith(TABLE_COMPACTION_CONFIGURER_OPTS.getKey()))
|| key.startsWith(TABLE_CRYPTO_PREFIX.getKey()));
}
diff --git
a/core/src/main/java/org/apache/accumulo/core/spi/compaction/CompactionKind.java
b/core/src/main/java/org/apache/accumulo/core/spi/compaction/CompactionKind.java
index 97e64216fc..7667dcf577 100644
---
a/core/src/main/java/org/apache/accumulo/core/spi/compaction/CompactionKind.java
+++
b/core/src/main/java/org/apache/accumulo/core/spi/compaction/CompactionKind.java
@@ -18,8 +18,6 @@
*/
package org.apache.accumulo.core.spi.compaction;
-import org.apache.accumulo.core.client.admin.compaction.CompactionSelector;
-
/**
* @since 2.1.0
* @see org.apache.accumulo.core.spi.compaction
@@ -29,10 +27,6 @@ public enum CompactionKind {
* A system initiated routine compaction.
*/
SYSTEM,
- /**
- * Set of files selected by a {@link CompactionSelector} configured for a
table.
- */
- SELECTOR,
/**
* A user initiated a one time compaction using an Accumulo client.
*/
diff --
(accumulo) branch main updated: Deprecates per table selector compactions (#4398)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/main by this push:
new 5bea0f6b26 Deprecates per table selector compactions (#4398)
5bea0f6b26 is described below
commit 5bea0f6b2674188b5c242e78e177d1a8c11ba4b7
Author: Keith Turner
AuthorDate: Wed Apr 17 15:50:48 2024 -0400
Deprecates per table selector compactions (#4398)
See #4081 for details
---
.../admin/compaction/CompactionSelector.java | 3 +-
.../org/apache/accumulo/core/conf/Property.java| 5 ++-
.../schema/ExternalCompactionMetadata.java | 3 +-
.../core/spi/compaction/CompactionKind.java| 10 ++
.../spi/compaction/DefaultCompactionPlanner.java | 6 ++--
.../util/compaction/CompactionJobPrioritizer.java | 1 +
.../compaction/DeprecatedCompactionKind.java} | 25 -
.../util/compaction/CompactionPrioritizerTest.java | 4 +--
.../accumulo/tserver/tablet/CompactableImpl.java | 41 +-
.../accumulo/tserver/tablet/CompactableUtils.java | 17 +
.../apache/accumulo/tserver/tablet/MajCEnv.java| 1 +
.../tablet/CompactableImplFileManagerTest.java | 2 +-
.../tserver/tablet/CompactableImplTest.java| 9 ++---
.../test/compaction/CompactionExecutorIT.java | 1 +
14 files changed, 75 insertions(+), 53 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/CompactionSelector.java
b/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/CompactionSelector.java
index 281372d43d..713a451dc0 100644
---
a/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/CompactionSelector.java
+++
b/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/CompactionSelector.java
@@ -36,7 +36,8 @@ import
org.apache.accumulo.core.iterators.SortedKeyValueIterator;
/**
* This class selects which files a user compaction will compact. It can also
be configured per
- * table to periodically select files to compact.
+ * table to periodically select files to compact, although per table
functionality is deprecated.
+ * See {@link org.apache.accumulo.core.spi.compaction.CompactionKind#SELECTOR}
for details.
*
* @since 2.1.0
*/
diff --git a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
index f86e4cad3b..7d48c05ae1 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
@@ -916,10 +916,13 @@ public enum Property {
+ "specified time. If a system compaction cancels a hold and runs,
then the user compaction"
+ " can reselect and hold files after the system compaction runs.",
"2.1.0"),
+ @Deprecated(since = "3.1")
TABLE_COMPACTION_SELECTOR("table.compaction.selector", "",
PropertyType.CLASSNAME,
"A configurable selector for a table that can periodically select file
for mandatory "
- + "compaction, even if the files do not meet the compaction ratio.",
+ + "compaction, even if the files do not meet the compaction ratio.
This option was deprecated in "
+ + "3.1, see the CompactionKind.SELECTOR enum javadoc for details.",
"2.1.0"),
+ @Deprecated(since = "3.1")
TABLE_COMPACTION_SELECTOR_OPTS("table.compaction.selector.opts.", null,
PropertyType.PREFIX,
"Options for the table compaction dispatcher.", "2.1.0"),
TABLE_COMPACTION_CONFIGURER("table.compaction.configurer", "",
PropertyType.CLASSNAME,
diff --git
a/core/src/main/java/org/apache/accumulo/core/metadata/schema/ExternalCompactionMetadata.java
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/ExternalCompactionMetadata.java
index a5beddd7ad..4c7e4298ff 100644
---
a/core/src/main/java/org/apache/accumulo/core/metadata/schema/ExternalCompactionMetadata.java
+++
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/ExternalCompactionMetadata.java
@@ -31,6 +31,7 @@ import org.apache.accumulo.core.metadata.StoredTabletFile;
import org.apache.accumulo.core.spi.compaction.CompactionExecutorId;
import org.apache.accumulo.core.spi.compaction.CompactionKind;
import org.apache.accumulo.core.util.compaction.CompactionExecutorIdImpl;
+import org.apache.accumulo.core.util.compaction.DeprecatedCompactionKind;
public class ExternalCompactionMetadata {
@@ -50,7 +51,7 @@ public class ExternalCompactionMetadata {
CompactionExecutorId ceid, boolean propagateDeletes, boolean
initiallySelectedAll,
Long compactionId) {
if (!initiallySelectedAll && !propagateDeletes
-
(accumulo) branch elasticity updated: fixes continue scan in tablet mgmt iterator (#4457)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 6e0f2a14fa fixes continue scan in tablet mgmt iterator (#4457)
6e0f2a14fa is described below
commit 6e0f2a14fa051adb2c7873877bc59e1bcb0ad0b8
Author: Keith Turner
AuthorDate: Mon Apr 15 11:25:48 2024 -0400
fixes continue scan in tablet mgmt iterator (#4457)
Many places in the accumulo code will read a batch of key/values and
then use the last key in the batch to construct a range to get the next
batch. The last key in the batch will be a non inclusive start key for
the range. The tablet mgmt iterator was not handling this case and
returning the key that should have been excluded.
---
.../manager/state/TabletManagementIterator.java| 6 ++-
.../functional/TabletManagementIteratorIT.java | 51 +-
2 files changed, 44 insertions(+), 13 deletions(-)
diff --git
a/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletManagementIterator.java
b/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletManagementIterator.java
index b3ebe61c1d..3f5397a7a4 100644
---
a/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletManagementIterator.java
+++
b/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletManagementIterator.java
@@ -217,7 +217,11 @@ public class TabletManagementIterator extends
SkippingIterator {
// can pull this K,V pair from the results by looking at the colf.
TabletManagement.addActions(decodedRow, actions);
}
-topKey = decodedRow.firstKey();
+
+// This key is being created exactly the same way as the whole row
iterator creates keys.
+// This is important for ensuring that seek works as expected in the
continue case. See
+// WholeRowIterator seek function for details, it looks for keys w/o
columns.
+topKey = new Key(decodedRow.firstKey().getRow());
topValue = WholeRowIterator.encodeRow(new
ArrayList<>(decodedRow.keySet()),
new ArrayList<>(decodedRow.values()));
LOG.trace("Returning extent {} with reasons: {}", tm.getExtent(),
actions);
diff --git
a/test/src/main/java/org/apache/accumulo/test/functional/TabletManagementIteratorIT.java
b/test/src/main/java/org/apache/accumulo/test/functional/TabletManagementIteratorIT.java
index 6d74d0c16e..7af3e0ba00 100644
---
a/test/src/main/java/org/apache/accumulo/test/functional/TabletManagementIteratorIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/functional/TabletManagementIteratorIT.java
@@ -18,8 +18,13 @@
*/
package org.apache.accumulo.test.functional;
+import static
org.apache.accumulo.core.manager.state.TabletManagement.ManagementAction.BAD_STATE;
import static
org.apache.accumulo.core.manager.state.TabletManagement.ManagementAction.NEEDS_LOCATION_UPDATE;
+import static
org.apache.accumulo.core.manager.state.TabletManagement.ManagementAction.NEEDS_RECOVERY;
+import static
org.apache.accumulo.core.manager.state.TabletManagement.ManagementAction.NEEDS_SPLITTING;
+import static
org.apache.accumulo.core.manager.state.TabletManagement.ManagementAction.NEEDS_VOLUME_REPLACEMENT;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import java.io.IOException;
import java.time.Duration;
@@ -187,6 +192,10 @@ public class TabletManagementIteratorIT extends
AccumuloClusterHarness {
assertEquals(expected, findTabletsNeedingAttention(client, metaCopy1,
tabletMgmtParams),
"Should have four tablets with hosting availability changes");
+ // test continue scan functionality, this test needs a table and tablet
mgmt params that will
+ // return more than one tablet
+ testContinueScan(client, metaCopy1, tabletMgmtParams);
+
// test the assigned case (no location)
removeLocation(client, metaCopy1, t3);
expected =
@@ -210,18 +219,15 @@ public class TabletManagementIteratorIT extends
AccumuloClusterHarness {
// Test the recovery cases
createLogEntry(client, metaCopy5, t1);
setTabletAvailability(client, metaCopy5, t1,
TabletAvailability.UNHOSTED.name());
- expected = Map.of(endR1,
- Set.of(NEEDS_LOCATION_UPDATE,
TabletManagement.ManagementAction.NEEDS_RECOVERY));
+ expected = Map.of(endR1, Set.of(NEEDS_LOCATION_UPDATE, NEEDS_RECOVERY));
assertEquals(expected, findTabletsNeedingAttention(client, metaCopy5,
tabletMgmtParams),
"Only 1 of 2 tablets in table t1 should be returned");
setTabletAvailability(client, metaCopy5, t1,
TabletAvailability.ONDEMAND.name());
- expected = Map.of(endR
(accumulo) branch elasticity updated: refactored findTabletsNeedingAttention test to handle new return type. (#4418)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 370a7ded78 refactored findTabletsNeedingAttention test to handle new
return type. (#4418)
370a7ded78 is described below
commit 370a7ded78379ad312e3dc486f671d0c053792d0
Author: Arbaaz Khan
AuthorDate: Fri Apr 12 17:20:47 2024 -0400
refactored findTabletsNeedingAttention test to handle new return type.
(#4418)
---
.../functional/TabletManagementIteratorIT.java | 93 --
1 file changed, 68 insertions(+), 25 deletions(-)
diff --git
a/test/src/main/java/org/apache/accumulo/test/functional/TabletManagementIteratorIT.java
b/test/src/main/java/org/apache/accumulo/test/functional/TabletManagementIteratorIT.java
index 22ae8c6535..6d74d0c16e 100644
---
a/test/src/main/java/org/apache/accumulo/test/functional/TabletManagementIteratorIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/functional/TabletManagementIteratorIT.java
@@ -18,12 +18,14 @@
*/
package org.apache.accumulo.test.functional;
+import static
org.apache.accumulo.core.manager.state.TabletManagement.ManagementAction.NEEDS_LOCATION_UPDATE;
import static org.junit.jupiter.api.Assertions.assertEquals;
import java.io.IOException;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collections;
+import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
@@ -141,15 +143,30 @@ public class TabletManagementIteratorIT extends
AccumuloClusterHarness {
// examine a clone of the metadata table, so we can manipulate it
copyTable(client, AccumuloTable.METADATA.tableName(), metaCopy1);
+ var tableId1 = getServerContext().getTableId(t1);
+ var tableId3 = getServerContext().getTableId(t3);
+ var tableId4 = getServerContext().getTableId(t4);
+
+ // Create expected KeyExtents to test output of
findTabletsNeedingAttention
+ KeyExtent endR1 = new KeyExtent(tableId1, new Text("some split"), null);
+ KeyExtent endR3 = new KeyExtent(tableId3, new Text("some split"), null);
+ KeyExtent endR4 = new KeyExtent(tableId4, new Text("some split"), null);
+ KeyExtent prevR1 = new KeyExtent(tableId1, null, new Text("some split"));
+ KeyExtent prevR3 = new KeyExtent(tableId3, null, new Text("some split"));
+ KeyExtent prevR4 = new KeyExtent(tableId4, null, new Text("some split"));
+ Map> expected;
+
TabletManagementParameters tabletMgmtParams = createParameters(client);
- int tabletsInFlux = findTabletsNeedingAttention(client, metaCopy1,
tabletMgmtParams);
- while (tabletsInFlux > 0) {
+ Map> tabletsInFlux =
+ findTabletsNeedingAttention(client, metaCopy1, tabletMgmtParams);
+ while (!tabletsInFlux.isEmpty()) {
log.debug("Waiting for {} tablets for {}", tabletsInFlux, metaCopy1);
UtilWaitThread.sleep(500);
copyTable(client, AccumuloTable.METADATA.tableName(), metaCopy1);
tabletsInFlux = findTabletsNeedingAttention(client, metaCopy1,
tabletMgmtParams);
}
- assertEquals(0, findTabletsNeedingAttention(client, metaCopy1,
tabletMgmtParams),
+ expected = Map.of();
+ assertEquals(expected, findTabletsNeedingAttention(client, metaCopy1,
tabletMgmtParams),
"No tables should need attention");
// The metadata table stabilized and metaCopy1 contains a copy suitable
for testing. Before
@@ -165,72 +182,93 @@ public class TabletManagementIteratorIT extends
AccumuloClusterHarness {
// t3 is hosted, setting to never will generate a change to unhost
tablets
setTabletAvailability(client, metaCopy1, t3,
TabletAvailability.UNHOSTED.name());
tabletMgmtParams = createParameters(client);
- assertEquals(4, findTabletsNeedingAttention(client, metaCopy1,
tabletMgmtParams),
+ expected = Map.of(endR1, Set.of(NEEDS_LOCATION_UPDATE), prevR1,
Set.of(NEEDS_LOCATION_UPDATE),
+ endR3, Set.of(NEEDS_LOCATION_UPDATE), prevR3,
Set.of(NEEDS_LOCATION_UPDATE));
+ assertEquals(expected, findTabletsNeedingAttention(client, metaCopy1,
tabletMgmtParams),
"Should have four tablets with hosting availability changes");
// test the assigned case (no location)
removeLocation(client, metaCopy1, t3);
- assertEquals(2, findTabletsNeedingAttention(client, metaCopy1,
tabletMgmtParams),
+ expected =
+ Map.of(endR1, Set.of(NEEDS_LOCATION_UPDATE), prevR1,
Set.of(NEEDS_LOCATION_UPDATE));
+ assertEquals(expected, findTabletsNeedingAttention(client, metaCopy1,
tabletMgmtParams),
"Should have two tablets without a loc");
// Test setting the operation i
(accumulo) branch elasticity updated: removes TODO that is done
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 4081f460ae removes TODO that is done
4081f460ae is described below
commit 4081f460aea41342904929b0317edd5b650b69fb
Author: Keith Turner
AuthorDate: Thu Apr 11 20:12:29 2024 -0400
removes TODO that is done
---
.../base/src/main/java/org/apache/accumulo/server/split/SplitUtils.java | 2 --
1 file changed, 2 deletions(-)
diff --git
a/server/base/src/main/java/org/apache/accumulo/server/split/SplitUtils.java
b/server/base/src/main/java/org/apache/accumulo/server/split/SplitUtils.java
index eb5ceec1bf..8f64c461bc 100644
--- a/server/base/src/main/java/org/apache/accumulo/server/split/SplitUtils.java
+++ b/server/base/src/main/java/org/apache/accumulo/server/split/SplitUtils.java
@@ -203,8 +203,6 @@ public class SplitUtils {
var threshold = tableConf.getAsBytes(Property.TABLE_SPLIT_THRESHOLD);
var maxEndRowSize = tableConf.getAsBytes(Property.TABLE_MAX_END_ROW_SIZE);
-// ELASTICITY_TODO rename and deprecate property. This is not executing in
the tablet server
-// anymore.
int maxFilesToOpen = tableConf.getCount(Property.SPLIT_MAXOPEN);
var estimatedSize = tabletMetadata.getFileSize();
(accumulo) 01/01: Merge branch 'main' into elasticity
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit f337357a527b1d8beab4ad6573fdde3229bd101d
Merge: 9fd7338ba4 859694aba8
Author: Keith Turner
AuthorDate: Thu Apr 11 20:07:16 2024 -0400
Merge branch 'main' into elasticity
assemble/bin/accumulo-cluster| 2 +-
.../java/org/apache/accumulo/core/conf/Property.java | 16
.../server/manager/state/TabletManagementIterator.java | 4 ++--
.../org/apache/accumulo/server/split/SplitUtils.java | 4 ++--
4 files changed, 13 insertions(+), 13 deletions(-)
diff --cc assemble/bin/accumulo-cluster
index 82e53dbd4a,f525093404..b157f04c1e
--- a/assemble/bin/accumulo-cluster
+++ b/assemble/bin/accumulo-cluster
@@@ -156,9 -131,9 +156,9 @@@ function control_service()
for ((inst_id = 1; inst_id <= last_instance_id; inst_id++)); do
ACCUMULO_SERVICE_INSTANCE=""
-[[ $service == "tserver" && ${NUM_TSERVERS:-1} -gt 1 ]] &&
ACCUMULO_SERVICE_INSTANCE=${inst_id}
-[[ $service == "compactor" ]] &&
ACCUMULO_SERVICE_INSTANCE="${inst_id}_${5}"
-[[ $service == "sserver" ]] && ACCUMULO_SERVICE_INSTANCE="${inst_id}_${5}"
+[[ $service == "tserver" && ${NUM_TSERVERS:-1} -gt 1 ]] &&
ACCUMULO_SERVICE_INSTANCE="_${group}_${inst_id}"
+[[ $service == "compactor" ]] &&
ACCUMULO_SERVICE_INSTANCE="_${group}_${inst_id}"
- [[ $service == "sserver" && ${NUM_SSERVERS:-1} -gt 1 ]] &&
ACCUMULO_SERVICE_INSTANCE="_${group}_${inst_id}"
++[[ $service == "sserver" ]] &&
ACCUMULO_SERVICE_INSTANCE="_${group}_${inst_id}"
if [[ $host == localhost || $host == "$(hostname -s)" || $host ==
"$(hostname -f)" || "$(hostname -I)" =~ $host ]]; then
#
diff --cc core/src/main/java/org/apache/accumulo/core/conf/Property.java
index d5a9504f28,f86e4cad3b..e33eed2748
--- a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
@@@ -433,13 -395,15 +433,21 @@@ public enum Property
+ "indefinitely. Default is 0 to block indefinitely. Only valid
when tserver available "
+ "threshold is set greater than 0.",
"1.10.0"),
+ MANAGER_SPLIT_WORKER_THREADS("manager.split.inspection.threadpool.size",
"8", PropertyType.COUNT,
+ "The number of threads used to inspect tablets files to find split
points.", "4.0.0"),
+
+
MANAGER_COMPACTION_SERVICE_PRIORITY_QUEUE_SIZE("manager.compaction.major.service.queue.size",
+ // ELASTICITY_TODO: It might be good to note that there is a priority
queue per compactor
+ // resource group
+ "1", PropertyType.COUNT, "The max size of the priority queue.",
"4.0"),
+ SPLIT_PREFIX("split.", null, PropertyType.PREFIX,
+ "System wide properties related to splitting tablets.", "3.1.0"),
+ SPLIT_MAXOPEN("split.files.max", "300", PropertyType.COUNT,
+ "To find a tablets split points, all RFiles are opened and their
indexes"
+ + " are read. This setting determines how many RFiles can be opened
at once."
- + " When there are more RFiles than this setting multiple passes
must be"
- + " made, which is slower. However opening too many RFiles at once
can cause"
- + " problems.",
++ + " When there are more RFiles than this setting the tablet will be
marked"
++ + " as un-splittable.",
+ "3.1.0"),
// properties that are specific to scan server behavior
@Experimental
SSERV_PREFIX("sserver.", null, PropertyType.PREFIX,
@@@ -545,14 -505,16 +553,6 @@@
TSERV_TOTAL_MUTATION_QUEUE_MAX("tserver.total.mutation.queue.max", "5%",
PropertyType.MEMORY,
"The amount of memory used to store write-ahead-log mutations before
flushing them.",
"1.7.0"),
- @ReplacedBy(property = SPLIT_MAXOPEN)
- @Deprecated(since = "3.1")
--
TSERV_TABLET_SPLIT_FINDMIDPOINT_MAXOPEN("tserver.tablet.split.midpoint.files.max",
"300",
-- PropertyType.COUNT,
-- "To find a tablets split points, all RFiles are opened and their
indexes"
-- + " are read. This setting determines how many RFiles can be opened
at once."
-- + " When there are more RFiles than this setting multiple passes
must be"
-- + " made,
(accumulo) branch elasticity updated (9fd7338ba4 -> f337357a52)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from 9fd7338ba4 Documents causes of error in ConditionalTabletsMutatorImpl (#4449) add 61e81c43d8 falls back to deprecated compaction prop (#4451) add c39599d75d Renames split property to avoid tserver (#4450) add 3885a25df6 corrects illegal json in javadoc (#4446) add 7c6572caf3 fixes bug with starting multiple group of scan servers (#4445) add 859694aba8 Merge branch '2.1' new f337357a52 Merge branch 'main' into elasticity The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: assemble/bin/accumulo-cluster| 2 +- .../java/org/apache/accumulo/core/conf/Property.java | 16 .../server/manager/state/TabletManagementIterator.java | 4 ++-- .../org/apache/accumulo/server/split/SplitUtils.java | 4 ++-- 4 files changed, 13 insertions(+), 13 deletions(-)
(accumulo) 01/01: Merge branch '2.1'
This is an automated email from the ASF dual-hosted git repository. kturner pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git commit 859694aba88b64ce3f9c5559c0f772f987dbb0e4 Merge: c39599d75d 7c6572caf3 Author: Keith Turner AuthorDate: Thu Apr 11 19:22:12 2024 -0400 Merge branch '2.1' assemble/bin/accumulo-cluster | 2 +- .../apache/accumulo/core/spi/compaction/DefaultCompactionPlanner.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
(accumulo) branch main updated (c39599d75d -> 859694aba8)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git from c39599d75d Renames split property to avoid tserver (#4450) add 3885a25df6 corrects illegal json in javadoc (#4446) add 7c6572caf3 fixes bug with starting multiple group of scan servers (#4445) new 859694aba8 Merge branch '2.1' The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: assemble/bin/accumulo-cluster | 2 +- .../apache/accumulo/core/spi/compaction/DefaultCompactionPlanner.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
(accumulo) branch 2.1 updated: fixes bug with starting multiple group of scan servers (#4445)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch 2.1
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/2.1 by this push:
new 7c6572caf3 fixes bug with starting multiple group of scan servers
(#4445)
7c6572caf3 is described below
commit 7c6572caf336cd76bb794befe7a31dce3503cd4b
Author: Keith Turner
AuthorDate: Thu Apr 11 19:19:36 2024 -0400
fixes bug with starting multiple group of scan servers (#4445)
When trying to start and stop multiple groups of scan servers using the
accumulo-cluster script the groups would interfere with each other.
This change fixes that by making the the scan servers use the group in
the instance id like compactors do.
Ran into this while workiing on # and pulled it out as a stand alone
fix.
---
assemble/bin/accumulo-cluster | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/assemble/bin/accumulo-cluster b/assemble/bin/accumulo-cluster
index 5dd9de7e4e..d31d65989c 100755
--- a/assemble/bin/accumulo-cluster
+++ b/assemble/bin/accumulo-cluster
@@ -141,7 +141,7 @@ function control_service() {
ACCUMULO_SERVICE_INSTANCE=""
[[ $service == "tserver" && ${NUM_TSERVERS:-1} -gt 1 ]] &&
ACCUMULO_SERVICE_INSTANCE=${inst_id}
[[ $service == "compactor" ]] &&
ACCUMULO_SERVICE_INSTANCE="${inst_id}_${5}"
-[[ $service == "sserver" && ${NUM_SSERVERS:-1} -gt 1 ]] &&
ACCUMULO_SERVICE_INSTANCE=${inst_id}
+[[ $service == "sserver" ]] && ACCUMULO_SERVICE_INSTANCE="${inst_id}_${5}"
if [[ $host == localhost || $host == "$(hostname -s)" || $host ==
"$(hostname -f)" || "$(hostname -I)" =~ $host ]]; then
#
(accumulo) branch elasticity updated: Documents causes of error in ConditionalTabletsMutatorImpl (#4449)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 9fd7338ba4 Documents causes of error in ConditionalTabletsMutatorImpl
(#4449)
9fd7338ba4 is described below
commit 9fd7338ba4381cd21bc471bc0bf8d37c04a9b0b3
Author: Keith Turner
AuthorDate: Thu Apr 11 19:18:27 2024 -0400
Documents causes of error in ConditionalTabletsMutatorImpl (#4449)
---
.../accumulo/server/metadata/ConditionalTabletsMutatorImpl.java | 9 +
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git
a/server/base/src/main/java/org/apache/accumulo/server/metadata/ConditionalTabletsMutatorImpl.java
b/server/base/src/main/java/org/apache/accumulo/server/metadata/ConditionalTabletsMutatorImpl.java
index 4401654313..2e999033fa 100644
---
a/server/base/src/main/java/org/apache/accumulo/server/metadata/ConditionalTabletsMutatorImpl.java
+++
b/server/base/src/main/java/org/apache/accumulo/server/metadata/ConditionalTabletsMutatorImpl.java
@@ -273,9 +273,6 @@ public class ConditionalTabletsMutatorImpl implements
Ample.ConditionalTabletsMu
private void
ensureAllExtentsSeen(HashMap resultsMap,
Set extentsSet) {
if (!resultsMap.keySet().equals(Set.copyOf(extents.values( {
- // ELASTICITY_TODO this check can trigger if someone forgets to submit,
could check for
- // that
-
Sets.difference(resultsMap.keySet(), extentsSet)
.forEach(extent -> log.error("Unexpected extent seen in in result
{}", extent));
@@ -286,7 +283,11 @@ public class ConditionalTabletsMutatorImpl implements
Ample.ConditionalTabletsMu
log.error("result seen {} {}", keyExtent, new
Text(result.getMutation().getRow()));
});
- throw new AssertionError("Not all extents were seen, this is
unexpected");
+ // There are at least two possible causes for this condition. First
there is a bug in the
+ // ConditionalWriter, and it did not return a result for a mutation that
was given to it.
+ // Second, Ample code was not used correctly and mutating an extent was
started but never
+ // submitted.
+ throw new IllegalStateException("Not all extents were seen, this is
unexpected.");
}
}
(accumulo) branch 2.1 updated: corrects illegal json in javadoc (#4446)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch 2.1
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/2.1 by this push:
new 3885a25df6 corrects illegal json in javadoc (#4446)
3885a25df6 is described below
commit 3885a25df6f448846fa33f9f337336e49585f73e
Author: Keith Turner
AuthorDate: Thu Apr 11 19:18:11 2024 -0400
corrects illegal json in javadoc (#4446)
---
.../apache/accumulo/core/spi/compaction/DefaultCompactionPlanner.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlanner.java
b/core/src/main/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlanner.java
index 5a30556f63..73952b7900 100644
---
a/core/src/main/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlanner.java
+++
b/core/src/main/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlanner.java
@@ -99,7 +99,7 @@ import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
* [
* {"name":"small", "type": "internal", "maxSize":"100M","numThreads":3},
* {"name":"medium", "type": "internal", "maxSize":"500M","numThreads":3},
- * {"name": "large", "type": "external", "queue", "Queue1"}
+ * {"name":"large", "type":"external", "queue":"Queue1"}
* ]}
*
*
(accumulo) branch main updated: Renames split property to avoid tserver (#4450)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/main by this push:
new c39599d75d Renames split property to avoid tserver (#4450)
c39599d75d is described below
commit c39599d75d9eed7a00d2bb506d2a3a35090be00a
Author: Keith Turner
AuthorDate: Thu Apr 11 19:17:43 2024 -0400
Renames split property to avoid tserver (#4450)
In 4.0.0 split code will no longer run in the tablet server. This
change deprecates a split related property that includes tablet server
in the property perfix and replaces it with a property that does
not include tablet sever.
---
.../main/java/org/apache/accumulo/core/conf/Property.java | 11 +++
.../main/java/org/apache/accumulo/server/util/FileUtil.java | 13 +
2 files changed, 20 insertions(+), 4 deletions(-)
diff --git a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
index e2afb7defb..f86e4cad3b 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
@@ -395,6 +395,15 @@ public enum Property {
+ "indefinitely. Default is 0 to block indefinitely. Only valid when
tserver available "
+ "threshold is set greater than 0.",
"1.10.0"),
+ SPLIT_PREFIX("split.", null, PropertyType.PREFIX,
+ "System wide properties related to splitting tablets.", "3.1.0"),
+ SPLIT_MAXOPEN("split.files.max", "300", PropertyType.COUNT,
+ "To find a tablets split points, all RFiles are opened and their indexes"
+ + " are read. This setting determines how many RFiles can be opened
at once."
+ + " When there are more RFiles than this setting multiple passes
must be"
+ + " made, which is slower. However opening too many RFiles at once
can cause"
+ + " problems.",
+ "3.1.0"),
// properties that are specific to scan server behavior
@Experimental
SSERV_PREFIX("sserver.", null, PropertyType.PREFIX,
@@ -496,6 +505,8 @@ public enum Property {
TSERV_TOTAL_MUTATION_QUEUE_MAX("tserver.total.mutation.queue.max", "5%",
PropertyType.MEMORY,
"The amount of memory used to store write-ahead-log mutations before
flushing them.",
"1.7.0"),
+ @ReplacedBy(property = SPLIT_MAXOPEN)
+ @Deprecated(since = "3.1")
TSERV_TABLET_SPLIT_FINDMIDPOINT_MAXOPEN("tserver.tablet.split.midpoint.files.max",
"300",
PropertyType.COUNT,
"To find a tablets split points, all RFiles are opened and their indexes"
diff --git
a/server/base/src/main/java/org/apache/accumulo/server/util/FileUtil.java
b/server/base/src/main/java/org/apache/accumulo/server/util/FileUtil.java
index a600dff94b..dd1dd91ed2 100644
--- a/server/base/src/main/java/org/apache/accumulo/server/util/FileUtil.java
+++ b/server/base/src/main/java/org/apache/accumulo/server/util/FileUtil.java
@@ -33,6 +33,7 @@ import java.util.SortedMap;
import java.util.TreeMap;
import java.util.stream.Collectors;
+import org.apache.accumulo.core.conf.AccumuloConfiguration;
import org.apache.accumulo.core.conf.Property;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.PartialKey;
@@ -214,8 +215,7 @@ public class FileUtil {
Path tmpDir = null;
-int maxToOpen =
-
context.getConfiguration().getCount(Property.TSERV_TABLET_SPLIT_FINDMIDPOINT_MAXOPEN);
+int maxToOpen = getMaxFilesToOpen(context.getConfiguration());
ArrayList readers = new ArrayList<>(dataFiles.size());
try {
@@ -276,6 +276,12 @@ public class FileUtil {
}
}
+ @SuppressWarnings("deprecation")
+ private static int getMaxFilesToOpen(AccumuloConfiguration conf) {
+return conf.getCount(
+conf.resolve(Property.SPLIT_MAXOPEN,
Property.TSERV_TABLET_SPLIT_FINDMIDPOINT_MAXOPEN));
+ }
+
/**
*
* @param dataFiles - list of data files to find the mid point key
@@ -294,8 +300,7 @@ public class FileUtil {
Path tmpDir = null;
-int maxToOpen =
-
context.getConfiguration().getCount(Property.TSERV_TABLET_SPLIT_FINDMIDPOINT_MAXOPEN);
+int maxToOpen = getMaxFilesToOpen(context.getConfiguration());
ArrayList readers = new ArrayList<>(dataFiles.size());
try {
(accumulo) branch main updated: falls back to deprecated compaction prop (#4451)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/main by this push:
new 61e81c43d8 falls back to deprecated compaction prop (#4451)
61e81c43d8 is described below
commit 61e81c43d852be13f270d8e0638d6b4b9b4c5510
Author: Keith Turner
AuthorDate: Thu Apr 11 19:17:22 2024 -0400
falls back to deprecated compaction prop (#4451)
---
.../org/apache/accumulo/server/compaction/CompactionWatcher.java | 8 +++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git
a/server/base/src/main/java/org/apache/accumulo/server/compaction/CompactionWatcher.java
b/server/base/src/main/java/org/apache/accumulo/server/compaction/CompactionWatcher.java
index 4790440ff6..ff90bd5880 100644
---
a/server/base/src/main/java/org/apache/accumulo/server/compaction/CompactionWatcher.java
+++
b/server/base/src/main/java/org/apache/accumulo/server/compaction/CompactionWatcher.java
@@ -62,6 +62,12 @@ public class CompactionWatcher implements Runnable {
this.config = config;
}
+ @SuppressWarnings("deprecation")
+ private static long getCompactionWarnTime(AccumuloConfiguration config) {
+return config.getTimeInMillis(
+config.resolve(Property.COMPACTION_WARN_TIME,
Property.TSERV_COMPACTION_WARN_TIME));
+ }
+
@Override
public void run() {
List runningCompactions =
FileCompactor.getRunningCompactions();
@@ -98,7 +104,7 @@ public class CompactionWatcher implements Runnable {
// remove any compaction that completed or made progress
observedCompactions.keySet().retainAll(newKeys);
-long warnTime = config.getTimeInMillis(Property.COMPACTION_WARN_TIME);
+long warnTime = getCompactionWarnTime(config);
// check for stuck compactions
for (ObservedCompactionInfo oci : observedCompactions.values()) {
(accumulo) branch elasticity updated: fixes conditional mutation bug when loading tablet (#4447)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new f2416680de fixes conditional mutation bug when loading tablet (#4447)
f2416680de is described below
commit f2416680de2f1923800a2c81c75ffefcf952d673
Author: Keith Turner
AuthorDate: Thu Apr 11 19:18:00 2024 -0400
fixes conditional mutation bug when loading tablet (#4447)
In #4436 a tablet update was moved to use conditional mutations. The
changed checked tablets current location in the conditional update.
However when this code was called in the Tablet constructor the tablet
server was the future location. Adjusted to code to check for the
future location when calling from the constructor.
---
.../java/org/apache/accumulo/tserver/tablet/ScanfileManager.java| 5 ++---
.../src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java| 6 --
2 files changed, 6 insertions(+), 5 deletions(-)
diff --git
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/ScanfileManager.java
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/ScanfileManager.java
index 5ab4780a39..a0b00127af 100644
---
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/ScanfileManager.java
+++
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/ScanfileManager.java
@@ -144,7 +144,7 @@ class ScanfileManager {
}
}
- void removeFilesAfterScan(Collection scanFiles) {
+ void removeFilesAfterScan(Collection scanFiles, Location
location) {
if (scanFiles.isEmpty()) {
return;
}
@@ -163,8 +163,7 @@ class ScanfileManager {
if (!filesToDelete.isEmpty()) {
log.debug("Removing scan refs from metadata {} {}", tablet.getExtent(),
filesToDelete);
- var currLoc =
Location.current(tablet.getTabletServer().getTabletSession());
- removeScanFiles(tablet.getExtent(), filesToDelete, tablet.getContext(),
currLoc,
+ removeScanFiles(tablet.getExtent(), filesToDelete, tablet.getContext(),
location,
tablet.getTabletServer().getLock());
}
}
diff --git
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
index 0565a00bbf..2e87b25305 100644
---
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
+++
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
@@ -326,7 +326,8 @@ public class Tablet extends TabletBase {
computeNumEntries();
-getScanfileManager().removeFilesAfterScan(metadata.getScans());
+getScanfileManager().removeFilesAfterScan(metadata.getScans(),
+Location.future(tabletServer.getTabletSession()));
}
public TabletMetadata getMetadata() {
@@ -1614,7 +1615,8 @@ public class Tablet extends TabletBase {
}
if (refreshPurpose == RefreshPurpose.REFRESH_RPC) {
- scanfileManager.removeFilesAfterScan(getMetadata().getScans());
+ scanfileManager.removeFilesAfterScan(getMetadata().getScans(),
+ Location.current(tabletServer.getTabletSession()));
}
}
(accumulo) branch elasticity updated: removes TODO that is done (#4448)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 4e8440a0cc removes TODO that is done (#4448)
4e8440a0cc is described below
commit 4e8440a0ccb449edc26dadfae6f2c01bf977627e
Author: Keith Turner
AuthorDate: Wed Apr 10 22:27:01 2024 -0400
removes TODO that is done (#4448)
This todo was already done in #4072
---
.../tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java | 1 -
1 file changed, 1 deletion(-)
diff --git
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
index 414fedd105..0565a00bbf 100644
---
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
+++
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java
@@ -1647,7 +1647,6 @@ public class Tablet extends TabletBase {
getTabletMemory().getCommitSession().updateMaxCommittedTime(timestamp);
-// ELASTICITY_TODO this needs to be persisted in the metadata table or
walog
return OptionalLong.of(timestamp);
}
}
(accumulo) branch elasticity updated: updates bulk import and compaction logging (#4440)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 212e3cd27c updates bulk import and compaction logging (#4440)
212e3cd27c is described below
commit 212e3cd27cf9701e750ca8c3a2566a74d88867f3
Author: Keith Turner
AuthorDate: Wed Apr 10 10:56:01 2024 -0400
updates bulk import and compaction logging (#4440)
---
.../coordinator/commit/CommitCompaction.java | 20
.../manager/tableOps/bulkVer2/LoadFiles.java | 54 +-
.../manager/tableOps/compact/CompactionDriver.java | 2 -
3 files changed, 41 insertions(+), 35 deletions(-)
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/commit/CommitCompaction.java
b/server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/commit/CommitCompaction.java
index cc0432d4a0..7add060466 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/commit/CommitCompaction.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/commit/CommitCompaction.java
@@ -130,10 +130,11 @@ public class CommitCompaction extends ManagerRepo {
tabletMutator
.submit(tabletMetadata ->
!tabletMetadata.getExternalCompactions().containsKey(ecid));
-// TODO expensive logging
-LOG.debug("Compaction completed {} added {} removed {}",
tablet.getExtent(), newDatafile,
-ecm.getJobFiles().stream().map(AbstractTabletFile::getFileName)
-.collect(Collectors.toList()));
+if (LOG.isDebugEnabled()) {
+ LOG.debug("Compaction completed {} added {} removed {}",
tablet.getExtent(), newDatafile,
+ ecm.getJobFiles().stream().map(AbstractTabletFile::getFileName)
+ .collect(Collectors.toList()));
+}
var result = tabletsMutator.process().get(getExtent());
if (result.getStatus() == Ample.ConditionalResult.Status.ACCEPTED) {
@@ -159,7 +160,7 @@ public class CommitCompaction extends ManagerRepo {
private void updateTabletForCompaction(TCompactionStats stats,
ExternalCompactionId ecid,
TabletMetadata tablet, Optional newDatafile,
CompactionMetadata ecm,
Ample.ConditionalTabletMutator tabletMutator) {
-// ELASTICITY_TODO improve logging adapt to use existing tablet files
logging
+
if (ecm.getKind() == CompactionKind.USER) {
if (tablet.getSelectedFiles().getFiles().equals(ecm.getJobFiles())) {
// all files selected for the user compactions are finished, so the
tablet is finish and
@@ -171,8 +172,7 @@ public class CommitCompaction extends ManagerRepo {
"Tablet %s unexpected has selected files and compacted columns for
%s",
tablet.getExtent(), fateId);
-// TODO set to trace
-LOG.debug("All selected files compacted for {} setting compacted for
{}",
+LOG.trace("All selected files compacted for {} setting compacted for
{}",
tablet.getExtent(), tablet.getSelectedFiles().getFateId());
tabletMutator.deleteSelectedFiles();
@@ -187,14 +187,12 @@ public class CommitCompaction extends ManagerRepo {
newSelectedFileSet.removeAll(ecm.getJobFiles());
if (newDatafile.isPresent()) {
- // TODO set to trace
- LOG.debug(
+ LOG.trace(
"Not all selected files for {} are done, adding new selected
file {} from compaction",
tablet.getExtent(),
newDatafile.orElseThrow().getPath().getName());
newSelectedFileSet.add(newDatafile.orElseThrow().insert());
} else {
- // TODO set to trace
- LOG.debug(
+ LOG.trace(
"Not all selected files for {} are done, compaction produced no
output so not adding to selected set.",
tablet.getExtent());
}
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/bulkVer2/LoadFiles.java
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/bulkVer2/LoadFiles.java
index d336358700..1689ae4359 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/bulkVer2/LoadFiles.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/bulkVer2/LoadFiles.java
@@ -32,6 +32,7 @@ import java.util.List;
import java.util.Map;
import java.util.NoSuchElementException;
import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
import java.util.stream.Collectors;
import org.apache.accumulo.core.clientImpl.bulk.Bulk;
@@ -44,9 +45,11 @@ import org.apache.accumulo.core.dataImpl.KeyExtent;
import org.apache.accumulo.core.dataImpl.thrift.TKeyExtent;
import org.apa
(accumulo) branch elasticity updated: verifies tablets are seen by compaction driver (#4434)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 9503ebe22f verifies tablets are seen by compaction driver (#4434)
9503ebe22f is described below
commit 9503ebe22fef88ee3ecfbecbfecb9f5bfcc86e9b
Author: Keith Turner
AuthorDate: Mon Apr 8 12:37:39 2024 -0400
verifies tablets are seen by compaction driver (#4434)
---
.../accumulo/manager/tableOps/compact/CompactionDriver.java | 11 +--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/compact/CompactionDriver.java
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/compact/CompactionDriver.java
index 0f224736a0..3790d981c3 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/compact/CompactionDriver.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/compact/CompactionDriver.java
@@ -61,10 +61,13 @@ import
org.apache.accumulo.manager.tableOps.delete.PreDeleteTable;
import org.apache.accumulo.server.ServerContext;
import org.apache.accumulo.server.compaction.CompactionConfigStorage;
import org.apache.accumulo.server.compaction.CompactionPluginUtils;
+import org.apache.hadoop.io.Text;
import org.apache.zookeeper.KeeperException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import com.google.common.base.Preconditions;
+
class CompactionDriver extends ManagerRepo {
private static final Logger log =
LoggerFactory.getLogger(CompactionDriver.class);
@@ -300,6 +303,12 @@ class CompactionDriver extends ManagerRepo {
long t2 = System.currentTimeMillis();
+// The Fate operation gets a table lock that prevents the table from being
deleted while this is
+// running, so seeing zero tablets in the metadata table is unexpected.
+Preconditions.checkState(total > 0,
+"No tablets were seen for table %s in the compaction range %s %s",
tableId,
+new Text(startRow), new Text(endRow));
+
log.debug(
"{} tablet stats, total:{} complete:{} selected_now:{}
selected_prev:{} selected_by_other:{} "
+ "no_files:{} none_selected:{} user_compaction_requested:{}
user_compaction_waiting:{} "
@@ -314,8 +323,6 @@ class CompactionDriver extends ManagerRepo {
}
return total - complete;
-
-// ELASTICITIY_TODO need to handle seeing zero tablets
}
@Override
(accumulo) branch elasticity updated: conditionally removes tables scan files (#4436)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 5a669c9d1f conditionally removes tables scan files (#4436)
5a669c9d1f is described below
commit 5a669c9d1fb36e3989da0b2c7b3889fd9f496443
Author: Keith Turner
AuthorDate: Mon Apr 8 12:36:20 2024 -0400
conditionally removes tables scan files (#4436)
---
.../accumulo/server/util/MetadataTableUtil.java| 9 --
.../accumulo/tserver/tablet/ScanfileManager.java | 35 ++
2 files changed, 30 insertions(+), 14 deletions(-)
diff --git
a/server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
b/server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
index e5db26344f..53dc257ef9 100644
---
a/server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
+++
b/server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
@@ -37,7 +37,6 @@ import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
-import java.util.Set;
import java.util.SortedMap;
import java.util.TreeMap;
import java.util.concurrent.TimeUnit;
@@ -152,14 +151,6 @@ public class MetadataTableUtil {
return newFiles;
}
- public static void removeScanFiles(KeyExtent extent, Set
scanFiles,
- ServerContext context, ServiceLock zooLock) {
-TabletMutator tablet = context.getAmple().mutateTablet(extent);
-scanFiles.forEach(tablet::deleteScan);
-tablet.putZooLock(context.getZooKeeperRoot(), zooLock);
-tablet.mutate();
- }
-
public static void deleteTable(TableId tableId, boolean insertDeletes,
ServerContext context,
ServiceLock lock) throws AccumuloException {
try (
diff --git
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/ScanfileManager.java
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/ScanfileManager.java
index e53bbf2553..5ab4780a39 100644
---
a/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/ScanfileManager.java
+++
b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/ScanfileManager.java
@@ -20,21 +20,28 @@ package org.apache.accumulo.tserver.tablet;
import java.io.IOException;
import java.util.Collection;
+import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
+import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.lock.ServiceLock;
import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.Ample;
import org.apache.accumulo.core.metadata.schema.DataFileValue;
+import org.apache.accumulo.core.metadata.schema.TabletMetadata.Location;
import org.apache.accumulo.core.util.MapCounter;
import org.apache.accumulo.core.util.Pair;
+import org.apache.accumulo.server.ServerContext;
import org.apache.accumulo.server.fs.VolumeManager;
-import org.apache.accumulo.server.util.MetadataTableUtil;
import org.apache.hadoop.fs.Path;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import com.google.common.base.Preconditions;
+
class ScanfileManager {
private final Logger log = LoggerFactory.getLogger(ScanfileManager.class);
private final Tablet tablet;
@@ -54,6 +61,23 @@ class ScanfileManager {
}
}
+ static void removeScanFiles(KeyExtent extent, Set
scanFiles,
+ ServerContext context, Location currLocation, ServiceLock zooLock) {
+try (var mutator = context.getAmple().conditionallyMutateTablets()) {
+ var tabletMutator =
mutator.mutateTablet(extent).requireLocation(currLocation);
+
+ scanFiles.forEach(tabletMutator::deleteScan);
+ tabletMutator.putZooLock(context.getZooKeeperRoot(), zooLock);
+
+ tabletMutator
+ .submit(tabletMetadata -> Collections.disjoint(scanFiles,
tabletMetadata.getScans()));
+
+ var result = mutator.process().get(extent);
+ Preconditions.checkState(result.getStatus() ==
Ample.ConditionalResult.Status.ACCEPTED,
+ "Failed to remove scan file entries for %s", extent);
+}
+ }
+
Pair> reserveFilesForScan() {
synchronized (tablet) {
@@ -112,8 +136,9 @@ class ScanfileManager {
// file is in the set filesToDelete that means it was removed from
filesToDeleteAfterScan
// and would never be added back.
log.debug("Removing scan refs from metadata {} {}",
tablet.getExtent(), filesToDelete);
-// ELASTICTIY_TODO use conditional mutation
-MetadataTableUtil.removeScanFiles(tablet.getExtent(), filesToDelete,
tablet.getContext(),
+
+var currLoc =
Location.current(tablet.getTabletServer().getTabletSession());
+removeScanFiles(tablet.getExtent(), filesToDelete,
ta
(accumulo) branch elasticity updated (0386506c1d -> cb097aee5f)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from 0386506c1d moves ample filters out of public API (#4431) add b912506d43 avoids acquiring recovery lock when tablet has no wals (#4429) new cb097aee5f Merge branch 'main' into elasticity The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/accumulo/tserver/AssignmentHandler.java | 6 +- .../java/org/apache/accumulo/tserver/TabletServer.java | 16 2 files changed, 9 insertions(+), 13 deletions(-)
(accumulo) 01/01: Merge branch 'main' into elasticity
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit cb097aee5f36fb91f04c09db084595ef624cd2c8
Merge: 0386506c1d b912506d43
Author: Keith Turner
AuthorDate: Fri Apr 5 15:19:05 2024 -0400
Merge branch 'main' into elasticity
.../org/apache/accumulo/tserver/AssignmentHandler.java | 6 +-
.../java/org/apache/accumulo/tserver/TabletServer.java | 16
2 files changed, 9 insertions(+), 13 deletions(-)
diff --cc
server/tserver/src/main/java/org/apache/accumulo/tserver/AssignmentHandler.java
index cef355cbf1,552d9f40a9..ec56611a03
---
a/server/tserver/src/main/java/org/apache/accumulo/tserver/AssignmentHandler.java
+++
b/server/tserver/src/main/java/org/apache/accumulo/tserver/AssignmentHandler.java
@@@ -131,13 -156,12 +131,11 @@@ class AssignmentHandler implements Runn
Tablet tablet = null;
boolean successful = false;
- try {
- server.acquireRecoveryMemory(extent);
-
+ try (var recoveryMemory = server.acquireRecoveryMemory(tabletMetadata)) {
TabletResourceManager trm =
server.resourceManager.createTabletResourceManager(extent,
server.getTableConfiguration(extent));
- TabletData data = new TabletData(tabletMetadata);
- tablet = new Tablet(server, extent, trm, data);
+ tablet = new Tablet(server, extent, trm, tabletMetadata);
// If a minor compaction starts after a tablet opens, this indicates a
log recovery
// occurred. This recovered data must be minor compacted.
// There are three reasons to wait for this minor compaction to finish
before placing the
diff --cc
server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
index 63fcbd4c5b,678b1294c5..026c904ca2
--- a/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
+++ b/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
@@@ -87,7 -91,8 +87,8 @@@ import org.apache.accumulo.core.manager
import org.apache.accumulo.core.manager.thrift.TabletServerStatus;
import org.apache.accumulo.core.metadata.AccumuloTable;
import org.apache.accumulo.core.metadata.TServerInstance;
+import org.apache.accumulo.core.metadata.schema.Ample.TabletsMutator;
+ import org.apache.accumulo.core.metadata.schema.TabletMetadata;
-import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
import org.apache.accumulo.core.metrics.MetricsUtil;
import org.apache.accumulo.core.rpc.ThriftUtil;
import org.apache.accumulo.core.rpc.clients.ThriftClientTypes;
(accumulo) branch main updated: avoids acquiring recovery lock when tablet has no wals (#4429)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/main by this push:
new b912506d43 avoids acquiring recovery lock when tablet has no wals
(#4429)
b912506d43 is described below
commit b912506d43fbac9477aa3efb9ea81d39ce4d1a5c
Author: Keith Turner
AuthorDate: Fri Apr 5 13:01:07 2024 -0400
avoids acquiring recovery lock when tablet has no wals (#4429)
Tablet servers have a lock for recovery that has the purpose of
preventing concurrent recovery of tablets using too much memory.
This lock is acquired for tablets that have no walogs and will therefore
do no recovery. This commit changes the code to only obtain the lock
when there are walogs.
Noticed this while reviewing #4404 and while researching found
[ACCUMULO-1543](https://issues.apache.org/jira/browse/ACCUMULO-1543)
which is an existing issue describing this change.
---
.../org/apache/accumulo/tserver/AssignmentHandler.java | 6 +-
.../java/org/apache/accumulo/tserver/TabletServer.java | 16
2 files changed, 9 insertions(+), 13 deletions(-)
diff --git
a/server/tserver/src/main/java/org/apache/accumulo/tserver/AssignmentHandler.java
b/server/tserver/src/main/java/org/apache/accumulo/tserver/AssignmentHandler.java
index 8b4f117a21..552d9f40a9 100644
---
a/server/tserver/src/main/java/org/apache/accumulo/tserver/AssignmentHandler.java
+++
b/server/tserver/src/main/java/org/apache/accumulo/tserver/AssignmentHandler.java
@@ -156,9 +156,7 @@ class AssignmentHandler implements Runnable {
Tablet tablet = null;
boolean successful = false;
-try {
- server.acquireRecoveryMemory(extent);
-
+try (var recoveryMemory = server.acquireRecoveryMemory(tabletMetadata)) {
TabletResourceManager trm =
server.resourceManager.createTabletResourceManager(extent,
server.getTableConfiguration(extent));
TabletData data = new TabletData(tabletMetadata);
@@ -205,8 +203,6 @@ class AssignmentHandler implements Runnable {
TableId tableId = extent.tableId();
ProblemReports.getInstance(server.getContext()).report(new
ProblemReport(tableId, TABLET_LOAD,
extent.getUUID().toString(), server.getClientAddressString(), e));
-} finally {
- server.releaseRecoveryMemory(extent);
}
if (successful) {
diff --git
a/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
b/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
index b9c9a39359..678b1294c5 100644
--- a/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
+++ b/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
@@ -91,6 +91,7 @@ import org.apache.accumulo.core.manager.thrift.TableInfo;
import org.apache.accumulo.core.manager.thrift.TabletServerStatus;
import org.apache.accumulo.core.metadata.AccumuloTable;
import org.apache.accumulo.core.metadata.TServerInstance;
+import org.apache.accumulo.core.metadata.schema.TabletMetadata;
import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
import org.apache.accumulo.core.metrics.MetricsUtil;
import org.apache.accumulo.core.rpc.ThriftUtil;
@@ -516,15 +517,14 @@ public class TabletServer extends AbstractServer
implements TabletHostingServer
managerMessages.addLast(m);
}
- void acquireRecoveryMemory(KeyExtent extent) {
-if (!extent.isMeta()) {
- recoveryLock.lock();
-}
- }
+ private static final AutoCloseable NOOP_CLOSEABLE = () -> {};
- void releaseRecoveryMemory(KeyExtent extent) {
-if (!extent.isMeta()) {
- recoveryLock.unlock();
+ AutoCloseable acquireRecoveryMemory(TabletMetadata tabletMetadata) {
+if (tabletMetadata.getExtent().isMeta() ||
tabletMetadata.getLogs().isEmpty()) {
+ return NOOP_CLOSEABLE;
+} else {
+ recoveryLock.lock();
+ return recoveryLock::unlock;
}
}
(accumulo) branch elasticity updated: moves ample filters out of public API (#4431)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 0386506c1d moves ample filters out of public API (#4431)
0386506c1d is described below
commit 0386506c1d91a5edfcc524df17ee472992adc46d
Author: Keith Turner
AuthorDate: Fri Apr 5 12:05:38 2024 -0400
moves ample filters out of public API (#4431)
Ample filters were in a package that is in the public API. Since Ample
itself is not in the public API, these filters should also not be in the
public API. This commit moves the filters to another packages that is
not in the public API
---
.../org/apache/accumulo/core/metadata/schema/TabletsMetadata.java | 2 +-
.../{iterators/user => metadata/schema/filters}/GcWalsFilter.java | 5 +
.../user => metadata/schema/filters}/HasCurrentFilter.java | 2 +-
.../schema/filters}/HasExternalCompactionsFilter.java | 2 +-
.../user => metadata/schema/filters}/TabletMetadataFilter.java | 3 ++-
.../java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogs.java | 2 +-
.../manager/compaction/coordinator/CompactionCoordinator.java | 2 +-
.../manager/compaction/coordinator/DeadCompactionDetector.java | 2 +-
.../org/apache/accumulo/monitor/rest/tables/TablesResource.java | 2 +-
.../apache/accumulo/test/functional/AmpleConditionalWriterIT.java | 6 +++---
.../apache/accumulo/test/functional/TestTabletMetadataFilter.java | 2 +-
11 files changed, 14 insertions(+), 16 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletsMetadata.java
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletsMetadata.java
index 14b8181e78..48a1160513 100644
---
a/core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletsMetadata.java
+++
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletsMetadata.java
@@ -63,7 +63,6 @@ import org.apache.accumulo.core.data.TableId;
import org.apache.accumulo.core.dataImpl.KeyExtent;
import org.apache.accumulo.core.fate.zookeeper.ZooCache;
import org.apache.accumulo.core.fate.zookeeper.ZooReader;
-import org.apache.accumulo.core.iterators.user.TabletMetadataFilter;
import org.apache.accumulo.core.iterators.user.WholeRowIterator;
import org.apache.accumulo.core.metadata.AccumuloTable;
import org.apache.accumulo.core.metadata.RootTable;
@@ -85,6 +84,7 @@ import
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.Sp
import
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.SuspendLocationColumn;
import
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.UserCompactionRequestedColumnFamily;
import org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType;
+import org.apache.accumulo.core.metadata.schema.filters.TabletMetadataFilter;
import org.apache.accumulo.core.security.Authorizations;
import org.apache.accumulo.core.util.ColumnFQ;
import org.apache.hadoop.io.Text;
diff --git
a/core/src/main/java/org/apache/accumulo/core/iterators/user/GcWalsFilter.java
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/filters/GcWalsFilter.java
similarity index 94%
rename from
core/src/main/java/org/apache/accumulo/core/iterators/user/GcWalsFilter.java
rename to
core/src/main/java/org/apache/accumulo/core/metadata/schema/filters/GcWalsFilter.java
index a30302ce59..4e1aca1a06 100644
---
a/core/src/main/java/org/apache/accumulo/core/iterators/user/GcWalsFilter.java
+++
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/filters/GcWalsFilter.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.accumulo.core.iterators.user;
+package org.apache.accumulo.core.metadata.schema.filters;
import java.io.IOException;
import java.util.Arrays;
@@ -41,9 +41,6 @@ import com.google.common.collect.Sets;
* A filter used by the Accumulo GC to find tablets that either have walogs or
are assigned to a
* dead tablet server.
*/
-
-// ELASTICITY_TODO Move TabletMetadataFilter and its subclasses out of public
API. It use internal
-// types that are not user facing.
public class GcWalsFilter extends TabletMetadataFilter {
private Map options = null;
diff --git
a/core/src/main/java/org/apache/accumulo/core/iterators/user/HasCurrentFilter.java
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/filters/HasCurrentFilter.java
similarity index 96%
rename from
core/src/main/java/org/apache/accumulo/core/iterators/user/HasCurrentFilter.java
rename to
core/src/main/java/org/apache/accumulo/core/metadata/schema/filters/HasCurrentFilter.java
index ca58922306..0d00b59837 100644
---
a/core/src/main/java/org/apache/accumulo/core/iterators/user/HasCurrentFilter.java
+++
b/core
(accumulo) branch elasticity updated: avoid unnecessary metadata reads in client tablet cache (#4432)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new f586e7f9c9 avoid unnecessary metadata reads in client tablet cache
(#4432)
f586e7f9c9 is described below
commit f586e7f9c9a82c0f01cba1ac279a7d583c7be3c4
Author: Keith Turner
AuthorDate: Fri Apr 5 12:05:21 2024 -0400
avoid unnecessary metadata reads in client tablet cache (#4432)
For ondemand tablets the client tablet cache caches tablets w/o a
location. There was a bug fixed in #4280 where the cache would do a
metadata table lookup for each mutation when tablets had no location.
The fix in #4280 only partially fixed the problem, after that change
more metadata lookups than needed were still being done.
Also there was a bug with the batchscanner that #4280 did not address.
Before this change when tablets had no location, the batch scanner would
do a metadata lookup for each range passed to the batch scanner (well
the client tablet cache would these metadata lookups on behalf of the
batch scanner). For example before this change if the batch scanner was
given 10K ranges that all fell in a single tablet w/o a location, it
would do 10K metadata lookups. After this change for that situation it
will do a single metadata lookup.
This change minimizes the metadata lookups done by the batch writer and
batch scanner. The fix is to make sure that cached entries populated by
looking up one range or mutation are used by subsequent range or
mutations lookups, even if there is no location present. This is done by
always reusing cache entries that were created after work started on a
batch of mutations or ranges. Cache entries w/o a location that existed
before work started on a batch are ignored. By reusing cache entries
created after starting work on a batch we minimize metadata lookups.
A test was also added to ensure the client tablet cache does not do
excessive metadata table lookups. If this test had existed, it would
have caught the problem.
---
.../core/clientImpl/ClientTabletCache.java | 8 +-
.../core/clientImpl/ClientTabletCacheImpl.java | 66 ++
.../core/clientImpl/ClientTabletCacheImplTest.java | 137 +
3 files changed, 181 insertions(+), 30 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCache.java
b/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCache.java
index 2564fb0063..a31ca2418b 100644
---
a/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCache.java
+++
b/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCache.java
@@ -20,7 +20,6 @@ package org.apache.accumulo.core.clientImpl;
import static com.google.common.base.Preconditions.checkArgument;
-import java.time.Duration;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
@@ -46,6 +45,7 @@ import org.apache.accumulo.core.singletons.SingletonManager;
import org.apache.accumulo.core.singletons.SingletonService;
import org.apache.accumulo.core.util.Interner;
import org.apache.accumulo.core.util.UtilWaitThread;
+import org.apache.accumulo.core.util.time.NanoTime;
import org.apache.hadoop.io.Text;
import com.google.common.base.Preconditions;
@@ -311,7 +311,7 @@ public abstract class ClientTabletCache {
private final TabletAvailability availability;
private final boolean hostingRequested;
-private final Long creationTime = System.nanoTime();
+private final NanoTime creationTime = NanoTime.now();
public CachedTablet(KeyExtent tablet_extent, String tablet_location,
String session,
TabletAvailability availability, boolean hostingRequested) {
@@ -392,8 +392,8 @@ public abstract class ClientTabletCache {
return this.availability;
}
-public Duration getAge() {
- return Duration.ofNanos(System.nanoTime() - creationTime);
+public NanoTime getCreationTime() {
+ return creationTime;
}
public boolean wasHostingRequested() {
diff --git
a/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java
b/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java
index 10fb3aa21e..eb5225 100644
---
a/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java
+++
b/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java
@@ -60,6 +60,7 @@ import org.apache.accumulo.core.trace.TraceUtil;
import org.apache.accumulo.core.util.OpTimer;
import org.apache.accumulo.core.util.Pair;
import org.apache.accumulo.core.util.TextUtil;
+import org.apache.accumulo.core.util.time.NanoTime;
import org.apache.hadoop.io.Text;
import
(accumulo) branch elasticity updated (cebef4bdbc -> ebfa35b49e)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from cebef4bdbc cleaned up a few TODOs in the split code (#4401) add 7947c2c2df adds PluginEnv support to client side iterators (#4283) add eaa1f678c9 Merge branch '2.1' add ebfa35b49e Merge branch 'main' into elasticity No new revisions were added by this update. Summary of changes: .../core/client/ClientSideIteratorScanner.java | 39 +++- .../accumulo/core/clientImpl/OfflineIterator.java | 27 +- .../accumulo/core/clientImpl/ScannerImpl.java | 10 ++ .../apache/accumulo/test/ClientSideIteratorIT.java | 108 + 4 files changed, 178 insertions(+), 6 deletions(-)
(accumulo) branch main updated (0ad96b1dc0 -> eaa1f678c9)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git from 0ad96b1dc0 Merge remote-tracking branch 'upstream/2.1' add 7947c2c2df adds PluginEnv support to client side iterators (#4283) new eaa1f678c9 Merge branch '2.1' The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../core/client/ClientSideIteratorScanner.java | 39 +++- .../accumulo/core/clientImpl/OfflineIterator.java | 27 +- .../accumulo/core/clientImpl/ScannerImpl.java | 10 ++ .../apache/accumulo/test/ClientSideIteratorIT.java | 108 + 4 files changed, 178 insertions(+), 6 deletions(-)
(accumulo) 01/01: Merge branch '2.1'
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit eaa1f678c96b135534fba89fe585107c4f242e54
Merge: 0ad96b1dc0 7947c2c2df
Author: Keith Turner
AuthorDate: Wed Mar 20 18:39:26 2024 -0400
Merge branch '2.1'
.../core/client/ClientSideIteratorScanner.java | 39 +++-
.../accumulo/core/clientImpl/OfflineIterator.java | 27 +-
.../accumulo/core/clientImpl/ScannerImpl.java | 10 ++
.../apache/accumulo/test/ClientSideIteratorIT.java | 108 +
4 files changed, 178 insertions(+), 6 deletions(-)
diff --cc
core/src/main/java/org/apache/accumulo/core/client/ClientSideIteratorScanner.java
index a50561263f,362c4e85d6..7e300e917d
---
a/core/src/main/java/org/apache/accumulo/core/client/ClientSideIteratorScanner.java
+++
b/core/src/main/java/org/apache/accumulo/core/client/ClientSideIteratorScanner.java
@@@ -122,6 -133,22 +133,16 @@@ public class ClientSideIteratorScanner
public SamplerConfiguration getSamplerConfiguration() {
return samplerConfig;
}
+
-@Deprecated(since = "2.1.0")
-@Override
-public ServiceEnvironment getServiceEnv() {
- return new ClientServiceEnvironmentImpl(context.get());
-}
-
+ @Override
+ public PluginEnvironment getPluginEnv() {
+ return new ClientServiceEnvironmentImpl(context.get());
+ }
+
+ @Override
+ public TableId getTableId() {
+ return tableId.get();
+ }
}
/**
diff --cc
core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineIterator.java
index 0e7a3fd4d2,a03cc811ab..9200591ff2
---
a/core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineIterator.java
+++
b/core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineIterator.java
@@@ -135,7 -153,24 +140,18 @@@ class OfflineIterator implements Iterat
if (sampleConf == null) {
throw new SampleNotPresentException();
}
- return new OfflineIteratorEnvironment(authorizations, conf, true,
sampleConf);
+ return new OfflineIteratorEnvironment(context, tableId, authorizations,
conf, true,
+ sampleConf);
+ }
+
-@Deprecated(since = "2.1.0")
-@Override
-public ServiceEnvironment getServiceEnv() {
- return new ClientServiceEnvironmentImpl(context);
-}
-
+ @Override
+ public PluginEnvironment getPluginEnv() {
+ return new ClientServiceEnvironmentImpl(context);
+ }
+
+ @Override
+ public TableId getTableId() {
+ return tableId;
}
}
(accumulo) branch elasticity updated: cleaned up a few TODOs in the split code (#4401)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new cebef4bdbc cleaned up a few TODOs in the split code (#4401)
cebef4bdbc is described below
commit cebef4bdbce59c9f33da8409d64b590b59166195
Author: Keith Turner
AuthorDate: Wed Mar 20 18:26:20 2024 -0400
cleaned up a few TODOs in the split code (#4401)
Removed TODOs in the code related to #3415, #3709, #3412. These todos
were done or had issues opened.
Added some best effot checks to avoid doing splits if a table is
offline. The checks have race conditions as the table lock is not
acquired by the split fate op. However as outlined in a comment
on #3412 the offline code that waits could be made more
comprehensive to account for this.
One TODO was handled by using
`Ample.RejectionHandler.acceptAbsentTablet()` which likely did not
exists when the TODO was written.
---
.../accumulo/manager/FateServiceHandler.java | 6 --
.../manager/tableOps/split/FindSplits.java | 7 +++
.../accumulo/manager/tableOps/split/PreSplit.java | 20 ---
.../manager/tableOps/split/UpdateTablets.java | 23 +++---
4 files changed, 27 insertions(+), 29 deletions(-)
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/FateServiceHandler.java
b/server/manager/src/main/java/org/apache/accumulo/manager/FateServiceHandler.java
index 5c2ae952d0..3e45faca23 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/FateServiceHandler.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/FateServiceHandler.java
@@ -719,10 +719,6 @@ class FateServiceHandler implements FateService.Iface {
case TABLE_SPLIT: {
TableOperation tableOp = TableOperation.SPLIT;
-// ELASTICITY_TODO this does not check if table is offline for now,
that is usually done in
-// FATE operation with a table lock. Deferring that check for now as
its possible tablet
-// locks may not be needed.
-
int SPLIT_OFFSET = 3; // offset where split data begins in arguments
list
if (arguments.size() < (SPLIT_OFFSET + 1)) {
throw new ThriftTableOperationException(null, null, tableOp,
@@ -752,8 +748,6 @@ class FateServiceHandler implements FateService.Iface {
endRow = endRow.getLength() == 0 ? null : endRow;
prevEndRow = prevEndRow.getLength() == 0 ? null : prevEndRow;
-// ELASTICITY_TODO create table stores splits in a file, maybe this
operation should do the
-// same
SortedSet splits = arguments.subList(SPLIT_OFFSET,
arguments.size()).stream()
.map(ByteBufferUtil::toText).collect(Collectors.toCollection(TreeSet::new));
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/FindSplits.java
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/FindSplits.java
index 7e3acd8d7f..f72d26cb05 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/FindSplits.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/FindSplits.java
@@ -28,6 +28,7 @@ import java.util.function.Consumer;
import org.apache.accumulo.core.dataImpl.KeyExtent;
import org.apache.accumulo.core.fate.FateId;
import org.apache.accumulo.core.fate.Repo;
+import org.apache.accumulo.core.manager.state.tables.TableState;
import org.apache.accumulo.core.metadata.schema.Ample.ConditionalResult;
import org.apache.accumulo.core.metadata.schema.Ample.ConditionalResult.Status;
import org.apache.accumulo.core.metadata.schema.UnSplittableMetadata;
@@ -88,6 +89,12 @@ public class FindSplits extends ManagerRepo {
return null;
}
+if (manager.getContext().getTableState(extent.tableId()) !=
TableState.ONLINE) {
+ // The table is offline, do not bother finding splits
+ log.debug("Not splitting {} because the table is not online",
tabletMetadata.getExtent());
+ return null;
+}
+
SortedSet splits = SplitUtils.findSplits(manager.getContext(),
tabletMetadata);
if (extent.endRow() != null) {
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/PreSplit.java
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/PreSplit.java
index d956821d18..2bf5007d8c 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/PreSplit.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/PreSplit.java
@@ -30,9 +30,13 @@ import java.util.Objects;
import java.util.Optional;
import java.util.SortedSet;
+import
org.apache.accumulo.core.clientImpl.AcceptableThriftTableOperationExcep
(accumulo) branch 2.1 updated: adds PluginEnv support to client side iterators (#4283)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch 2.1
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/2.1 by this push:
new 7947c2c2df adds PluginEnv support to client side iterators (#4283)
7947c2c2df is described below
commit 7947c2c2dfb80fb280420b729b4197547be22124
Author: Keith Turner
AuthorDate: Wed Mar 20 18:25:43 2024 -0400
adds PluginEnv support to client side iterators (#4283)
Accumulo code that ran user iterators client side would throw an unsupported
operations exception when attempting to access the PluginEnv. This commit
adds support for PluginEnv in situations where iterators are run client
side.
---
.../core/client/ClientSideIteratorScanner.java | 46 -
.../accumulo/core/clientImpl/OfflineIterator.java | 34 ++-
.../accumulo/core/clientImpl/ScannerImpl.java | 10 ++
.../apache/accumulo/test/ClientSideIteratorIT.java | 108 +
4 files changed, 192 insertions(+), 6 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/client/ClientSideIteratorScanner.java
b/core/src/main/java/org/apache/accumulo/core/client/ClientSideIteratorScanner.java
index 542273d470..362c4e85d6 100644
---
a/core/src/main/java/org/apache/accumulo/core/client/ClientSideIteratorScanner.java
+++
b/core/src/main/java/org/apache/accumulo/core/client/ClientSideIteratorScanner.java
@@ -29,15 +29,20 @@ import java.util.Map.Entry;
import java.util.Set;
import java.util.TreeMap;
import java.util.TreeSet;
+import java.util.function.Supplier;
import org.apache.accumulo.core.Constants;
import org.apache.accumulo.core.client.sample.SamplerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.clientImpl.ClientServiceEnvironmentImpl;
+import org.apache.accumulo.core.clientImpl.ScannerImpl;
import org.apache.accumulo.core.clientImpl.ScannerOptions;
import org.apache.accumulo.core.data.ArrayByteSequence;
import org.apache.accumulo.core.data.ByteSequence;
import org.apache.accumulo.core.data.Column;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.dataImpl.thrift.IterInfo;
import org.apache.accumulo.core.iterators.IteratorAdapter;
@@ -47,6 +52,7 @@ import
org.apache.accumulo.core.iterators.SortedKeyValueIterator;
import org.apache.accumulo.core.iteratorsImpl.IteratorBuilder;
import org.apache.accumulo.core.iteratorsImpl.IteratorConfigUtil;
import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.core.spi.common.ServiceEnvironment;
import org.apache.hadoop.io.Text;
/**
@@ -70,6 +76,7 @@ import org.apache.hadoop.io.Text;
* server side) and to the client side scanner (which will execute client
side).
*/
public class ClientSideIteratorScanner extends ScannerOptions implements
Scanner {
+
private int size;
private Range range;
@@ -77,6 +84,9 @@ public class ClientSideIteratorScanner extends ScannerOptions
implements Scanner
private long readaheadThreshold =
Constants.SCANNER_DEFAULT_READAHEAD_THRESHOLD;
private SamplerConfiguration iteratorSamplerConfig;
+ private final Supplier context;
+ private final Supplier tableId;
+
private class ClientSideIteratorEnvironment implements IteratorEnvironment {
private SamplerConfiguration samplerConfig;
@@ -94,7 +104,9 @@ public class ClientSideIteratorScanner extends
ScannerOptions implements Scanner
@Override
public boolean isFullMajorCompaction() {
- return false;
+ // The javadocs state this method will throw an ISE when scope is not
majc
+ throw new IllegalStateException(
+ "Asked about major compaction type when scope is " +
getIteratorScope());
}
@Override
@@ -121,6 +133,22 @@ public class ClientSideIteratorScanner extends
ScannerOptions implements Scanner
public SamplerConfiguration getSamplerConfiguration() {
return samplerConfig;
}
+
+@Deprecated(since = "2.1.0")
+@Override
+public ServiceEnvironment getServiceEnv() {
+ return new ClientServiceEnvironmentImpl(context.get());
+}
+
+@Override
+public PluginEnvironment getPluginEnv() {
+ return new ClientServiceEnvironmentImpl(context.get());
+}
+
+@Override
+public TableId getTableId() {
+ return tableId.get();
+}
}
/**
@@ -220,6 +248,22 @@ public class ClientSideIteratorScanner extends
ScannerOptions implements Scanner
if (samplerConfig != null) {
setSamplerConfiguration(samplerConfig);
}
+
+if (scanner instanceof ScannerImpl) {
+ var scannerImpl = (ScannerImpl) scanner;
+ this.context = () -> scannerImpl.getClientContext();
+ this.tableId = () -&
(accumulo) branch elasticity updated: removed queued jobs for a tablet if no new jobs are seen (#4394)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 36df8cee36 removed queued jobs for a tablet if no new jobs are seen
(#4394)
36df8cee36 is described below
commit 36df8cee360081ca445f615cbc354e9e2ff2e6fc
Author: Keith Turner
AuthorDate: Mon Mar 18 13:51:20 2024 -0400
removed queued jobs for a tablet if no new jobs are seen (#4394)
fixes #3528
---
.../accumulo/manager/TabletGroupWatcher.java | 2 +
.../queue/CompactionJobPriorityQueue.java | 56 -
.../compaction/queue/CompactionJobQueues.java | 35 +++-
.../compaction/CompactionCoordinatorTest.java | 1 +
.../queue/CompactionJobPriorityQueueTest.java | 14 +-
.../compaction/queue/CompactionJobQueuesTest.java | 229 -
6 files changed, 314 insertions(+), 23 deletions(-)
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/TabletGroupWatcher.java
b/server/manager/src/main/java/org/apache/accumulo/manager/TabletGroupWatcher.java
index 8779568916..3e8f1fdc5b 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/TabletGroupWatcher.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/TabletGroupWatcher.java
@@ -678,7 +678,9 @@ abstract class TabletGroupWatcher extends
AccumuloDaemonThread {
eventHandler.clearNeedsFullScan();
iter = store.iterator(tableMgmtParams);
+
manager.getCompactionCoordinator().getJobQueues().beginFullScan(store.getLevel());
var tabletMgmtStats = manageTablets(iter, tableMgmtParams,
currentTServers, true);
+
manager.getCompactionCoordinator().getJobQueues().endFullScan(store.getLevel());
// If currently looking for volume replacements, determine if the next
round needs to look.
if (lookForTabletsNeedingVolReplacement) {
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/compaction/queue/CompactionJobPriorityQueue.java
b/server/manager/src/main/java/org/apache/accumulo/manager/compaction/queue/CompactionJobPriorityQueue.java
index e4aa059e95..4dfd6868ad 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/compaction/queue/CompactionJobPriorityQueue.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/compaction/queue/CompactionJobPriorityQueue.java
@@ -23,18 +23,23 @@ import static
com.google.common.base.Preconditions.checkState;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
+import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
+import java.util.Set;
import java.util.TreeMap;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;
import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.metadata.schema.Ample;
import org.apache.accumulo.core.metadata.schema.TabletMetadata;
import org.apache.accumulo.core.spi.compaction.CompactionJob;
import org.apache.accumulo.core.spi.compaction.CompactorGroupId;
import org.apache.accumulo.core.util.compaction.CompactionJobPrioritizer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import com.google.common.base.Preconditions;
@@ -49,6 +54,8 @@ import com.google.common.base.Preconditions;
*/
public class CompactionJobPriorityQueue {
+ private static final Logger log =
LoggerFactory.getLogger(CompactionJobPriorityQueue.class);
+
private final CompactorGroupId groupId;
private class CjpqKey implements Comparable {
@@ -99,9 +106,19 @@ public class CompactionJobPriorityQueue {
private final AtomicLong rejectedJobs;
private final AtomicLong dequeuedJobs;
+ private static class TabletJobs {
+final long generation;
+final HashSet jobs;
+
+private TabletJobs(long generation, HashSet jobs) {
+ this.generation = generation;
+ this.jobs = jobs;
+}
+ }
+
// This map tracks what jobs a tablet currently has in the queue. Its used
to efficiently remove
// jobs in the queue when new jobs are queued for a tablet.
- private final Map> tabletJobs;
+ private final Map tabletJobs;
private final AtomicLong nextSeq = new AtomicLong(0);
@@ -116,10 +133,32 @@ public class CompactionJobPriorityQueue {
this.dequeuedJobs = new AtomicLong(0);
}
+ public synchronized void removeOlderGenerations(Ample.DataLevel level, long
currGeneration) {
+if (closed.get()) {
+ return;
+}
+
+List removals = new ArrayList<>();
+
+tabletJobs.forEach((extent, jobs) -> {
+ if (Ample.DataLevel.of(extent.tableId()) == level && jobs.generation <
currGeneration) {
+removals.add(extent);
+ }
+});
+
+if (!removals.isEmpty()) {
+ log.trace("Removed {} queued
(accumulo) branch elasticity updated: forgo fabricating fake fateid for system compation (#4392)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new f05e44335b forgo fabricating fake fateid for system compation (#4392)
f05e44335b is described below
commit f05e44335be91fc60097d336b98b3eed7f979a5c
Author: Keith Turner
AuthorDate: Sat Mar 16 14:52:44 2024 -0400
forgo fabricating fake fateid for system compation (#4392)
---
.../manager/compaction/coordinator/CompactionCoordinator.java | 11 +++
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java
b/server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java
index 174d741abb..178b4f1e95 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java
@@ -42,7 +42,6 @@ import java.util.Map;
import java.util.Map.Entry;
import java.util.Optional;
import java.util.Set;
-import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.ScheduledThreadPoolExecutor;
@@ -584,8 +583,11 @@ public class CompactionCoordinator
dfv.getTime());
}).collect(toList());
-FateInstanceType type =
FateInstanceType.fromTableId(metaJob.getTabletMetadata().getTableId());
-FateId fateId = FateId.from(type, UUID.randomUUID());
+// The fateId here corresponds to the Fate transaction that is driving a
user initiated
+// compaction. A system initiated compaction has no Fate transaction
driving it so its ok to set
+// it to null. If anything tries to use the id for a system compaction and
triggers a NPE it's
+// probably a bug that needs to be fixed.
+FateId fateId = null;
if (metaJob.getJob().getKind() == CompactionKind.USER) {
fateId = metaJob.getTabletMetadata().getSelectedFiles().getFateId();
}
@@ -593,7 +595,8 @@ public class CompactionCoordinator
return new TExternalCompactionJob(externalCompactionId,
metaJob.getTabletMetadata().getExtent().toThrift(), files,
iteratorSettings,
ecm.getCompactTmpName().getNormalizedPathStr(),
ecm.getPropagateDeletes(),
-TCompactionKind.valueOf(ecm.getKind().name()), fateId.toThrift(),
overrides);
+TCompactionKind.valueOf(ecm.getKind().name()), fateId == null ? null :
fateId.toThrift(),
+overrides);
}
@Override
(accumulo) branch elasticity updated: Adds test for illegal fate ids (#4391)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 7d7c28c1a1 Adds test for illegal fate ids (#4391)
7d7c28c1a1 is described below
commit 7d7c28c1a133aebc8e914e8b27aabeb6632eeb59
Author: Keith Turner
AuthorDate: Sat Mar 16 14:52:19 2024 -0400
Adds test for illegal fate ids (#4391)
---
.../org/apache/accumulo/core/fate/FateIdTest.java | 66 ++
1 file changed, 66 insertions(+)
diff --git a/core/src/test/java/org/apache/accumulo/core/fate/FateIdTest.java
b/core/src/test/java/org/apache/accumulo/core/fate/FateIdTest.java
new file mode 100644
index 00..3de10825e2
--- /dev/null
+++ b/core/src/test/java/org/apache/accumulo/core/fate/FateIdTest.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.fate;
+
+import static org.apache.accumulo.core.fate.FateInstanceType.META;
+import static org.apache.accumulo.core.fate.FateInstanceType.USER;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import java.util.List;
+import java.util.UUID;
+
+import org.junit.jupiter.api.Test;
+
+public class FateIdTest {
+
+ @Test
+ public void testIllegal() {
+var uuid = UUID.randomUUID();
+
+var legalId = FateId.from(USER, uuid);
+
+// test the test assumptions about what is legal
+assertNotNull(FateId.from("FATE:" + META + ":" + uuid));
+assertEquals(legalId, FateId.from(legalId.canonical()));
+
+// The fate id has three fields, try making each field invalid and having
the wrong number of
+// fields
+for (var illegalId : new String[] {"1234567890", "FATE:" + uuid,
"FATE:GOLANG:" + uuid,
+META + ":" + uuid, USER + ":" + uuid, "RUST:" + META + ":" + uuid,
+"RUST:" + USER + ":" + uuid, legalId + ":JAVA", "JAVA:" + legalId,
"FATE:" + legalId}) {
+ assertThrows(IllegalArgumentException.class, () ->
FateId.from(illegalId));
+ assertFalse(FateId.isFateId(illegalId));
+}
+
+// Try different illegal uuids in the fate id
+for (var illegalUuid : List.of("3366-4485-92ab-6961bbd6d3f4",
"075cd820-4485-92ab-6961bbd6d3f4",
+"075cd820-3366-4485-92ab", "075C++20-3366-4485-92ab-6961bbd6d3f4",
+"075cd820--4485-92ab-6961bbd6d3f4",
"42b64e8-4307-4f7d-8466-a11e81eb56c7",
+"842b64e8-307-4f7d-8466-a11e81eb56c7",
"842b64e8-4307-f7d-8466-a11e81eb56c7",
+"842b64e8-4307-4f7d-466-a11e81eb56c7",
"842b64e8-4307-4f7d-8466-11e81eb56c7",
+"842b64e843074f7d8466a11e81eb56c7")) {
+ var illegalId = "FATE:" + USER + ":" + illegalUuid;
+ assertThrows(IllegalArgumentException.class, () ->
FateId.from(illegalId));
+ assertFalse(FateId.isFateId(illegalId));
+}
+ }
+}
(accumulo) branch elasticity updated: fixes compile error after clean merge of #4388 (#4390)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new c5f4b9b678 fixes compile error after clean merge of #4388 (#4390)
c5f4b9b678 is described below
commit c5f4b9b678abdc993c0f32274b210212b2458710
Author: Keith Turner
AuthorDate: Sat Mar 16 13:02:07 2024 -0400
fixes compile error after clean merge of #4388 (#4390)
---
.../test/java/org/apache/accumulo/core/util/FastFormatTest.java| 7 +--
1 file changed, 1 insertion(+), 6 deletions(-)
diff --git
a/core/src/test/java/org/apache/accumulo/core/util/FastFormatTest.java
b/core/src/test/java/org/apache/accumulo/core/util/FastFormatTest.java
index 8e68d87a34..545f3feaf2 100644
--- a/core/src/test/java/org/apache/accumulo/core/util/FastFormatTest.java
+++ b/core/src/test/java/org/apache/accumulo/core/util/FastFormatTest.java
@@ -24,7 +24,6 @@ import static org.junit.jupiter.api.Assertions.assertThrows;
import java.util.Arrays;
-import org.apache.accumulo.core.fate.FateId;
import org.junit.jupiter.api.Test;
public class FastFormatTest {
@@ -121,14 +120,10 @@ public class FastFormatTest {
@Test
public void testHexString() {
-String prefix = "FATE:USER:";
-assertEquals(FateId.formatTid(987654321L),
FastFormat.toHexString(987654321L));
-long txid = FateId.from(prefix + "2e429160071c63d8").getTid();
-assertEquals(prefix + "2e429160071c63d8", FastFormat.toHexString(prefix,
txid, ""));
assertEquals(String.format("%016x", 64L), FastFormat.toHexString(64L));
assertEquals(String.format("%016x", 0X2e429160071c63d8L),
FastFormat.toHexString(0X2e429160071c63d8L));
-
+assertEquals(String.format("%016x", 987654321L),
FastFormat.toHexString(987654321L));
assertEquals("-0040-", FastFormat.toHexString("-", 64L, "-"));
assertEquals("-075bcd15", FastFormat.toHexString("-", 123456789L,
""));
assertEquals("000a", FastFormat.toHexString(0XaL));
(accumulo) branch elasticity updated: FateId to use UUID instead of long (#4388)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new eb71e3e4a6 FateId to use UUID instead of long (#4388)
eb71e3e4a6 is described below
commit eb71e3e4a6af3c37ee27c9f5cb3b26277d74b05b
Author: Kevin Rathbun <43969518+kevinrr...@users.noreply.github.com>
AuthorDate: Sat Mar 16 12:25:26 2024 -0400
FateId to use UUID instead of long (#4388)
- FateId now uses a 128bit UUID instead of a (64bit) long
- FateIds created by ZooStore and AccumuloStore now use a v4 UUID
(random UUID)
- FateIds created by FateIdGenerator now use a v3 UUID (name based)
so the same FateKey gives the same UUID
- Necessary updates to classes and tests which used the long id
---
.../accumulo/core/fate/AbstractFateStore.java | 12 +-
.../java/org/apache/accumulo/core/fate/FateId.java | 56 ---
.../org/apache/accumulo/core/fate/ZooStore.java| 8 +-
.../accumulo/core/fate/accumulo/AccumuloStore.java | 8 +-
.../core/fate/zookeeper/ZooReservation.java| 2 +-
.../accumulo/core/manager/thrift/TFateId.java | 125
core/src/main/thrift/manager.thrift| 2 +-
.../org/apache/accumulo/core/fate/TestStore.java | 4 +-
.../core/metadata/schema/SelectedFilesTest.java| 19 ++-
.../core/metadata/schema/TabletMetadataTest.java | 61
.../server/compaction/CompactionConfigStorage.java | 4 +-
.../constraints/MetadataConstraintsTest.java | 52 +++
.../server/manager/state/TabletManagementTest.java | 8 +-
.../server/util/fateCommand/SummaryReportTest.java | 3 +-
.../server/util/fateCommand/TxnDetailsTest.java| 14 +-
.../accumulo/manager/FateServiceHandler.java | 8 +-
.../coordinator/CompactionCoordinator.java | 3 +-
.../compaction/CompactionCoordinatorTest.java | 6 +-
.../manager/tableOps/ShutdownTServerTest.java | 3 +-
.../manager/tableOps/merge/MergeTabletsTest.java | 4 +-
.../manager/tableOps/split/UpdateTabletsTest.java | 16 +-
.../org/apache/accumulo/test/ScanServerIT.java | 3 +-
.../test/fate/accumulo/AccumuloFateIT.java | 2 +-
.../test/fate/accumulo/AccumuloStoreIT.java| 19 ++-
.../test/fate/accumulo/FateMutatorImplIT.java | 10 +-
.../accumulo/test/fate/accumulo/FateStoreIT.java | 12 +-
.../test/fate/zookeeper/ZooStoreFateIT.java| 2 +-
.../test/fate/zookeeper/ZookeeperFateIT.java | 2 +-
.../test/functional/AmpleConditionalWriterIT.java | 161 ++---
.../test/functional/ManagerAssignmentIT.java | 3 +-
.../apache/accumulo/test/functional/MergeIT.java | 4 +-
.../accumulo/test/functional/SplitRecoveryIT.java | 3 +-
.../functional/TabletManagementIteratorIT.java | 5 +-
33 files changed, 334 insertions(+), 310 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/fate/AbstractFateStore.java
b/core/src/main/java/org/apache/accumulo/core/fate/AbstractFateStore.java
index d805b230b2..0bec78d196 100644
--- a/core/src/main/java/org/apache/accumulo/core/fate/AbstractFateStore.java
+++ b/core/src/main/java/org/apache/accumulo/core/fate/AbstractFateStore.java
@@ -35,6 +35,7 @@ import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
+import java.util.UUID;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;
@@ -48,8 +49,6 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.common.base.Preconditions;
-import com.google.common.hash.HashCode;
-import com.google.common.hash.Hashing;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
@@ -64,9 +63,8 @@ public abstract class AbstractFateStore implements
FateStore {
public static final FateIdGenerator DEFAULT_FATE_ID_GENERATOR = new
FateIdGenerator() {
@Override
public FateId fromTypeAndKey(FateInstanceType instanceType, FateKey
fateKey) {
- HashCode hashCode =
Hashing.murmur3_128().hashBytes(fateKey.getSerialized());
- long tid = hashCode.asLong() & 0x7fffL;
- return FateId.from(instanceType, tid);
+ UUID txUUID = UUID.nameUUIDFromBytes(fateKey.getSerialized());
+ return FateId.from(instanceType, txUUID);
}
};
@@ -271,9 +269,9 @@ public abstract class AbstractFateStore implements
FateStore {
// mean a collision
if (status == TStatus.NEW) {
Preconditions.checkState(tFateKey.isPresent(), "Tx Key is missing from
tid %s",
-fateId.getTid());
+fateId.getTxUUIDStr());
Preconditions.checkState(fateKey.equals(tFateKey.orElseThrow()),
-"Collision detected for tid %s", fateId.getT
(accumulo) branch elasticity updated: Deletes FateTxId (#4370)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new b9867c4c8c Deletes FateTxId (#4370)
b9867c4c8c is described below
commit b9867c4c8cd97b23800b1e129bfa492381ca74c2
Author: Kevin Rathbun <43969518+kevinrr...@users.noreply.github.com>
AuthorDate: Fri Mar 15 19:08:52 2024 -0400
Deletes FateTxId (#4370)
FateTxId has now been fully replaced by FateId, and can be safely
deleted
---
.../org/apache/accumulo/core/fate/FateTxId.java| 62 --
.../apache/accumulo/core/util/FastFormatTest.java | 13 ++---
2 files changed, 5 insertions(+), 70 deletions(-)
diff --git a/core/src/main/java/org/apache/accumulo/core/fate/FateTxId.java
b/core/src/main/java/org/apache/accumulo/core/fate/FateTxId.java
deleted file mode 100644
index ad0d5740af..00
--- a/core/src/main/java/org/apache/accumulo/core/fate/FateTxId.java
+++ /dev/null
@@ -1,62 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * https://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.accumulo.core.fate;
-
-import java.util.regex.Pattern;
-
-import org.apache.accumulo.core.util.FastFormat;
-
-import com.google.common.base.Preconditions;
-
-public class FateTxId {
-
- private static final String PREFIX = "FATE[";
- private static final String SUFFIX = "]";
-
- private final static Pattern PATTERN =
- Pattern.compile(Pattern.quote(PREFIX) + "[0-9a-fA-F]+" +
Pattern.quote(SUFFIX));
-
- private static String getHex(String fmtTid) {
-return fmtTid.substring(PREFIX.length(), fmtTid.length() -
SUFFIX.length());
- }
-
- /**
- * @return true if string was created by {@link #formatTid(long)} and false
otherwise.
- */
- public static boolean isFormatedTid(String fmtTid) {
-return PATTERN.matcher(fmtTid).matches();
- }
-
- /**
- * Reverses {@link #formatTid(long)}
- */
- public static long fromString(String fmtTid) {
-Preconditions.checkArgument(fmtTid.startsWith(PREFIX) &&
fmtTid.endsWith(SUFFIX));
-return Long.parseLong(getHex(fmtTid), 16);
- }
-
- /**
- * Formats transaction ids in a consistent way that is useful for logging
and persisting.
- */
- public static String formatTid(long tid) {
-// do not change how this formats without considering implications for
persistence
-return FastFormat.toHexString(PREFIX, tid, SUFFIX);
- }
-
-}
diff --git
a/core/src/test/java/org/apache/accumulo/core/util/FastFormatTest.java
b/core/src/test/java/org/apache/accumulo/core/util/FastFormatTest.java
index 40da603e4a..8e68d87a34 100644
--- a/core/src/test/java/org/apache/accumulo/core/util/FastFormatTest.java
+++ b/core/src/test/java/org/apache/accumulo/core/util/FastFormatTest.java
@@ -24,7 +24,7 @@ import static org.junit.jupiter.api.Assertions.assertThrows;
import java.util.Arrays;
-import org.apache.accumulo.core.fate.FateTxId;
+import org.apache.accumulo.core.fate.FateId;
import org.junit.jupiter.api.Test;
public class FastFormatTest {
@@ -121,13 +121,10 @@ public class FastFormatTest {
@Test
public void testHexString() {
-final String PREFIX = "FATE[";
-final String SUFFIX = "]";
-String formattedTxId = FateTxId.formatTid(64L);
-String hexStr = FastFormat.toHexString(PREFIX, 64L, SUFFIX);
-assertEquals(formattedTxId, hexStr);
-long txid = FateTxId.fromString("FATE[2e429160071c63d8]");
-assertEquals("FATE[2e429160071c63d8]", FastFormat.toHexString(PREFIX,
txid, SUFFIX));
+String prefix = "FATE:USER:";
+assertEquals(FateId.formatTid(987654321L),
FastFormat.toHexString(987654321L));
+long txid = FateId.from(prefix + "2e429160071c63d8").getTid();
+assertEquals(prefix + "2e429160071c63d8", FastFormat.toHexString(prefix,
txid, ""));
assertEquals(String.format("%016x", 64L), FastFormat.toHexString(64L));
assertEquals(String.format("%016x", 0X2e429160071c63d8L),
FastFormat.toHexString(0X2e429160071c63d8L));
(accumulo) 01/01: Merge branch 'main' into elasticity
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit 9550da7c838994f9f6e9547819566b01e4b1dc6e
Merge: ae49a7ff02 9839e4d42f
Author: Keith Turner
AuthorDate: Thu Mar 14 10:29:24 2024 -0400
Merge branch 'main' into elasticity
.../accumulo/core/fate/AbstractFateStore.java | 12 ++-
.../org/apache/accumulo/core/fate/AdminUtil.java | 6 +++---
.../java/org/apache/accumulo/core/fate/Fate.java | 15 +++---
.../org/apache/accumulo/core/fate/FateCleaner.java | 3 ++-
.../org/apache/accumulo/core/fate/FateStore.java | 6 +++---
.../accumulo/core/fate/WrappedFateTxStore.java | 6 +++---
.../apache/accumulo/core/fate/FateCleanerTest.java | 21 ++--
.../org/apache/accumulo/core/fate/TestStore.java | 4 ++--
.../test/compaction/ExternalCompaction_1_IT.java | 6 +++---
.../accumulo/test/fate/accumulo/FateStoreIT.java | 23 +++---
10 files changed, 53 insertions(+), 49 deletions(-)
diff --cc
core/src/main/java/org/apache/accumulo/core/fate/AbstractFateStore.java
index 0cad25f857,00..d805b230b2
mode 100644,00..100644
--- a/core/src/main/java/org/apache/accumulo/core/fate/AbstractFateStore.java
+++ b/core/src/main/java/org/apache/accumulo/core/fate/AbstractFateStore.java
@@@ -1,494 -1,0 +1,496 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.fate;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+import java.io.UncheckedIOException;
+import java.time.Duration;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
++import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.fate.Fate.TxInfo;
+import org.apache.accumulo.core.util.Pair;
+import org.apache.accumulo.core.util.time.NanoTime;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+import com.google.common.hash.HashCode;
+import com.google.common.hash.Hashing;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+public abstract class AbstractFateStore implements FateStore {
+
+ private static final Logger log =
LoggerFactory.getLogger(AbstractFateStore.class);
+
+ // Default maximum size of 100,000 transactions before deferral is stopped
and
+ // all existing transactions are processed immediately again
+ public static final int DEFAULT_MAX_DEFERRED = 100_000;
+
+ public static final FateIdGenerator DEFAULT_FATE_ID_GENERATOR = new
FateIdGenerator() {
+@Override
+public FateId fromTypeAndKey(FateInstanceType instanceType, FateKey
fateKey) {
+ HashCode hashCode =
Hashing.murmur3_128().hashBytes(fateKey.getSerialized());
+ long tid = hashCode.asLong() & 0x7fffL;
+ return FateId.from(instanceType, tid);
+}
+ };
+
+ protected final Set reserved;
+ protected final Map deferred;
+ private final int maxDeferred;
+ private final AtomicBoolean deferredOverflow = new AtomicBoolean();
+ private final FateIdGenerator fateIdGenerator;
+
+ // This is incremented each time a transaction was unreserved that was non
new
+ protected final SignalCount unreservedNonNewCount = new SignalCount();
+
+ // This is incremented each time a transaction is unreserved that was
runnable
+ protected final SignalCount unreservedRunnableCount = new SignalCount();
+
+ public AbstractFateStore() {
+this(DEFAULT_MAX_DEFERRED, DEFAULT_FATE_ID_GENERATOR);
+ }
+
+ public AbstractFateStore(int maxDeferred, FateIdG
(accumulo) branch elasticity updated (ae49a7ff02 -> 9550da7c83)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from ae49a7ff02 Merge branch 'main' into elasticity add 23e17129de Revert #4358 - Replace long + TimeUnit with Duration in ReadOnlyTStore.unreserve() add 05c2f45042 Reduced warning logs under normal conditions in compaction coordinator (#4362) add 9839e4d42f Merge branch '2.1' new 9550da7c83 Merge branch 'main' into elasticity The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../accumulo/core/fate/AbstractFateStore.java | 12 ++- .../org/apache/accumulo/core/fate/AdminUtil.java | 6 +++--- .../java/org/apache/accumulo/core/fate/Fate.java | 15 +++--- .../org/apache/accumulo/core/fate/FateCleaner.java | 3 ++- .../org/apache/accumulo/core/fate/FateStore.java | 6 +++--- .../accumulo/core/fate/WrappedFateTxStore.java | 6 +++--- .../apache/accumulo/core/fate/FateCleanerTest.java | 21 ++-- .../org/apache/accumulo/core/fate/TestStore.java | 4 ++-- .../test/compaction/ExternalCompaction_1_IT.java | 6 +++--- .../accumulo/test/fate/accumulo/FateStoreIT.java | 23 +++--- 10 files changed, 53 insertions(+), 49 deletions(-)
(accumulo) branch elasticity updated (8384b9be6a -> ae49a7ff02)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from 8384b9be6a Fate Op Command Updates and Tests (#4350) add 162b8effb1 Replace long + TimeUnit with Duration in ReadOnlyTStore.unreserve() (#4358) add 92331ea113 Throw error when non-standard chars exist (#4348) add 171a1e144c Merge branch '2.1' add ae6085c346 Adds NanoTime wrapper for System.nanoTime (#4364) new ae49a7ff02 Merge branch 'main' into elasticity The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../accumulo/core/fate/AbstractFateStore.java | 24 +-- .../org/apache/accumulo/core/fate/AdminUtil.java | 6 +- .../java/org/apache/accumulo/core/fate/Fate.java | 15 +- .../org/apache/accumulo/core/fate/FateCleaner.java | 3 +- .../org/apache/accumulo/core/fate/FateStore.java | 6 +- .../accumulo/core/fate/WrappedFateTxStore.java | 6 +- .../accumulo/core/file/rfile/GenerateSplits.java | 32 +++- .../apache/accumulo/core/util/time/NanoTime.java | 104 + .../apache/accumulo/core/fate/FateCleanerTest.java | 21 ++- .../org/apache/accumulo/core/fate/TestStore.java | 4 +- .../core/file/rfile/GenerateSplitsTest.java| 64 +++- .../accumulo/core/util/time/NanoTimeTest.java | 162 + .../java/org/apache/accumulo/manager/Manager.java | 9 +- .../test/compaction/ExternalCompaction_1_IT.java | 6 +- .../accumulo/test/fate/accumulo/FateStoreIT.java | 23 ++- 15 files changed, 410 insertions(+), 75 deletions(-) create mode 100644 core/src/main/java/org/apache/accumulo/core/util/time/NanoTime.java create mode 100644 core/src/test/java/org/apache/accumulo/core/util/time/NanoTimeTest.java
(accumulo) 01/01: Merge branch 'main' into elasticity
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit ae49a7ff027178a387d39f048e1090bcffe11162
Merge: 8384b9be6a ae6085c346
Author: Keith Turner
AuthorDate: Wed Mar 13 15:12:41 2024 -0400
Merge branch 'main' into elasticity
.../accumulo/core/fate/AbstractFateStore.java | 24 +--
.../org/apache/accumulo/core/fate/AdminUtil.java | 6 +-
.../java/org/apache/accumulo/core/fate/Fate.java | 15 +-
.../org/apache/accumulo/core/fate/FateCleaner.java | 3 +-
.../org/apache/accumulo/core/fate/FateStore.java | 6 +-
.../accumulo/core/fate/WrappedFateTxStore.java | 6 +-
.../accumulo/core/file/rfile/GenerateSplits.java | 32 +++-
.../apache/accumulo/core/util/time/NanoTime.java | 104 +
.../apache/accumulo/core/fate/FateCleanerTest.java | 21 ++-
.../org/apache/accumulo/core/fate/TestStore.java | 4 +-
.../core/file/rfile/GenerateSplitsTest.java| 64 +++-
.../accumulo/core/util/time/NanoTimeTest.java | 162 +
.../java/org/apache/accumulo/manager/Manager.java | 9 +-
.../test/compaction/ExternalCompaction_1_IT.java | 6 +-
.../accumulo/test/fate/accumulo/FateStoreIT.java | 23 ++-
15 files changed, 410 insertions(+), 75 deletions(-)
diff --cc
core/src/main/java/org/apache/accumulo/core/fate/AbstractFateStore.java
index 702313f5ab,00..0cad25f857
mode 100644,00..100644
--- a/core/src/main/java/org/apache/accumulo/core/fate/AbstractFateStore.java
+++ b/core/src/main/java/org/apache/accumulo/core/fate/AbstractFateStore.java
@@@ -1,494 -1,0 +1,494 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.fate;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+import java.io.UncheckedIOException;
++import java.time.Duration;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
- import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.fate.Fate.TxInfo;
+import org.apache.accumulo.core.util.Pair;
++import org.apache.accumulo.core.util.time.NanoTime;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+import com.google.common.hash.HashCode;
+import com.google.common.hash.Hashing;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+public abstract class AbstractFateStore implements FateStore {
+
+ private static final Logger log =
LoggerFactory.getLogger(AbstractFateStore.class);
+
+ // Default maximum size of 100,000 transactions before deferral is stopped
and
+ // all existing transactions are processed immediately again
+ public static final int DEFAULT_MAX_DEFERRED = 100_000;
+
+ public static final FateIdGenerator DEFAULT_FATE_ID_GENERATOR = new
FateIdGenerator() {
+@Override
+public FateId fromTypeAndKey(FateInstanceType instanceType, FateKey
fateKey) {
+ HashCode hashCode =
Hashing.murmur3_128().hashBytes(fateKey.getSerialized());
+ long tid = hashCode.asLong() & 0x7fffL;
+ return FateId.from(instanceType, tid);
+}
+ };
+
+ protected final Set reserved;
- protected final Map deferred;
++ protected final Map deferred;
+ private final int maxDeferred;
+ private final AtomicBoolean deferredOverflow = new AtomicBoolean();
+ private final FateIdGenerator fateIdGenerator;
+
+ // This is incremented each time a transaction was unreserved that was non
new
+ protected final SignalCount unreservedNonNewCount = new SignalCount();
+
+ // This
(accumulo) branch main updated: Adds NanoTime wrapper for System.nanoTime (#4364)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/main by this push:
new ae6085c346 Adds NanoTime wrapper for System.nanoTime (#4364)
ae6085c346 is described below
commit ae6085c34677e752949ef093cb350281b41a275f
Author: Keith Turner
AuthorDate: Wed Mar 13 14:08:17 2024 -0400
Adds NanoTime wrapper for System.nanoTime (#4364)
* Adds NanoTime wrapper for System.nanoTime
Adds a strong type for System.nanoTime() and uses it in a few places.
Could be used in many more places if this is merged.
Co-authored-by: EdColeman
---
.../org/apache/accumulo/core/fate/ZooStore.java| 15 +-
.../apache/accumulo/core/util/time/NanoTime.java | 104 +
.../accumulo/core/util/time/NanoTimeTest.java | 162 +
.../java/org/apache/accumulo/manager/Manager.java | 9 +-
4 files changed, 280 insertions(+), 10 deletions(-)
diff --git a/core/src/main/java/org/apache/accumulo/core/fate/ZooStore.java
b/core/src/main/java/org/apache/accumulo/core/fate/ZooStore.java
index 941c04c241..c3de5f29df 100644
--- a/core/src/main/java/org/apache/accumulo/core/fate/ZooStore.java
+++ b/core/src/main/java/org/apache/accumulo/core/fate/ZooStore.java
@@ -39,12 +39,12 @@ import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
-import java.util.concurrent.TimeUnit;
import org.apache.accumulo.core.fate.zookeeper.ZooReaderWriter;
import org.apache.accumulo.core.fate.zookeeper.ZooUtil.NodeExistsPolicy;
import org.apache.accumulo.core.fate.zookeeper.ZooUtil.NodeMissingPolicy;
import org.apache.accumulo.core.util.FastFormat;
+import org.apache.accumulo.core.util.time.NanoTime;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.KeeperException.NoNodeException;
import org.apache.zookeeper.KeeperException.NodeExistsException;
@@ -64,7 +64,7 @@ public class ZooStore implements TStore {
private ZooReaderWriter zk;
private String lastReserved = "";
private Set reserved;
- private Map deferred; // use Long here to properly handle
System.nanoTime()
+ private Map deferred;
private long statusChangeEvents = 0;
private int reservationsWaiting = 0;
@@ -164,7 +164,7 @@ public class ZooStore implements TStore {
}
if (deferred.containsKey(tid)) {
- if (deferred.get(tid) - System.nanoTime() < 0) {
+ if (deferred.get(tid).elapsed().compareTo(Duration.ZERO) > 0) {
deferred.remove(tid);
} else {
continue;
@@ -203,10 +203,11 @@ public class ZooStore implements TStore {
if (deferred.isEmpty()) {
this.wait(5000);
} else {
- final long now = System.nanoTime();
- long minWait = deferred.values().stream().mapToLong(l -> l -
now).min().orElseThrow();
+ var now = NanoTime.now();
+ long minWait = deferred.values().stream()
+ .mapToLong(nanoTime ->
nanoTime.subtract(now).toMillis()).min().orElseThrow();
if (minWait > 0) {
-this.wait(Math.min(TimeUnit.NANOSECONDS.toMillis(minWait),
5000));
+this.wait(Math.min(minWait, 5000));
}
}
}
@@ -284,7 +285,7 @@ public class ZooStore implements TStore {
}
if (deferTime.compareTo(Duration.ZERO) > 0) {
-deferred.put(tid, deferTime.toNanos() + System.nanoTime());
+deferred.put(tid, NanoTime.nowPlus(deferTime));
}
this.notifyAll();
diff --git
a/core/src/main/java/org/apache/accumulo/core/util/time/NanoTime.java
b/core/src/main/java/org/apache/accumulo/core/util/time/NanoTime.java
new file mode 100644
index 00..f081278589
--- /dev/null
+++ b/core/src/main/java/org/apache/accumulo/core/util/time/NanoTime.java
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.ut
(accumulo) branch elasticity updated: Fate Op Command Updates and Tests (#4350)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 8384b9be6a Fate Op Command Updates and Tests (#4350)
8384b9be6a is described below
commit 8384b9be6abe786ad855d5457eba6bceb3868025
Author: Kevin Rathbun <43969518+kevinrr...@users.noreply.github.com>
AuthorDate: Wed Mar 13 14:07:33 2024 -0400
Fate Op Command Updates and Tests (#4350)
Changes:
- Update summary FateOpsCommand
- Prints full FateId not just the long
- Option to filter based on FateInstanceType
- Option to filter based on FateId
- Originally, the summary command printed all Fate
transactions, thought it might be good to optionally allow filtering by certain
Fate transactions like print allows
- Update print FateOpsCommand
- Prints full FateId not just the long
- Option to filter based on FateInstanceType
- Option to filter based on FateId
- Originally, could filter by long id, but with the
replacement of long transaction id by FateId, this had to be updated.
- Instead of receiving "" or "FATE[]" on
cmd line, now expects a FateId
- Update cancel, delete, and fail FateOpsCommand
- Now work with both ZooStore and AccumuloStore by taking the full
FateId on the command line and determining the store based on the
FateInstanceType of the FateId
- Added tests for all 5 FateOpsCommands (FateOpsCommandsIT)
- Tests using both AccumuloStore (AccumuloFateOpsCommandsIT) and
ZooStore (ZookeeperFateOpsCommandsIT)
- Deleted FateSummaryIT, moved the test to FateOpsCommands
* Changes:
- Wait for condition after stopping ServerType.COMPACTOR instead of sleep
- Cleaned up tests to use fewer assertions
- Added attempt to --delete and --fail a transaction when the Manager is
still alive in testFateDeleteCommand() and testFateFailCommand()
---
.../org/apache/accumulo/core/fate/AdminUtil.java | 239 +
.../org/apache/accumulo/core/fate/FateTxId.java| 8 -
.../org/apache/accumulo/server/util/Admin.java | 110 ++--
.../server/util/fateCommand/FateSummaryReport.java | 34 +-
.../server/util/fateCommand/FateTxnDetails.java| 24 +-
.../server/util/fateCommand/SummaryReportTest.java | 11 +-
.../server/util/fateCommand/TxnDetailsTest.java| 7 +-
.../manager/metrics/fate/FateMetricValues.java | 2 +-
.../org/apache/accumulo/test/FateSummaryIT.java| 156 --
.../accumulo/test/fate/FateOpsCommandsIT.java | 564 +
.../fate/accumulo/AccumuloFateOpsCommandsIT.java | 32 ++
.../fate/zookeeper/ZookeeperFateOpsCommandsIT.java | 37 ++
.../test/functional/FateConcurrencyIT.java | 14 +-
.../test/functional/FunctionalTestUtils.java | 2 +-
14 files changed, 888 insertions(+), 352 deletions(-)
diff --git a/core/src/main/java/org/apache/accumulo/core/fate/AdminUtil.java
b/core/src/main/java/org/apache/accumulo/core/fate/AdminUtil.java
index 2a436b3444..c18defb1ac 100644
--- a/core/src/main/java/org/apache/accumulo/core/fate/AdminUtil.java
+++ b/core/src/main/java/org/apache/accumulo/core/fate/AdminUtil.java
@@ -46,7 +46,6 @@ import
org.apache.accumulo.core.fate.zookeeper.ZooReaderWriter;
import org.apache.accumulo.core.fate.zookeeper.ZooUtil.NodeMissingPolicy;
import org.apache.accumulo.core.lock.ServiceLock;
import org.apache.accumulo.core.lock.ServiceLock.ServiceLockPath;
-import org.apache.accumulo.core.util.FastFormat;
import org.apache.zookeeper.KeeperException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -75,7 +74,7 @@ public class AdminUtil {
*/
public static class TransactionStatus {
-private final long txid;
+private final FateId fateId;
private final FateInstanceType instanceType;
private final TStatus status;
private final String txName;
@@ -84,10 +83,10 @@ public class AdminUtil {
private final String top;
private final long timeCreated;
-private TransactionStatus(Long tid, FateInstanceType instanceType, TStatus
status,
+private TransactionStatus(FateId fateId, FateInstanceType instanceType,
TStatus status,
String txName, List hlocks, List wlocks, String top,
Long timeCreated) {
- this.txid = tid;
+ this.fateId = fateId;
this.instanceType = instanceType;
this.status = status;
this.txName = txName;
@@ -102,8 +101,8 @@ public class AdminUtil {
* @return This fate operations transaction id, formatted in the same way
as FATE transactions
* are in the Accumulo logs.
*/
-public String getTxid() {
- return FastFormat.toHexString(txid);
+public FateId getFateI
(accumulo) branch elasticity updated: partially avoids splitting and compacting offline tablets (#4343)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 744dc5286d partially avoids splitting and compacting offline tablets
(#4343)
744dc5286d is described below
commit 744dc5286ddc92cf8007c46d664e1deef9afe56d
Author: Keith Turner
AuthorDate: Tue Mar 12 11:18:46 2024 -0400
partially avoids splitting and compacting offline tablets (#4343)
Modifies TabletManagementIterator so that it does not return tablets
that need split of compaction of if the table is offline. This
partially prevents those tablets from splitting. Still need to handle
things that were queued for split of compaction when a table is taken
offline. Once those other changes are made this change will still avoid
queuing and processing tablets for work that does not need to be done.
---
.../manager/state/TabletManagementIterator.java| 2 +-
.../functional/TabletManagementIteratorIT.java | 29 +++---
2 files changed, 27 insertions(+), 4 deletions(-)
diff --git
a/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletManagementIterator.java
b/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletManagementIterator.java
index 4704f691c8..2e6627c78e 100644
---
a/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletManagementIterator.java
+++
b/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletManagementIterator.java
@@ -270,7 +270,7 @@ public class TabletManagementIterator extends
SkippingIterator {
reasonsToReturnThisTablet.add(ManagementAction.NEEDS_LOCATION_UPDATE);
}
-if (tm.getOperationId() == null
+if (tm.getOperationId() == null &&
tabletMgmtParams.isTableOnline(tm.getTableId())
&& Collections.disjoint(REASONS_NOT_TO_SPLIT_OR_COMPACT,
reasonsToReturnThisTablet)) {
try {
if (shouldReturnDueToSplit(tm,
this.env.getPluginEnv().getConfiguration(tm.getTableId( {
diff --git
a/test/src/main/java/org/apache/accumulo/test/functional/TabletManagementIteratorIT.java
b/test/src/main/java/org/apache/accumulo/test/functional/TabletManagementIteratorIT.java
index c88989d1c4..34723f3fae 100644
---
a/test/src/main/java/org/apache/accumulo/test/functional/TabletManagementIteratorIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/functional/TabletManagementIteratorIT.java
@@ -50,6 +50,7 @@ import org.apache.accumulo.core.client.TableNotFoundException;
import org.apache.accumulo.core.client.admin.NewTableConfiguration;
import org.apache.accumulo.core.client.admin.TabletAvailability;
import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Mutation;
import org.apache.accumulo.core.data.Range;
@@ -112,7 +113,7 @@ public class TabletManagementIteratorIT extends
AccumuloClusterHarness {
try (AccumuloClient client =
Accumulo.newClient().from(getClientProps()).build()) {
- String[] tables = getUniqueNames(9);
+ String[] tables = getUniqueNames(10);
final String t1 = tables[0];
final String t2 = tables[1];
final String t3 = tables[2];
@@ -122,6 +123,7 @@ public class TabletManagementIteratorIT extends
AccumuloClusterHarness {
final String metaCopy3 = tables[6];
final String metaCopy4 = tables[7];
final String metaCopy5 = tables[8];
+ final String metaCopy6 = tables[9];
// create some metadata
createTable(client, t1, true);
@@ -156,6 +158,7 @@ public class TabletManagementIteratorIT extends
AccumuloClusterHarness {
copyTable(client, metaCopy1, metaCopy3);
copyTable(client, metaCopy1, metaCopy4);
copyTable(client, metaCopy1, metaCopy5);
+ copyTable(client, metaCopy1, metaCopy6);
// t1 is unassigned, setting to always will generate a change to host
tablets
setTabletAvailability(client, metaCopy1, t1,
TabletAvailability.HOSTED.name());
@@ -240,8 +243,27 @@ public class TabletManagementIteratorIT extends
AccumuloClusterHarness {
assertEquals(1, findTabletsNeedingAttention(client, metaCopy4,
tabletMgmtParams),
"Should have one tablet that needs a volume replacement");
+ // In preparation for split an offline testing ensure nothing needs
attention
+ tabletMgmtParams = createParameters(client);
+ addFiles(client, metaCopy6, t4);
+ assertEquals(0, findTabletsNeedingAttention(client, metaCopy6,
tabletMgmtParams),
+ "No tablets should need attention");
+ // Lower the split threshold for the table, should cause the files added
to need attention.
+ client.tableOperations().setProperty(tables[3],
P
(accumulo) branch elasticity updated: Updates merge code to handle all tablet metadata columns (#4352)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 3fe1115014 Updates merge code to handle all tablet metadata columns
(#4352)
3fe1115014 is described below
commit 3fe11150145fab6d920aae8582340ad7e07a8fba
Author: Keith Turner
AuthorDate: Fri Mar 8 19:59:27 2024 -0500
Updates merge code to handle all tablet metadata columns (#4352)
Adds tests for all tablet metadata columns to ensure the merge code
handles them. Some columns were not handled, so new code was added to
handle them.
In the process of adding unit test for the merge code, also added test
for some odd conditions in the code that would be hard to test in an
integration test.
---
.../manager/tableOps/merge/DeleteRows.java | 11 +-
.../manager/tableOps/merge/MergeTablets.java | 70 +++-
.../manager/tableOps/merge/MergeTabletsTest.java | 466 +
.../manager/tableOps/split/UpdateTabletsTest.java | 2 +-
4 files changed, 527 insertions(+), 22 deletions(-)
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/merge/DeleteRows.java
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/merge/DeleteRows.java
index f071785ece..2c5221b7ce 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/merge/DeleteRows.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/merge/DeleteRows.java
@@ -18,11 +18,6 @@
*/
package org.apache.accumulo.manager.tableOps.merge;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.FILES;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.LOCATION;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.LOGS;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.OPID;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.PREV_ROW;
import static
org.apache.accumulo.manager.tableOps.merge.MergeTablets.validateTablet;
import java.util.ArrayList;
@@ -88,9 +83,9 @@ public class DeleteRows extends ManagerRepo {
var opid = TabletOperationId.from(TabletOperationType.MERGING, fateId);
try (
-var tabletsMetadata = manager.getContext().getAmple().readTablets()
-.forTable(range.tableId()).overlapping(range.prevEndRow(),
range.endRow())
-.fetch(OPID, LOCATION, FILES, PREV_ROW,
LOGS).checkConsistency().build();
+var tabletsMetadata =
+
manager.getContext().getAmple().readTablets().forTable(range.tableId())
+.overlapping(range.prevEndRow(),
range.endRow()).checkConsistency().build();
var tabletsMutator =
manager.getContext().getAmple().conditionallyMutateTablets()) {
KeyExtent firstCompleteContained = null;
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/merge/MergeTablets.java
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/merge/MergeTablets.java
index d48528a3f7..50703849c6 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/merge/MergeTablets.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/merge/MergeTablets.java
@@ -18,16 +18,6 @@
*/
package org.apache.accumulo.manager.tableOps.merge;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.AVAILABILITY;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.DIR;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.ECOMP;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.FILES;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.LOCATION;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.LOGS;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.MERGED;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.OPID;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.PREV_ROW;
-import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.TIME;
import static
org.apache.accumulo.manager.tableOps.merge.DeleteRows.verifyAccepted;
import java.util.ArrayList;
@@ -87,12 +77,12 @@ public class MergeTablets extends ManagerRepo {
TabletMetadata lastTabletMeta = null;
try (var tabletsMetadata = manager.getContext().getAmple().readTablets()
-.forTable(range.tableId()).overlapping(range.prevEndRow(),
range.endRow())
-.fetch(OPID, LOCATION, AVAILABILITY, FILES, TIME, DIR, ECOMP,
PREV_ROW, LOGS, MERGED)
-.build
(accumulo) branch elasticity updated: Stop waiting on fate when reporting compaction complete (#4339)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 5c5150242d Stop waiting on fate when reporting compaction complete
(#4339)
5c5150242d is described below
commit 5c5150242d160160bfcf8aed3cfe6fdcefd81da5
Author: Keith Turner
AuthorDate: Fri Mar 8 19:58:45 2024 -0500
Stop waiting on fate when reporting compaction complete (#4339)
Compactors were waiting on the Fate transaction that does compaction
commit. They were doing this so that the dead compaction detector
would not kill the compaction during commit. This change removes
the need for compactors to wait on compaction commit, freeing them
to run another compaction.
To accomplish this the dead compaction detector was modified to look
for fate operations commiting compactions. This work leveraged the
new FateKey. A bonus for this changes is that duplicate RPC messages
reporting a compaction as completed will no longer spin up duplicate
Fate operations to commit. The Fate operations would have likely
been harmless, but would have consumed resources.
A new method was added to Fate to list transactions with a certain
FateKeyType. Tests were added for this new Fate method.
An integration test was added to ensure the dead compaction detector
does not delete compactions that no compactors knows of but do have
a Fate transaction committing them.
---
.../java/org/apache/accumulo/core/fate/Fate.java | 8 ++
.../accumulo/core/fate/ReadOnlyFateStore.java | 5 ++
.../org/apache/accumulo/core/fate/ZooStore.java| 6 ++
.../accumulo/core/fate/accumulo/AccumuloStore.java | 14
.../apache/accumulo/core/logging/FateLogger.java | 5 ++
.../org/apache/accumulo/core/fate/TestStore.java | 5 ++
.../coordinator/CompactionCoordinator.java | 33 ++--
.../coordinator/DeadCompactionDetector.java| 97 --
.../coordinator/commit/PutGcCandidates.java| 11 +--
.../coordinator/commit/RefreshTablet.java | 7 +-
.../test/compaction/ExternalCompaction_1_IT.java | 97 ++
.../accumulo/test/fate/accumulo/FateStoreIT.java | 65 +++
12 files changed, 318 insertions(+), 35 deletions(-)
diff --git a/core/src/main/java/org/apache/accumulo/core/fate/Fate.java
b/core/src/main/java/org/apache/accumulo/core/fate/Fate.java
index 0b82f73f11..e5be68dbb2 100644
--- a/core/src/main/java/org/apache/accumulo/core/fate/Fate.java
+++ b/core/src/main/java/org/apache/accumulo/core/fate/Fate.java
@@ -42,6 +42,7 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.TransferQueue;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.Function;
+import java.util.stream.Stream;
import
org.apache.accumulo.core.clientImpl.AcceptableThriftTableOperationException;
import org.apache.accumulo.core.conf.AccumuloConfiguration;
@@ -482,6 +483,13 @@ public class Fate {
}
}
+ /**
+ * Lists transctions for a given fate key type.
+ */
+ public Stream list(FateKey.FateKeyType type) {
+return store.list(type);
+ }
+
/**
* Initiates shutdown of background threads and optionally waits on them.
*/
diff --git
a/core/src/main/java/org/apache/accumulo/core/fate/ReadOnlyFateStore.java
b/core/src/main/java/org/apache/accumulo/core/fate/ReadOnlyFateStore.java
index bdbb7739f9..b2aa4999b2 100644
--- a/core/src/main/java/org/apache/accumulo/core/fate/ReadOnlyFateStore.java
+++ b/core/src/main/java/org/apache/accumulo/core/fate/ReadOnlyFateStore.java
@@ -138,6 +138,11 @@ public interface ReadOnlyFateStore {
*/
Stream list();
+ /**
+ * list transaction in the store that have a given fate key type.
+ */
+ Stream list(FateKey.FateKeyType type);
+
/**
* Finds all fate ops that are (IN_PROGRESS, SUBMITTED, or
FAILED_IN_PROGRESS) and unreserved. Ids
* that are found are passed to the consumer. This method will block until
at least one runnable
diff --git a/core/src/main/java/org/apache/accumulo/core/fate/ZooStore.java
b/core/src/main/java/org/apache/accumulo/core/fate/ZooStore.java
index 6813e727c5..af6fd233de 100644
--- a/core/src/main/java/org/apache/accumulo/core/fate/ZooStore.java
+++ b/core/src/main/java/org/apache/accumulo/core/fate/ZooStore.java
@@ -377,6 +377,12 @@ public class ZooStore extends AbstractFateStore {
}
}
+ @Override
+ public Stream list(FateKey.FateKeyType type) {
+return getTransactions().flatMap(fis -> getKey(fis.getFateId()).stream())
+.filter(fateKey -> fateKey.getType() == type);
+ }
+
protected static class NodeValue {
final TStatus status;
final Optional fateKey;
diff --git
a/core/src/main/java/org/apache/accumulo/core/fate/accumulo/AccumuloStore.j
(accumulo) branch elasticity updated: fixes bug and removes unneeded wait in delete table (#4346)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new c7ae69b881 fixes bug and removes unneeded wait in delete table (#4346)
c7ae69b881 is described below
commit c7ae69b881da00113f2ceef642f5f470818d569d
Author: Keith Turner
AuthorDate: Thu Mar 7 13:31:48 2024 -0500
fixes bug and removes unneeded wait in delete table (#4346)
Noticed an integration test was taking 0 to 5 seconds to delete a table.
Investigated this and found some code in delete table that was waiting
on the TabletGroupWatcher to cycle as the cause of the delay.
Investigated why this wait was there and could not find a definite
answer. Speculating that the purpose of the wait was to deal with
possible concurrent writes to tablet metadata by the tablet group
watcher in the case it has not yet noticed the table state is DELETING.
Changes in elasticity make this wait unnecessary because the delete table
fate sets an operation id on each tablet that should prevent any writes
to the tablet after its set. After being set this would prevent any
in flight writes by the tablet group watcher from succeeding. Before
elasticity there was no way for the delete table fate to prevent writes
by the tablet group watcher, so thinking it just waited until it was
certain the tablet group watcher was aware of the new table state and
would therefore do no writes.
While researching I noticed the delete table code in 2.1 logged per
tablet information about any tablet its was waiting on. This was not
being done in elasticity, so added logging for this.
With these changes TabletManagementIteratorIT went from 1 min runtime to
30 second runtime. The test deleted 9 tables.
---
.../java/org/apache/accumulo/manager/Manager.java | 10 --
.../apache/accumulo/manager/state/TableStats.java | 4
.../accumulo/manager/tableOps/delete/CleanUp.java | 23 --
.../manager/tableOps/delete/DeleteTable.java | 2 +-
.../manager/tableOps/delete/ReserveTablets.java| 13 +++-
5 files changed, 13 insertions(+), 39 deletions(-)
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/Manager.java
b/server/manager/src/main/java/org/apache/accumulo/manager/Manager.java
index 935ca7d262..95fb00777e 100644
--- a/server/manager/src/main/java/org/apache/accumulo/manager/Manager.java
+++ b/server/manager/src/main/java/org/apache/accumulo/manager/Manager.java
@@ -520,16 +520,6 @@ public class Manager extends AbstractServer
}
}
- public boolean hasCycled(long time) {
-for (TabletGroupWatcher watcher : watchers) {
- if (watcher.stats.lastScanFinished() < time) {
-return false;
- }
-}
-
-return true;
- }
-
public void clearMigrations(TableId tableId) {
synchronized (migrations) {
migrations.keySet().removeIf(extent -> extent.tableId().equals(tableId));
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/state/TableStats.java
b/server/manager/src/main/java/org/apache/accumulo/manager/state/TableStats.java
index 3e2380a57c..c0d9a4f5e3 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/state/TableStats.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/state/TableStats.java
@@ -76,10 +76,6 @@ public class TableStats {
return endScan - startScan;
}
- public synchronized long lastScanFinished() {
-return endScan;
- }
-
@Override
public String toString() {
return new StringBuilder().append("last:
").append(last.toString()).toString();
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/delete/CleanUp.java
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/delete/CleanUp.java
index 9d8d2b86f6..9dd234eba9 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/delete/CleanUp.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/delete/CleanUp.java
@@ -61,32 +61,9 @@ class CleanUp extends ManagerRepo {
private long creationTime;
- private void readObject(java.io.ObjectInputStream in) throws IOException,
ClassNotFoundException {
-in.defaultReadObject();
-
-// handle the case where we start executing on a new machine where the
current time is in the
-// past relative to the previous machine
-// if the new machine has time in the future, that will work ok w/
hasCycled
-if (System.currentTimeMillis() < creationTime) {
- creationTime = System.currentTimeMillis();
-}
-
- }
-
public CleanUp(TableId tableId, NamespaceId namespaceId) {
this.tableId = tableId;
this.namespaceId = namespaceId;
-creationTime = Syste
(accumulo) branch elasticity updated: Updates compaction to use TabletLogger (#4333)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 202198a588 Updates compaction to use TabletLogger (#4333)
202198a588 is described below
commit 202198a588aae1096611ff9b5513075054e2a876
Author: Keith Turner
AuthorDate: Mon Mar 4 10:55:50 2024 -0500
Updates compaction to use TabletLogger (#4333)
---
.../apache/accumulo/core/logging/TabletLogger.java | 49 +++---
.../coordinator/CompactionCoordinator.java | 9 ++--
.../coordinator/commit/CommitCompaction.java | 4 ++
.../manager/tableOps/compact/CompactionDriver.java | 22 --
test/src/main/resources/log4j2-test.properties | 3 ++
5 files changed, 55 insertions(+), 32 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/logging/TabletLogger.java
b/core/src/main/java/org/apache/accumulo/core/logging/TabletLogger.java
index 349d29b19f..e76c62a6c9 100644
--- a/core/src/main/java/org/apache/accumulo/core/logging/TabletLogger.java
+++ b/core/src/main/java/org/apache/accumulo/core/logging/TabletLogger.java
@@ -27,14 +27,16 @@ import java.util.SortedSet;
import java.util.UUID;
import java.util.concurrent.TimeUnit;
-import org.apache.accumulo.core.client.admin.CompactionConfig;
import org.apache.accumulo.core.client.admin.compaction.CompactableFile;
import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.fate.FateId;
import org.apache.accumulo.core.metadata.CompactableFileImpl;
+import org.apache.accumulo.core.metadata.ReferencedTabletFile;
import org.apache.accumulo.core.metadata.StoredTabletFile;
import org.apache.accumulo.core.metadata.TServerInstance;
import org.apache.accumulo.core.metadata.TabletFile;
import org.apache.accumulo.core.metadata.schema.ExternalCompactionId;
+import org.apache.accumulo.core.metadata.schema.TabletMetadata;
import org.apache.accumulo.core.spi.compaction.CompactionJob;
import org.apache.accumulo.core.spi.compaction.CompactionKind;
import org.apache.accumulo.core.tabletserver.log.LogEntry;
@@ -127,40 +129,36 @@ public class TabletLogger {
cf -> CompactableFileImpl.toStoredTabletFile(cf).toMinimalString());
}
- public static void selected(KeyExtent extent, CompactionKind kind,
+ public static void selected(FateId fateId, KeyExtent extent,
Collection inputs) {
-fileLog.trace("{} changed compaction selection set for {} new set {}",
extent, kind,
+fileLog.trace("Selected files {} {} {}", extent, fateId,
Collections2.transform(inputs, StoredTabletFile::toMinimalString));
}
- public static void compacting(KeyExtent extent, CompactionJob job,
CompactionConfig config) {
+ public static void compacting(TabletMetadata tabletMetadata,
ExternalCompactionId cid,
+ String compactorAddress, CompactionJob job) {
if (fileLog.isDebugEnabled()) {
- if (config == null) {
-fileLog.debug("Compacting {} on {} for {} from {} size {}", extent,
job.getGroup(),
-job.getKind(), asMinimalString(job.getFiles()),
getSize(job.getFiles()));
+ if (job.getKind() == CompactionKind.USER) {
+var fateId = tabletMetadata.getSelectedFiles().getFateId();
+fileLog.debug(
+"Compacting {} driver:{} id:{} group:{} compactor:{} priority:{}
size:{} kind:{} files:{}",
+tabletMetadata.getExtent(), fateId, cid, job.getGroup(),
compactorAddress,
+job.getPriority(), getSize(job.getFiles()), job.getKind(),
+asMinimalString(job.getFiles()));
} else {
-fileLog.debug("Compacting {} on {} for {} from {} size {} config {}",
extent,
-job.getGroup(), job.getKind(), asMinimalString(job.getFiles()),
getSize(job.getFiles()),
-config);
+fileLog.debug(
+"Compacting {} id:{} group:{} compactor:{} priority:{} size:{}
kind:{} files:{}",
+tabletMetadata.getExtent(), cid, job.getGroup(), compactorAddress,
job.getPriority(),
+getSize(job.getFiles()), job.getKind(),
asMinimalString(job.getFiles()));
}
}
}
- public static void compacted(KeyExtent extent, CompactionJob job,
StoredTabletFile output) {
-fileLog.debug("Compacted {} for {} created {} from {}", extent,
job.getKind(), output,
-asMinimalString(job.getFiles()));
- }
-
- public static void compactionFailed(KeyExtent extent, CompactionJob job,
- CompactionConfig config) {
-fileLog.debug("Failed to compact: extent: {}, input files: {}, iterators:
{}", extent,
-asMinimalString(job.getFiles()), config.getIterators());
- }
-
- public static void externalCompactionFailed(KeyExtent extent,
ExternalCompactionId id,
- CompactionJob job, CompactionConf
(accumulo) branch elasticity updated: Considers all tablet metadata columns in split code (#4323)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 1d7f6937fc Considers all tablet metadata columns in split code (#4323)
1d7f6937fc is described below
commit 1d7f6937fcd4a122978486c10832b37c62effea1
Author: Keith Turner
AuthorDate: Mon Mar 4 10:31:48 2024 -0500
Considers all tablet metadata columns in split code (#4323)
Made the following changes to the split code that adds new tablets and
updates the existing tablet.
* fixed potential NPE w/ tablet operation id check by reversing order
of equals check
* Throws IllegalStateException when attempting to split tablet with
merged or cloned markers
* Removed adding wals when creating new tablets in split, its not
expected that the parent tablet would have wals and this is checked
earlier
* Deleted any user compaction requested, hosting requested, suspended,
or last columns in the parent tablet
Added a unit test that attempts to exercise the split code with all
tablet columns. The unit test also has a set of tablet columns that
were verified to work with split and it is checked against the set of
columns in the code. The purpose of this test is to fail when a new
column is added to ensure that split is considered.
Was a bit uncertain about deleting the last location and suspend. Those
columns either need to be deleted from the parent tablet or added to the
new tablets being created. The current code was doing neither. Decided
to delete them as the new tablets have a different range and are
conceptually different tablets than the parent.
---
.../manager/tableOps/split/UpdateTablets.java | 42 ++-
.../manager/tableOps/split/UpdateTabletsTest.java | 328 +
.../apache/accumulo/test/functional/SplitIT.java | 54
3 files changed, 421 insertions(+), 3 deletions(-)
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/UpdateTablets.java
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/UpdateTablets.java
index 2fe3c56399..f2d1501ae5 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/UpdateTablets.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/UpdateTablets.java
@@ -22,6 +22,7 @@ import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
import java.util.Set;
import java.util.SortedSet;
import java.util.TreeMap;
@@ -82,7 +83,7 @@ public class UpdateTablets extends ManagerRepo {
}
}
-Preconditions.checkState(tabletMetadata.getOperationId().equals(opid),
+Preconditions.checkState(opid.equals(tabletMetadata.getOperationId()),
"Tablet %s does not have expected operation id %s it has %s",
splitInfo.getOriginal(), opid,
tabletMetadata.getOperationId());
@@ -94,6 +95,13 @@ public class UpdateTablets extends ManagerRepo {
"Tablet unexpectedly had walogs %s %s %s", fateId,
tabletMetadata.getLogs(),
tabletMetadata.getExtent());
+Preconditions.checkState(!tabletMetadata.hasMerged(),
+"Tablet unexpectedly has a merged marker %s %s", fateId,
tabletMetadata.getExtent());
+
+Preconditions.checkState(tabletMetadata.getCloned() == null,
+"Tablet unexpectedly has a cloned marker %s %s %s", fateId,
tabletMetadata.getCloned(),
+tabletMetadata.getExtent());
+
var newTablets = splitInfo.getTablets();
var newTabletsFiles = getNewTabletFiles(newTablets, tabletMetadata,
@@ -120,7 +128,7 @@ public class UpdateTablets extends ManagerRepo {
newTablets.forEach(extent -> tabletsFiles.put(extent, new HashMap<>()));
-// determine while files overlap which tablets and their estimated sizes
+// determine which files overlap which tablets and their estimated sizes
tabletMetadata.getFilesMap().forEach((file, dataFileValue) -> {
FileUtil.FileInfo fileInfo = fileInfoProvider.apply(file);
@@ -187,6 +195,7 @@ public class UpdateTablets extends ManagerRepo {
mutator.putTime(tabletMetadata.getTime());
tabletMetadata.getFlushId().ifPresent(mutator::putFlushId);
mutator.putPrevEndRow(newExtent.prevEndRow());
+
tabletMetadata.getCompacted().forEach(mutator::putCompacted);
tabletMetadata.getCompacted().forEach(compactedFateId -> log
@@ -195,7 +204,6 @@ public class UpdateTablets extends ManagerRepo {
mutator.putTabletAvailability(tabletMetadata.getTabletAvailability());
tabletMetadata.getLoaded().forEach((k, v) ->
mutator.putBulkFile(k.getTabletFile(), v));
(accumulo) branch elasticity updated: Unit test compaction reservation and deny offline (#4334)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 01b0ff9356 Unit test compaction reservation and deny offline (#4334)
01b0ff9356 is described below
commit 01b0ff9356b851d8f47368989407fa610441c23d
Author: Keith Turner
AuthorDate: Mon Mar 4 10:30:10 2024 -0500
Unit test compaction reservation and deny offline (#4334)
Changes the compaction coordinator to not return jobs for a table that
is offline. The point where this check was added to the coordinator
needed unit test, so also added the unit test.
---
.../coordinator/CompactionCoordinator.java | 18 ++-
.../compaction/CompactionCoordinatorTest.java | 148 +
2 files changed, 160 insertions(+), 6 deletions(-)
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java
b/server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java
index ebd5d8f764..a07ff50bc4 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java
@@ -74,6 +74,7 @@ import org.apache.accumulo.core.fate.FateInstanceType;
import org.apache.accumulo.core.fate.zookeeper.ZooReaderWriter;
import org.apache.accumulo.core.iterators.user.HasExternalCompactionsFilter;
import org.apache.accumulo.core.iteratorsImpl.system.SystemIteratorUtil;
+import org.apache.accumulo.core.manager.state.tables.TableState;
import org.apache.accumulo.core.metadata.CompactableFileImpl;
import org.apache.accumulo.core.metadata.ReferencedTabletFile;
import org.apache.accumulo.core.metadata.StoredTabletFile;
@@ -124,6 +125,7 @@ import com.github.benmanes.caffeine.cache.Cache;
import com.github.benmanes.caffeine.cache.CacheLoader;
import com.github.benmanes.caffeine.cache.LoadingCache;
import com.github.benmanes.caffeine.cache.Weigher;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
import com.google.common.collect.Sets;
import com.google.common.net.HostAndPort;
@@ -382,9 +384,9 @@ public class CompactionCoordinator
}
- // ELASTICITY_TODO unit test this code
- private boolean canReserveCompaction(TabletMetadata tablet, CompactionJob
job,
- Set jobFiles) {
+ @VisibleForTesting
+ public static boolean canReserveCompaction(TabletMetadata tablet,
CompactionKind kind,
+ Set jobFiles, ServerContext ctx) {
if (tablet == null) {
// the tablet no longer exist
@@ -395,6 +397,10 @@ public class CompactionCoordinator
return false;
}
+if (ctx.getTableState(tablet.getTableId()) != TableState.ONLINE) {
+ return false;
+}
+
if (!tablet.getFiles().containsAll(jobFiles)) {
return false;
}
@@ -406,7 +412,7 @@ public class CompactionCoordinator
return false;
}
-switch (job.getKind()) {
+switch (kind) {
case SYSTEM:
var userRequestedCompactions =
tablet.getUserCompactionsRequested().size();
if (userRequestedCompactions > 0) {
@@ -427,7 +433,7 @@ public class CompactionCoordinator
}
break;
default:
-throw new UnsupportedOperationException("Not currently handling " +
job.getKind());
+throw new UnsupportedOperationException("Not currently handling " +
kind);
}
return true;
@@ -508,7 +514,7 @@ public class CompactionCoordinator
try (var tabletsMutator = ctx.getAmple().conditionallyMutateTablets()) {
var extent = metaJob.getTabletMetadata().getExtent();
-if (!canReserveCompaction(tabletMetadata, metaJob.getJob(), jobFiles))
{
+if (!canReserveCompaction(tabletMetadata, metaJob.getJob().getKind(),
jobFiles, ctx)) {
return null;
}
diff --git
a/server/manager/src/test/java/org/apache/accumulo/manager/compaction/CompactionCoordinatorTest.java
b/server/manager/src/test/java/org/apache/accumulo/manager/compaction/CompactionCoordinatorTest.java
index 770405863e..4c0b2b1d52 100644
---
a/server/manager/src/test/java/org/apache/accumulo/manager/compaction/CompactionCoordinatorTest.java
+++
b/server/manager/src/test/java/org/apache/accumulo/manager/compaction/CompactionCoordinatorTest.java
@@ -18,10 +18,18 @@
*/
package org.apache.accumulo.manager.compaction;
+import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.ECOMP;
+import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.OPID;
+import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.S
(accumulo) branch elasticity updated: updates IdleProcessMetricsIT to verify config instead of set it. (#4331)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 8798fccd5c updates IdleProcessMetricsIT to verify config instead of
set it. (#4331)
8798fccd5c is described below
commit 8798fccd5c32874cfd2d8f255781820280534e0f
Author: Keith Turner
AuthorDate: Fri Mar 1 13:42:50 2024 -0500
updates IdleProcessMetricsIT to verify config instead of set it. (#4331)
IdleProcessMetricsIT was setting compactor config to the same as the
default. Changed the test to verify the config instead of setting it.
---
.../apache/accumulo/test/functional/IdleProcessMetricsIT.java | 11 +++
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git
a/test/src/main/java/org/apache/accumulo/test/functional/IdleProcessMetricsIT.java
b/test/src/main/java/org/apache/accumulo/test/functional/IdleProcessMetricsIT.java
index c2d2f649ae..36d16b8d2d 100644
---
a/test/src/main/java/org/apache/accumulo/test/functional/IdleProcessMetricsIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/functional/IdleProcessMetricsIT.java
@@ -18,11 +18,14 @@
*/
package org.apache.accumulo.test.functional;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
import java.time.Duration;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.accumulo.core.Constants;
import org.apache.accumulo.core.conf.Property;
import org.apache.accumulo.core.metrics.MetricsProducer;
import org.apache.accumulo.harness.MiniClusterConfigurationCallback;
@@ -42,10 +45,10 @@ public class IdleProcessMetricsIT extends
SharedMiniClusterBase {
@Override
public void configureMiniCluster(MiniAccumuloConfigImpl cfg, Configuration
coreSite) {
- // Configure all compaction planners to use the default resource group so
- // that only 1 compactor is started by MiniAccumuloCluster
- cfg.setProperty(Property.COMPACTION_SERVICE_DEFAULT_GROUPS.getKey(),
- "[{'group':'default'}]".replaceAll("'", "\""));
+ // Verify expectations about the default config. Want to ensure there no
other resource groups
+ // configured.
+ assertEquals(Map.of(Constants.DEFAULT_COMPACTION_SERVICE_NAME, 1),
+ cfg.getClusterServerConfiguration().getCompactorConfiguration());
// Disable the default scan servers and compactors, just start 1
// tablet server in the default group to host the root and metadata
(accumulo) branch elasticity updated: update IdleProcessMetricsIT to work w/ recent changes (#4326)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 5bec92248c update IdleProcessMetricsIT to work w/ recent changes
(#4326)
5bec92248c is described below
commit 5bec92248ce3c980681dd5925a207eb9b866df3e
Author: Keith Turner
AuthorDate: Thu Feb 29 18:40:37 2024 -0500
update IdleProcessMetricsIT to work w/ recent changes (#4326)
---
.../org/apache/accumulo/test/functional/IdleProcessMetricsIT.java | 6 +-
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git
a/test/src/main/java/org/apache/accumulo/test/functional/IdleProcessMetricsIT.java
b/test/src/main/java/org/apache/accumulo/test/functional/IdleProcessMetricsIT.java
index 169d54a3c2..c2d2f649ae 100644
---
a/test/src/main/java/org/apache/accumulo/test/functional/IdleProcessMetricsIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/functional/IdleProcessMetricsIT.java
@@ -44,11 +44,7 @@ public class IdleProcessMetricsIT extends
SharedMiniClusterBase {
// Configure all compaction planners to use the default resource group so
// that only 1 compactor is started by MiniAccumuloCluster
- cfg.setProperty("compaction.service.root.planner.opts.groups",
- "[{'group':'default'}]".replaceAll("'", "\""));
- cfg.setProperty("compaction.service.meta.planner.opts.groups",
- "[{'group':'default'}]".replaceAll("'", "\""));
- cfg.setProperty("compaction.service.default.planner.opts.groups",
+ cfg.setProperty(Property.COMPACTION_SERVICE_DEFAULT_GROUPS.getKey(),
"[{'group':'default'}]".replaceAll("'", "\""));
// Disable the default scan servers and compactors, just start 1
(accumulo) branch elasticity updated: sets initial fate table config (#4320)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new a6f1073ac9 sets initial fate table config (#4320)
a6f1073ac9 is described below
commit a6f1073ac9b95a4a36508e018ac6f24fa499c37f
Author: Keith Turner
AuthorDate: Wed Feb 28 16:32:19 2024 -0500
sets initial fate table config (#4320)
---
.../accumulo/server/init/InitialConfiguration.java | 44 +++---
1 file changed, 23 insertions(+), 21 deletions(-)
diff --git
a/server/base/src/main/java/org/apache/accumulo/server/init/InitialConfiguration.java
b/server/base/src/main/java/org/apache/accumulo/server/init/InitialConfiguration.java
index e932e0af15..81f2876db5 100644
---
a/server/base/src/main/java/org/apache/accumulo/server/init/InitialConfiguration.java
+++
b/server/base/src/main/java/org/apache/accumulo/server/init/InitialConfiguration.java
@@ -49,26 +49,30 @@ class InitialConfiguration {
this.hadoopConf = hadoopConf;
this.siteConf = siteConf;
-
initialRootMetaConf.put(Property.TABLE_FILE_COMPRESSED_BLOCK_SIZE.getKey(),
"32K");
-initialRootMetaConf.put(Property.TABLE_FILE_REPLICATION.getKey(), "5");
-initialRootMetaConf.put(Property.TABLE_DURABILITY.getKey(), "sync");
-initialRootMetaConf.put(Property.TABLE_MAJC_RATIO.getKey(), "1");
-initialRootMetaConf.put(Property.TABLE_SPLIT_THRESHOLD.getKey(), "64M");
-initialRootMetaConf.put(Property.TABLE_CONSTRAINT_PREFIX.getKey() + "1",
-MetadataConstraints.class.getName());
-initialRootMetaConf.put(Property.TABLE_ITERATOR_PREFIX.getKey() +
"scan.vers",
+// config common to all Accumulo tables
+Map commonConfig = new HashMap<>();
+commonConfig.put(Property.TABLE_FILE_COMPRESSED_BLOCK_SIZE.getKey(),
"32K");
+commonConfig.put(Property.TABLE_FILE_REPLICATION.getKey(), "5");
+commonConfig.put(Property.TABLE_DURABILITY.getKey(), "sync");
+commonConfig.put(Property.TABLE_MAJC_RATIO.getKey(), "1");
+commonConfig.put(Property.TABLE_ITERATOR_PREFIX.getKey() + "scan.vers",
"10," + VersioningIterator.class.getName());
-initialRootMetaConf.put(Property.TABLE_ITERATOR_PREFIX.getKey() +
"scan.vers.opt.maxVersions",
-"1");
-initialRootMetaConf.put(Property.TABLE_ITERATOR_PREFIX.getKey() +
"minc.vers",
+commonConfig.put(Property.TABLE_ITERATOR_PREFIX.getKey() +
"scan.vers.opt.maxVersions", "1");
+commonConfig.put(Property.TABLE_ITERATOR_PREFIX.getKey() + "minc.vers",
"10," + VersioningIterator.class.getName());
-initialRootMetaConf.put(Property.TABLE_ITERATOR_PREFIX.getKey() +
"minc.vers.opt.maxVersions",
-"1");
-initialRootMetaConf.put(Property.TABLE_ITERATOR_PREFIX.getKey() +
"majc.vers",
+commonConfig.put(Property.TABLE_ITERATOR_PREFIX.getKey() +
"minc.vers.opt.maxVersions", "1");
+commonConfig.put(Property.TABLE_ITERATOR_PREFIX.getKey() + "majc.vers",
"10," + VersioningIterator.class.getName());
-initialRootMetaConf.put(Property.TABLE_ITERATOR_PREFIX.getKey() +
"majc.vers.opt.maxVersions",
-"1");
-initialRootMetaConf.put(Property.TABLE_FAILURES_IGNORE.getKey(), "false");
+commonConfig.put(Property.TABLE_ITERATOR_PREFIX.getKey() +
"majc.vers.opt.maxVersions", "1");
+commonConfig.put(Property.TABLE_FAILURES_IGNORE.getKey(), "false");
+commonConfig.put(Property.TABLE_DEFAULT_SCANTIME_VISIBILITY.getKey(), "");
+commonConfig.put(Property.TABLE_INDEXCACHE_ENABLED.getKey(), "true");
+commonConfig.put(Property.TABLE_BLOCKCACHE_ENABLED.getKey(), "true");
+
+initialRootMetaConf.putAll(commonConfig);
+initialRootMetaConf.put(Property.TABLE_SPLIT_THRESHOLD.getKey(), "64M");
+initialRootMetaConf.put(Property.TABLE_CONSTRAINT_PREFIX.getKey() + "1",
+MetadataConstraints.class.getName());
initialRootMetaConf.put(Property.TABLE_LOCALITY_GROUP_PREFIX.getKey() +
"tablet",
String.format("%s,%s",
MetadataSchema.TabletsSection.TabletColumnFamily.NAME,
MetadataSchema.TabletsSection.CurrentLocationColumnFamily.NAME));
@@ -78,11 +82,9 @@ class InitialConfiguration {
MetadataSchema.TabletsSection.ServerColumnFamily.NAME,
MetadataSchema.TabletsSection.FutureLocationColumnFamily.NAME));
initialRootMetaConf.put(Property.TABLE_LOCALITY_GROUPS.getKey(),
"tablet,server");
-
initialRootMetaConf.put(Property.TABLE_DEFAULT_SCANTIME_V
(accumulo) branch elasticity updated: Added check for tablets with errors, when looking for volume replacements. (#4287)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new cca3683e5b Added check for tablets with errors, when looking for
volume replacements. (#4287)
cca3683e5b is described below
commit cca3683e5b51f66787569e9b968b4599848e136e
Author: Arbaaz Khan
AuthorDate: Tue Feb 27 17:18:36 2024 -0500
Added check for tablets with errors, when looking for volume replacements.
(#4287)
Added check for tablets with errors, when looking for volume replacements.
fixes #4234
---
.../org/apache/accumulo/manager/TabletGroupWatcher.java| 14 +-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/TabletGroupWatcher.java
b/server/manager/src/main/java/org/apache/accumulo/manager/TabletGroupWatcher.java
index e8bdd794a4..e7ea20413a 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/TabletGroupWatcher.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/TabletGroupWatcher.java
@@ -353,6 +353,7 @@ abstract class TabletGroupWatcher extends
AccumuloDaemonThread {
int[] counts = new int[TabletState.values().length];
private int totalUnloaded;
private long totalVolumeReplacements;
+private int tabletsWithErrors;
}
private TableMgmtStats manageTablets(Iterator iter,
@@ -398,6 +399,7 @@ abstract class TabletGroupWatcher extends
AccumuloDaemonThread {
"Error on TabletServer trying to get Tablet management information
for extent: {}. Error message: {}",
tm.getExtent(), mtiError);
this.metrics.incrementTabletGroupWatcherError(this.store.getLevel());
+tableMgmtStats.tabletsWithErrors++;
continue;
}
@@ -670,7 +672,17 @@ abstract class TabletGroupWatcher extends
AccumuloDaemonThread {
iter = store.iterator(tableMgmtParams);
var tabletMgmtStats = manageTablets(iter, tableMgmtParams,
currentTServers, true);
-lookForTabletsNeedingVolReplacement =
tabletMgmtStats.totalVolumeReplacements != 0;
+
+// If currently looking for volume replacements, determine if the next
round needs to look.
+if (lookForTabletsNeedingVolReplacement) {
+ // Continue to look for tablets needing volume replacement if there
was an error
+ // processing tablets in the call to manageTablets() or if we are
still performing volume
+ // replacement. We only want to stop looking for tablets that need
volume replacement when
+ // we have successfully processed all tablet metadata and no more
volume replacements are
+ // being performed.
+ lookForTabletsNeedingVolReplacement =
tabletMgmtStats.totalVolumeReplacements != 0
+ || tabletMgmtStats.tabletsWithErrors != 0;
+}
// provide stats after flushing changes to avoid race conditions w/
delete table
stats.end(managerState);
(accumulo) branch elasticity updated: Adds race condtion test for CompactionJobQueues (#4310)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new f320ca36d9 Adds race condtion test for CompactionJobQueues (#4310)
f320ca36d9 is described below
commit f320ca36d9b84cea97522ac4d21b327b3fe11cc6
Author: Keith Turner
AuthorDate: Tue Feb 27 14:30:40 2024 -0500
Adds race condtion test for CompactionJobQueues (#4310)
Adds a unit test that concurrently calls add() and poll() on
CompactionJobsQueues.
This test validates that race conditions between these methods are handled.
Fixed
a race condition found by running the test.
---
.../queue/CompactionJobPriorityQueue.java | 6 +
.../compaction/queue/CompactionJobQueues.java | 11 +-
.../queue/CompactionJobPriorityQueueTest.java | 2 +-
.../compaction/queue/CompactionJobQueuesTest.java | 121 +
4 files changed, 135 insertions(+), 5 deletions(-)
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/compaction/queue/CompactionJobPriorityQueue.java
b/server/manager/src/main/java/org/apache/accumulo/manager/compaction/queue/CompactionJobPriorityQueue.java
index 6c28fe5ecf..e4aa059e95 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/compaction/queue/CompactionJobPriorityQueue.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/compaction/queue/CompactionJobPriorityQueue.java
@@ -116,8 +116,14 @@ public class CompactionJobPriorityQueue {
this.dequeuedJobs = new AtomicLong(0);
}
+ /**
+ * @return the number of jobs added. If the queue is closed returns -1
+ */
public synchronized int add(TabletMetadata tabletMetadata,
Collection jobs) {
Preconditions.checkArgument(jobs.stream().allMatch(job ->
job.getGroup().equals(groupId)));
+if (closed.get()) {
+ return -1;
+}
removePreviousSubmissions(tabletMetadata.getExtent());
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/compaction/queue/CompactionJobQueues.java
b/server/manager/src/main/java/org/apache/accumulo/manager/compaction/queue/CompactionJobQueues.java
index 77395fd253..b47ec92675 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/compaction/queue/CompactionJobQueues.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/compaction/queue/CompactionJobQueues.java
@@ -29,6 +29,8 @@ import
org.apache.accumulo.core.spi.compaction.CompactorGroupId;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import com.google.common.base.Preconditions;
+
public class CompactionJobQueues {
private static final Logger log =
LoggerFactory.getLogger(CompactionJobQueues.class);
@@ -145,16 +147,17 @@ public class CompactionJobQueues {
private void add(TabletMetadata tabletMetadata, CompactorGroupId groupId,
Collection jobs) {
-// TODO log level
-if (log.isDebugEnabled()) {
- log.debug("Adding jobs to queue {} {} {}", groupId,
tabletMetadata.getExtent(),
+if (log.isTraceEnabled()) {
+ log.trace("Adding jobs to queue {} {} {}", groupId,
tabletMetadata.getExtent(),
jobs.stream().map(job -> "#files:" + job.getFiles().size() +
",prio:" + job.getPriority()
+ ",kind:" + job.getKind()).collect(Collectors.toList()));
}
var pq = priorityQueues.computeIfAbsent(groupId,
gid -> new CompactionJobPriorityQueue(gid, queueSize));
-while (pq.add(tabletMetadata, jobs) == 0 && pq.isClosed()) {
+while (pq.add(tabletMetadata, jobs) < 0) {
+ // When entering this loop its expected the queue is closed
+ Preconditions.checkState(pq.isClosed());
// This loop handles race condition where poll() closes empty priority
queues. The queue could
// be closed after its obtained from the map and before add is called.
pq = priorityQueues.computeIfAbsent(groupId,
diff --git
a/server/manager/src/test/java/org/apache/accumulo/manager/compaction/queue/CompactionJobPriorityQueueTest.java
b/server/manager/src/test/java/org/apache/accumulo/manager/compaction/queue/CompactionJobPriorityQueueTest.java
index a7bbc914cf..268751dc85 100644
---
a/server/manager/src/test/java/org/apache/accumulo/manager/compaction/queue/CompactionJobPriorityQueueTest.java
+++
b/server/manager/src/test/java/org/apache/accumulo/manager/compaction/queue/CompactionJobPriorityQueueTest.java
@@ -258,7 +258,7 @@ public class CompactionJobPriorityQueueTest {
assertTrue(queue.closeIfEmpty());
-assertEquals(2, queue.add(tm, List.of(cj1, cj2)));
+assertEquals(-1, queue.add(tm, List.of(cj1, cj2)));
}
diff --git
a/server/manager/src/test/java/org/apache/accumulo/manager/compaction/queue/CompactionJobQueuesTest.java
b/ser
(accumulo) branch elasticity updated (3310612bc8 -> 0345175549)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from 3310612bc8 Add new metadata column to prevent user compaction starvation (#4254) add 0345175549 fetches all columns in tablet mgmt iterator (#4307) No new revisions were added by this update. Summary of changes: .../core/manager/state/TabletManagement.java | 5 + .../manager/state/TabletManagementIterator.java| 26 ++ 2 files changed, 3 insertions(+), 28 deletions(-)
(accumulo) branch elasticity updated (58be770cb5 -> d7348ae521)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from 58be770cb5 Merge branch 'main' into elasticity add bdec3a7c3a fixes clearing suspension for offline tables (#4295) add ea5afb9d85 Merge branch '2.1' new d7348ae521 Merge branch 'main' into elasticity The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../server/manager/state/MetaDataStateStore.java | 9 +- .../accumulo/test/manager/SuspendedTabletsIT.java | 160 ++--- 2 files changed, 112 insertions(+), 57 deletions(-)
(accumulo) 01/01: Merge branch 'main' into elasticity
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit d7348ae5219e40f7e960eb585b14960a1a242bcd
Merge: 58be770cb5 ea5afb9d85
Author: Keith Turner
AuthorDate: Thu Feb 22 19:24:44 2024 -0500
Merge branch 'main' into elasticity
.../server/manager/state/MetaDataStateStore.java | 9 +-
.../accumulo/test/manager/SuspendedTabletsIT.java | 160 ++---
2 files changed, 112 insertions(+), 57 deletions(-)
diff --cc
server/base/src/main/java/org/apache/accumulo/server/manager/state/MetaDataStateStore.java
index 66f9b87a0a,ccb5dc2747..689a667e4b
---
a/server/base/src/main/java/org/apache/accumulo/server/manager/state/MetaDataStateStore.java
+++
b/server/base/src/main/java/org/apache/accumulo/server/manager/state/MetaDataStateStore.java
@@@ -64,22 -67,29 +64,21 @@@ class MetaDataStateStore extends Abstra
}
@Override
- public void setLocations(Collection assignments) throws
DistributedStoreException {
-try (var tabletsMutator = ample.mutateTablets()) {
- for (Assignment assignment : assignments) {
-TabletMutator tabletMutator =
tabletsMutator.mutateTablet(assignment.tablet);
-tabletMutator.putLocation(Location.current(assignment.server));
-ManagerMetadataUtil.updateLastForAssignmentMode(context,
tabletMutator, assignment.server,
-assignment.lastLocation);
-tabletMutator.deleteLocation(Location.future(assignment.server));
-tabletMutator.deleteSuspension();
-tabletMutator.mutate();
+ public void unsuspend(Collection tablets) throws
DistributedStoreException {
+try (var tabletsMutator = ample.conditionallyMutateTablets()) {
+ for (TabletMetadata tm : tablets) {
+if (tm.getSuspend() != null) {
- continue;
++ // ELASTICITY_TODO add conditional mutation check that tls.suspend
is what currently
++ // exists in the tablet
++
tabletsMutator.mutateTablet(tm.getExtent()).requireAbsentOperation().deleteSuspension()
++ .submit(tabletMetadata -> tabletMetadata.getSuspend() == null);
+}
-
- // ELASTICITY_TODO pending #3314, add conditional mutation check that
tls.suspend exists
-
tabletsMutator.mutateTablet(tm.getExtent()).requireAbsentOperation().deleteSuspension()
- .submit(tabletMetadata -> tabletMetadata.getSuspend() == null);
}
-} catch (RuntimeException ex) {
- throw new DistributedStoreException(ex);
-}
- }
- @Override
- public void setFutureLocations(Collection assignments)
- throws DistributedStoreException {
-try (var tabletsMutator = ample.mutateTablets()) {
- for (Assignment assignment : assignments) {
-tabletsMutator.mutateTablet(assignment.tablet).deleteSuspension()
-.putLocation(Location.future(assignment.server)).mutate();
+ boolean unacceptedConditions =
tabletsMutator.process().values().stream()
+ .anyMatch(conditionalResult -> conditionalResult.getStatus() !=
Status.ACCEPTED);
+ if (unacceptedConditions) {
+throw new DistributedStoreException("Some mutations failed to satisfy
conditions");
}
} catch (RuntimeException ex) {
throw new DistributedStoreException(ex);
diff --cc
test/src/main/java/org/apache/accumulo/test/manager/SuspendedTabletsIT.java
index 9c8dfccc0b,8ebb75378b..03f42fa253
---
a/test/src/main/java/org/apache/accumulo/test/manager/SuspendedTabletsIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/manager/SuspendedTabletsIT.java
@@@ -19,11 -19,16 +19,12 @@@
package org.apache.accumulo.test.manager;
import static java.util.concurrent.TimeUnit.SECONDS;
-import static org.apache.accumulo.core.util.LazySingletons.RANDOM;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+ import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assertions.fail;
-import java.net.UnknownHostException;
import java.time.Duration;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
@@@ -77,18 -78,14 +78,17 @@@ import com.google.common.collect.HashMu
import com.google.common.collect.SetMultimap;
import com.google.common.net.HostAndPort;
-public class SuspendedTabletsIT extends ConfigurableMacBase {
+public class SuspendedTabletsIT extends AccumuloClusterHarness {
private static final Logger log =
LoggerFactory.getLogger(SuspendedTabletsIT.class);
private static ExecutorService THREAD_POOL;
+ private static final String TEST_GROUP_NAME = "SUSPEND_TEST";
public static final int TSERVERS = 3;
- public static final long SUSP
(accumulo) 01/01: Merge branch '2.1'
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit ea5afb9d85e342834554504cdf541a43dcc79a84
Merge: 9ffc450b0a bdec3a7c3a
Author: Keith Turner
AuthorDate: Thu Feb 22 19:00:28 2024 -0500
Merge branch '2.1'
.../server/manager/state/MetaDataStateStore.java | 3 +-
.../manager/state/TabletStateChangeIterator.java | 3 +
.../accumulo/test/manager/SuspendedTabletsIT.java | 249 +
3 files changed, 157 insertions(+), 98 deletions(-)
diff --cc
test/src/main/java/org/apache/accumulo/test/manager/SuspendedTabletsIT.java
index ec99094346,764bc1dfc9..8ebb75378b
---
a/test/src/main/java/org/apache/accumulo/test/manager/SuspendedTabletsIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/manager/SuspendedTabletsIT.java
@@@ -333,6 -300,95 +303,93 @@@ public class SuspendedTabletsIT extend
throws Exception;
}
+ private class ShutdownTserverKiller implements TServerKiller {
+
+ @Override
+ public void eliminateTabletServers(ClientContext ctx, TabletLocations
locs, int count)
+ throws Exception {
-
+ Set tserverSet = new HashSet<>();
+ Set metadataServerSet = new HashSet<>();
+
- TabletLocator tl = TabletLocator.getLocator(ctx, MetadataTable.ID);
++ TabletLocator tl = TabletLocator.getLocator(ctx,
AccumuloTable.METADATA.tableId());
+ for (TabletLocationState tls : locs.locationStates.values()) {
+ if (tls.current != null) {
+ // add to set of all servers
+ tserverSet.add(tls.current.getServerInstance());
+
+ // get server that the current tablets metadata is on
+ TabletLocator.TabletLocation tab =
+ tl.locateTablet(ctx, tls.extent.toMetaRow(), false, false);
+ // add it to the set of servers with metadata
- metadataServerSet
- .add(new TServerInstance(tab.tablet_location,
Long.valueOf(tab.tablet_session, 16)));
++ metadataServerSet.add(new TServerInstance(tab.getTserverLocation(),
++ Long.valueOf(tab.getTserverSession(), 16)));
+ }
+ }
+
+ // remove servers with metadata on them from the list of servers to be
shutdown
+ assertEquals(1, metadataServerSet.size(), "Expecting a single tServer
in metadataServerSet");
+ tserverSet.removeAll(metadataServerSet);
+
+ assertEquals(TSERVERS - 1, tserverSet.size(),
+ "Expecting " + (TSERVERS - 1) + " tServers in shutdown-list");
+
+ List tserversList = new ArrayList<>(tserverSet);
- Collections.shuffle(tserversList, random);
++ Collections.shuffle(tserversList, RANDOM.get());
+
+ for (int i1 = 0; i1 < count; ++i1) {
+ final String tserverName = tserversList.get(i1).getHostPortSession();
+ ThriftClientTypes.MANAGER.executeVoid(ctx, client -> {
+ log.info("Sending shutdown command to {} via ManagerClientService",
tserverName);
+ client.shutdownTabletServer(null, ctx.rpcCreds(), tserverName,
false);
+ });
+ }
+
+ log.info("Waiting for tserver process{} to die", count == 1 ? "" :
"es");
+ for (int i2 = 0; i2 < 10; ++i2) {
+ List deadProcs = new ArrayList<>();
+ for (ProcessReference pr1 :
getCluster().getProcesses().get(ServerType.TABLET_SERVER)) {
+ Process p = pr1.getProcess();
+ if (!p.isAlive()) {
+ deadProcs.add(pr1);
+ }
+ }
+ for (ProcessReference pr2 : deadProcs) {
+ log.info("Process {} is dead, informing cluster control about
this", pr2.getProcess());
+
getCluster().getClusterControl().killProcess(ServerType.TABLET_SERVER, pr2);
+ --count;
+ }
+ if (count == 0) {
+ return;
+ } else {
+ Thread.sleep(SECONDS.toMillis(2));
+ }
+ }
+ throw new IllegalStateException("Tablet servers didn't die!");
-
+ }
+ }
+
+ private class CrashTserverKiller implements TServerKiller {
+
+ @Override
+ public void eliminateTabletServers(ClientContext ctx, TabletLocations
locs, int count)
+ throws Exception {
+ // Exclude the tablet server hosting the metadata table from the list
and only
+ // kill tablet servers that are not hosting the metadata table.
+ List procs =
getCluster().getProcesses().get(ServerType.TABLET_SERVER)
+ .stream().filter(p ->
!metadataTserverProcess.equals(p)).collect(Collectors.toList());
- Collections.shuffle(procs, random);
++ Collections.shuffle(procs, RANDOM.get());
+ assertEquals(TSERVERS - 1, procs.size(), "Not enough tservers exist");
+ assertTrue(procs.size() >= count, "Attempting
(accumulo) branch main updated (9ffc450b0a -> ea5afb9d85)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git from 9ffc450b0a Merge branch '2.1' add bdec3a7c3a fixes clearing suspension for offline tables (#4295) new ea5afb9d85 Merge branch '2.1' The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../server/manager/state/MetaDataStateStore.java | 3 +- .../manager/state/TabletStateChangeIterator.java | 3 + .../accumulo/test/manager/SuspendedTabletsIT.java | 249 + 3 files changed, 157 insertions(+), 98 deletions(-)
(accumulo) branch 2.1 updated: fixes clearing suspension for offline tables (#4295)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch 2.1
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/2.1 by this push:
new bdec3a7c3a fixes clearing suspension for offline tables (#4295)
bdec3a7c3a is described below
commit bdec3a7c3a5d373cf95fd7123363dd35679a0ab8
Author: Keith Turner
AuthorDate: Thu Feb 22 18:35:10 2024 -0500
fixes clearing suspension for offline tables (#4295)
There was code in the manager that seemed to have the intent of clearing
suspension markers in tablets for an offline table. This code was not
working or tested. This commit fixes this code and adds a tests that
validates that suspension markers are removed when a table is taken
offline.
fixes #3314
---
.../server/manager/state/MetaDataStateStore.java | 3 +-
.../manager/state/TabletStateChangeIterator.java | 3 +
.../accumulo/test/manager/SuspendedTabletsIT.java | 251 +
3 files changed, 159 insertions(+), 98 deletions(-)
diff --git
a/server/base/src/main/java/org/apache/accumulo/server/manager/state/MetaDataStateStore.java
b/server/base/src/main/java/org/apache/accumulo/server/manager/state/MetaDataStateStore.java
index 8c7bc888eb..9af443895a 100644
---
a/server/base/src/main/java/org/apache/accumulo/server/manager/state/MetaDataStateStore.java
+++
b/server/base/src/main/java/org/apache/accumulo/server/manager/state/MetaDataStateStore.java
@@ -150,9 +150,8 @@ class MetaDataStateStore implements TabletStateStore {
try (var tabletsMutator = ample.mutateTablets()) {
for (TabletLocationState tls : tablets) {
if (tls.suspend != null) {
- continue;
+ tabletsMutator.mutateTablet(tls.extent).deleteSuspension().mutate();
}
-tabletsMutator.mutateTablet(tls.extent).deleteSuspension().mutate();
}
} catch (RuntimeException ex) {
throw new DistributedStoreException(ex);
diff --git
a/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletStateChangeIterator.java
b/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletStateChangeIterator.java
index c4c9e611de..42fd9cb05c 100644
---
a/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletStateChangeIterator.java
+++
b/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletStateChangeIterator.java
@@ -209,6 +209,9 @@ public class TabletStateChangeIterator extends
SkippingIterator {
case ASSIGNED_TO_DEAD_SERVER:
return;
case SUSPENDED:
+ // Always return data about suspended tablets. Need to clear the
suspension stats when the
+ // tablet is offline. May need to assign the tablet when the tablet
is online.
+ return;
case UNASSIGNED:
if (shouldBeOnline) {
return;
diff --git
a/test/src/main/java/org/apache/accumulo/test/manager/SuspendedTabletsIT.java
b/test/src/main/java/org/apache/accumulo/test/manager/SuspendedTabletsIT.java
index 46e81fb055..764bc1dfc9 100644
---
a/test/src/main/java/org/apache/accumulo/test/manager/SuspendedTabletsIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/manager/SuspendedTabletsIT.java
@@ -83,7 +83,6 @@ public class SuspendedTabletsIT extends ConfigurableMacBase {
private static ExecutorService THREAD_POOL;
public static final int TSERVERS = 3;
- public static final long SUSPEND_DURATION = 80;
public static final int TABLETS = 30;
private ProcessReference metadataTserverProcess;
@@ -95,7 +94,6 @@ public class SuspendedTabletsIT extends ConfigurableMacBase {
@Override
public void configure(MiniAccumuloConfigImpl cfg, Configuration fsConf) {
-cfg.setProperty(Property.TABLE_SUSPEND_DURATION, SUSPEND_DURATION + "s");
cfg.setClientProperty(ClientProperty.INSTANCE_ZOOKEEPERS_TIMEOUT, "5s");
cfg.setProperty(Property.INSTANCE_ZK_TIMEOUT, "5s");
// Start with 1 tserver, we'll increase that later
@@ -149,91 +147,45 @@ public class SuspendedTabletsIT extends
ConfigurableMacBase {
getCluster().start();
}
+ enum AfterSuspendAction {
+RESUME("80s"),
+// Set a long suspend time for testing offline table, want the suspension
to be cleared because
+// the tablet went offline and not the because the suspension timed out.
+OFFLINE("800s");
+
+public final String suspendTime;
+
+AfterSuspendAction(String suspendTime) {
+ this.suspendTime = suspendTime;
+}
+ }
+
@Test
public void crashAndResumeTserver() throws Exception {
// Run the test body. When we get to the point where we need a tserver to
go away, get rid of it
// via crashing
-suspensionTestBody((ctx, locs, count) -> {
- // Exclude the tablet server hosting the metadata table from the list
and only
- //
(accumulo) 01/01: Merge branch 'main' into elasticity
This is an automated email from the ASF dual-hosted git repository. kturner pushed a commit to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git commit 58be770cb54331081965ef5689e5722a0d47797e Merge: a4af4f0155 9ffc450b0a Author: Keith Turner AuthorDate: Thu Feb 22 16:15:55 2024 -0500 Merge branch 'main' into elasticity .../apache/accumulo/test/ScanConsistencyIT.java| 33 -- 1 file changed, 18 insertions(+), 15 deletions(-)
(accumulo) branch elasticity updated (a4af4f0155 -> 58be770cb5)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from a4af4f0155 Merge branch 'main' into elasticity add f2adaebba2 Removed extra braces in string added as part of #4289 add 9deb242ff6 Merge branch '2.1' add 1a40aee541 fixes intermittent failure in ScanConsistencyIT (#4292) add 9ffc450b0a Merge branch '2.1' new 58be770cb5 Merge branch 'main' into elasticity The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/accumulo/test/ScanConsistencyIT.java| 33 -- 1 file changed, 18 insertions(+), 15 deletions(-)
(accumulo) 01/01: Merge branch '2.1'
This is an automated email from the ASF dual-hosted git repository. kturner pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git commit 9ffc450b0a0c15b2fe16a070b323250e82bca00d Merge: 9deb242ff6 1a40aee541 Author: Keith Turner AuthorDate: Thu Feb 22 16:11:00 2024 -0500 Merge branch '2.1' .../apache/accumulo/test/ScanConsistencyIT.java| 33 -- 1 file changed, 18 insertions(+), 15 deletions(-)
(accumulo) branch main updated (9deb242ff6 -> 9ffc450b0a)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git from 9deb242ff6 Merge branch '2.1' add 1a40aee541 fixes intermittent failure in ScanConsistencyIT (#4292) new 9ffc450b0a Merge branch '2.1' The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/accumulo/test/ScanConsistencyIT.java| 33 -- 1 file changed, 18 insertions(+), 15 deletions(-)
(accumulo) branch 2.1 updated: fixes intermittent failure in ScanConsistencyIT (#4292)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch 2.1
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/2.1 by this push:
new 1a40aee541 fixes intermittent failure in ScanConsistencyIT (#4292)
1a40aee541 is described below
commit 1a40aee54116416181576f3f10198033fe3235f3
Author: Keith Turner
AuthorDate: Thu Feb 22 16:10:29 2024 -0500
fixes intermittent failure in ScanConsistencyIT (#4292)
The following failure was observed when running ScanConsistencyIT in the
elasticity branch.
```
java.util.concurrent.ExecutionException: java.util.NoSuchElementException
at
java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at
java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at
org.apache.accumulo.test.ScanConsistencyIT.testConcurrentScanConsistency(ScanConsistencyIT.java:186)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.NoSuchElementException
at
com.google.common.collect.MoreCollectors$ToOptionalState.getElement(MoreCollectors.java:163)
at
com.google.common.collect.MoreCollectors.lambda$static$1(MoreCollectors.java:75)
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:582)
at
org.apache.accumulo.test.ScanConsistencyIT$TableOpsTask.call(ScanConsistencyIT.java:685)
at
org.apache.accumulo.test.ScanConsistencyIT$TableOpsTask.call(ScanConsistencyIT.java:622)
... 4 more
```
This was caused by the test attempting to do a filter compaction when
there was currently no data to delete. Added a check for this case in
this commit.
---
.../apache/accumulo/test/ScanConsistencyIT.java| 33 --
1 file changed, 18 insertions(+), 15 deletions(-)
diff --git a/test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java
b/test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java
index a882547a05..0c03de99b0 100644
--- a/test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java
+++ b/test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java
@@ -659,21 +659,24 @@ public class ScanConsistencyIT extends
AccumuloClusterHarness {
// 1 in 20 chance of doing a filter compaction. This compaction will
delete a data set.
var deletes = tctx.dataTracker.getDeletes();
- // The row has the format :, the following
gets the generations
- // from the rows. Expect the generation to be the same for a set of
data to delete.
- String gen = deletes.stream().map(m -> new String(m.getRow(), UTF_8))
- .map(row ->
row.split(":")[1]).distinct().collect(MoreCollectors.onlyElement());
-
- IteratorSetting iterSetting =
- new IteratorSetting(100, "genfilter", GenerationFilter.class);
- iterSetting.addOptions(Map.of("generation", gen));
-
- // run a compaction that deletes every key with the specified
generation. Must wait on the
- // compaction because at the end of the test it will try to verify
deleted data is not
- // present. Must flush the table in case data to delete is still in
memory.
- tctx.client.tableOperations().compact(tctx.table, new
CompactionConfig().setFlush(true)
- .setWait(true).setIterators(List.of(iterSetting)));
- numFilters++;
+ if (!deletes.isEmpty()) {
+// The row has the format :, the
following gets the generations
+// from the rows. Expect the generation to be the same for a set
of data to delete.
+String gen = deletes.stream().map(m -> new String(m.getRow(),
UTF_8))
+.map(row ->
row.split(":")[1]).distinct().collect(MoreCollectors.onlyElement());
+
+IteratorSetting iterSetting =
+new IteratorSetting(100, "genfilter", GenerationFilter.class);
+iterSetting.addOptions(Map.of("generation", gen));
+
+// run a compaction that deletes every key with the specified
generation. Must wait on
+// the
+// compaction because at the end of the test it will try to verify
deleted data is not
+// present. Must flush the table in case data to delete is still
in memory.
+tctx.client.tableOperations().compact(tctx.table, new
CompactionConfig().setFlus
(accumulo) branch elasticity updated: improves TabletServerGivesUpIT (#4290)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new d766696152 improves TabletServerGivesUpIT (#4290)
d766696152 is described below
commit d7666961522c595ac6e8c408377c1c1664e62ee2
Author: Keith Turner
AuthorDate: Thu Feb 22 10:24:03 2024 -0500
improves TabletServerGivesUpIT (#4290)
TabletServerGivesUpIT tests that a tserver will eventually die when it
can not write to a walog. For walog write failures to occur, there must
be some thread in the tserver trying to write to the tserver. Changes in
elasticity have made it where the activitiy this test was doing did not
always cause a walog write. This change is an attempt to improve the
chance of a tserver thread attempting a walog write after DFS is killed.
---
.../accumulo/test/TabletServerGivesUpIT.java | 30 --
1 file changed, 22 insertions(+), 8 deletions(-)
diff --git
a/test/src/main/java/org/apache/accumulo/test/TabletServerGivesUpIT.java
b/test/src/main/java/org/apache/accumulo/test/TabletServerGivesUpIT.java
index 2c023859e7..ae4009ee07 100644
--- a/test/src/main/java/org/apache/accumulo/test/TabletServerGivesUpIT.java
+++ b/test/src/main/java/org/apache/accumulo/test/TabletServerGivesUpIT.java
@@ -21,16 +21,15 @@ package org.apache.accumulo.test;
import static java.util.concurrent.TimeUnit.SECONDS;
import java.time.Duration;
-import java.util.TreeSet;
import java.util.concurrent.atomic.AtomicReference;
import org.apache.accumulo.core.client.Accumulo;
import org.apache.accumulo.core.client.AccumuloClient;
import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Mutation;
import org.apache.accumulo.miniclusterImpl.MiniAccumuloConfigImpl;
import org.apache.accumulo.test.functional.ConfigurableMacBase;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.io.Text;
import org.junit.jupiter.api.Test;
// ACCUMULO-2480
@@ -60,25 +59,40 @@ public class TabletServerGivesUpIT extends
ConfigurableMacBase {
}
final String tableName = getUniqueNames(1)[0];
client.tableOperations().create(tableName);
+
+ // do an initial write to host the tablet and get its location in cache
as we may not be able
+ // to read the metadata table later
+ try (var writer = client.createBatchWriter(tableName)) {
+Mutation m = new Mutation("001");
+m.put("a", "b", "c");
+writer.addMutation(m);
+ }
+
// Kill dfs
cluster.getMiniDfs().shutdown();
// ask the tserver to do something
final AtomicReference ex = new AtomicReference<>();
- Thread splitter = new Thread(() -> {
+ Thread backgroundWriter = new Thread(() -> {
try {
- TreeSet splits = new TreeSet<>();
- splits.add(new Text("X"));
- client.tableOperations().addSplits(tableName, splits);
+ for (int i = 0; i < 100; i++) {
+// These writes should cause the tserver to attempt to write to
walog, which should
+// repeatedly fail. Relying on the client side cache to have the
tablet location so the
+// writes make it to the tserver where the wal write fails.
+try (var writer = client.createBatchWriter(tableName)) {
+ Mutation m = new Mutation("001");
+ m.put("a", "b", "c");
+ writer.addMutation(m);
+}
+ }
} catch (Exception e) {
ex.set(e);
}
});
- splitter.start();
+ backgroundWriter.start();
// wait for the tserver to give up on writing to the WAL
while (client.instanceOperations().getTabletServers().size() == 1) {
Thread.sleep(SECONDS.toMillis(1));
}
}
}
-
}
(accumulo) branch elasticity updated (5d8d9cd57f -> 78f6430141)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from 5d8d9cd57f moves finding split points into FATE (#4178) add 00bac7ae56 FateIT Repo Error Test (#4282) add e3d6204305 Merge remote-tracking branch 'upstream/2.1' add 78f6430141 Merge branch 'main' into elasticity No new revisions were added by this update. Summary of changes: .../java/org/apache/accumulo/test/fate/FateIT.java | 125 + 1 file changed, 125 insertions(+)
(accumulo) branch elasticity updated: moves finding split points into FATE (#4178)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 5d8d9cd57f moves finding split points into FATE (#4178)
5d8d9cd57f is described below
commit 5d8d9cd57ffcfdab21f42fb4b4a9dfa36a16cd26
Author: Keith Turner
AuthorDate: Wed Feb 21 17:37:20 2024 -0500
moves finding split points into FATE (#4178)
This commit makes the following changes
1. Moves finding split points from tablet from a custom thread pool in
the manager into FATE. This is done by the new Repo named FindSplits.
2. Stops tracking what splits are running in manager memory and moves
this into FATE. This is not implemented yet, plan to do that in
another PR
3. Stops tracking what tablets are unsplittable in manager memory. This
should be tracked in metadata table per #4177. This change can be
in its own commit. In this commit the in memory tracking is simply
removed.
These changes will reduce manager memory usage. These changes also will
make splitting tablets work much better if FATE is distributed for two
reasons. First it allows the computation to spread over multiple
manager processes. Second it allows tracking of what is splitting to
move from memory to persisted storage and not rely on a map in a single
process.
---
.../java/org/apache/accumulo/core/fate/Fate.java | 58 +++---
.../apache/accumulo/core/util/cache/Caches.java| 1 -
.../accumulo/core/util/threads/ThreadPools.java| 3 -
.../accumulo/manager/TabletGroupWatcher.java | 14 +--
.../accumulo/manager/split/SeedSplitTask.java | 62 +++
.../apache/accumulo/manager/split/SplitTask.java | 98
.../apache/accumulo/manager/split/Splitter.java| 122
.../manager/tableOps/split/FindSplits.java | 87 +++
.../accumulo/manager/tableOps/split/PreSplit.java | 7 +-
.../accumulo/manager/split/SplitterTest.java | 123 -
10 files changed, 214 insertions(+), 361 deletions(-)
diff --git a/core/src/main/java/org/apache/accumulo/core/fate/Fate.java
b/core/src/main/java/org/apache/accumulo/core/fate/Fate.java
index 3828bb80c4..0b82f73f11 100644
--- a/core/src/main/java/org/apache/accumulo/core/fate/Fate.java
+++ b/core/src/main/java/org/apache/accumulo/core/fate/Fate.java
@@ -57,6 +57,8 @@ import org.apache.thrift.TApplicationException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import com.google.common.base.Preconditions;
+
/**
* Fault tolerant executor
*/
@@ -334,6 +336,44 @@ public class Fate {
return store.create();
}
+ public Optional seedTransaction(String txName, FateKey fateKey,
Repo repo,
+ boolean autoCleanUp, String goalMessage) {
+
+Optional> optTxStore = store.createAndReserve(fateKey);
+
+return optTxStore.map(txStore -> {
+ var fateId = txStore.getID();
+ try {
+Preconditions.checkState(txStore.getStatus() == NEW);
+seedTransaction(txName, fateId, repo, autoCleanUp, goalMessage,
txStore);
+ } finally {
+txStore.unreserve(0, MILLISECONDS);
+ }
+ return fateId;
+});
+ }
+
+ private void seedTransaction(String txName, FateId fateId, Repo repo,
boolean autoCleanUp,
+ String goalMessage, FateTxStore txStore) {
+if (txStore.top() == null) {
+ try {
+log.info("Seeding {} {}", fateId, goalMessage);
+txStore.push(repo);
+ } catch (StackOverflowException e) {
+// this should not happen
+throw new IllegalStateException(e);
+ }
+}
+
+if (autoCleanUp) {
+ txStore.setTransactionInfo(TxInfo.AUTO_CLEAN, autoCleanUp);
+}
+
+txStore.setTransactionInfo(TxInfo.TX_NAME, txName);
+
+txStore.setStatus(SUBMITTED);
+ }
+
// start work in the transaction.. it is safe to call this
// multiple times for a transaction... but it will only seed once
public void seedTransaction(String txName, FateId fateId, Repo repo,
boolean autoCleanUp,
@@ -341,23 +381,7 @@ public class Fate {
FateTxStore txStore = store.reserve(fateId);
try {
if (txStore.getStatus() == NEW) {
-if (txStore.top() == null) {
- try {
-log.info("Seeding {} {}", fateId, goalMessage);
-txStore.push(repo);
- } catch (StackOverflowException e) {
-// this should not happen
-throw new IllegalStateException(e);
- }
-}
-
-if (autoCleanUp) {
- txStore.setTransactionInfo(TxInfo.AUTO_CLEAN, autoCleanUp);
-}
-
-txStore.setTransactionInfo(TxInfo.TX_NAME, txName);
-
-txStore.setStatus(SUBMITTED);
+seedTransaction(txName, fateId, repo, autoClea
(accumulo) branch elasticity updated: persist time to tablet in bulk update (#4072)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 4378e02420 persist time to tablet in bulk update (#4072)
4378e02420 is described below
commit 4378e0242046aecb448cfa7b8b79700cb85ea230
Author: Keith Turner
AuthorDate: Wed Feb 21 16:12:34 2024 -0500
persist time to tablet in bulk update (#4072)
When bulk import operations set time and a tablet was hosted the time
was not persisted. The bulk import fate operation now persist time in
tablet metadata. The tablet code assumed it was the only thing
updating a tablets time field. The tablet code was modified to
accomodate the bulk import code running in the manager updating the
tablets time column in the metadata table.
---
.../accumulo/core/metadata/schema/Ample.java | 2 +
.../core/metadata/schema/MetadataSchema.java | 8 ++
.../core/metadata/schema/TabletMetadata.java | 11 ++
.../metadata/schema/TabletMetadataBuilder.java | 8 ++
.../core/metadata/schema/TabletMutatorBase.java| 7 ++
.../server/constraints/MetadataConstraints.java| 1 +
.../apache/accumulo/server/tablets/TabletTime.java | 6 +-
.../accumulo/server/tablets/LogicalTimeTest.java | 4 +-
.../accumulo/server/tablets/MillisTimeTest.java| 4 +-
.../manager/tableOps/bulkVer2/LoadFiles.java | 16 ++-
.../org/apache/accumulo/tserver/tablet/Tablet.java | 117 -
.../apache/accumulo/test/functional/BulkNewIT.java | 15 +++
12 files changed, 141 insertions(+), 58 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/metadata/schema/Ample.java
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/Ample.java
index 47bd5e4d1d..18d54d1f25 100644
--- a/core/src/main/java/org/apache/accumulo/core/metadata/schema/Ample.java
+++ b/core/src/main/java/org/apache/accumulo/core/metadata/schema/Ample.java
@@ -358,6 +358,8 @@ public interface Ample {
T putFlushId(long flushId);
+T putFlushNonce(long flushNonce);
+
T putLocation(Location location);
T deleteLocation(Location location);
diff --git
a/core/src/main/java/org/apache/accumulo/core/metadata/schema/MetadataSchema.java
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/MetadataSchema.java
index d024fbfd59..852eb257c8 100644
---
a/core/src/main/java/org/apache/accumulo/core/metadata/schema/MetadataSchema.java
+++
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/MetadataSchema.java
@@ -235,6 +235,14 @@ public class MetadataSchema {
public static final String FLUSH_QUAL = "flush";
public static final ColumnFQ FLUSH_COLUMN = new ColumnFQ(NAME, new
Text(FLUSH_QUAL));
+ /**
+ * Holds a nonce that is written when a new flush file is added. The
nonce is used to check if
+ * the write was successful in failure cases. The value is a random
64bit integer.
+ */
+ public static final String FLUSH_NONCE_QUAL = "flonce";
+ public static final ColumnFQ FLUSH_NONCE_COLUMN =
+ new ColumnFQ(NAME, new Text(FLUSH_NONCE_QUAL));
+
/**
* Holds lock IDs to enable a sanity check to ensure that the TServer
writing to the metadata
* tablet is not dead
diff --git
a/core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletMetadata.java
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletMetadata.java
index 5b426952a4..693ae04b62 100644
---
a/core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletMetadata.java
+++
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletMetadata.java
@@ -19,6 +19,7 @@
package org.apache.accumulo.core.metadata.schema;
import static
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.ServerColumnFamily.DIRECTORY_QUAL;
+import static
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.ServerColumnFamily.FLUSH_NONCE_QUAL;
import static
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.ServerColumnFamily.FLUSH_QUAL;
import static
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.ServerColumnFamily.OPID_QUAL;
import static
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.ServerColumnFamily.SELECTED_QUAL;
@@ -108,6 +109,7 @@ public class TabletMetadata {
private String cloned;
private SortedMap keyValues;
private OptionalLong flush = OptionalLong.empty();
+ private OptionalLong flushNonce = OptionalLong.empty();
private List logs;
private Map extCompactions;
private boolean merged;
@@ -136,6 +138,7 @@ public class TabletMetadata {
TIME,
CLONED,
FLUSH_ID,
+FLUSH_NONCE,
LOGS,
SUSPEND,
ECOMP,
@@ -345,6 +348,11 @@ public class TabletMetadata
(accumulo) branch 2.1 updated: FateIT Repo Error Test (#4282)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch 2.1
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/2.1 by this push:
new 00bac7ae56 FateIT Repo Error Test (#4282)
00bac7ae56 is described below
commit 00bac7ae5633147502ce57b28532942323f05030
Author: Kevin Rathbun <43969518+kevinrr...@users.noreply.github.com>
AuthorDate: Wed Feb 21 16:03:08 2024 -0500
FateIT Repo Error Test (#4282)
---
.../accumulo/test/fate/zookeeper/FateIT.java | 141 +++--
1 file changed, 133 insertions(+), 8 deletions(-)
diff --git
a/test/src/main/java/org/apache/accumulo/test/fate/zookeeper/FateIT.java
b/test/src/main/java/org/apache/accumulo/test/fate/zookeeper/FateIT.java
index 5e153b21b1..2dde8fabca 100644
--- a/test/src/main/java/org/apache/accumulo/test/fate/zookeeper/FateIT.java
+++ b/test/src/main/java/org/apache/accumulo/test/fate/zookeeper/FateIT.java
@@ -36,6 +36,8 @@ import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assertions.fail;
import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
import java.util.UUID;
import java.util.concurrent.CountDownLatch;
@@ -115,6 +117,59 @@ public class FateIT {
}
+ public static class TestOperationFails extends ManagerRepo {
+private static final long serialVersionUID = 1L;
+private static final Logger LOG =
LoggerFactory.getLogger(TestOperationFails.class);
+private static List undoOrder = new ArrayList<>();
+private static final int TOTAL_NUM_OPS = 3;
+private int opNum;
+private final String opName;
+private final ExceptionLocation location;
+
+public TestOperationFails(int opNum, ExceptionLocation location) {
+ this.opNum = opNum;
+ this.opName = "OP" + opNum;
+ this.location = location;
+}
+
+@Override
+public long isReady(long tid, Manager environment) throws Exception {
+ LOG.debug("{} {} Entered isReady()", opName, FateTxId.formatTid(tid));
+ if (location == ExceptionLocation.IS_READY) {
+if (opNum < TOTAL_NUM_OPS) {
+ return 0;
+} else {
+ throw new Exception(
+ opName + " " + FateTxId.formatTid(tid) + " isReady() failed -
this is expected");
+}
+ } else {
+return 0;
+ }
+}
+
+@Override
+public void undo(long tid, Manager environment) throws Exception {
+ LOG.debug("{} {} Entered undo()", opName, FateTxId.formatTid(tid));
+ undoOrder.add(opName);
+ undoLatch.countDown();
+}
+
+@Override
+public Repo call(long tid, Manager environment) throws Exception {
+ LOG.debug("{} {} Entered call()", opName, FateTxId.formatTid(tid));
+ if (location == ExceptionLocation.CALL) {
+if (opNum < TOTAL_NUM_OPS) {
+ return new TestOperationFails(++opNum, location);
+} else {
+ throw new Exception(
+ opName + " " + FateTxId.formatTid(tid) + " call() failed - this
is expected");
+}
+ } else {
+return new TestOperationFails(++opNum, location);
+ }
+}
+ }
+
private static final Logger LOG = LoggerFactory.getLogger(FateIT.class);
@TempDir
@@ -128,6 +183,11 @@ public class FateIT {
private static CountDownLatch callStarted;
private static CountDownLatch finishCall;
+ private static CountDownLatch undoLatch;
+
+ private enum ExceptionLocation {
+CALL, IS_READY
+ };
@BeforeAll
public static void setup() throws Exception {
@@ -165,10 +225,6 @@ public class FateIT {
ConfigurationCopy config = new ConfigurationCopy();
config.set(Property.GENERAL_THREADPOOL_SIZE, "2");
config.set(Property.MANAGER_FATE_THREADPOOL_SIZE, "1");
- fate.startTransactionRunners(config);
-
- // Wait for the transaction runner to be scheduled.
- UtilWaitThread.sleep(3000);
callStarted = new CountDownLatch(1);
finishCall = new CountDownLatch(1);
@@ -177,6 +233,11 @@ public class FateIT {
assertEquals(TStatus.NEW, getTxStatus(zk, txid));
fate.seedTransaction("TestOperation", txid, new TestOperation(NS, TID),
true, "Test Op");
assertEquals(TStatus.SUBMITTED, getTxStatus(zk, txid));
+
+ fate.startTransactionRunners(config);
+ // Wait for the transaction runner to be scheduled.
+ UtilWaitThread.sleep(3000);
+
// wait for call() to be called
callStarted.await();
assertEquals(IN_PROGRESS, getTxStatus(zk, txid));
@@ -346,10 +407,6 @@ public class FateIT {
ConfigurationCopy config = new ConfigurationCopy();
config.set(Property.GENERAL_THREADPOOL_SIZE, "2");
config.set(Property.MANAGER_FATE_THREADPOOL_SIZE, "1");
- fa
(accumulo) branch elasticity updated: adds support for splitting tablets with walogs (#3914)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 08cdb0e8b2 adds support for splitting tablets with walogs (#3914)
08cdb0e8b2 is described below
commit 08cdb0e8b228118161b553e5cc529139d8611138
Author: Keith Turner
AuthorDate: Wed Feb 21 15:21:07 2024 -0500
adds support for splitting tablets with walogs (#3914)
Updates the FATE operations that split tablets to handle walogs. These
changes should wait for a tablet with walogs to recover before
proceeding. The changes to actually make recovery happen were already
done in #3904 with changes to TabletGoalState.compute().
fixes #3844
---
.../accumulo/core/metadata/schema/Ample.java | 2 ++
.../metadata/ConditionalTabletMutatorImpl.java | 8 +++
.../manager/tableOps/merge/DeleteTablets.java | 2 +-
.../manager/tableOps/merge/MergeTablets.java | 4 ++--
.../manager/tableOps/split/DeleteOperationIds.java | 2 +-
.../accumulo/manager/tableOps/split/PreSplit.java | 25 ++
.../manager/tableOps/split/UpdateTablets.java | 7 --
7 files changed, 35 insertions(+), 15 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/metadata/schema/Ample.java
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/Ample.java
index ce8f32ceb9..47bd5e4d1d 100644
--- a/core/src/main/java/org/apache/accumulo/core/metadata/schema/Ample.java
+++ b/core/src/main/java/org/apache/accumulo/core/metadata/schema/Ample.java
@@ -529,6 +529,8 @@ public interface Ample {
ConditionalTabletMutator requireSame(TabletMetadata tabletMetadata,
ColumnType type,
ColumnType... otherTypes);
+ConditionalTabletMutator requireAbsentLogs();
+
/**
*
* Ample provides the following features on top of the conditional writer
to help automate
diff --git
a/server/base/src/main/java/org/apache/accumulo/server/metadata/ConditionalTabletMutatorImpl.java
b/server/base/src/main/java/org/apache/accumulo/server/metadata/ConditionalTabletMutatorImpl.java
index bd35c00713..34ecdeeebd 100644
---
a/server/base/src/main/java/org/apache/accumulo/server/metadata/ConditionalTabletMutatorImpl.java
+++
b/server/base/src/main/java/org/apache/accumulo/server/metadata/ConditionalTabletMutatorImpl.java
@@ -247,6 +247,14 @@ public class ConditionalTabletMutatorImpl extends
TabletMutatorBase {
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/DeleteOperationIds.java
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/DeleteOperationIds.java
index dbf0d04f42..9748f2d21f 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/DeleteOperationIds.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/DeleteOperationIds.java
@@ -54,7 +54,7 @@ public class DeleteOperationIds extends ManagerRepo {
splitInfo.getTablets().forEach(extent -> {
tabletsMutator.mutateTablet(extent).requireOperation(opid).requireAbsentLocation()
-.deleteOperation().submit(rejectionHandler);
+.requireAbsentLogs().deleteOperation().submit(rejectionHandler);
});
var results = tabletsMutator.process();
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/PreSplit.java
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/PreSplit.java
index a973015ddc..f94afbdb1e 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/PreSplit.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/tableOps/split/PreSplit.java
@@ -19,6 +19,7 @@
package org.apache.accumulo.manager.tableOps.split;
import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.LOCATION;
+import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.LOGS;
import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.OPID;
import static
org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.PREV_ROW;
@@ -77,7 +78,7 @@ public class PreSplit extends ManagerRepo {
// through as quickly as possible.
var tabletMetadata =
manager.getContext().getAmple().readTablet(splitInfo.getOriginal(),
-PREV_ROW, LOCATION, OPID);
+PREV_ROW, LOCATION, OPID, LOGS);
log.trace("Attempting tablet split {} {} {}", fateId,
splitInfo.getOriginal(),
tabletMetadata == null ? null : tabletMetadata.getLocation());
@@ -87,28 +88,30 @@ public class PreSplit extends ManagerRepo {
// tablet no longer exists or is reserved by another operation
return 0;
} else if (opid.equals(tabletMetadata.getOperationId())) {
- if (tabletMetadata.getLocat
(accumulo) branch elasticity updated: fixes table import issue in comprehensiveIT (#4288)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new c65c90f67b fixes table import issue in comprehensiveIT (#4288)
c65c90f67b is described below
commit c65c90f67bdbb94209a234f2457c01c1622f54e2
Author: Keith Turner
AuthorDate: Wed Feb 21 14:27:20 2024 -0500
fixes table import issue in comprehensiveIT (#4288)
---
.../java/org/apache/accumulo/test/ComprehensiveIT.java | 14 --
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/test/src/main/java/org/apache/accumulo/test/ComprehensiveIT.java
b/test/src/main/java/org/apache/accumulo/test/ComprehensiveIT.java
index 31a4280063..e3c9d6cfe6 100644
--- a/test/src/main/java/org/apache/accumulo/test/ComprehensiveIT.java
+++ b/test/src/main/java/org/apache/accumulo/test/ComprehensiveIT.java
@@ -751,14 +751,14 @@ public class ComprehensiveIT extends
SharedMiniClusterBase {
write(client, everythingTable, generateMutations(0, 100, tr -> true));
- verifyEverythingTable(client, everythingTable);
+ verifyEverythingTable(client, everythingTable,
TabletAvailability.HOSTED);
// test cloning a table as part of this test because the table has lots
of customizations.
client.tableOperations().clone(everythingTable, everythingClone,
CloneConfiguration.builder().setFlush(true).build());
// check the clone has all the same config and data as the original table
- verifyEverythingTable(client, everythingClone);
+ verifyEverythingTable(client, everythingClone,
TabletAvailability.HOSTED);
// test compaction with an iterator that filters out col fam 3
CompactionConfig compactionConfig = new CompactionConfig();
@@ -792,7 +792,7 @@ public class ComprehensiveIT extends SharedMiniClusterBase {
generateKeys(0, 100, tr -> (tr.row <= 35 || tr.row > 40) && tr.fam
!= 3));
// the changes to the clone should not have affected the source table so
verify it again
- verifyEverythingTable(client, everythingTable);
+ verifyEverythingTable(client, everythingTable,
TabletAvailability.HOSTED);
// test renaming a table
String tableIdBeforeRename =
client.tableOperations().tableIdMap().get(everythingClone);
@@ -813,7 +813,7 @@ public class ComprehensiveIT extends SharedMiniClusterBase {
// customizations.
exportImport(client, everythingTable, everythingImport);
- verifyEverythingTable(client, everythingImport);
+ verifyEverythingTable(client, everythingImport,
TabletAvailability.ONDEMAND);
}
}
@@ -900,7 +900,8 @@ public class ComprehensiveIT extends SharedMiniClusterBase {
return ntc;
}
- private static void verifyEverythingTable(AccumuloClient client, String
table) throws Exception {
+ private static void verifyEverythingTable(AccumuloClient client, String
table,
+ TabletAvailability expectedAvailabilityForDefaultTable) throws Exception
{
assertEquals(TimeType.LOGICAL,
client.tableOperations().getTimeType(table));
assertEquals(everythingSampleConfig,
client.tableOperations().getSamplerConfiguration(table));
assertTrue(client.tableOperations().tableIdMap().keySet().contains(table));
@@ -920,7 +921,8 @@ public class ComprehensiveIT extends SharedMiniClusterBase {
client.tableOperations().getTabletInformation(table, new Range())) {
tabletInfo.forEach(tabletInformation -> {
if (tabletInformation.getTabletId().getEndRow() == null) {
- assertEquals(TabletAvailability.HOSTED,
tabletInformation.getTabletAvailability());
+ assertEquals(expectedAvailabilityForDefaultTable,
+ tabletInformation.getTabletAvailability());
} else {
assertEquals(TabletAvailability.ONDEMAND,
tabletInformation.getTabletAvailability());
}
(accumulo) branch elasticity updated: replaces hosting requested cache with in progress set (#4285)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a commit to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git The following commit(s) were added to refs/heads/elasticity by this push: new d31a1df405 replaces hosting requested cache with in progress set (#4285) d31a1df405 is described below commit d31a1df4058a6a73bf24781af9a6c87e41fe7f57 Author: Keith Turner AuthorDate: Wed Feb 21 10:26:12 2024 -0500 replaces hosting requested cache with in progress set (#4285) A test in MergeIT was running slower than expected. Debugged and saw the following happen in the manager logs. ``` 2024-02-20T22:21:40,578 80 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:21:40,587 80 [manager.EventCoordinator] DEBUG: Tablet hosting requested for 3 tablets in 1 2024-02-20T22:21:42,718 93 [manager.EventCoordinator] DEBUG: tablet 1;row_000250< was unloaded from localhost:35711 2024-02-20T22:21:45,136 85 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:21:45,136 85 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:21:45,242 98 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:21:45,242 98 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:21:45,431 99 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:21:45,431 99 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:21:45,797 62 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:21:45,797 62 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:21:46,480 71 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:21:46,480 71 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:21:47,885 88 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:21:47,886 88 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:21:50,539 89 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:21:50,539 89 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:21:55,802 80 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:21:55,802 80 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:22:01,034 75 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:22:01,034 75 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:22:06,421 87 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:22:06,421 87 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:22:11,126 60 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:22:11,126 60 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:22:16,457 67 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:22:16,457 67 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:22:21,816 80 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:22:21,816 80 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:22:27,104 62 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:22:27,104 62 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:22:32,472 82 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:22:32,472 82 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:22:37,224 99 [manager.Manager] INFO : Tablet hosting requested for: 1;row_000250< 2024-02-20T22:22:37,224 99 [manager.Manager] TRACE: Ignoring hosting request because it was recently requested 1;row_000250< 2024-02-20T22:22:41,811 87 [m
(accumulo) branch elasticity updated: fixes problems in BalanceInPresenceOfOfflineTableIT (#4281)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 448afc8a9d fixes problems in BalanceInPresenceOfOfflineTableIT (#4281)
448afc8a9d is described below
commit 448afc8a9d5cc36b188baf554eb02142dd622e2e
Author: Keith Turner
AuthorDate: Tue Feb 20 10:47:42 2024 -0500
fixes problems in BalanceInPresenceOfOfflineTableIT (#4281)
BalanceInPresenceOfOfflineTableIT had the following problems
* Was testing balancing of tablets, but did nothing to explicitly
ensure tablets were hosted. Reads and writes were done that would
cause hosting, however async split would unhost tablets and leave
them unhosted.
* The test seemed to expected that splits would happen in one of the
test tables it created, but never verified this.
* The test would sleep for 10s and then check for balancing. If the
check did not find what it was looking for, it would double the time.
The doubling was unbounded, so saw the test sleep for 320s before
timing out.
The following was done to fix the problems above.
1. The two tables created by the test were set to always host tablets.
This was done because balancing is being tested and balancing is
something that is done on hosted tablets.
2. A check was added to see if a table had an expected number of
splits.
3. The sleep times were adjusted to start lower and max out at 30s.
---
.../BalanceInPresenceOfOfflineTableIT.java | 27 --
1 file changed, 20 insertions(+), 7 deletions(-)
diff --git
a/test/src/main/java/org/apache/accumulo/test/functional/BalanceInPresenceOfOfflineTableIT.java
b/test/src/main/java/org/apache/accumulo/test/functional/BalanceInPresenceOfOfflineTableIT.java
index be51f31d4f..46cb89ba7b 100644
---
a/test/src/main/java/org/apache/accumulo/test/functional/BalanceInPresenceOfOfflineTableIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/functional/BalanceInPresenceOfOfflineTableIT.java
@@ -22,6 +22,7 @@ import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assumptions.assumeTrue;
import java.util.Arrays;
+import java.util.List;
import java.util.Map;
import java.util.SortedSet;
import java.util.TreeSet;
@@ -33,6 +34,8 @@ import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.AccumuloSecurityException;
import org.apache.accumulo.core.client.TableExistsException;
import org.apache.accumulo.core.client.TableNotFoundException;
+import org.apache.accumulo.core.client.admin.NewTableConfiguration;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
import org.apache.accumulo.core.clientImpl.ClientContext;
import org.apache.accumulo.core.clientImpl.Credentials;
import org.apache.accumulo.core.conf.Property;
@@ -103,15 +106,17 @@ public class BalanceInPresenceOfOfflineTableIT extends
AccumuloClusterHarness {
UNUSED_TABLE = names[0];
TEST_TABLE = names[1];
-// load into a table we won't use
-
-accumuloClient.tableOperations().create(UNUSED_TABLE);
+// load into a table we won't use. Balancing happens to hosted tablets.
Since this test is
+// testing balancing, create the table with a goal of HOSTED.
+accumuloClient.tableOperations().create(UNUSED_TABLE,
+new
NewTableConfiguration().withInitialTabletAvailability(TabletAvailability.HOSTED));
accumuloClient.tableOperations().addSplits(UNUSED_TABLE, splits);
// mark the table offline before it can rebalance.
accumuloClient.tableOperations().offline(UNUSED_TABLE);
// actual test table
-accumuloClient.tableOperations().create(TEST_TABLE);
+accumuloClient.tableOperations().create(TEST_TABLE,
+new
NewTableConfiguration().withInitialTabletAvailability(TabletAvailability.HOSTED));
accumuloClient.tableOperations().setProperty(TEST_TABLE,
Property.TABLE_SPLIT_THRESHOLD.getKey(), "10K");
}
@@ -134,13 +139,13 @@ public class BalanceInPresenceOfOfflineTableIT extends
AccumuloClusterHarness {
log.debug("waiting for balancing, up to ~5 minutes to allow for migration
cleanup.");
final long startTime = System.currentTimeMillis();
-long currentWait = 10_000;
+long currentWait = 1_000;
boolean balancingWorked = false;
Credentials creds = new Credentials(getAdminPrincipal(), getAdminToken());
while (!balancingWorked && (System.currentTimeMillis() - startTime) < ((5
* 60 + 15) * 1000)) {
Thread.sleep(currentWait);
- currentWait *= 2;
+ currentWait = Math.min(currentWait * 2, 30_000);
log.debug("fetch the list of tablets assigned to each tserver.&qu
(accumulo) branch elasticity updated: avoids a metadata lookup per mutation for batch writer (#4280)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new af9b0602db avoids a metadata lookup per mutation for batch writer
(#4280)
af9b0602db is described below
commit af9b0602dbb3debb34828d7d3116ea310e175163
Author: Keith Turner
AuthorDate: Fri Feb 16 19:01:03 2024 -0500
avoids a metadata lookup per mutation for batch writer (#4280)
When writing to unhosted ondemand tablets the batch writer was performing
a metadata lookup per mutation before it requested hosting. This was of
course extremely slow. This commit makes a quick change to perform a
metadata lookup per extent instead of per mutation. This could be
further improved because its still not as good as the pre elasticity
code. There is in ELASTICITY_TODO in the commit about further
improvements.
This is a massive improvement. Was running FateStarvationIT which
writes 100K rows. Without this change it took 53 secs. With this
change it took 1.7 secs.
This mostly solves #3708, there may be some further minor improvements
that could be made.
---
.../accumulo/core/clientImpl/ClientTabletCacheImpl.java | 15 +--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java
b/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java
index 4539a60b3a..10fb3aa21e 100644
---
a/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java
+++
b/core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java
@@ -234,12 +234,23 @@ public class ClientTabletCacheImpl extends
ClientTabletCache {
wLock.lock();
try {
+CachedTablet lastTablet = null;
for (T mutation : notInCache) {
row.set(mutation.getRow());
- CachedTablet tl =
- _findTablet(context, row, false, false, false, lcSession,
LocationNeed.REQUIRED);
+ // ELASTICITY_TODO using lastTablet avoids doing a metadata table
lookup per mutation.
+ // However this still does at least one metadata lookup per tablet.
This is not as good as
+ // the pre-elasticity code that would lookup N tablets at once and
use them to bin
+ // mutations. So there is further room for improvement in the way
this code interacts with
+ // cache and metadata table.
+ CachedTablet tl;
+ if (lastTablet != null && lastTablet.getExtent().contains(row)) {
+tl = lastTablet;
+ } else {
+tl = _findTablet(context, row, false, false, false, lcSession,
LocationNeed.REQUIRED);
+lastTablet = tl;
+ }
if (!addMutation(binnedMutations, mutation, tl, lcSession)) {
failures.add(mutation);
(accumulo) branch elasticity updated: fixes FateStarvationIT (#4278)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 3de72f63c4 fixes FateStarvationIT (#4278)
3de72f63c4 is described below
commit 3de72f63c434b977af43b7536d11e35123aa9347
Author: Keith Turner
AuthorDate: Fri Feb 16 16:45:19 2024 -0500
fixes FateStarvationIT (#4278)
This test had two problems. First it was using the offline operation to
check if all compactions were finished. However compactations that were
queued in fate when the offline started would start after the offline
and potentially have fate locks for the check at the end of the test.
Second the test starts a large number of compactions, but there was only
one compactor process. Started multiple compactor process to process
compactions faster.
---
.../accumulo/test/functional/FateStarvationIT.java | 43 --
1 file changed, 40 insertions(+), 3 deletions(-)
diff --git
a/test/src/main/java/org/apache/accumulo/test/functional/FateStarvationIT.java
b/test/src/main/java/org/apache/accumulo/test/functional/FateStarvationIT.java
index 92a2d8251c..6b6da95020 100644
---
a/test/src/main/java/org/apache/accumulo/test/functional/FateStarvationIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/functional/FateStarvationIT.java
@@ -23,21 +23,43 @@ import static
org.apache.accumulo.core.util.LazySingletons.RANDOM;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
import org.apache.accumulo.core.client.Accumulo;
import org.apache.accumulo.core.client.AccumuloClient;
import org.apache.accumulo.core.client.admin.NewTableConfiguration;
+import org.apache.accumulo.core.conf.Property;
import org.apache.accumulo.harness.AccumuloClusterHarness;
+import org.apache.accumulo.miniclusterImpl.MiniAccumuloConfigImpl;
import org.apache.accumulo.test.TestIngest;
import org.apache.accumulo.test.TestIngest.IngestParams;
+import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.Text;
import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
public class FateStarvationIT extends AccumuloClusterHarness {
+ private static final Logger log =
LoggerFactory.getLogger(FateStarvationIT.class);
+
@Override
protected Duration defaultTimeout() {
-return Duration.ofMinutes(2);
+return Duration.ofMinutes(4);
+ }
+
+ @Override
+ public void configureMiniCluster(MiniAccumuloConfigImpl cfg, Configuration
hadoopCoreSite) {
+var groupName = "user_small";
+// Add this check in case the config changes
+Preconditions.checkState(
+
Property.COMPACTION_SERVICE_DEFAULT_GROUPS.getDefaultValue().contains(groupName));
+// This test creates around ~1300 compaction task, so start more
compactors. There is randomness
+// so the exact number of task varies.
+cfg.getClusterServerConfiguration().addCompactorResourceGroup(groupName,
4);
}
@Test
@@ -53,19 +75,34 @@ public class FateStarvationIT extends
AccumuloClusterHarness {
params.dataSize = 50;
params.cols = 1;
TestIngest.ingest(c, params);
+ log.debug("Ingest complete");
c.tableOperations().flush(tableName, null, null, true);
+ log.debug("Flush complete");
List splits = new ArrayList<>(TestIngest.getSplitPoints(0, 10,
67));
+ List> futures = new ArrayList<>();
+ var executor = Executors.newCachedThreadPool();
+
for (int i = 0; i < 100; i++) {
int idx1 = RANDOM.get().nextInt(splits.size() - 1);
int idx2 = RANDOM.get().nextInt(splits.size() - (idx1 + 1)) + idx1 + 1;
-c.tableOperations().compact(tableName, splits.get(idx1),
splits.get(idx2), false, false);
+var future = executor.submit(() -> {
+ c.tableOperations().compact(tableName, splits.get(idx1),
splits.get(idx2), false, true);
+ return null;
+});
+
+futures.add(future);
}
- c.tableOperations().offline(tableName);
+ log.debug("Started compactions");
+
+ // wait for all compactions to complete
+ for (var future : futures) {
+future.get();
+ }
FunctionalTestUtils.assertNoDanglingFateLocks(getCluster());
}
(accumulo) branch elasticity updated: Globally Unique FATE Transaction Ids - Part 4 (#4258)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new a3ec20e41a Globally Unique FATE Transaction Ids - Part 4 (#4258)
a3ec20e41a is described below
commit a3ec20e41a8058106d829f73346f65f1734d1860
Author: Kevin Rathbun <43969518+kevinrr...@users.noreply.github.com>
AuthorDate: Fri Feb 16 14:24:58 2024 -0500
Globally Unique FATE Transaction Ids - Part 4 (#4258)
This addresses several previously deferred changes for issue #4044.
Changes:
- ZooReservation now uses FateId (used in Utils)
- TabletOperationId now uses FateId
- TExternalCompactionJob now uses FateId
- VolumeManager and VolumeManagerImpl now use FateId
- Utils.getLock() lockData now uses the full FateId
- TabletRefresher now uses FateId
- Classes which used the above classes updated
- Several test changes to reflect new changes
- Deferred a couple of changes (in Compactor and CompactionCoordinator)
(need pull/4247 merged first)
---
.../java/org/apache/accumulo/core/fate/FateId.java | 23 +++-
.../core/fate/zookeeper/ZooReservation.java| 24 ++--
.../core/metadata/schema/TabletOperationId.java| 12 +-
.../thrift/TExternalCompactionJob.java | 126 -
core/src/main/thrift/tabletserver.thrift | 2 +-
.../core/metadata/schema/TabletMetadataTest.java | 3 +-
.../server/constraints/MetadataConstraints.java| 2 +-
.../apache/accumulo/server/fs/VolumeManager.java | 5 +-
.../accumulo/server/fs/VolumeManagerImpl.java | 11 +-
.../constraints/MetadataConstraintsTest.java | 2 +-
.../org/apache/accumulo/compactor/Compactor.java | 13 +--
.../coordinator/CompactionCoordinator.java | 3 +-
.../apache/accumulo/manager/tableOps/Utils.java| 8 +-
.../manager/tableOps/bulkVer2/BulkImportMove.java | 3 +-
.../manager/tableOps/bulkVer2/RefreshTablets.java | 3 +-
.../manager/tableOps/bulkVer2/TabletRefresher.java | 10 +-
.../manager/tableOps/compact/CompactionDriver.java | 5 +-
.../manager/tableOps/compact/RefreshTablets.java | 5 +-
.../manager/tableOps/delete/ReserveTablets.java| 3 +-
.../manager/tableOps/merge/DeleteRows.java | 3 +-
.../manager/tableOps/merge/DeleteTablets.java | 3 +-
.../manager/tableOps/merge/FinishTableRangeOp.java | 3 +-
.../manager/tableOps/merge/MergeTablets.java | 3 +-
.../manager/tableOps/merge/ReserveTablets.java | 3 +-
.../manager/tableOps/split/DeleteOperationIds.java | 3 +-
.../accumulo/manager/tableOps/split/PreSplit.java | 6 +-
.../manager/tableOps/split/UpdateTablets.java | 3 +-
.../tableOps/tableImport/MoveExportedFiles.java| 3 +-
.../compaction/CompactionCoordinatorTest.java | 10 +-
.../org/apache/accumulo/test/ScanServerIT.java | 6 +-
.../test/functional/AmpleConditionalWriterIT.java | 18 ++-
.../test/functional/ManagerAssignmentIT.java | 10 +-
.../functional/TabletManagementIteratorIT.java | 6 +-
33 files changed, 191 insertions(+), 152 deletions(-)
diff --git a/core/src/main/java/org/apache/accumulo/core/fate/FateId.java
b/core/src/main/java/org/apache/accumulo/core/fate/FateId.java
index 5be742d2fd..8907c6879c 100644
--- a/core/src/main/java/org/apache/accumulo/core/fate/FateId.java
+++ b/core/src/main/java/org/apache/accumulo/core/fate/FateId.java
@@ -24,6 +24,7 @@ import java.util.stream.Stream;
import org.apache.accumulo.core.data.AbstractId;
import org.apache.accumulo.core.manager.thrift.TFateId;
+import org.apache.accumulo.core.manager.thrift.TFateInstanceType;
import org.apache.accumulo.core.util.FastFormat;
/**
@@ -107,7 +108,7 @@ public class FateId extends AbstractId {
* @param fateIdStr the string representation of the FateId
* @return true if the string is a valid FateId, false otherwise
*/
- public static boolean isFormattedTid(String fateIdStr) {
+ public static boolean isFateId(String fateIdStr) {
return FATEID_PATTERN.matcher(fateIdStr).matches();
}
@@ -133,6 +134,26 @@ public class FateId extends AbstractId {
return new FateId(PREFIX + type + ":" + formatTid(tid));
}
+ /**
+ *
+ * @return the TFateId equivalent of the FateId
+ */
+ public TFateId toThrift() {
+TFateInstanceType thriftType;
+FateInstanceType type = getType();
+switch (type) {
+ case USER:
+thriftType = TFateInstanceType.USER;
+break;
+ case META:
+thriftType = TFateInstanceType.META;
+break;
+ default:
+throw new IllegalArgumentException("Invalid FateInstanceType: " +
type);
+}
+return new TFateId(thriftType, getTid());
+ }
+
/**
* Returns the hex string equivalent of the tid
*/
diff --git
a/core/src/mai
(accumulo-access) branch main updated: moves thread local access outside of loop (#61)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-access.git
The following commit(s) were added to refs/heads/main by this push:
new d0b4423 moves thread local access outside of loop (#61)
d0b4423 is described below
commit d0b4423087faf78dd0e96e3042bbc12fa87d6f42
Author: Keith Turner
AuthorDate: Thu Feb 15 11:06:14 2024 -0500
moves thread local access outside of loop (#61)
A thread local was being accessed multiple times in a loop. The result
should be the same each time, so the access was moved out of the loop to
avoid any cost related to accessing a thread local.
---
src/main/java/org/apache/accumulo/access/AccessEvaluatorImpl.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/main/java/org/apache/accumulo/access/AccessEvaluatorImpl.java
b/src/main/java/org/apache/accumulo/access/AccessEvaluatorImpl.java
index 66adb96..a73f627 100644
--- a/src/main/java/org/apache/accumulo/access/AccessEvaluatorImpl.java
+++ b/src/main/java/org/apache/accumulo/access/AccessEvaluatorImpl.java
@@ -196,9 +196,9 @@ final class AccessEvaluatorImpl implements AccessEvaluator {
boolean evaluate(byte[] accessExpression) throws
IllegalAccessExpressionException {
var bytesWrapper = lookupWrappers.get();
+var tokenizer = tokenizers.get();
for (var auths : authorizedPredicates) {
- var tokenizer = tokenizers.get();
tokenizer.reset(accessExpression);
Predicate atp = authToken -> {
bytesWrapper.set(authToken.data, authToken.start, authToken.len);
(accumulo) branch elasticity updated: Adds unit test for SetEqualityIterator (#4227)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new b5afd3cc1c Adds unit test for SetEqualityIterator (#4227)
b5afd3cc1c is described below
commit b5afd3cc1cfcf05c05be79ec92be08ffae1416d7
Author: rsingh433 <74160026+rsingh...@users.noreply.github.com>
AuthorDate: Thu Feb 15 10:56:25 2024 -0500
Adds unit test for SetEqualityIterator (#4227)
---
.../accumulo/server/SetEqualityIteratorTest.java | 179 +
1 file changed, 179 insertions(+)
diff --git
a/server/base/src/test/java/org/apache/accumulo/server/SetEqualityIteratorTest.java
b/server/base/src/test/java/org/apache/accumulo/server/SetEqualityIteratorTest.java
new file mode 100644
index 00..3d6c11f9b8
--- /dev/null
+++
b/server/base/src/test/java/org/apache/accumulo/server/SetEqualityIteratorTest.java
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.server;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Set;
+import java.util.SortedMap;
+import java.util.TreeMap;
+
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.dataImpl.KeyExtent;
+import org.apache.accumulo.core.iteratorsImpl.system.SortedMapIterator;
+import org.apache.accumulo.core.metadata.ReferencedTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.DataFileValue;
+import org.apache.accumulo.core.metadata.schema.MetadataSchema;
+import org.apache.accumulo.core.metadata.schema.TabletMetadata;
+import org.apache.accumulo.server.metadata.iterators.SetEqualityIterator;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+public class SetEqualityIteratorTest {
+
+ private SetEqualityIterator setEqualityIterator;
+ private SetEqualityIterator setEqualityIteratorNoFiles;
+ private SetEqualityIterator setEqualityIteratorOneFile;
+ private SortedMapIterator sortedMapIterator;
+ private SortedMapIterator sortedMapIteratorNoFiles;
+ private SortedMapIterator sortedMapIteratorOneFile;
+
+ private KeyExtent extent = new KeyExtent(TableId.of("5"), new Text("df"),
new Text("da"));
+
+ private StoredTabletFile file1 =
+ new ReferencedTabletFile(new
Path("dfs://nn1/acc/tables/1/t-0001/sf1.rf")).insert();
+ private StoredTabletFile file2 =
+ new ReferencedTabletFile(new
Path("dfs://nn1/acc/tables/1/t-0001/sf2.rf")).insert();
+ private StoredTabletFile file3 =
+ new ReferencedTabletFile(new
Path("dfs://nn1/acc/tables/1/t-0001/sf3.rf")).insert();
+
+ @BeforeEach
+ public void setUp() throws IOException {
+
+// Create tablet metadata with no files
+TabletMetadata tmNoFiles =
TabletMetadata.builder(extent).putFlushId(7).build();
+
+// Create tablet metadata with one file
+StoredTabletFile singleFile =
+new ReferencedTabletFile(new
Path("dfs://nn1/acc/tables/1/t-0001/sf1.rf")).insert();
+TabletMetadata tmOneFile = TabletMetadata.builder(extent)
+.putFile(singleFile, new DataFileValue(100, 50)).putFlushId(8).build();
+
+// Create tablet metadata with multiple files
+TabletMetadata tmMultipleFiles = TabletMetadata.builder(extent)
+.putFile(file1, new DataFileValue(0, 0)).putFile(file2, new
DataFileValue(555, 23))
+.putFile(file3, new DataFileValue(234, 13)).putFlushId(6).build();
+
+var extent2 = new KeyExtent(extent.tableId(), null, extent.endRow());
+// create another tablet metadata using extent2 w/ diff files and add it
t
(accumulo) branch elasticity updated: Bug fix from PR #4247 (#4265)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 10d0824fac Bug fix from PR #4247 (#4265)
10d0824fac is described below
commit 10d0824fac42faefba3305d4dcf0dc676dffb54d
Author: Kevin Rathbun <43969518+kevinrr...@users.noreply.github.com>
AuthorDate: Thu Feb 15 09:55:28 2024 -0500
Bug fix from PR #4247 (#4265)
---
.../org/apache/accumulo/test/functional/AmpleConditionalWriterIT.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/test/src/main/java/org/apache/accumulo/test/functional/AmpleConditionalWriterIT.java
b/test/src/main/java/org/apache/accumulo/test/functional/AmpleConditionalWriterIT.java
index 892a507530..a193357597 100644
---
a/test/src/main/java/org/apache/accumulo/test/functional/AmpleConditionalWriterIT.java
+++
b/test/src/main/java/org/apache/accumulo/test/functional/AmpleConditionalWriterIT.java
@@ -631,7 +631,7 @@ public class AmpleConditionalWriterIT extends
AccumuloClusterHarness {
*
*
* {
- * "txid": "FATE:META:123456",
+ * "fateId": "FATE:META:123456",
* "selAll": true,
* "files": ["/path/to/file1.rf", "/path/to/file2.rf"]
* }
@@ -640,7 +640,7 @@ public class AmpleConditionalWriterIT extends
AccumuloClusterHarness {
public static String createSelectedFilesJson(FateId fateId, boolean selAll,
Collection paths) {
String filesJsonArray = GSON.get().toJson(paths);
-return ("{'txid':'" + fateId + "','selAll':" + selAll + ",'files':" +
filesJsonArray + "}")
+return ("{'fateId':'" + fateId + "','selAll':" + selAll + ",'files':" +
filesJsonArray + "}")
.replace('\'', '\"');
}
(accumulo) branch elasticity updated: fixes NPE with system compactions (#4263)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 8a30262e28 fixes NPE with system compactions (#4263)
8a30262e28 is described below
commit 8a30262e2878bd0d4c3374c06052fc670abba191
Author: Keith Turner
AuthorDate: Wed Feb 14 18:37:30 2024 -0500
fixes NPE with system compactions (#4263)
---
.../accumulo/core/metadata/schema/CompactionMetadata.java| 12 +---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/metadata/schema/CompactionMetadata.java
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/CompactionMetadata.java
index e3cd1f3347..3d0536d593 100644
---
a/core/src/main/java/org/apache/accumulo/core/metadata/schema/CompactionMetadata.java
+++
b/core/src/main/java/org/apache/accumulo/core/metadata/schema/CompactionMetadata.java
@@ -54,7 +54,12 @@ public class CompactionMetadata {
this.priority = priority;
this.cgid = Objects.requireNonNull(ceid);
this.propagateDeletes = propagateDeletes;
-this.fateId = fateId;
+if (kind == CompactionKind.SYSTEM) {
+ // its ok if this is null for system compactions because its not used.
+ this.fateId = fateId;
+} else {
+ this.fateId = Objects.requireNonNull(fateId);
+}
}
public Set getJobFiles() {
@@ -112,7 +117,7 @@ public class CompactionMetadata {
jData.groupId = cgid.toString();
jData.priority = priority;
jData.propDels = propagateDeletes;
-jData.fateId = fateId.canonical();
+jData.fateId = fateId == null ? null : fateId.canonical();
return GSON.get().toJson(jData);
}
@@ -122,7 +127,8 @@ public class CompactionMetadata {
return new
CompactionMetadata(jData.inputs.stream().map(StoredTabletFile::new).collect(toSet()),
StoredTabletFile.of(jData.tmp).getTabletFile(), jData.compactor,
CompactionKind.valueOf(jData.kind), jData.priority,
-CompactorGroupIdImpl.groupId(jData.groupId), jData.propDels,
FateId.from(jData.fateId));
+CompactorGroupIdImpl.groupId(jData.groupId), jData.propDels,
+jData.fateId == null ? null : FateId.from(jData.fateId));
}
@Override
(accumulo) branch elasticity updated: always sets needs recovery flag in tablet mgmt iterator (#4255)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 4e1b052904 always sets needs recovery flag in tablet mgmt iterator
(#4255)
4e1b052904 is described below
commit 4e1b0529043f8359cae8929e427c4c156ebefaf6
Author: Keith Turner
AuthorDate: Wed Feb 14 17:13:18 2024 -0500
always sets needs recovery flag in tablet mgmt iterator (#4255)
This commit fixes #4251. The WalSunnyDayIT was failing because the root
tablet needed recovery, but the tablet mgmt iterator was not indicating
this because the manager was in safe mode. Therefore recovery never
happened and the test timed out.
---
.../server/manager/state/TabletManagementIterator.java | 18 +++---
1 file changed, 11 insertions(+), 7 deletions(-)
diff --git
a/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletManagementIterator.java
b/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletManagementIterator.java
index 4e7bca7287..69fbbdd1b0 100644
---
a/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletManagementIterator.java
+++
b/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletManagementIterator.java
@@ -201,14 +201,18 @@ public class TabletManagementIterator extends
SkippingIterator {
actions.clear();
Exception error = null;
try {
-if (tabletMgmtParams.getManagerState() != ManagerState.NORMAL
-|| tabletMgmtParams.getOnlineTsevers().isEmpty()
-|| tabletMgmtParams.getOnlineTables().isEmpty()) {
- // when manager is in the process of starting up or shutting down
return everything.
- actions.add(ManagementAction.NEEDS_LOCATION_UPDATE);
-} else {
- LOG.trace("Evaluating extent: {}", tm);
+LOG.trace("Evaluating extent: {}", tm);
+if (tm.getExtent().isMeta()) {
computeTabletManagementActions(tm, actions);
+} else {
+ if (tabletMgmtParams.getManagerState() != ManagerState.NORMAL
+ || tabletMgmtParams.getOnlineTsevers().isEmpty()
+ || tabletMgmtParams.getOnlineTables().isEmpty()) {
+// when manager is in the process of starting up or shutting down
return everything.
+actions.add(ManagementAction.NEEDS_LOCATION_UPDATE);
+ } else {
+computeTabletManagementActions(tm, actions);
+ }
}
} catch (Exception e) {
LOG.error("Error computing tablet management actions for extent: {}",
tm.getExtent(), e);
(accumulo) branch elasticity updated: fixes SplitRecoveryIT to work with newer columns (#4260)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 92fbdaa228 fixes SplitRecoveryIT to work with newer columns (#4260)
92fbdaa228 is described below
commit 92fbdaa228dec64af4d970e494e51b732113c160
Author: Keith Turner
AuthorDate: Wed Feb 14 17:12:28 2024 -0500
fixes SplitRecoveryIT to work with newer columns (#4260)
SplitRecoveryIT is a tricky test to maintain because it is testing
upgrade code that deals with older metadata columns that Accumulo
no longer recognizes. This commit adjust the test to work with
recent changes to the metadata schema in the elasticity branch.
---
.../manager/upgrade/SplitRecovery12to13.java | 52 ---
.../accumulo/test/functional/SplitRecoveryIT.java | 73 +++---
2 files changed, 63 insertions(+), 62 deletions(-)
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/SplitRecovery12to13.java
b/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/SplitRecovery12to13.java
index 03e4180f8c..b339462503 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/SplitRecovery12to13.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/upgrade/SplitRecovery12to13.java
@@ -22,7 +22,6 @@ import static
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSec
import static
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.Upgrade12to13.SPLIT_RATIO_COLUMN;
import java.util.ArrayList;
-import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -33,8 +32,6 @@ import java.util.TreeMap;
import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.Scanner;
-import org.apache.accumulo.core.client.admin.TabletAvailability;
-import org.apache.accumulo.core.client.admin.TimeType;
import org.apache.accumulo.core.clientImpl.ScannerImpl;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Mutation;
@@ -43,19 +40,12 @@ import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.TableId;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.dataImpl.KeyExtent;
-import org.apache.accumulo.core.fate.FateId;
-import org.apache.accumulo.core.lock.ServiceLock;
-import org.apache.accumulo.core.metadata.ReferencedTabletFile;
import org.apache.accumulo.core.metadata.StoredTabletFile;
-import org.apache.accumulo.core.metadata.TServerInstance;
import org.apache.accumulo.core.metadata.schema.Ample;
-import org.apache.accumulo.core.metadata.schema.Ample.TabletMutator;
import org.apache.accumulo.core.metadata.schema.DataFileValue;
import org.apache.accumulo.core.metadata.schema.ExternalCompactionId;
import org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection;
import
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily;
-import org.apache.accumulo.core.metadata.schema.MetadataTime;
-import org.apache.accumulo.core.metadata.schema.TabletMetadata.Location;
import org.apache.accumulo.core.security.Authorizations;
import org.apache.accumulo.server.ServerContext;
import org.apache.accumulo.server.util.FileUtil;
@@ -68,37 +58,6 @@ public class SplitRecovery12to13 {
private static final Logger log =
LoggerFactory.getLogger(SplitRecovery12to13.class);
- public static void addNewTablet(ServerContext context, KeyExtent extent,
String dirName,
- TServerInstance tServerInstance, Map
datafileSizes,
- Map> bulkLoadedFiles,
MetadataTime time,
- long lastFlushID) {
-
-TabletMutator tablet = context.getAmple().mutateTablet(extent);
-tablet.putPrevEndRow(extent.prevEndRow());
-tablet.putDirName(dirName);
-tablet.putTime(time);
-
-if (lastFlushID > 0) {
- tablet.putFlushId(lastFlushID);
-}
-
-if (tServerInstance != null) {
- tablet.putLocation(Location.current(tServerInstance));
- tablet.deleteLocation(Location.future(tServerInstance));
-}
-
-datafileSizes.forEach((key, value) -> tablet.putFile(key, value));
-
-for (Entry> entry :
bulkLoadedFiles
-.entrySet()) {
- for (ReferencedTabletFile ref : entry.getValue()) {
-tablet.putBulkFile(ref, entry.getKey());
- }
-}
-
-tablet.mutate();
- }
-
public static KeyExtent fixSplit(ServerContext context, Text metadataEntry)
throws AccumuloException {
var tableId = KeyExtent.fromMetaRow(metadataEntry).tableId();
@@ -286,15 +245,4 @@ public class SplitRecovery12to13 {
finishSplit(extent.toMetaRow(), datafileSizes, highDatafilesToRemove,
context);
}
- public static void addTablet(KeyExtent extent, String path, ServerContext
context,
-
(accumulo) branch elasticity updated (5d12597b4d -> f8ab378ef9)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from 5d12597b4d Add LogSorter to Compactor and ScanServer (#4239) add f8ab378ef9 Globally Unique FATE Transaction Ids - Part 3 (#4247) No new revisions were added by this update. Summary of changes: .../java/org/apache/accumulo/core/fate/FateId.java | 29 +++ .../accumulo/core/metadata/schema/Ample.java | 11 +-- .../core/metadata/schema/CompactionMetadata.java | 17 ++-- .../core/metadata/schema/MetadataSchema.java | 15 ++-- .../core/metadata/schema/SelectedFiles.java| 26 +++--- .../core/metadata/schema/TabletMetadata.java | 16 ++-- .../metadata/schema/TabletMetadataBuilder.java | 11 +-- .../core/metadata/schema/TabletMutatorBase.java| 14 +-- .../core/metadata/schema/SelectedFilesTest.java| 43 +- .../core/metadata/schema/TabletMetadataTest.java | 36 +--- .../server/compaction/CompactionConfigStorage.java | 40 + .../server/compaction/CompactionJobGenerator.java | 7 +- .../server/constraints/MetadataConstraints.java| 4 +- .../manager/state/TabletManagementParameters.java | 22 +++-- .../metadata/ConditionalTabletMutatorImpl.java | 3 +- .../accumulo/server/metadata/ServerAmpleImpl.java | 6 +- .../accumulo/server/util/MetadataTableUtil.java| 9 +- .../constraints/MetadataConstraintsTest.java | 51 ++- .../state/TabletManagementParametersTest.java | 3 +- .../server/manager/state/TabletManagementTest.java | 11 ++- .../org/apache/accumulo/compactor/Compactor.java | 10 ++- .../java/org/apache/accumulo/manager/Manager.java | 5 +- .../coordinator/CompactionCoordinator.java | 23 ++--- .../coordinator/commit/CommitCompaction.java | 20 ++--- .../manager/tableOps/bulkVer2/BulkImportMove.java | 3 +- .../tableOps/bulkVer2/CleanUpBulkImport.java | 9 +- .../manager/tableOps/bulkVer2/LoadFiles.java | 3 +- .../manager/tableOps/bulkVer2/RefreshTablets.java | 2 +- .../accumulo/manager/tableOps/compact/CleanUp.java | 10 +-- .../manager/tableOps/compact/CompactRange.java | 6 +- .../manager/tableOps/compact/CompactionDriver.java | 45 -- .../tableOps/compact/cancel/CancelCompactions.java | 4 +- .../manager/tableOps/split/UpdateTablets.java | 9 +- .../manager/upgrade/SplitRecovery12to13.java | 5 +- .../compaction/CompactionCoordinatorTest.java | 6 +- .../test/fate/accumulo/FateMutatorImplIT.java | 4 +- .../test/functional/AmpleConditionalWriterIT.java | 99 +- .../apache/accumulo/test/functional/MergeIT.java | 6 +- .../accumulo/test/functional/SplitRecoveryIT.java | 20 +++-- 39 files changed, 373 insertions(+), 290 deletions(-)
(accumulo) branch elasticity updated (e375190772 -> 101ae815f1)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from e375190772 fixes volume replacement failures (#4257) add d91178f44c Replaced System.currentTimeMillis() with System.nanoTime() in DfsLogger (#4250) add f28b55dc25 Fixes CompactionIT.testMultiStepCompactionThatDeletesAll() (#4259) add 978f8b9eca Merge branch '2.1' new 101ae815f1 Merge branch 'main' into elasticity The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/accumulo/tserver/log/DfsLogger.java | 10 ++ .../accumulo/test/functional/CompactionIT.java | 20 +--- 2 files changed, 19 insertions(+), 11 deletions(-)
(accumulo) 01/01: Merge branch 'main' into elasticity
This is an automated email from the ASF dual-hosted git repository. kturner pushed a commit to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git commit 101ae815f129efaf26cfc95eff0266861b561b49 Merge: e375190772 978f8b9eca Author: Keith Turner AuthorDate: Tue Feb 13 18:03:52 2024 + Merge branch 'main' into elasticity .../org/apache/accumulo/tserver/log/DfsLogger.java | 10 ++ .../accumulo/test/functional/CompactionIT.java | 20 +--- 2 files changed, 19 insertions(+), 11 deletions(-) diff --cc test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java index 896929654a,18fdff715d..a310b1e08c --- a/test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java +++ b/test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java @@@ -72,10 -79,12 +72,11 @@@ import org.apache.accumulo.core.iterato import org.apache.accumulo.core.iterators.IteratorEnvironment; import org.apache.accumulo.core.iterators.IteratorUtil.IteratorScope; import org.apache.accumulo.core.iterators.SortedKeyValueIterator; -import org.apache.accumulo.core.iterators.user.AgeOffFilter; import org.apache.accumulo.core.iterators.user.GrepIterator; import org.apache.accumulo.core.metadata.AccumuloTable; +import org.apache.accumulo.core.metadata.RootTable; import org.apache.accumulo.core.metadata.StoredTabletFile; -import org.apache.accumulo.core.metadata.schema.Ample; + import org.apache.accumulo.core.metadata.schema.MetadataSchema; import org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily; import org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.TabletColumnFamily; import org.apache.accumulo.core.metadata.schema.TabletMetadata; @@@ -350,10 -703,8 +351,10 @@@ public class CompactionIT extends Accum assertEquals(0, scanner.stream().count()); } - var finalCount = countFiles(c); + var finalCount = countFiles(c, tableName); assertTrue(finalCount <= beforeCount); + + ExternalCompactionTestUtils.assertNoCompactionMetadata(getServerContext(), tableName); } }
(accumulo) branch main updated (d91178f44c -> 978f8b9eca)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git from d91178f44c Replaced System.currentTimeMillis() with System.nanoTime() in DfsLogger (#4250) add f28b55dc25 Fixes CompactionIT.testMultiStepCompactionThatDeletesAll() (#4259) new 978f8b9eca Merge branch '2.1' The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../accumulo/test/functional/CompactionIT.java | 20 +--- 1 file changed, 13 insertions(+), 7 deletions(-)
(accumulo) 01/01: Merge branch '2.1'
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo.git
commit 978f8b9ecaac2518ad3f2559512cf11d5fdd3901
Merge: d91178f44c f28b55dc25
Author: Keith Turner
AuthorDate: Tue Feb 13 17:12:35 2024 +
Merge branch '2.1'
.../accumulo/test/functional/CompactionIT.java | 20 +---
1 file changed, 13 insertions(+), 7 deletions(-)
diff --cc
test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java
index 6df72f17d9,46aa609e2c..18fdff715d
--- a/test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java
+++ b/test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java
@@@ -81,9 -69,9 +81,10 @@@ import org.apache.accumulo.core.iterato
import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
import org.apache.accumulo.core.iterators.user.AgeOffFilter;
import org.apache.accumulo.core.iterators.user.GrepIterator;
-import org.apache.accumulo.core.metadata.MetadataTable;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
import org.apache.accumulo.core.metadata.schema.Ample;
+ import org.apache.accumulo.core.metadata.schema.MetadataSchema;
import
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily;
import
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.TabletColumnFamily;
import org.apache.accumulo.core.metadata.schema.TabletMetadata;
@@@ -985,9 -715,14 +986,14 @@@ public class CompactionIT extends Accum
}
}
- private int countFiles(AccumuloClient c) throws Exception {
+ /**
+* Counts the number of tablets and files in a table.
+*/
+ private int countFiles(AccumuloClient c, String tableName) throws Exception
{
+ var tableId = getCluster().getServerContext().getTableId(tableName);
-try (Scanner s = c.createScanner(MetadataTable.NAME,
Authorizations.EMPTY)) {
+try (Scanner s = c.createScanner(AccumuloTable.METADATA.tableName(),
Authorizations.EMPTY)) {
- s.fetchColumnFamily(new Text(TabletColumnFamily.NAME));
+ s.setRange(MetadataSchema.TabletsSection.getRange(tableId));
+ TabletColumnFamily.PREV_ROW_COLUMN.fetch(s);
s.fetchColumnFamily(new Text(DataFileColumnFamily.NAME));
return Iterators.size(s.iterator());
}
(accumulo) branch 2.1 updated: Fixes CompactionIT.testMultiStepCompactionThatDeletesAll() (#4259)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch 2.1
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/2.1 by this push:
new f28b55dc25 Fixes CompactionIT.testMultiStepCompactionThatDeletesAll()
(#4259)
f28b55dc25 is described below
commit f28b55dc254051fea8a328fee9868f800a5d
Author: Keith Turner
AuthorDate: Tue Feb 13 16:29:06 2024 +
Fixes CompactionIT.testMultiStepCompactionThatDeletesAll() (#4259)
CompactionIT.countFiles() was counting in an overly broad way. In the
elasticity branch that included the new +fate table and the new availability
columns in ~tab. This was causing a test failure. Narrowed the scope of
countFiles to fix this. Even though the test was not broken in 2.1, the
test fixes were made there because its a general improvement to the test.
---
.../accumulo/test/functional/CompactionIT.java | 20 +---
1 file changed, 13 insertions(+), 7 deletions(-)
diff --git
a/test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java
b/test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java
index 93c8c1713e..46aa609e2c 100644
--- a/test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java
+++ b/test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java
@@ -71,6 +71,7 @@ import org.apache.accumulo.core.iterators.user.AgeOffFilter;
import org.apache.accumulo.core.iterators.user.GrepIterator;
import org.apache.accumulo.core.metadata.MetadataTable;
import org.apache.accumulo.core.metadata.schema.Ample;
+import org.apache.accumulo.core.metadata.schema.MetadataSchema;
import
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily;
import
org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.TabletColumnFamily;
import org.apache.accumulo.core.metadata.schema.TabletMetadata;
@@ -531,7 +532,7 @@ public class CompactionIT extends AccumuloClusterHarness {
FunctionalTestUtils.createRFiles(c, fs, testrf.toString(), 50, 59,
4);
c.tableOperations().importDirectory(testrf.toString()).to(tableName).load();
- int beforeCount = countFiles(c);
+ int beforeCount = countFiles(c, tableName);
final AtomicBoolean fail = new AtomicBoolean(false);
final int THREADS = 5;
@@ -561,7 +562,7 @@ public class CompactionIT extends AccumuloClusterHarness {
"Failed to successfully run all threads, Check the test output for
error");
}
- int finalCount = countFiles(c);
+ int finalCount = countFiles(c, tableName);
assertTrue(finalCount < beforeCount);
try {
getClusterControl().adminStopAll();
@@ -588,7 +589,7 @@ public class CompactionIT extends AccumuloClusterHarness {
c.tableOperations().setProperty(tableName,
Property.TABLE_FILE_MAX.getKey(), "1001");
c.tableOperations().setProperty(tableName,
Property.TABLE_MAJC_RATIO.getKey(), "100.0");
- var beforeCount = countFiles(c);
+ var beforeCount = countFiles(c, tableName);
final int NUM_ENTRIES_AND_FILES = 60;
@@ -607,7 +608,7 @@ public class CompactionIT extends AccumuloClusterHarness {
assertEquals(NUM_ENTRIES_AND_FILES, scanner.stream().count());
}
- var afterCount = countFiles(c);
+ var afterCount = countFiles(c, tableName);
assertTrue(afterCount >= beforeCount + NUM_ENTRIES_AND_FILES);
@@ -623,7 +624,7 @@ public class CompactionIT extends AccumuloClusterHarness {
assertEquals(0, scanner.stream().count());
}
- var finalCount = countFiles(c);
+ var finalCount = countFiles(c, tableName);
assertTrue(finalCount <= beforeCount);
}
}
@@ -714,9 +715,14 @@ public class CompactionIT extends AccumuloClusterHarness {
}
}
- private int countFiles(AccumuloClient c) throws Exception {
+ /**
+ * Counts the number of tablets and files in a table.
+ */
+ private int countFiles(AccumuloClient c, String tableName) throws Exception {
+var tableId = getCluster().getServerContext().getTableId(tableName);
try (Scanner s = c.createScanner(MetadataTable.NAME,
Authorizations.EMPTY)) {
- s.fetchColumnFamily(new Text(TabletColumnFamily.NAME));
+ s.setRange(MetadataSchema.TabletsSection.getRange(tableId));
+ TabletColumnFamily.PREV_ROW_COLUMN.fetch(s);
s.fetchColumnFamily(new Text(DataFileColumnFamily.NAME));
return Iterators.size(s.iterator());
}
(accumulo) branch elasticity updated (0739f7ee6d -> e375190772)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from 0739f7ee6d Merge branch 'main' into elasticity add e375190772 fixes volume replacement failures (#4257) No new revisions were added by this update. Summary of changes: .../server/manager/state/TabletGoalState.java | 31 +++--- .../manager/state/TabletManagementIterator.java| 12 - .../accumulo/manager/TabletGroupWatcher.java | 4 +-- 3 files changed, 23 insertions(+), 24 deletions(-)
(accumulo) branch elasticity updated: fixes clean shutdown bug in manager (#4248)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 9d80b65f0e fixes clean shutdown bug in manager (#4248)
9d80b65f0e is described below
commit 9d80b65f0eec60ad6d137cc8400798380cc55fe7
Author: Keith Turner
AuthorDate: Fri Feb 9 18:04:17 2024 -0500
fixes clean shutdown bug in manager (#4248)
This commits fixes #4209 by shutting down fate before unassigning any
tablets on which fate depends.
---
.../java/org/apache/accumulo/core/fate/Fate.java | 47 +-
.../accumulo/core/fate/ReadOnlyFateStore.java | 2 +
.../org/apache/accumulo/core/fate/ZooStore.java| 5 +++
.../accumulo/core/fate/accumulo/AccumuloStore.java | 5 +++
.../apache/accumulo/core/logging/FateLogger.java | 6 +++
.../org/apache/accumulo/core/fate/TestStore.java | 5 +++
.../java/org/apache/accumulo/manager/Manager.java | 9 -
.../java/org/apache/accumulo/test/fate/FateIT.java | 12 +++---
8 files changed, 74 insertions(+), 17 deletions(-)
diff --git a/core/src/main/java/org/apache/accumulo/core/fate/Fate.java
b/core/src/main/java/org/apache/accumulo/core/fate/Fate.java
index 35807ee0fc..3828bb80c4 100644
--- a/core/src/main/java/org/apache/accumulo/core/fate/Fate.java
+++ b/core/src/main/java/org/apache/accumulo/core/fate/Fate.java
@@ -459,20 +459,45 @@ public class Fate {
}
/**
- * Flags that FATE threadpool to clear out and end. Does not actively stop
running FATE processes.
+ * Initiates shutdown of background threads and optionally waits on them.
*/
- public void shutdown() {
-keepRunning.set(false);
-fatePoolWatcher.shutdown();
-if (executor != null) {
+ public void shutdown(long timeout, TimeUnit timeUnit) {
+if (keepRunning.compareAndSet(true, false)) {
+ fatePoolWatcher.shutdown();
executor.shutdown();
+ workFinder.interrupt();
}
-workFinder.interrupt();
-try {
- workFinder.join();
-} catch (InterruptedException e) {
- throw new RuntimeException(e);
+
+if (timeout > 0) {
+ long start = System.nanoTime();
+
+ while ((System.nanoTime() - start) < timeUnit.toNanos(timeout)
+ && (workFinder.isAlive() || !executor.isTerminated())) {
+try {
+ if (!executor.awaitTermination(1, SECONDS)) {
+log.debug("Fate {} is waiting for worker threads to terminate",
store.type());
+continue;
+ }
+
+ workFinder.join(1_000);
+ if (workFinder.isAlive()) {
+log.debug("Fate {} is waiting for work finder thread to
terminate", store.type());
+workFinder.interrupt();
+ }
+} catch (InterruptedException e) {
+ throw new RuntimeException(e);
+}
+ }
+
+ if (workFinder.isAlive() || !executor.isTerminated()) {
+log.warn(
+"Waited for {}ms for all fate {} background threads to stop, but
some are still running. workFinder:{} executor:{}",
+TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start),
store.type(),
+workFinder.isAlive(), !executor.isTerminated());
+ }
}
- }
+// interrupt the background threads
+executor.shutdownNow();
+ }
}
diff --git
a/core/src/main/java/org/apache/accumulo/core/fate/ReadOnlyFateStore.java
b/core/src/main/java/org/apache/accumulo/core/fate/ReadOnlyFateStore.java
index deb79413c9..bdbb7739f9 100644
--- a/core/src/main/java/org/apache/accumulo/core/fate/ReadOnlyFateStore.java
+++ b/core/src/main/java/org/apache/accumulo/core/fate/ReadOnlyFateStore.java
@@ -158,4 +158,6 @@ public interface ReadOnlyFateStore {
* @return the current number of transactions that have been deferred
*/
int getDeferredCount();
+
+ FateInstanceType type();
}
diff --git a/core/src/main/java/org/apache/accumulo/core/fate/ZooStore.java
b/core/src/main/java/org/apache/accumulo/core/fate/ZooStore.java
index d0ef960054..6813e727c5 100644
--- a/core/src/main/java/org/apache/accumulo/core/fate/ZooStore.java
+++ b/core/src/main/java/org/apache/accumulo/core/fate/ZooStore.java
@@ -118,6 +118,11 @@ public class ZooStore extends AbstractFateStore {
return new Pair<>(node.status, node.fateKey);
}
+ @Override
+ public FateInstanceType type() {
+return fateInstanceType;
+ }
+
private class FateTxStoreImpl extends AbstractFateTxStoreImpl {
private FateTxStoreImpl(FateId fateId, boolean isReserved) {
diff --git
a/core/src/main/java/org/apache/accumulo/core/fate/accumulo/AccumuloStore.java
b/core/src/main/java/org/apache/accumulo/core/fate/accumulo/AccumuloStore.java
index 328560b150..7fd4b967cb 100644
---
a/core/src/main/java/org/apache/accumulo/core/fate/accumulo/AccumuloStore.java
+++
b/core/src/main/java/org/apa
(accumulo) branch elasticity updated: Fixes GC WAL bug (#4196)
This is an automated email from the ASF dual-hosted git repository.
kturner pushed a commit to branch elasticity
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/elasticity by this push:
new 8b8a49f573 Fixes GC WAL bug (#4196)
8b8a49f573 is described below
commit 8b8a49f5730e2960593f6650216a1419e8157c51
Author: Keith Turner
AuthorDate: Thu Feb 8 18:43:18 2024 -0500
Fixes GC WAL bug (#4196)
This fixes a GC bug that only exists in the elastictiy branch. When
garbage collecting write ahead logs the metadata table should be
inspected to look for tablets with WALs and tablets assigned to dead
tablet servers. The GC code was only looking for tablets with WALs.
This commits makes it also look for tablets assigned to dead
tservers.
This bug was noticed because CleanWalIT was sometimes failing.
Sometimes the GC was collecting a WAL after the metadata table recovered
but before user tablets could be assigned the WAL. This caused the user
tablets to silently loose data. The test caught this data loss.
In the course of fixing this bug a change was made to how RowFilter was
used and a pre-existing bug with RowFilter was found and fixed. This
bug fix will be back ported separately.
---
.../accumulo/core/iterators/user/GcWalsFilter.java | 101 +
.../core/iterators/user/HasWalsFilter.java | 45 -
.../core/iterators/user/TabletMetadataFilter.java | 5 +
.../core/metadata/schema/TabletsMetadata.java | 2 +
.../accumulo/gc/GarbageCollectWriteAheadLogs.java | 14 +--
.../gc/GarbageCollectWriteAheadLogsTest.java | 9 +-
.../test/functional/AmpleConditionalWriterIT.java | 44 -
7 files changed, 161 insertions(+), 59 deletions(-)
diff --git
a/core/src/main/java/org/apache/accumulo/core/iterators/user/GcWalsFilter.java
b/core/src/main/java/org/apache/accumulo/core/iterators/user/GcWalsFilter.java
new file mode 100644
index 00..a30302ce59
--- /dev/null
+++
b/core/src/main/java/org/apache/accumulo/core/iterators/user/GcWalsFilter.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.iterators.user;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.IteratorEnvironment;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.metadata.TServerInstance;
+import org.apache.accumulo.core.metadata.TabletState;
+import org.apache.accumulo.core.metadata.schema.TabletMetadata;
+import org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Sets;
+
+/**
+ * A filter used by the Accumulo GC to find tablets that either have walogs or
are assigned to a
+ * dead tablet server.
+ */
+
+// ELASTICITY_TODO Move TabletMetadataFilter and its subclasses out of public
API. It use internal
+// types that are not user facing.
+public class GcWalsFilter extends TabletMetadataFilter {
+
+ private Map options = null;
+
+ private Predicate filter = null;
+
+ private static final String LIVE_TSERVER_OPT = "liveTservers";
+
+ public GcWalsFilter() {}
+
+ public GcWalsFilter(Set liveTservers) {
+String lts = liveTservers.stream().map(TServerInstance::toString).peek(tsi
-> {
+ if (tsi.contains(",")) {
+throw new IllegalArgumentException(tsi);
+ }
+}).collect(Collectors.joining(","));
+this.options = Map.of(LIVE_TSERVER_OPT, lts);
+ }
+
+ @Override
+ public void init(SortedKeyValueIterator source,
Map options,
+ IteratorEnvironment env) throws IOException {
+super.init(source, options, env);
+var encodedLiveTservers = options.get(LIVE_TSERVER_OPT);
+Set liveTservers;
+if (encodedLiveTservers.isBlank()) {
+ liveT
(accumulo) 01/01: Merge branch 'main' into elasticity
This is an automated email from the ASF dual-hosted git repository. kturner pushed a commit to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git commit eb7524214bc06ce6e38617df76830a63424d32be Merge: a356e4977b 113e495dbf Author: Keith Turner AuthorDate: Thu Feb 8 13:58:29 2024 -0500 Merge branch 'main' into elasticity .../server/manager/recovery/RecoveryPath.java | 18 ++--- .../org/apache/accumulo/compactor/Compactor.java | 2 +- .../org/apache/accumulo/tserver/log/DfsLogger.java | 83 +- .../accumulo/tserver/log/TabletServerLogger.java | 4 +- .../org/apache/accumulo/tserver/tablet/Tablet.java | 2 +- .../accumulo/tserver/WalRemovalOrderTest.java | 24 +-- 6 files changed, 66 insertions(+), 67 deletions(-)
(accumulo) branch elasticity updated (a356e4977b -> eb7524214b)
This is an automated email from the ASF dual-hosted git repository. kturner pushed a change to branch elasticity in repository https://gitbox.apache.org/repos/asf/accumulo.git from a356e4977b Merge branch 'main' into elasticity add 62d2a6f4f5 Refactored DfsLogger constructors to static methods (#4192) add 1443a7edaf Make RecoveryPath more strict (#4243) add 896e6277e1 Adjusts compactor error logging because of race (#4151) add 113e495dbf Merge branch '2.1' new eb7524214b Merge branch 'main' into elasticity The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../server/manager/recovery/RecoveryPath.java | 18 ++--- .../org/apache/accumulo/compactor/Compactor.java | 2 +- .../org/apache/accumulo/tserver/log/DfsLogger.java | 83 +- .../accumulo/tserver/log/TabletServerLogger.java | 4 +- .../org/apache/accumulo/tserver/tablet/Tablet.java | 2 +- .../accumulo/tserver/WalRemovalOrderTest.java | 24 +-- 6 files changed, 66 insertions(+), 67 deletions(-)
