[accumulo] branch no-chop-merge updated (6fadff50bf -> 90944c5939)
This is an automated email from the ASF dual-hosted git repository. cshannon pushed a change to branch no-chop-merge in repository https://gitbox.apache.org/repos/asf/accumulo.git from 6fadff50bf update comments add 75140d8965 Add Message Prefix for File deletion actions in GCRun (#3745) add f7e59d3653 Merge branch '2.1' add 6c34c848c4 reverts #3747 that caused tablet split to fail (#3760) add 48d69f30a1 Merge branch '2.1' new 90944c5939 Merge branch 'main' into no-chop-merge The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../main/java/org/apache/accumulo/gc/GCRun.java| 24 +- .../accumulo/tserver/tablet/DatafileManager.java | 5 ++--- .../org/apache/accumulo/tserver/tablet/Tablet.java | 5 ++--- 3 files changed, 18 insertions(+), 16 deletions(-)
[accumulo] 01/01: Merge branch 'main' into no-chop-merge
This is an automated email from the ASF dual-hosted git repository. cshannon pushed a commit to branch no-chop-merge in repository https://gitbox.apache.org/repos/asf/accumulo.git commit 90944c593960c06aefc87ebd7d154c0aa0129586 Merge: 6fadff50bf 48d69f30a1 Author: Christopher L. Shannon (cshannon) AuthorDate: Mon Sep 18 18:17:17 2023 -0400 Merge branch 'main' into no-chop-merge .../main/java/org/apache/accumulo/gc/GCRun.java| 24 +- .../accumulo/tserver/tablet/DatafileManager.java | 5 ++--- .../org/apache/accumulo/tserver/tablet/Tablet.java | 5 ++--- 3 files changed, 18 insertions(+), 16 deletions(-)
[accumulo] 01/01: Merge branch '2.1'
This is an automated email from the ASF dual-hosted git repository. ctubbsii pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git commit 48d69f30a18da524e03ad068f733906e7428c5ea Merge: f7e59d3653 6c34c848c4 Author: Christopher Tubbs AuthorDate: Mon Sep 18 14:44:33 2023 -0400 Merge branch '2.1' .../java/org/apache/accumulo/tserver/tablet/DatafileManager.java | 5 ++--- .../src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java | 5 ++--- 2 files changed, 4 insertions(+), 6 deletions(-)
[accumulo] branch main updated (f7e59d3653 -> 48d69f30a1)
This is an automated email from the ASF dual-hosted git repository. ctubbsii pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git from f7e59d3653 Merge branch '2.1' add 6c34c848c4 reverts #3747 that caused tablet split to fail (#3760) new 48d69f30a1 Merge branch '2.1' The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../java/org/apache/accumulo/tserver/tablet/DatafileManager.java | 5 ++--- .../src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java | 5 ++--- 2 files changed, 4 insertions(+), 6 deletions(-)
[accumulo] branch 2.1 updated (75140d8965 -> 6c34c848c4)
This is an automated email from the ASF dual-hosted git repository. ctubbsii pushed a change to branch 2.1 in repository https://gitbox.apache.org/repos/asf/accumulo.git from 75140d8965 Add Message Prefix for File deletion actions in GCRun (#3745) add 6c34c848c4 reverts #3747 that caused tablet split to fail (#3760) No new revisions were added by this update. Summary of changes: .../java/org/apache/accumulo/tserver/tablet/DatafileManager.java | 5 ++--- .../src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java | 5 ++--- 2 files changed, 4 insertions(+), 6 deletions(-)
[accumulo] branch main updated (e4127a5014 -> f7e59d3653)
This is an automated email from the ASF dual-hosted git repository. ddanielr pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git from e4127a5014 adds code lost in merge from 2.1 add 75140d8965 Add Message Prefix for File deletion actions in GCRun (#3745) new f7e59d3653 Merge branch '2.1' The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../main/java/org/apache/accumulo/gc/GCRun.java| 24 +- 1 file changed, 14 insertions(+), 10 deletions(-)
[accumulo] 01/01: Merge branch '2.1'
This is an automated email from the ASF dual-hosted git repository. ddanielr pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/accumulo.git commit f7e59d3653edddc9bc4554fea5dbcba386d04fab Merge: e4127a5014 75140d8965 Author: Daniel Roberts AuthorDate: Mon Sep 18 18:24:46 2023 + Merge branch '2.1' .../main/java/org/apache/accumulo/gc/GCRun.java| 24 +- 1 file changed, 14 insertions(+), 10 deletions(-)
[accumulo] branch 2.1 updated: Add Message Prefix for File deletion actions in GCRun (#3745)
This is an automated email from the ASF dual-hosted git repository.
ddanielr pushed a commit to branch 2.1
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/2.1 by this push:
new 75140d8965 Add Message Prefix for File deletion actions in GCRun
(#3745)
75140d8965 is described below
commit 75140d8965009d9a9cbc5b59e704ea9f7e8826b7
Author: Daniel Roberts
AuthorDate: Mon Sep 18 14:12:58 2023 -0400
Add Message Prefix for File deletion actions in GCRun (#3745)
* Changes the logger name and removes the space and "class" info from the
name.
* Moves datalevel to the end of the logger name to ensure it shows up
in log messages.
* Add comment regarding logger creation
Co-authored-by: EdColeman
* Adds a message prefix which can be picked up by a regexFilter.
* Update log error message in GCRun.java
Remove redundant `e.getMessage()` in log message
Co-authored-by: Christopher Tubbs
* Makes fileActionPrefix final
Co-authored-by: Christopher Tubbs
* Update non-descriptive log message
-
Co-authored-by: EdColeman
Co-authored-by: Christopher Tubbs
---
.../main/java/org/apache/accumulo/gc/GCRun.java| 24 +-
1 file changed, 14 insertions(+), 10 deletions(-)
diff --git a/server/gc/src/main/java/org/apache/accumulo/gc/GCRun.java
b/server/gc/src/main/java/org/apache/accumulo/gc/GCRun.java
index 515feb6fbc..4af97d3a25 100644
--- a/server/gc/src/main/java/org/apache/accumulo/gc/GCRun.java
+++ b/server/gc/src/main/java/org/apache/accumulo/gc/GCRun.java
@@ -87,7 +87,9 @@ import com.google.protobuf.InvalidProtocolBufferException;
* A single garbage collection performed on a table (Root, MD) or all User
tables.
*/
public class GCRun implements GarbageCollectionEnvironment {
+ // loggers are not static to support unique naming by level
private final Logger log;
+ private static final String fileActionPrefix = "FILE-ACTION:";
private final Ample.DataLevel level;
private final ServerContext context;
private final AccumuloConfiguration config;
@@ -97,7 +99,7 @@ public class GCRun implements GarbageCollectionEnvironment {
private long errors = 0;
public GCRun(Ample.DataLevel level, ServerContext context) {
-this.log = LoggerFactory.getLogger(level.name() + GCRun.class);
+this.log = LoggerFactory.getLogger(GCRun.class.getName() + "." +
level.name());
this.level = level;
this.context = context;
this.config = context.getConfiguration();
@@ -236,9 +238,9 @@ public class GCRun implements GarbageCollectionEnvironment {
System.out.println("SAFEMODE: There are " + confirmedDeletes.size()
+ " data file candidates marked for deletion in " + metadataLocation
+ ".\n"
+ " Examine the log files to identify them.\n");
- log.info("SAFEMODE: Listing all data file candidates for deletion");
+ log.info("{} SAFEMODE: Listing all data file candidates for deletion",
fileActionPrefix);
for (String s : confirmedDeletes.values()) {
-log.info("SAFEMODE: {}", s);
+log.info("{} SAFEMODE: {}", fileActionPrefix, s);
}
log.info("SAFEMODE: End candidates for deletion");
return;
@@ -278,7 +280,7 @@ public class GCRun implements GarbageCollectionEnvironment {
}
for (Path pathToDel : GcVolumeUtil.expandAllVolumesUri(fs,
fullPath)) {
-log.debug("Deleting {}", pathToDel);
+log.debug("{} Deleting {}", fileActionPrefix, pathToDel);
if (moveToTrash(pathToDel) || fs.deleteRecursively(pathToDel)) {
// delete succeeded, still want to delete
@@ -288,7 +290,8 @@ public class GCRun implements GarbageCollectionEnvironment {
// leave the entry in the metadata; we'll try again later
removeFlag = false;
errors++;
- log.warn("File exists, but was not deleted for an unknown
reason: {}", pathToDel);
+ log.warn("{} File exists, but was not deleted for an unknown
reason: {}",
+ fileActionPrefix, pathToDel);
break;
} else {
// this failure, we still want to remove the metadata entry
@@ -303,11 +306,11 @@ public class GCRun implements
GarbageCollectionEnvironment {
if (tableState != null && tableState != TableState.DELETING) {
// clone directories don't always exist
if (!tabletDir.startsWith(Constants.CLONE_PREFIX)) {
-log.debug("File doesn't exist: {}", pathToDel);
+log.debug("{} File doesn't exist: {}", fileActionPrefix,
pathToDel);
}
}
} else {
-log.warn("Very strange path name: {}", delete);
+log.warn("{} Invalid file
[accumulo-website] branch asf-site updated: Automatic Site Publish by Buildbot
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/accumulo-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 3d66d846 Automatic Site Publish by Buildbot 3d66d846 is described below commit 3d66d8463ef52708b0957c1b891484258be1e49e Author: buildbot AuthorDate: Mon Sep 18 13:05:08 2023 + Automatic Site Publish by Buildbot --- output/api/index.html | 8 +- output/docs/2.x/administration/multivolume.html| 2 +- output/docs/2.x/administration/replication.html| 40 ++--- output/docs/2.x/configuration/overview.html| 6 +- output/docs/2.x/development/summaries.html | 122 +++-- output/docs/2.x/getting-started/clients.html | 10 +- output/docs/2.x/getting-started/shell.html | 87 +- .../2.x/getting-started/table_configuration.html | 188 ++--- output/docs/2.x/security/authentication.html | 24 +-- output/docs/2.x/security/kerberos.html | 64 +++ output/docs/2.x/security/permissions.html | 10 +- output/feed.xml| 4 +- output/search_data.json| 4 +- output/tour/authorizations-code/index.html | 42 ++--- output/tour/authorizations/index.html | 8 +- output/tour/basic-read-write/index.html| 20 +-- output/tour/batch-scanner-code/index.html | 12 +- output/tour/batch-scanner/index.html | 4 +- output/tour/client/index.html | 12 +- output/tour/conditional-writer-code/index.html | 8 +- output/tour/conditional-writer/index.html | 8 +- output/tour/data-model-code/index.html | 24 +-- output/tour/data-model/index.html | 6 +- output/tour/getting-started/index.html | 20 +-- output/tour/ranges-splits/index.html | 12 +- output/tour/using-iterators/index.html | 32 ++-- 26 files changed, 383 insertions(+), 394 deletions(-) diff --git a/output/api/index.html b/output/api/index.html index e1d874b3..88f066f7 100644 --- a/output/api/index.html +++ b/output/api/index.html @@ -171,14 +171,14 @@ imports in a project using Accumulo. For 1.x: -import\s+org\.apache\.accumulo\.(.*\.(impl|thrift|crypto)\..*|(?!(core\.(client|data|security)|minicluster)\.).*) - +import\s+org\.apache\.accumulo\.(.*\.(impl|thrift|crypto)\..*|(?!(core\.(client|data|security)|minicluster)\.).*) + For 2.0 and later, this can be simplified, because sub-packages not intended for public API were relocated, and also altered to include the new MapReduce module: -import\s+org\.apache\.accumulo\.(?!(core\.(client|data|iterators|security)|minicluster|hadoop)\.).* - +import\s+org\.apache\.accumulo\.(?!(core\.(client|data|iterators|security)|minicluster|hadoop)\.).* + See the blog post about using the checkstyle plugin for more explicit non-API detection. diff --git a/output/docs/2.x/administration/multivolume.html b/output/docs/2.x/administration/multivolume.html index 0d43e71c..7c7bcb48 100644 --- a/output/docs/2.x/administration/multivolume.html +++ b/output/docs/2.x/administration/multivolume.html @@ -474,7 +474,7 @@ in its metadata. The example configuration below will replace ns1 with nsA and ns2 with nsB in Accumulo metadata. For this property to take affect, Accumulo will need to be restarted. -instance.volumes.replacements=hdfs://ns1:9001 hdfs://nsA:9001, hdfs://ns2:9001 hdfs://nsB:9001 +instance.volumes.replacements=hdfs://ns1:9001 hdfs://nsA:9001, hdfs://ns2:9001 hdfs://nsB:9001 Using viewfs or HA namenode, introduced in Hadoop 2, offers another option for diff --git a/output/docs/2.x/administration/replication.html b/output/docs/2.x/administration/replication.html index 4885cebc..438c7f5b 100644 --- a/output/docs/2.x/administration/replication.html +++ b/output/docs/2.x/administration/replication.html @@ -499,17 +499,17 @@ change, it can be useful to set this information using the Accumulo shell. and a ZooKeeper quorum of 10.0.0.1,10.0.2.1,10.0.3.1, invoke the following command in the shell. -root@accumulo_primary config -s -replication.peer.peer1=org.apache.accumulo.tserver.replication.AccumuloReplicaSystem,accumulo_peer,10.0.0.1,10.0.2.1,10.0.3.1 - +root@accumulo_primary config -s +replication.peer.peer1=org.apache.accumulo.tserver.replication.AccumuloReplicaSystem,accumulo_peer,10.0.0.1,10.0.2.1,10.0.3.1 + Since this is an Accumulo system, we also want to set a username and password to use when authenticating with this peer. On our peer, we make a special user which has permission to write to the tables we want to replicate data into, “replication” with a password of “password”. We then need to record this in the primary’s configuration.
[accumulo-website] branch main updated: Fix code block styling (#407)
This is an automated email from the ASF dual-hosted git repository. domgarguilo pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/accumulo-website.git The following commit(s) were added to refs/heads/main by this push: new 5947324f Fix code block styling (#407) 5947324f is described below commit 5947324f9788b88ff65e1733f1909d99ab1086ab Author: Dom G AuthorDate: Mon Sep 18 09:04:45 2023 -0400 Fix code block styling (#407) --- _docs-2/administration/multivolume.md | 2 +- _docs-2/administration/replication.md | 20 ++-- _docs-2/configuration/overview.md | 2 +- _docs-2/development/summaries.md | 126 + _docs-2/getting-started/clients.md | 6 +- _docs-2/getting-started/shell.md | 61 ++-- _docs-2/getting-started/table_configuration.md | 12 +-- _docs-2/security/authentication.md | 8 +- _docs-2/security/kerberos.md | 8 +- _docs-2/security/permissions.md| 6 +- pages/api.md | 4 +- tour/authorizations-code.md| 24 ++--- tour/authorizations.md | 4 +- tour/basic-read-write.md | 10 +- tour/batch-scanner-code.md | 6 +- tour/batch-scanner.md | 2 +- tour/client.md | 6 +- tour/conditional-writer-code.md| 4 +- tour/conditional-writer.md | 4 +- tour/data-model-code.md| 12 +-- tour/data-model.md | 2 +- tour/getting-started.md| 10 +- tour/ranges-splits.md | 4 +- tour/using-iterators.md| 16 ++-- 24 files changed, 183 insertions(+), 176 deletions(-) diff --git a/_docs-2/administration/multivolume.md b/_docs-2/administration/multivolume.md index 3aed973e..eb71d6b3 100644 --- a/_docs-2/administration/multivolume.md +++ b/_docs-2/administration/multivolume.md @@ -48,7 +48,7 @@ in its metadata. The example configuration below will replace ns1 with nsA and ns2 with nsB in Accumulo metadata. For this property to take affect, Accumulo will need to be restarted. -```xml +``` instance.volumes.replacements=hdfs://ns1:9001 hdfs://nsA:9001, hdfs://ns2:9001 hdfs://nsB:9001 ``` diff --git a/_docs-2/administration/replication.md b/_docs-2/administration/replication.md index cb18a616..91cb4b35 100644 --- a/_docs-2/administration/replication.md +++ b/_docs-2/administration/replication.md @@ -73,7 +73,7 @@ To configure a peer with the name `peer1` which is an Accumulo system with an in and a ZooKeeper quorum of `10.0.0.1,10.0.2.1,10.0.3.1`, invoke the following command in the shell. -``` +```console root@accumulo_primary> config -s replication.peer.peer1=org.apache.accumulo.tserver.replication.AccumuloReplicaSystem,accumulo_peer,10.0.0.1,10.0.2.1,10.0.3.1 ``` @@ -83,7 +83,7 @@ to use when authenticating with this peer. On our peer, we make a special user which has permission to write to the tables we want to replicate data into, "replication" with a password of "password". We then need to record this in the primary's configuration. -``` +```console root@accumulo_primary> config -s replication.peer.user.peer1=replication root@accumulo_primary> config -s replication.peer.password.peer1=password ``` @@ -93,7 +93,7 @@ file per peer can be configured instead of a password. The provided keytabs must by the unix user running Accumulo. They keytab for a peer can be unique from the keytab used by Accumulo or any keytabs for other peers. -``` +```console accum...@example.com@accumulo_primary> config -s replication.peer.user.peer1=replicat...@example.com accum...@example.com@accumulo_primary> config -s replication.peer.keytab.peer1=/path/to/replication.keytab ``` @@ -107,7 +107,7 @@ cluster, this is a table ID. In this example, we want to enable replication on `my_table` and configure our peer `accumulo_peer` as a target, sending the data to the table with an ID of `2` in `accumulo_peer`. -``` +```console root@accumulo_primary> config -t my_table -s table.replication=true root@accumulo_primary> config -t my_table -s table.replication.target.accumulo_peer=2 ``` @@ -225,7 +225,7 @@ The rest of the configuration is dynamic and is best configured on the fly (in Z The next series of command are to be run on the peer system. Create a user account for the primary instance called "peer". The password for this account will need to be saved in the configuration on the primary -``` +```console root@peer> createtable my_table root@peer> createuser peer root@peer> grant -t my_table -u peer Table.WRITE @@ -241,7 +241,7 @@ Next, configure the primary instance. Set up the table -``` +```console root@primary>
[accumulo] branch no-chop-merge updated: update comments
This is an automated email from the ASF dual-hosted git repository.
cshannon pushed a commit to branch no-chop-merge
in repository https://gitbox.apache.org/repos/asf/accumulo.git
The following commit(s) were added to refs/heads/no-chop-merge by this push:
new 6fadff50bf update comments
6fadff50bf is described below
commit 6fadff50bf32e6d6015cab3bb8977780240a7aeb
Author: Christopher L. Shannon (cshannon)
AuthorDate: Mon Sep 18 07:52:31 2023 -0400
update comments
---
.../org/apache/accumulo/manager/TabletGroupWatcher.java | 13 -
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git
a/server/manager/src/main/java/org/apache/accumulo/manager/TabletGroupWatcher.java
b/server/manager/src/main/java/org/apache/accumulo/manager/TabletGroupWatcher.java
index c0b4e3b785..3990015bb3 100644
---
a/server/manager/src/main/java/org/apache/accumulo/manager/TabletGroupWatcher.java
+++
b/server/manager/src/main/java/org/apache/accumulo/manager/TabletGroupWatcher.java
@@ -846,9 +846,10 @@ abstract class TabletGroupWatcher extends
AccumuloDaemonThread {
previousKeyExtent = lastExtent;
}
-// Special case for now to handle the highest/stop tablet which is
where files are
-// merged to so we need to handle the deletes on update here as it
won't be handled later
-// TODO: Can this be re-written to not have a special edge case and
make it simpler?
+// Special case to handle the highest/stop tablet, which is where
files are
+// merged to. The existing merge code won't delete files from this
tablet
+// so we need to handle the deletes in this tablet when fencing files.
+// We may be able to make this simpler in the future.
if (keyExtent.equals(stopExtent)) {
if (previousKeyExtent != null
&& key.getColumnFamily().equals(DataFileColumnFamily.NAME)) {
@@ -876,10 +877,12 @@ abstract class TabletGroupWatcher extends
AccumuloDaemonThread {
continue;
}
+// Handle metadata updates for all other tablets except the highest
tablet
+// Ranges are created for the files and then added to the highest
tablet in
+// the merge range. Deletes are handled later for the old files when
the tablets
+// are removed.
if (key.getColumnFamily().equals(DataFileColumnFamily.NAME)) {
final StoredTabletFile existing =
StoredTabletFile.of(key.getColumnQualifier());
- // TODO: Should we try and be smart and eventually collapse
overlapping ranges to reduce
- // the metadata? The fenced reader will already collapse ranges when
reading.
// Fence off files by the previous tablet and current tablet that is
being merged
// The end row should be inclusive for the current tablet and the
previous end row should
[accumulo] branch accumulo-1327-itest deleted (was b84cb74598)
This is an automated email from the ASF dual-hosted git repository. cshannon pushed a change to branch accumulo-1327-itest in repository https://gitbox.apache.org/repos/asf/accumulo.git was b84cb74598 Merge branch 'no-chop-merge' into accumulo-1327-itest The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[accumulo] branch no-chop-merge-prototype deleted (was 204010ed29)
This is an automated email from the ASF dual-hosted git repository. cshannon pushed a change to branch no-chop-merge-prototype in repository https://gitbox.apache.org/repos/asf/accumulo.git was 204010ed29 Merge branch 'main' into no-chop-merge-prototype This change permanently discards the following revisions: discard 204010ed29 Merge branch 'main' into no-chop-merge-prototype
