This is an automated email from the ASF dual-hosted git repository.
dlmarion pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-classloaders.git
The following commit(s) were added to refs/heads/main by this push:
new b4c3f7d Add tools for ContextDefinition create and cleanup (#33)
b4c3f7d is described below
commit b4c3f7d7cda49759af71c4f655b244a35ee59de3
Author: Dave Marion <[email protected]>
AuthorDate: Fri Jan 16 10:14:53 2026 -0500
Add tools for ContextDefinition create and cleanup (#33)
Modified ContextDefinition to implement KeywordExecutable so that
the user can use `accumulo create-context-definition` to create the
json required for the ContextDefinition file.
Added a JMX MXBean that will return the list of files in the local cache
directory that are referenced by classloaders created by the factory
in this module and are still in use in the JVM. Users can connect to
the JMX MXBean to retrieve the list to aid in the cleanup of the local
cache directory.
Co-authored-by: Christopher Tubbs <[email protected]>
Co-authored-by: Keith Turner <[email protected]>
---
.github/workflows/maven-on-demand.yaml | 2 +-
.github/workflows/maven.yaml | 2 +-
modules/local-caching-classloader/README.md | 31 ++++++--
modules/local-caching-classloader/pom.xml | 17 ++++
.../lcc/LocalCachingContextClassLoaderFactory.java | 35 +++++++-
.../lcc/definition/ContextDefinition.java | 52 +++++++++++-
.../classloader/lcc/jmx/ContextClassLoaders.java | 33 ++++++++
.../lcc/jmx/ContextClassLoadersMXBean.java | 35 ++++++++
.../classloader/lcc/util/DeduplicationCache.java | 10 +++
.../MiniAccumuloClusterClassLoaderFactoryTest.java | 92 +++++++++++++++++++++-
pom.xml | 15 +++-
11 files changed, 307 insertions(+), 17 deletions(-)
diff --git a/.github/workflows/maven-on-demand.yaml
b/.github/workflows/maven-on-demand.yaml
index 5bdd47c..ca60d24 100644
--- a/.github/workflows/maven-on-demand.yaml
+++ b/.github/workflows/maven-on-demand.yaml
@@ -50,7 +50,7 @@ jobs:
timeout-minutes: 345
run: mvn -B -V -e -ntp "-Dstyle.color=always" ${{
github.event.inputs.goals }}
env:
- MAVEN_OPTS: -Djansi.force=true
+ MAVEN_OPTS: -Djansi.force=true -Dapache.snapshots=true
- name: Upload unit test results
if: ${{ failure() }}
uses: actions/upload-artifact@v4
diff --git a/.github/workflows/maven.yaml b/.github/workflows/maven.yaml
index 099e04f..434a20e 100644
--- a/.github/workflows/maven.yaml
+++ b/.github/workflows/maven.yaml
@@ -45,7 +45,7 @@ jobs:
- name: Build with Maven (verify javadoc:jar)
run: mvn -B -V -e -ntp "-Dstyle.color=always" verify javadoc:jar
-DskipFormat -DverifyFormat
env:
- MAVEN_OPTS: -Djansi.force=true
+ MAVEN_OPTS: -Djansi.force=true -Dapache.snapshots=true
- name: Upload unit test results
if: ${{ failure() }}
uses: actions/upload-artifact@v4
diff --git a/modules/local-caching-classloader/README.md
b/modules/local-caching-classloader/README.md
index 0e66de6..cb5589c 100644
--- a/modules/local-caching-classloader/README.md
+++ b/modules/local-caching-classloader/README.md
@@ -122,9 +122,18 @@ download, and any modification will likely cause
unexpected behavior.
## Creating a ContextDefinition file
Users may take advantage of the `ContextDefinition.create(int,URL[])` method to
-construct a `ContextDefinition` object. This will calculate the checksums of
-the classpath elements. `ContextDefinition.toJson()` can be used to serialize
-the `ContextDefinition` to a `String` to store in a file.
+construct a `ContextDefinition` object, programmatically. This will calculate
+the checksums of the classpath elements. `ContextDefinition.toJson()` can be
+used to serialize the `ContextDefinition` to a `String` to store in a file.
+
+Alternatively, if this library's jar is built and placed onto Accumulo's
+`CLASSPATH`, then one can run `bin/accumulo create-context-definition` to
+create the ContextDefinition json file using the command-line. The resulting
+json is printed to stdout and can be redirected to a file. The command takes
+two arguments:
+
+1. the monitor interval, in seconds (e.g. `-i 300`), and
+2. a list of file URLs (e.g. `hdfs://host:port/path/to/one.jar
file://host/path/to/two.jar`)
## Updating a ContextDefinition file
@@ -177,12 +186,22 @@ unused files from the cache. While the context definition
JSON files are always
safe to delete, it is not recommended to do so for any that are still in use,
because they can be useful for troubleshooting.
+To aid in this task, a JMX MXBean has been created to expose the files that are
+still referenced by the classloaders that have been created by this factory and
+currently still exist in the system. For an example of how to use this MXBean,
+please see the test method
+`MiniAccumuloClusterClassLoaderFactoryTest.getReferencedFiles`. This method
+attaches to the local Accumulo JVM processes to get the set of referenced
+files. It should be safe to delete files that are located in the local cache
+directory (set by property `general.custom.classloader.lcc.cache.dir`) that are
+NOT in the set of referenced files, so long as no new classloaders have been
+created that reference the files being deleted.
+
**IMPORTANT**: as mentioned earlier, it is not safe to delete resource files
that are still referenced by any `ClassLoader` instances. Each `ClassLoader`
instance assumes that the locally cached resources exist and can be read. They
-will not attempt to download any files. Downloading and verifying files only
-occurs when `ClassLoader` instances are initially created for a context
-definition.
+will not attempt to download any files. Downloading files only occurs when
+`ClassLoader` instances are initially created for a context definition.
## Accumulo Configuration
diff --git a/modules/local-caching-classloader/pom.xml
b/modules/local-caching-classloader/pom.xml
index 8b39c34..0428499 100644
--- a/modules/local-caching-classloader/pom.xml
+++ b/modules/local-caching-classloader/pom.xml
@@ -51,6 +51,17 @@
<artifactId>spotbugs-annotations</artifactId>
<optional>true</optional>
</dependency>
+ <dependency>
+ <!-- needed for annotation processor during compile, but not after -->
+ <groupId>com.google.auto.service</groupId>
+ <artifactId>auto-service</artifactId>
+ <optional>true</optional>
+ </dependency>
+ <dependency>
+ <groupId>com.beust</groupId>
+ <artifactId>jcommander</artifactId>
+ <scope>provided</scope>
+ </dependency>
<dependency>
<groupId>com.github.ben-manes.caffeine</groupId>
<artifactId>caffeine</artifactId>
@@ -82,6 +93,12 @@
<artifactId>accumulo-core</artifactId>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <!-- provided by accumulo -->
+ <groupId>org.apache.accumulo</groupId>
+ <artifactId>accumulo-start</artifactId>
+ <scope>provided</scope>
+ </dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-api</artifactId>
diff --git
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactory.java
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactory.java
index 6e5ff22..94fed63 100644
---
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactory.java
+++
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactory.java
@@ -22,12 +22,15 @@ import static java.util.Objects.requireNonNull;
import java.io.IOException;
import java.io.UncheckedIOException;
+import java.lang.management.ManagementFactory;
import java.net.URISyntaxException;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.Path;
import java.time.Duration;
+import java.util.ArrayList;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executors;
@@ -36,8 +39,16 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Supplier;
+import javax.management.InstanceAlreadyExistsException;
+import javax.management.MBeanRegistrationException;
+import javax.management.MBeanServer;
+import javax.management.MalformedObjectNameException;
+import javax.management.NotCompliantMBeanException;
+
import org.apache.accumulo.classloader.lcc.definition.ContextDefinition;
import org.apache.accumulo.classloader.lcc.definition.Resource;
+import org.apache.accumulo.classloader.lcc.jmx.ContextClassLoaders;
+import org.apache.accumulo.classloader.lcc.jmx.ContextClassLoadersMXBean;
import org.apache.accumulo.classloader.lcc.util.DeduplicationCache;
import org.apache.accumulo.classloader.lcc.util.LccUtils;
import org.apache.accumulo.classloader.lcc.util.LocalStore;
@@ -97,7 +108,7 @@ public class LocalCachingContextClassLoaderFactory
implements ContextClassLoader
// to keep this coherent with the contextDefs, updates to this should be
done in the compute
// method of contextDefs
- private final DeduplicationCache<String,URL[],URLClassLoader> classloaders =
+ private static final DeduplicationCache<String,URL[],URLClassLoader>
classloaders =
new DeduplicationCache<>(LccUtils::createClassLoader,
Duration.ofHours(24), null);
private final AtomicReference<LocalStore> localStore = new
AtomicReference<>();
@@ -144,6 +155,16 @@ public class LocalCachingContextClassLoaderFactory
implements ContextClassLoader
throw new UncheckedIOException("Unable to create the local storage area
at " + baseCacheDir,
e);
}
+ try {
+ MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
+ mbs.registerMBean(new ContextClassLoaders(),
ContextClassLoadersMXBean.getObjectName());
+ } catch (MalformedObjectNameException | MBeanRegistrationException
+ | NotCompliantMBeanException e) {
+ throw new IllegalStateException("Error registering MBean", e);
+ } catch (InstanceAlreadyExistsException e) {
+ // instance was re-init'd. This is likely to happen during tests
+ // can ignore as no issue here
+ }
}
@Override
@@ -279,4 +300,16 @@ public class LocalCachingContextClassLoaderFactory
implements ContextClassLoader
}
+ public static Map<String,List<String>> getReferencedFiles() {
+ final Map<String,List<String>> referencedContexts = new HashMap<>();
+ classloaders.forEach((cacheKey, cl) -> {
+ List<String> files = new ArrayList<>();
+ for (URL u : cl.getURLs()) {
+ files.add(u.toString());
+ }
+ referencedContexts.put(cacheKey, files);
+ });
+ return referencedContexts;
+ }
+
}
diff --git
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/ContextDefinition.java
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/ContextDefinition.java
index 356bde8..8c13fee 100644
---
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/ContextDefinition.java
+++
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/ContextDefinition.java
@@ -28,21 +28,41 @@ import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URL;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashSet;
+import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.function.Supplier;
import org.apache.accumulo.classloader.lcc.resolvers.FileResolver;
+import org.apache.accumulo.core.cli.Help;
+import org.apache.accumulo.start.spi.KeywordExecutable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
+import com.beust.jcommander.Parameter;
+import com.google.auto.service.AutoService;
import com.google.common.base.Preconditions;
import com.google.common.base.Suppliers;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
-public class ContextDefinition {
+@AutoService(KeywordExecutable.class)
+public class ContextDefinition implements KeywordExecutable {
+ static class Opts extends Help {
+ @Parameter(names = {"-i", "--interval"}, required = true,
+ description = "monitor interval (in seconds)", arity = 1, order = 1)
+ int monitorInterval;
+
+ @Parameter(required = true, description = "classpath element URL (<url>[
<url>...])",
+ arity = -1, order = 2)
+ public List<String> files = new ArrayList<>();
+ }
+
+ // pretty-print uses Unix newline
private static final Gson GSON =
new GsonBuilder().disableJdkUnsafe().setPrettyPrinting().create();
@@ -122,6 +142,34 @@ public class ContextDefinition {
}
public String toJson() {
- return GSON.toJson(this);
+ // GSON pretty print uses Unix line-endings, and may or may not have a
trailing one, so
+ // ensure a trailing one exists, so it's included in checksum computations
and in
+ // any files written from this (for better readability)
+ return GSON.toJson(this).stripTrailing() + "\n";
+ }
+
+ @Override
+ public String keyword() {
+ return "create-context-definition";
+ }
+
+ @Override
+ public String description() {
+ return "Creates and prints a Context Definition";
+ }
+
+ @Override
+ public void execute(String[] args) throws Exception {
+ URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory(new
Configuration()));
+
+ Opts opts = new Opts();
+ opts.parseArgs(ContextDefinition.class.getName(), args);
+ URL[] urls = new URL[opts.files.size()];
+ int count = 0;
+ for (String f : opts.files) {
+ urls[count++] = new URL(f);
+ }
+ ContextDefinition def = create(opts.monitorInterval, urls);
+ System.out.print(def.toJson());
}
}
diff --git
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/jmx/ContextClassLoaders.java
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/jmx/ContextClassLoaders.java
new file mode 100644
index 0000000..4a2a0a9
--- /dev/null
+++
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/jmx/ContextClassLoaders.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.lcc.jmx;
+
+import java.util.List;
+import java.util.Map;
+
+import
org.apache.accumulo.classloader.lcc.LocalCachingContextClassLoaderFactory;
+
+public class ContextClassLoaders implements ContextClassLoadersMXBean {
+
+ @Override
+ public Map<String,List<String>> getReferencedFiles() {
+ return LocalCachingContextClassLoaderFactory.getReferencedFiles();
+ }
+
+}
diff --git
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/jmx/ContextClassLoadersMXBean.java
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/jmx/ContextClassLoadersMXBean.java
new file mode 100644
index 0000000..753f3b9
--- /dev/null
+++
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/jmx/ContextClassLoadersMXBean.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.lcc.jmx;
+
+import java.util.List;
+import java.util.Map;
+
+import javax.management.MalformedObjectNameException;
+import javax.management.ObjectName;
+
+public interface ContextClassLoadersMXBean {
+
+ static ObjectName getObjectName() throws MalformedObjectNameException {
+ return new
ObjectName("org.apache.accumulo.classloader:type=ContextClassLoaders");
+ }
+
+ Map<String,List<String>> getReferencedFiles();
+
+}
diff --git
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/DeduplicationCache.java
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/DeduplicationCache.java
index 54b3a1f..4edaa46 100644
---
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/DeduplicationCache.java
+++
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/DeduplicationCache.java
@@ -21,6 +21,7 @@ package org.apache.accumulo.classloader.lcc.util;
import static java.util.Objects.requireNonNull;
import java.time.Duration;
+import java.util.function.BiConsumer;
import java.util.function.BiFunction;
import java.util.function.Predicate;
import java.util.function.Supplier;
@@ -75,4 +76,13 @@ public class DeduplicationCache<KEY,PARAMS,VALUE> {
return
canonicalWeakValuesCache.asMap().keySet().stream().anyMatch(keyPredicate);
}
+ public void forEach(final BiConsumer<KEY,VALUE> consumer) {
+ canonicalWeakValuesCache.cleanUp();
+ canonicalWeakValuesCache.asMap().forEach((k, v) -> {
+ if (v != null) {
+ consumer.accept(k, v);
+ }
+ });
+ }
+
}
diff --git
a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/MiniAccumuloClusterClassLoaderFactoryTest.java
b/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/MiniAccumuloClusterClassLoaderFactoryTest.java
index deb259e..07909eb 100644
---
a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/MiniAccumuloClusterClassLoaderFactoryTest.java
+++
b/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/MiniAccumuloClusterClassLoaderFactoryTest.java
@@ -31,6 +31,7 @@ import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.io.File;
+import java.io.IOException;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Path;
@@ -42,13 +43,25 @@ import java.nio.file.attribute.PosixFilePermissions;
import java.util.Arrays;
import java.util.Collections;
import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.HashSet;
import java.util.List;
+import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import java.util.TreeSet;
import java.util.stream.Collectors;
+import javax.management.JMX;
+import javax.management.MalformedObjectNameException;
+import javax.management.remote.JMXConnectorFactory;
+import javax.management.remote.JMXServiceURL;
+
import org.apache.accumulo.classloader.lcc.definition.ContextDefinition;
+import org.apache.accumulo.classloader.lcc.definition.Resource;
+import org.apache.accumulo.classloader.lcc.jmx.ContextClassLoadersMXBean;
+import org.apache.accumulo.classloader.lcc.resolvers.FileResolver;
+import org.apache.accumulo.classloader.lcc.util.LocalStore;
import org.apache.accumulo.core.client.Accumulo;
import org.apache.accumulo.core.client.AccumuloClient;
import org.apache.accumulo.core.client.AccumuloException;
@@ -77,14 +90,24 @@ import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.sun.tools.attach.AttachNotSupportedException;
+import com.sun.tools.attach.VirtualMachine;
+import com.sun.tools.attach.VirtualMachineDescriptor;
public class MiniAccumuloClusterClassLoaderFactoryTest extends
SharedMiniClusterBase {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(MiniAccumuloClusterClassLoaderFactoryTest.class);
+
private static class TestMACConfiguration implements
MiniClusterConfigurationCallback {
@Override
public void configureMiniCluster(MiniAccumuloConfigImpl cfg,
org.apache.hadoop.conf.Configuration coreSite) {
+ cfg.getJvmOptions().add("-XX:-PerfDisableSharedMem");
cfg.setNumTservers(3);
cfg.setProperty(Property.TSERV_NATIVEMAP_ENABLED.getKey(), "false");
cfg.setProperty(Property.GENERAL_CONTEXT_CLASSLOADER_FACTORY.getKey(),
@@ -130,9 +153,9 @@ public class MiniAccumuloClusterClassLoaderFactoryTest
extends SharedMiniCluster
@Test
public void testClassLoader() throws Exception {
-
- var baseDirPath = tempDir.resolve("base");
- var jsonDirPath = baseDirPath.resolve("contextFiles");
+ final var baseDirPath = tempDir.resolve("base");
+ final var resourcesDirPath = baseDirPath.resolve("resources");
+ final var jsonDirPath = tempDir.resolve("simulatedRemoteContextFiles");
Files.createDirectory(jsonDirPath, PERMISSIONS);
// Create a context definition that only references jar A
@@ -143,6 +166,10 @@ public class MiniAccumuloClusterClassLoaderFactoryTest
extends SharedMiniCluster
Files.writeString(testContextDefFile.toPath(), testContextDefJson,
StandardOpenOption.CREATE);
assertTrue(Files.exists(testContextDefFile.toPath()));
+ Resource jarAResource = testContextDef.getResources().iterator().next();
+ String jarALocalFileName = LocalStore.localResourceName(
+ FileResolver.resolve(jarAResource.getLocation()).getFileName(),
jarAResource.getChecksum());
+
final String[] names = this.getUniqueNames(1);
try (AccumuloClient client =
Accumulo.newClient().from(getCluster().getClientProperties()).build())
{
@@ -198,6 +225,10 @@ public class MiniAccumuloClusterClassLoaderFactoryTest
extends SharedMiniCluster
// before applying the iterator
final byte[] jarAValueBytes = "foo".getBytes(UTF_8);
assertEquals(0, countExpectedValues(client, tableName, jarAValueBytes));
+ Set<String> refFiles = getReferencedFiles();
+ assertEquals(1, refFiles.size());
+ assertTrue(refFiles
+
.contains(resourcesDirPath.resolve(jarALocalFileName).toUri().toURL().toString()));
// Attach a scan iterator to the table
IteratorSetting is = new IteratorSetting(101, "example",
ITER_CLASS_NAME);
@@ -209,6 +240,10 @@ public class MiniAccumuloClusterClassLoaderFactoryTest
extends SharedMiniCluster
while (count != 1000) {
count = countExpectedValues(client, tableName, jarAValueBytes);
}
+ refFiles = getReferencedFiles();
+ assertEquals(1, refFiles.size());
+ assertTrue(refFiles
+
.contains(resourcesDirPath.resolve(jarALocalFileName).toUri().toURL().toString()));
// Update the context definition to point to jar B
final ContextDefinition testContextDefUpdate =
@@ -218,6 +253,11 @@ public class MiniAccumuloClusterClassLoaderFactoryTest
extends SharedMiniCluster
StandardOpenOption.TRUNCATE_EXISTING);
assertTrue(Files.exists(testContextDefFile.toPath()));
+ Resource jarBResource =
testContextDefUpdate.getResources().iterator().next();
+ String jarBLocalFileName = LocalStore.localResourceName(
+ FileResolver.resolve(jarBResource.getLocation()).getFileName(),
+ jarBResource.getChecksum());
+
// Wait 2x the monitor interval
Thread.sleep(2 * MONITOR_INTERVAL_SECS * 1000);
@@ -226,6 +266,12 @@ public class MiniAccumuloClusterClassLoaderFactoryTest
extends SharedMiniCluster
// by the iterator
final byte[] jarBValueBytes = "bar".getBytes(UTF_8);
assertEquals(1000, countExpectedValues(client, tableName,
jarBValueBytes));
+ refFiles = getReferencedFiles();
+ assertEquals(2, refFiles.size());
+ assertTrue(refFiles
+
.contains(resourcesDirPath.resolve(jarALocalFileName).toUri().toURL().toString()));
+ assertTrue(refFiles
+
.contains(resourcesDirPath.resolve(jarBLocalFileName).toUri().toURL().toString()));
// Copy jar A, create a context definition using the copy, then
// remove the copy so that it's not found when the context classloader
@@ -252,9 +298,15 @@ public class MiniAccumuloClusterClassLoaderFactoryTest
extends SharedMiniCluster
Thread.sleep(2 * MONITOR_INTERVAL_SECS * 1000);
// Rescan and confirm that all values get transformed to "bar"
- // by the iterator. The previous class is still being used after
+ // by the iterator. The previous classloader is still being used after
// the monitor interval because the jar referenced does not exist.
assertEquals(1000, countExpectedValues(client, tableName,
jarBValueBytes));
+ refFiles = getReferencedFiles();
+ assertEquals(2, refFiles.size());
+ assertTrue(refFiles
+
.contains(resourcesDirPath.resolve(jarALocalFileName).toUri().toURL().toString()));
+ assertTrue(refFiles
+
.contains(resourcesDirPath.resolve(jarBLocalFileName).toUri().toURL().toString()));
// Wait 2 minutes, 2 times the UPDATE_FAILURE_GRACE_PERIOD_MINS
Thread.sleep(120_000);
@@ -268,6 +320,38 @@ public class MiniAccumuloClusterClassLoaderFactoryTest
extends SharedMiniCluster
}
}
+ private Set<String> getReferencedFiles() {
+ final Map<String,List<String>> referencedFiles = new HashMap<>();
+ for (VirtualMachineDescriptor vmd : VirtualMachine.list()) {
+ if (vmd.displayName().contains("org.apache.accumulo.start.Main")
+ && !vmd.displayName().contains("zookeeper")) {
+ LOG.info("Attempting to connect to {}", vmd.displayName());
+ try {
+ var vm = VirtualMachine.attach(vmd);
+ String connectorAddress = vm.getAgentProperties()
+
.getProperty("com.sun.management.jmxremote.localConnectorAddress");
+ if (connectorAddress == null) {
+ connectorAddress = vm.startLocalManagementAgent();
+ }
+ var url = new JMXServiceURL(connectorAddress);
+ try (var connector = JMXConnectorFactory.connect(url)) {
+ var mbsc = connector.getMBeanServerConnection();
+ var proxy = JMX.newMXBeanProxy(mbsc,
ContextClassLoadersMXBean.getObjectName(),
+ ContextClassLoadersMXBean.class);
+ referencedFiles.putAll(proxy.getReferencedFiles());
+ }
+ } catch (MalformedObjectNameException | AttachNotSupportedException |
IOException e) {
+ LOG.error("Error getting referenced files from {}",
vmd.displayName(), e);
+ }
+ }
+ }
+ Set<String> justTheFiles = new HashSet<>();
+ referencedFiles.values().forEach(justTheFiles::addAll);
+ LOG.info("Referenced files with contexts: {}", referencedFiles);
+ LOG.info("Referenced files: {}", justTheFiles);
+ return justTheFiles;
+ }
+
private int countExpectedValues(AccumuloClient client, String table, byte[]
expectedValue)
throws TableNotFoundException, AccumuloSecurityException,
AccumuloException {
Scanner scanner = client.createScanner(table);
diff --git a/pom.xml b/pom.xml
index 1829a88..0335e9d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -135,7 +135,7 @@ under the License.
<!-- most dependencies will be provided by the accumulo installation
-->
<groupId>org.apache.accumulo</groupId>
<artifactId>accumulo-project</artifactId>
- <version>2.1.4</version>
+ <version>2.1.5-SNAPSHOT</version>
<type>pom</type>
<scope>import</scope>
</dependency>
@@ -367,7 +367,13 @@ under the License.
</goals>
<configuration>
<failOnWarning>true</failOnWarning>
+ <ignoredUsedUndeclaredDependencies>
+ <!-- auto-service-annotations is transitive via auto-service
-->
+
<undeclared>com.google.auto.service:auto-service-annotations:jar:*</undeclared>
+ </ignoredUsedUndeclaredDependencies>
<ignoredUnusedDeclaredDependencies>
+ <!-- auto-service used by the compiler for annotation
processing, not by code -->
+ <unused>com.google.auto.service:auto-service:jar:*</unused>
<!-- ignore false positive runtime dependencies -->
<unused>org.apache.commons:commons-vfs2-hdfs:*</unused>
<unused>org.apache.httpcomponents.client5:httpclient5:*</unused>
@@ -419,7 +425,12 @@ under the License.
<reactorModuleConvergence />
<banDuplicatePomDependencyVersions />
<dependencyConvergence />
- <banDynamicVersions />
+ <banDynamicVersions>
+ <ignores>
+ <!-- allow Accumulo snapshots -->
+ <ignore>org.apache.accumulo:*:*-SNAPSHOT</ignore>
+ </ignores>
+ </banDynamicVersions>
<bannedDependencies>
<excludes>
<!-- we redirect logging to log4j2, so we should have
those bridges instead -->
