This is an automated email from the ASF dual-hosted git repository.
ctubbsii pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-classloaders.git
The following commit(s) were added to refs/heads/main by this push:
new 438b16b Replace resolvers with agnostic URL support (#51)
438b16b is described below
commit 438b16b54f645240b97b99888a71d71810cdea8b
Author: Christopher Tubbs <[email protected]>
AuthorDate: Wed Jan 21 20:58:05 2026 -0500
Replace resolvers with agnostic URL support (#51)
Closes #47
* Register new URLStreamHandlerProvider
* Register a provider for the hdfs: URL stream handling, rather than
override the system factory
* Also fix SpotBugs and rename classes to match the naming convention
from the classes they override (e.g. "URLStreamHandler" instead of
"UrlStreamHandler")
* Remove unneeded call to constructor in test
* Replace resolvers with agnostic URL support
* Remove resolvers (keep test coverage for file, http, and hdfs URL
types)
* Use url.openStream()
* Be agnostic to different URL types
* Depend on an HdfsURLStreamHandlerProvider for hdfs URL support
* Place hdfs provider in its own module/jar
* Clean up related POM and README stuff
This fixes #47
This supersedes #48 and #49 alternate fixes
* Remove redundant precondition
* Fix test and remove redundant log
* Make the DeduplicationCacheTest more strict, and keep a strong
reference in a background thread, so the weak values don't get garbage
collected too early
* Use Object to test DeduplicationCache, since the values are not
actually used
* Move a log message to avoid a redundant message about discontinued
monitoring of a ContextDefinition location
* Be less aggressive with System.gc in test
---------
Co-authored-by: Dave Marion <[email protected]>
---
modules/hdfs-urlstreamhandler-provider/.gitignore | 36 ++++++++
modules/hdfs-urlstreamhandler-provider/README.md | 31 +++++++
modules/hdfs-urlstreamhandler-provider/pom.xml | 62 ++++++++++++++
.../HdfsURLStreamHandlerProvider.java | 96 ++++++++++++++++++++++
modules/local-caching-classloader/README.md | 13 ++-
modules/local-caching-classloader/pom.xml | 33 ++++----
.../lcc/LocalCachingContextClassLoaderFactory.java | 11 +--
.../lcc/definition/ContextDefinition.java | 17 ++--
.../classloader/lcc/definition/Resource.java | 13 +++
.../classloader/lcc/resolvers/FileResolver.java | 80 ------------------
.../lcc/resolvers/HdfsFileResolver.java | 60 --------------
.../lcc/resolvers/HttpFileResolver.java | 45 ----------
.../lcc/resolvers/LocalFileResolver.java | 62 --------------
.../accumulo/classloader/lcc/util/LocalStore.java | 21 +++--
.../LocalCachingContextClassLoaderFactoryTest.java | 41 +++++----
.../MiniAccumuloClusterClassLoaderFactoryTest.java | 8 +-
.../apache/accumulo/classloader/lcc/TestUtils.java | 18 +++-
.../FileResolversTest.java => URLTypesTest.java} | 59 ++++---------
.../lcc/util/DeduplicationCacheTest.java | 68 ++++++++-------
.../classloader/lcc/util/LocalStoreTest.java | 70 +++++++++-------
modules/vfs-class-loader/pom.xml | 19 +++++
pom.xml | 12 ++-
22 files changed, 451 insertions(+), 424 deletions(-)
diff --git a/modules/hdfs-urlstreamhandler-provider/.gitignore
b/modules/hdfs-urlstreamhandler-provider/.gitignore
new file mode 100644
index 0000000..55d7f58
--- /dev/null
+++ b/modules/hdfs-urlstreamhandler-provider/.gitignore
@@ -0,0 +1,36 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Maven ignores
+/target/
+
+# IDE ignores
+/.settings/
+/.project
+/.classpath
+/.pydevproject
+/.idea
+/*.iml
+/*.ipr
+/*.iws
+/nbproject/
+/nbactions.xml
+/nb-configuration.xml
+/.vscode/
+/.factorypath
diff --git a/modules/hdfs-urlstreamhandler-provider/README.md
b/modules/hdfs-urlstreamhandler-provider/README.md
new file mode 100644
index 0000000..a3b27ea
--- /dev/null
+++ b/modules/hdfs-urlstreamhandler-provider/README.md
@@ -0,0 +1,31 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# URLStreamHandlerProvider
+
+This library contains a single class that implements
+[URLStreamHandlerProvider][1], to provide a stream handler for `hdfs:` URL
+types that uses a default Hadoop `Configuration` from your class path. This can
+be placed on your classpath to automatically be registered using the Java
+`ServiceLoader` when handling `hdfs:` URLs. This libary is unnecessary if
+another library on your classpath provides an equivalent handler.
+
+
+[1]:
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/spi/URLStreamHandlerProvider.html
diff --git a/modules/hdfs-urlstreamhandler-provider/pom.xml
b/modules/hdfs-urlstreamhandler-provider/pom.xml
new file mode 100644
index 0000000..fb6114d
--- /dev/null
+++ b/modules/hdfs-urlstreamhandler-provider/pom.xml
@@ -0,0 +1,62 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
https://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.accumulo</groupId>
+ <artifactId>classloader-extras</artifactId>
+ <version>1.0.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+ <artifactId>hdfs-urlstreamhandler-provider</artifactId>
+ <name>HDFS URLStreamHandlerProvider</name>
+ <dependencies>
+ <dependency>
+ <!-- needed for build checks, but not for runtime -->
+ <groupId>com.github.spotbugs</groupId>
+ <artifactId>spotbugs-annotations</artifactId>
+ <optional>true</optional>
+ </dependency>
+ <dependency>
+ <!-- needed for annotation processor during compile, but not after -->
+ <groupId>com.google.auto.service</groupId>
+ <artifactId>auto-service</artifactId>
+ <optional>true</optional>
+ </dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-api</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <!-- provided by accumulo -->
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
+</project>
diff --git
a/modules/hdfs-urlstreamhandler-provider/src/main/java/org/apache/accumulo/classloader/hdfsurlstreamhandlerprovider/HdfsURLStreamHandlerProvider.java
b/modules/hdfs-urlstreamhandler-provider/src/main/java/org/apache/accumulo/classloader/hdfsurlstreamhandlerprovider/HdfsURLStreamHandlerProvider.java
new file mode 100644
index 0000000..aff19e1
--- /dev/null
+++
b/modules/hdfs-urlstreamhandler-provider/src/main/java/org/apache/accumulo/classloader/hdfsurlstreamhandlerprovider/HdfsURLStreamHandlerProvider.java
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.classloader.hdfsurlstreamhandlerprovider;
+
+import static java.util.Objects.requireNonNull;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLConnection;
+import java.net.URLStreamHandler;
+import java.net.spi.URLStreamHandlerProvider;
+import java.util.function.Supplier;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.auto.service.AutoService;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Suppliers;
+
+@AutoService(URLStreamHandlerProvider.class)
+public class HdfsURLStreamHandlerProvider extends URLStreamHandlerProvider {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(HdfsURLStreamHandlerProvider.class);
+
+ private static class HdfsURLConnection extends URLConnection {
+
+ private final Supplier<Configuration> conf =
Suppliers.memoize(Configuration::new);
+
+ private InputStream is;
+
+ public HdfsURLConnection(URL url) {
+ super(requireNonNull(url, "null url argument"));
+ }
+
+ @Override
+ public void connect() throws IOException {
+ Preconditions.checkState(is == null, "Already connected");
+ LOG.debug("Connecting to {}", url);
+ final URI uri;
+ try {
+ uri = url.toURI();
+ } catch (URISyntaxException e) {
+ // this came from a URL that should be RFC2396-compliant, so it
shouldn't happen
+ throw new IllegalArgumentException(e);
+ }
+ var fs = FileSystem.get(uri, conf.get());
+ is = fs.open(new Path(uri));
+ }
+
+ @Override
+ public InputStream getInputStream() throws IOException {
+ if (is == null) {
+ connect();
+ }
+ return is;
+ }
+ }
+
+ private static class HdfsURLStreamHandler extends URLStreamHandler {
+ @Override
+ protected HdfsURLConnection openConnection(URL url) throws IOException {
+ return new HdfsURLConnection(url);
+ }
+ }
+
+ private final Supplier<URLStreamHandler> handler =
Suppliers.memoize(HdfsURLStreamHandler::new);
+
+ @Override
+ public URLStreamHandler createURLStreamHandler(String protocol) {
+ return protocol.equals("hdfs") ? handler.get() : null;
+ }
+
+}
diff --git a/modules/local-caching-classloader/README.md
b/modules/local-caching-classloader/README.md
index cb5589c..8aecacf 100644
--- a/modules/local-caching-classloader/README.md
+++ b/modules/local-caching-classloader/README.md
@@ -47,8 +47,12 @@ This context definition file must then be stored somewhere
where this factory
can download it, and use the URL to that context definition file as the
`context` parameter for this factory's `getClassLoader(String context)` method.
-This factory can handle context and resource URLs that use the `file`, `hdfs`,
-`http`, or `https` URL scheme.
+This factory can handle context and resource URLs of any type that are
+supported by your application via a registered [URLStreamHandlerProvider][1],
+such as the built-in `file:` and `http:` types. A provider that handles `hdfs:`
+URL types must be provided by the user. This may be provided by the Apache
+Hadoop project, or by another library. A reference implementation is available
+[elsewhere in this project][2].
Here is an example context definition file:
@@ -210,6 +214,9 @@ To use this with Accumulo:
1. Set the following Accumulo properties:
*
`general.context.class.loader.factory=org.apache.accumulo.classloader.lcc.LocalCachingContextClassLoaderFactory`
* `general.custom.classloader.lcc.cache.dir=file://path/to/some/directory`
-2. Set the following table property:
+2. Set the following table property to link to a context definition file. For
example:
*
`table.class.loader.context=(file|hdfs|http|https)://path/to/context/definition.json`
+
+[1]:
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/spi/URLStreamHandlerProvider.html
+[2]:
https://github.com/apache/accumulo-classloaders/tree/main/modules/hdfs-urlstreamhandler-provider
diff --git a/modules/local-caching-classloader/pom.xml
b/modules/local-caching-classloader/pom.xml
index 0428499..5f0ad98 100644
--- a/modules/local-caching-classloader/pom.xml
+++ b/modules/local-caching-classloader/pom.xml
@@ -29,21 +29,6 @@
</parent>
<artifactId>local-caching-classloader</artifactId>
<name>classloader-extras-local-caching</name>
- <properties>
- <accumulo.build.extraTestArgs>--add-opens java.base/java.lang=ALL-UNNAMED
--add-opens java.base/java.util=ALL-UNNAMED --add-opens
java.base/java.io=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED
--add-opens java.management/java.lang.management=ALL-UNNAMED --add-opens
java.management/sun.management=ALL-UNNAMED --add-opens
java.base/java.security=ALL-UNNAMED --add-opens
java.base/java.lang.reflect=ALL-UNNAMED --add-opens
java.base/java.util.concurrent=ALL-UNNAMED --add-opens ja [...]
-
<eclipseFormatterStyle>../../src/build/eclipse-codestyle.xml</eclipseFormatterStyle>
- <failsafe.excludedGroups />
- <failsafe.failIfNoSpecifiedTests>false</failsafe.failIfNoSpecifiedTests>
- <failsafe.forkCount>1</failsafe.forkCount>
- <failsafe.groups />
- <failsafe.reuseForks>false</failsafe.reuseForks>
-
<maven.test.redirectTestOutputToFile>true</maven.test.redirectTestOutputToFile>
- <surefire.excludedGroups />
- <surefire.failIfNoSpecifiedTests>false</surefire.failIfNoSpecifiedTests>
- <surefire.forkCount>1</surefire.forkCount>
- <surefire.groups />
- <surefire.reuseForks>false</surefire.reuseForks>
- </properties>
<dependencies>
<dependency>
<!-- needed for build checks, but not for runtime -->
@@ -110,6 +95,12 @@
<artifactId>slf4j-api</artifactId>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.accumulo</groupId>
+ <artifactId>hdfs-urlstreamhandler-provider</artifactId>
+ <scope>runtime</scope>
+ <optional>true</optional>
+ </dependency>
<dependency>
<groupId>org.apache.accumulo</groupId>
<artifactId>accumulo-minicluster</artifactId>
@@ -181,6 +172,18 @@
</artifactItems>
</configuration>
</execution>
+ <execution>
+ <id>analyze</id>
+ <goals>
+ <goal>analyze-only</goal>
+ </goals>
+ <configuration>
+ <ignoredUnusedDeclaredDependencies combine.children="append">
+ <!-- ignore false positive runtime dependencies -->
+
<unused>org.apache.accumulo:hdfs-urlstreamhandler-provider:*</unused>
+ </ignoredUnusedDeclaredDependencies>
+ </configuration>
+ </execution>
</executions>
</plugin>
<plugin>
diff --git
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactory.java
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactory.java
index 2c185a6..13d6a38 100644
---
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactory.java
+++
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactory.java
@@ -71,8 +71,8 @@ import com.google.common.base.Stopwatch;
* discontinue until the next use of that context. Each resource is defined by
a URL to the file and
* an expected SHA-256 checksum.
* <p>
- * The URLs supplied for the context definition file and for the resources can
use one of the
- * following URL schemes: file, http, https, or hdfs.
+ * The URLs supplied for the context definition file and for the resources may
use any URL type with
+ * a registered provider in your application, such as: file, http, https, or
hdfs.
* <p>
* As this class processes the ContextDefinition, it fetches the contents of
the resource from the
* resource URL and caches it in a directory on the local filesystem. This
class uses the value of
@@ -249,6 +249,10 @@ public class LocalCachingContextClassLoaderFactory
implements ContextClassLoader
ContextDefinition currentDef =
contextDefs.compute(contextLocation, (contextLocationKey,
previousDefinition) -> {
if (previousDefinition == null) {
+ // context has been removed from the map, no need to check for
update
+ LOG.debug(
+ "ContextDefinition for context {} not present, no longer
monitoring for changes",
+ contextLocation);
return null;
}
// check for any classloaders still in the cache that were created
for a context
@@ -262,9 +266,6 @@ public class LocalCachingContextClassLoaderFactory
implements ContextClassLoader
return previousDefinition;
});
if (currentDef == null) {
- // context has been removed from the map, no need to check for update
- LOG.debug("ContextDefinition for context {} not present, no longer
monitoring for changes",
- contextLocation);
return;
}
long nextInterval = interval;
diff --git
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/ContextDefinition.java
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/ContextDefinition.java
index 8c13fee..4278e25 100644
---
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/ContextDefinition.java
+++
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/ContextDefinition.java
@@ -36,11 +36,8 @@ import java.util.Objects;
import java.util.Set;
import java.util.function.Supplier;
-import org.apache.accumulo.classloader.lcc.resolvers.FileResolver;
import org.apache.accumulo.core.cli.Help;
import org.apache.accumulo.start.spi.KeywordExecutable;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
import com.beust.jcommander.Parameter;
import com.google.auto.service.AutoService;
@@ -49,6 +46,8 @@ import com.google.common.base.Suppliers;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
@AutoService(KeywordExecutable.class)
public class ContextDefinition implements KeywordExecutable {
@@ -66,12 +65,13 @@ public class ContextDefinition implements KeywordExecutable
{
private static final Gson GSON =
new GsonBuilder().disableJdkUnsafe().setPrettyPrinting().create();
+ @SuppressFBWarnings(value = "URLCONNECTION_SSRF_FD",
+ justification = "user-supplied URL is the intended functionality")
public static ContextDefinition create(int monitorIntervalSecs, URL...
sources)
throws IOException {
LinkedHashSet<Resource> resources = new LinkedHashSet<>();
for (URL u : sources) {
- FileResolver resolver = FileResolver.resolve(u);
- try (InputStream is = resolver.getInputStream()) {
+ try (InputStream is = u.openStream()) {
String checksum = DIGESTER.digestAsHex(is);
resources.add(new Resource(u, checksum));
}
@@ -79,9 +79,10 @@ public class ContextDefinition implements KeywordExecutable {
return new ContextDefinition(monitorIntervalSecs, resources);
}
+ @SuppressFBWarnings(value = "URLCONNECTION_SSRF_FD",
+ justification = "user-supplied URL is the intended functionality")
public static ContextDefinition fromRemoteURL(final URL url) throws
IOException {
- final FileResolver resolver = FileResolver.resolve(url);
- try (InputStream is = resolver.getInputStream()) {
+ try (InputStream is = url.openStream()) {
var def = GSON.fromJson(new InputStreamReader(is, UTF_8),
ContextDefinition.class);
if (def == null) {
throw new EOFException("InputStream does not contain a valid
ContextDefinition at " + url);
@@ -160,8 +161,6 @@ public class ContextDefinition implements KeywordExecutable
{
@Override
public void execute(String[] args) throws Exception {
- URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory(new
Configuration()));
-
Opts opts = new Opts();
opts.parseArgs(ContextDefinition.class.getName(), args);
URL[] urls = new URL[opts.files.size()];
diff --git
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/Resource.java
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/Resource.java
index d31694b..70afdde 100644
---
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/Resource.java
+++
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/definition/Resource.java
@@ -19,6 +19,7 @@
package org.apache.accumulo.classloader.lcc.definition;
import java.net.URL;
+import java.nio.file.Path;
import java.util.Objects;
public class Resource {
@@ -37,6 +38,18 @@ public class Resource {
return location;
}
+ public String getFileName() {
+ var nameAsPath = Path.of(location.getPath()).getFileName();
+ if (nameAsPath == null) {
+ return "unknown";
+ }
+ String name = nameAsPath.toString();
+ if (name.isBlank()) {
+ return "unknown";
+ }
+ return name;
+ }
+
public String getChecksum() {
return checksum;
}
diff --git
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/resolvers/FileResolver.java
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/resolvers/FileResolver.java
deleted file mode 100644
index e32db52..0000000
---
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/resolvers/FileResolver.java
+++ /dev/null
@@ -1,80 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * https://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.accumulo.classloader.lcc.resolvers;
-
-import static java.util.Objects.hash;
-import static java.util.Objects.requireNonNull;
-
-import java.io.IOException;
-import java.io.InputStream;
-import java.net.URL;
-import java.util.Objects;
-
-public abstract class FileResolver {
-
- public static FileResolver resolve(URL url) throws IOException {
- requireNonNull(url, "URL must be supplied");
- switch (url.getProtocol()) {
- case "http":
- case "https":
- return new HttpFileResolver(url);
- case "file":
- return new LocalFileResolver(url);
- case "hdfs":
- return new HdfsFileResolver(url);
- default:
- throw new IOException("Unhandled protocol: " + url.getProtocol());
- }
- }
-
- private final URL url;
-
- protected FileResolver(URL url) {
- this.url = url;
- }
-
- protected URL getURL() {
- return this.url;
- }
-
- public abstract String getFileName();
-
- public abstract InputStream getInputStream() throws IOException;
-
- @Override
- public int hashCode() {
- return hash(url);
- }
-
- @Override
- public boolean equals(Object obj) {
- if (this == obj) {
- return true;
- }
- if (obj == null) {
- return false;
- }
- if (getClass() != obj.getClass()) {
- return false;
- }
- FileResolver other = (FileResolver) obj;
- return Objects.equals(url, other.url);
- }
-
-}
diff --git
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/resolvers/HdfsFileResolver.java
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/resolvers/HdfsFileResolver.java
deleted file mode 100644
index 6eda92d..0000000
---
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/resolvers/HdfsFileResolver.java
+++ /dev/null
@@ -1,60 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * https://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.accumulo.classloader.lcc.resolvers;
-
-import java.io.IOException;
-import java.io.InputStream;
-import java.net.URI;
-import java.net.URISyntaxException;
-import java.net.URL;
-
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-
-public final class HdfsFileResolver extends FileResolver {
-
- private final Configuration hadoopConf = new Configuration();
- private final FileSystem fs;
- private final Path path;
-
- protected HdfsFileResolver(URL url) throws IOException {
- super(url);
- try {
- final URI uri = url.toURI();
- this.fs = FileSystem.get(uri, hadoopConf);
- this.path = fs.makeQualified(new Path(uri));
- if (!fs.exists(this.path)) {
- throw new IOException("File: " + url + " does not exist.");
- }
- } catch (URISyntaxException e) {
- throw new IOException("Error creating URI from url: " + url, e);
- }
- }
-
- @Override
- public String getFileName() {
- return this.path.getName();
- }
-
- @Override
- public InputStream getInputStream() throws IOException {
- return fs.open(path);
- }
-}
diff --git
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/resolvers/HttpFileResolver.java
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/resolvers/HttpFileResolver.java
deleted file mode 100644
index 68da39c..0000000
---
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/resolvers/HttpFileResolver.java
+++ /dev/null
@@ -1,45 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * https://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.accumulo.classloader.lcc.resolvers;
-
-import java.io.IOException;
-import java.io.InputStream;
-import java.net.URL;
-
-import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
-
-public final class HttpFileResolver extends FileResolver {
-
- protected HttpFileResolver(URL url) throws IOException {
- super(url);
- }
-
- @Override
- public String getFileName() {
- String path = getURL().getPath();
- return path.substring(path.lastIndexOf("/") + 1);
- }
-
- @Override
- @SuppressFBWarnings(value = "URLCONNECTION_SSRF_FD",
- justification = "user-supplied URL is the intended functionality")
- public InputStream getInputStream() throws IOException {
- return getURL().openStream();
- }
-}
diff --git
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/resolvers/LocalFileResolver.java
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/resolvers/LocalFileResolver.java
deleted file mode 100644
index ece5914..0000000
---
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/resolvers/LocalFileResolver.java
+++ /dev/null
@@ -1,62 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * https://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.accumulo.classloader.lcc.resolvers;
-
-import java.io.BufferedInputStream;
-import java.io.File;
-import java.io.IOException;
-import java.io.InputStream;
-import java.net.URI;
-import java.net.URISyntaxException;
-import java.net.URL;
-import java.nio.file.Files;
-import java.nio.file.Path;
-
-public final class LocalFileResolver extends FileResolver {
-
- private final File file;
-
- public LocalFileResolver(URL url) throws IOException {
- super(url);
- if (url.getHost() != null && !url.getHost().isBlank()) {
- throw new IOException(
- "Unsupported file url, only local files are supported. host = " +
url.getHost());
- }
- try {
- final URI uri = url.toURI();
- final Path path = Path.of(uri);
- if (Files.notExists(Path.of(uri))) {
- throw new IOException("File: " + url + " does not exist.");
- }
- file = path.toFile();
- } catch (URISyntaxException e) {
- throw new IOException("Error creating URI from url: " + url, e);
- }
- }
-
- @Override
- public String getFileName() {
- return file.getName();
- }
-
- @Override
- public InputStream getInputStream() throws IOException {
- return new BufferedInputStream(Files.newInputStream(file.toPath()));
- }
-}
diff --git
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/LocalStore.java
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/LocalStore.java
index 96e1ed5..329fac4 100644
---
a/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/LocalStore.java
+++
b/modules/local-caching-classloader/src/main/java/org/apache/accumulo/classloader/lcc/util/LocalStore.java
@@ -50,10 +50,11 @@ import java.util.regex.Pattern;
import org.apache.accumulo.classloader.lcc.definition.ContextDefinition;
import org.apache.accumulo.classloader.lcc.definition.Resource;
-import org.apache.accumulo.classloader.lcc.resolvers.FileResolver;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
/**
* A simple storage service backed by a local file system for storing
downloaded
* {@link ContextDefinition} files and the {@link Resource} objects it
references.
@@ -112,9 +113,10 @@ public final class LocalStore {
// extension, and will instead just append the checksum to the original file
name
private static Pattern fileNamesWithExtensionPattern =
Pattern.compile("^(.*[^.].*)[.]([^.]+)$");
- public static String localResourceName(String remoteFileName, String
checksum) {
- requireNonNull(remoteFileName);
- requireNonNull(checksum);
+ public static String localResourceName(Resource r) {
+ requireNonNull(r);
+ String remoteFileName = r.getFileName();
+ String checksum = r.getChecksum();
var matcher = fileNamesWithExtensionPattern.matcher(remoteFileName);
if (matcher.matches()) {
return String.format("%s-%s.%s", matcher.group(1), checksum,
matcher.group(2));
@@ -195,8 +197,7 @@ public final class LocalStore {
*/
private Path storeResource(final Resource resource) throws IOException {
final URL url = resource.getLocation();
- final FileResolver source = FileResolver.resolve(url);
- final String baseName = localResourceName(source.getFileName(),
resource.getChecksum());
+ final String baseName = localResourceName(resource);
final Path destinationPath = resourcesDir.resolve(baseName);
final Path tempPath = resourcesDir.resolve(tempName(baseName));
final Path downloadingProgressPath = resourcesDir.resolve("." + baseName +
".downloading");
@@ -232,7 +233,7 @@ public final class LocalStore {
return null;
}
- var task = new FutureTask<Void>(() -> downloadFile(source, tempPath,
resource), null);
+ var task = new FutureTask<Void>(() -> downloadFile(tempPath, resource),
null);
var t = new Thread(task);
t.setDaemon(true);
t.setName("downloading " + url + " to " + tempPath);
@@ -292,11 +293,13 @@ public final class LocalStore {
private static final int DL_BUFF_SIZE = 1024 * 1024;
- private void downloadFile(FileResolver source, Path tempPath, Resource
resource) {
+ @SuppressFBWarnings(value = "URLCONNECTION_SSRF_FD",
+ justification = "user-supplied URL is the intended functionality")
+ private void downloadFile(Path tempPath, Resource resource) {
// CREATE_NEW ensures the temporary file name is unique for this attempt
// SYNC ensures file integrity on each write, in case of system failure.
Buffering minimizes
// system calls te read/write data which minimizes the number of syncs.
- try (var in = new BufferedInputStream(source.getInputStream(),
DL_BUFF_SIZE);
+ try (var in = new BufferedInputStream(resource.getLocation().openStream(),
DL_BUFF_SIZE);
var out = new BufferedOutputStream(Files.newOutputStream(tempPath,
CREATE_NEW, WRITE, SYNC),
DL_BUFF_SIZE)) {
in.transferTo(out);
diff --git
a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactoryTest.java
b/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactoryTest.java
index 47bd6d7..8a0cb31 100644
---
a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactoryTest.java
+++
b/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/LocalCachingContextClassLoaderFactoryTest.java
@@ -33,7 +33,7 @@ import static org.junit.jupiter.api.Assertions.fail;
import java.io.EOFException;
import java.io.File;
-import java.io.IOException;
+import java.io.FileNotFoundException;
import java.io.UncheckedIOException;
import java.net.MalformedURLException;
import java.net.URL;
@@ -52,13 +52,11 @@ import java.util.stream.Collectors;
import org.apache.accumulo.classloader.lcc.TestUtils.TestClassInfo;
import org.apache.accumulo.classloader.lcc.definition.ContextDefinition;
import org.apache.accumulo.classloader.lcc.definition.Resource;
-import org.apache.accumulo.classloader.lcc.resolvers.FileResolver;
import org.apache.accumulo.classloader.lcc.util.LocalStore;
import org.apache.accumulo.core.conf.ConfigurationCopy;
import
org.apache.accumulo.core.spi.common.ContextClassLoaderFactory.ContextClassLoaderException;
import org.apache.accumulo.core.util.ConfigurationImpl;
import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.eclipse.jetty.server.Server;
import org.junit.jupiter.api.AfterAll;
@@ -115,7 +113,6 @@ public class LocalCachingContextClassLoaderFactoryTest {
// Put B into HDFS
hdfs = TestUtils.getMiniCluster();
- URL.setURLStreamHandlerFactory(new
FsUrlStreamHandlerFactory(hdfs.getConfiguration(0)));
fs = hdfs.getFileSystem();
assertTrue(fs.mkdirs(new org.apache.hadoop.fs.Path("/contextB")));
@@ -277,7 +274,16 @@ public class LocalCachingContextClassLoaderFactoryTest {
var ex = assertThrows(ContextClassLoaderException.class,
() -> FACTORY.getClassLoader(initialDefUrl.toString()));
- assertTrue(ex.getMessage().endsWith("jarACopy.jar does not exist."),
ex::getMessage);
+ boolean foundExpectedException = false;
+ var cause = ex.getCause();
+ do {
+ foundExpectedException =
+ cause instanceof FileNotFoundException &&
cause.getMessage().contains("jarACopy.jar");
+ if (cause != null) {
+ cause = cause.getCause();
+ }
+ } while (!foundExpectedException && cause != null);
+ assertTrue(foundExpectedException, "Could not find expected
FileNotFoundException");
}
@Test
@@ -714,8 +720,16 @@ public class LocalCachingContextClassLoaderFactoryTest {
var ex = assertThrows(ContextClassLoaderException.class,
() -> localFactory.getClassLoader(updateDefUrl.toString()));
- assertTrue(ex.getMessage().endsWith("jarACopy.jar does not exist."),
ex::getMessage);
-
+ boolean foundExpectedException = false;
+ var cause = ex.getCause();
+ do {
+ foundExpectedException =
+ cause instanceof FileNotFoundException &&
cause.getMessage().contains("jarACopy.jar");
+ if (cause != null) {
+ cause = cause.getCause();
+ }
+ } while (!foundExpectedException && cause != null);
+ assertTrue(foundExpectedException, "Could not find expected
FileNotFoundException");
}
@Test
@@ -733,16 +747,9 @@ public class LocalCachingContextClassLoaderFactoryTest {
testClassLoads(cl, classD);
var resources = tempDir.resolve("base").resolve("resources");
- List<Path> files = def.getResources().stream().map(r -> {
- String basename;
- try {
- basename = FileResolver.resolve(r.getLocation()).getFileName();
- } catch (IOException e) {
- throw new UncheckedIOException(e);
- }
- var checksum = r.getChecksum();
- return resources.resolve(LocalStore.localResourceName(basename,
checksum));
- }).limit(2).collect(Collectors.toList());
+ List<Path> files =
+ def.getResources().stream().map(r ->
resources.resolve(LocalStore.localResourceName(r)))
+ .limit(2).collect(Collectors.toList());
assertEquals(2, files.size());
// overwrite one downloaded jar with others content
diff --git
a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/MiniAccumuloClusterClassLoaderFactoryTest.java
b/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/MiniAccumuloClusterClassLoaderFactoryTest.java
index 07909eb..b364354 100644
---
a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/MiniAccumuloClusterClassLoaderFactoryTest.java
+++
b/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/MiniAccumuloClusterClassLoaderFactoryTest.java
@@ -60,7 +60,6 @@ import javax.management.remote.JMXServiceURL;
import org.apache.accumulo.classloader.lcc.definition.ContextDefinition;
import org.apache.accumulo.classloader.lcc.definition.Resource;
import org.apache.accumulo.classloader.lcc.jmx.ContextClassLoadersMXBean;
-import org.apache.accumulo.classloader.lcc.resolvers.FileResolver;
import org.apache.accumulo.classloader.lcc.util.LocalStore;
import org.apache.accumulo.core.client.Accumulo;
import org.apache.accumulo.core.client.AccumuloClient;
@@ -167,8 +166,7 @@ public class MiniAccumuloClusterClassLoaderFactoryTest
extends SharedMiniCluster
assertTrue(Files.exists(testContextDefFile.toPath()));
Resource jarAResource = testContextDef.getResources().iterator().next();
- String jarALocalFileName = LocalStore.localResourceName(
- FileResolver.resolve(jarAResource.getLocation()).getFileName(),
jarAResource.getChecksum());
+ String jarALocalFileName = LocalStore.localResourceName(jarAResource);
final String[] names = this.getUniqueNames(1);
try (AccumuloClient client =
@@ -254,9 +252,7 @@ public class MiniAccumuloClusterClassLoaderFactoryTest
extends SharedMiniCluster
assertTrue(Files.exists(testContextDefFile.toPath()));
Resource jarBResource =
testContextDefUpdate.getResources().iterator().next();
- String jarBLocalFileName = LocalStore.localResourceName(
- FileResolver.resolve(jarBResource.getLocation()).getFileName(),
- jarBResource.getChecksum());
+ String jarBLocalFileName = LocalStore.localResourceName(jarBResource);
// Wait 2x the monitor interval
Thread.sleep(2 * MONITOR_INTERVAL_SECS * 1000);
diff --git
a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/TestUtils.java
b/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/TestUtils.java
index ca0043f..d625a68 100644
---
a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/TestUtils.java
+++
b/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/TestUtils.java
@@ -30,8 +30,11 @@ import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URL;
+import java.nio.file.Files;
import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import org.apache.commons.io.IOUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
@@ -173,9 +176,18 @@ public class TestUtils {
}
}
- public static String getFileName(URL url) {
- String path = url.getPath();
- return path.substring(path.lastIndexOf("/") + 1);
+ @SuppressFBWarnings(value = "URLCONNECTION_SSRF_FD",
+ justification = "user-supplied URL is the intended functionality")
+ public static long getFileSize(URL url) throws IOException {
+ try (InputStream is = url.openStream()) {
+ return IOUtils.consume(is);
+ }
+ }
+ public static long getFileSize(Path p) throws IOException {
+ try (InputStream is = Files.newInputStream(p, StandardOpenOption.READ)) {
+ return IOUtils.consume(is);
+ }
}
+
}
diff --git
a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/resolvers/FileResolversTest.java
b/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/URLTypesTest.java
similarity index 55%
rename from
modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/resolvers/FileResolversTest.java
rename to
modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/URLTypesTest.java
index 8be9907..54a122f 100644
---
a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/resolvers/FileResolversTest.java
+++
b/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/URLTypesTest.java
@@ -16,74 +16,51 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.accumulo.classloader.lcc.resolvers;
+package org.apache.accumulo.classloader.lcc;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.io.IOException;
-import java.io.InputStream;
import java.net.URL;
-import java.nio.file.Files;
import java.nio.file.Path;
-import java.nio.file.StandardOpenOption;
-import org.apache.accumulo.classloader.lcc.TestUtils;
-import org.apache.commons.io.IOUtils;
import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.eclipse.jetty.server.Server;
import org.junit.jupiter.api.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-public class FileResolversTest {
-
- private static final Logger LOG =
LoggerFactory.getLogger(FileResolversTest.class);
-
- private long getFileSize(Path p) throws IOException {
- try (InputStream is = Files.newInputStream(p, StandardOpenOption.READ)) {
- return IOUtils.consume(is);
- }
- }
+/**
+ * Test a variety of URL types. Specifically, test file:, http:, and hdfs:
types.
+ */
+public class URLTypesTest {
- private long getFileSize(FileResolver resolver) throws IOException {
- try (InputStream is = resolver.getInputStream()) {
- return IOUtils.consume(is);
- }
- }
+ private static final Logger LOG =
LoggerFactory.getLogger(URLTypesTest.class);
@Test
public void testLocalFile() throws Exception {
- URL jarPath = FileResolversTest.class.getResource("/HelloWorld.jar");
+ URL jarPath = URLTypesTest.class.getResource("/HelloWorld.jar");
assertNotNull(jarPath);
var p = Path.of(jarPath.toURI());
- final long origFileSize = getFileSize(p);
- FileResolver resolver = FileResolver.resolve(jarPath);
- assertTrue(resolver instanceof LocalFileResolver);
- assertEquals(jarPath, resolver.getURL());
- assertEquals("HelloWorld.jar", resolver.getFileName());
- assertEquals(origFileSize, getFileSize(resolver));
+ final long origFileSize = TestUtils.getFileSize(p);
+ assertEquals(origFileSize, TestUtils.getFileSize(jarPath));
}
@Test
public void testHttpFile() throws Exception {
- URL jarPath = FileResolversTest.class.getResource("/HelloWorld.jar");
+ URL jarPath = URLTypesTest.class.getResource("/HelloWorld.jar");
assertNotNull(jarPath);
var p = Path.of(jarPath.toURI());
- final long origFileSize = getFileSize(p);
+ final long origFileSize = TestUtils.getFileSize(p);
Server jetty = TestUtils.getJetty(p.getParent());
LOG.debug("Jetty listening at: {}", jetty.getURI());
URL httpPath = jetty.getURI().resolve("HelloWorld.jar").toURL();
- FileResolver resolver = FileResolver.resolve(httpPath);
- assertTrue(resolver instanceof HttpFileResolver);
- assertEquals(httpPath, resolver.getURL());
- assertEquals("HelloWorld.jar", resolver.getFileName());
- assertEquals(origFileSize, getFileSize(resolver));
+ assertEquals(origFileSize, TestUtils.getFileSize(httpPath));
jetty.stop();
jetty.join();
@@ -92,10 +69,10 @@ public class FileResolversTest {
@Test
public void testHdfsFile() throws Exception {
- URL jarPath = FileResolversTest.class.getResource("/HelloWorld.jar");
+ URL jarPath = URLTypesTest.class.getResource("/HelloWorld.jar");
assertNotNull(jarPath);
var p = Path.of(jarPath.toURI());
- final long origFileSize = getFileSize(p);
+ final long origFileSize = TestUtils.getFileSize(p);
MiniDFSCluster cluster = TestUtils.getMiniCluster();
try {
@@ -105,16 +82,10 @@ public class FileResolversTest {
fs.copyFromLocalFile(new org.apache.hadoop.fs.Path(jarPath.toURI()),
dst);
assertTrue(fs.exists(dst));
- URL.setURLStreamHandlerFactory(new
FsUrlStreamHandlerFactory(cluster.getConfiguration(0)));
-
URL fullPath = new URL(fs.getUri().toString() + dst.toUri().toString());
LOG.info("Path to hdfs file: {}", fullPath);
- FileResolver resolver = FileResolver.resolve(fullPath);
- assertTrue(resolver instanceof HdfsFileResolver);
- assertEquals(fullPath, resolver.getURL());
- assertEquals("HelloWorld.jar", resolver.getFileName());
- assertEquals(origFileSize, getFileSize(resolver));
+ assertEquals(origFileSize, TestUtils.getFileSize(fullPath));
} catch (IOException e) {
throw new RuntimeException("Error setting up mini cluster", e);
diff --git
a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/util/DeduplicationCacheTest.java
b/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/util/DeduplicationCacheTest.java
index 723ce61..5d17fea 100644
---
a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/util/DeduplicationCacheTest.java
+++
b/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/util/DeduplicationCacheTest.java
@@ -18,57 +18,63 @@
*/
package org.apache.accumulo.classloader.lcc.util;
-import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
-import java.net.URL;
-import java.net.URLClassLoader;
import java.time.Duration;
-import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.LinkedBlockingQueue;
import org.junit.jupiter.api.Test;
import com.github.benmanes.caffeine.cache.RemovalCause;
-import com.github.benmanes.caffeine.cache.RemovalListener;
public class DeduplicationCacheTest {
@Test
public void testCollectionNotification() throws Exception {
- final AtomicBoolean listenerCalled = new AtomicBoolean(false);
-
- final RemovalListener<String,URLClassLoader> removalListener =
- (key, value, cause) -> listenerCalled.compareAndSet(false, cause ==
RemovalCause.COLLECTED);
-
- final DeduplicationCache<String,URL[],URLClassLoader> cache = new
DeduplicationCache<>(
- LccUtils::createClassLoader, Duration.ofSeconds(5), removalListener);
-
- final URL jarAOrigLocation =
-
DeduplicationCacheTest.class.getResource("/ClassLoaderTestA/TestA.jar");
- assertNotNull(jarAOrigLocation);
-
- Thread t = new Thread(() -> cache.computeIfAbsent("TEST", () -> new URL[]
{jarAOrigLocation}));
+ final var endBackgroundThread = new CountDownLatch(1);
+ final var removalCauseQueue = new LinkedBlockingQueue<RemovalCause>();
+
+ final var cache = new DeduplicationCache<String,Object,Object>((k, p) ->
new Object(),
+ Duration.ofSeconds(1), (key, value, cause) ->
removalCauseQueue.add(cause));
+
+ var createdCacheEntry = new CountDownLatch(1);
+ Thread t = new Thread(() -> {
+ var value = cache.computeIfAbsent("TEST", () -> new Object());
+ createdCacheEntry.countDown();
+ try {
+ // hold a strong reference in the background thread long enough to
check the weak cache
+ endBackgroundThread.await();
+ } catch (InterruptedException e) {
+ throw new IllegalStateException(e);
+ }
+ assertNotNull(value);
+ });
t.start();
- t.join();
+ createdCacheEntry.await();
- boolean exists = cache.anyMatch("TEST"::equals);
- assertTrue(exists);
+ assertTrue(cache.anyMatch("TEST"::equals));
- // sleep twice as long as the access time duration in the strong reference
cache
- Thread.sleep(10_000);
- exists = cache.anyMatch("TEST"::equals);
- assertTrue(exists); // This is true because it's coming from the weak
reference cache
+ // wait for expiration from strong cache
+ assertEquals(RemovalCause.EXPIRED, removalCauseQueue.take());
+ assertTrue(removalCauseQueue.isEmpty());
- // sleep twice as long as the access time duration in the strong reference
cache
- Thread.sleep(10_000);
- System.gc();
+ // should still exist in the weak values cache
+ assertTrue(cache.anyMatch("TEST"::equals));
+ endBackgroundThread.countDown();
- exists = cache.anyMatch("TEST"::equals);
- assertFalse(exists);
- assertTrue(listenerCalled.get());
+ t.join();
+ // wait for it to be garbage collected (checking the cache triggers
cleanup)
+ while (cache.anyMatch("TEST"::equals)) {
+ System.gc();
+ Thread.sleep(50);
+ }
+ assertEquals(RemovalCause.COLLECTED, removalCauseQueue.take());
+ assertTrue(removalCauseQueue.isEmpty());
}
}
diff --git
a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/util/LocalStoreTest.java
b/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/util/LocalStoreTest.java
index 995ff77..86e6d94 100644
---
a/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/util/LocalStoreTest.java
+++
b/modules/local-caching-classloader/src/test/java/org/apache/accumulo/classloader/lcc/util/LocalStoreTest.java
@@ -40,7 +40,6 @@ import
org.apache.accumulo.classloader.lcc.TestUtils.TestClassInfo;
import org.apache.accumulo.classloader.lcc.definition.ContextDefinition;
import org.apache.accumulo.classloader.lcc.definition.Resource;
import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.eclipse.jetty.server.Server;
import org.junit.jupiter.api.AfterAll;
@@ -83,7 +82,6 @@ public class LocalStoreTest {
final var dst = new org.apache.hadoop.fs.Path("/contextB/TestB.jar");
fs.copyFromLocalFile(new
org.apache.hadoop.fs.Path(jarBOrigLocation.toURI()), dst);
assertTrue(fs.exists(dst));
- URL.setURLStreamHandlerFactory(new
FsUrlStreamHandlerFactory(hdfs.getConfiguration(0)));
final URL jarBNewLocation = new URL(fs.getUri().toString() +
dst.toUri().toString());
// Put C into Jetty
@@ -153,33 +151,47 @@ public class LocalStoreTest {
assertTrue(Files.exists(baseCacheDir));
}
+ private static Resource rsrc(String filename, String checksum) {
+ return new Resource() {
+ @Override
+ public String getFileName() {
+ return filename;
+ }
+
+ @Override
+ public String getChecksum() {
+ return checksum;
+ }
+ };
+ }
+
@Test
public void testLocalFileName() {
// regular war
- assertEquals("f1-chk1.war", LocalStore.localResourceName("f1.war",
"chk1"));
+ assertEquals("f1-chk1.war", LocalStore.localResourceName(rsrc("f1.war",
"chk1")));
// dotfile war
- assertEquals(".f1-chk1.war", LocalStore.localResourceName(".f1.war",
"chk1"));
+ assertEquals(".f1-chk1.war", LocalStore.localResourceName(rsrc(".f1.war",
"chk1")));
// regular jar (has multiple dots)
- assertEquals("f2-1.0-chk2.jar", LocalStore.localResourceName("f2-1.0.jar",
"chk2"));
+ assertEquals("f2-1.0-chk2.jar",
LocalStore.localResourceName(rsrc("f2-1.0.jar", "chk2")));
// dotfile jar (has multiple dots)
- assertEquals(".f2-1.0-chk2.jar",
LocalStore.localResourceName(".f2-1.0.jar", "chk2"));
+ assertEquals(".f2-1.0-chk2.jar",
LocalStore.localResourceName(rsrc(".f2-1.0.jar", "chk2")));
// regular file with no suffix
- assertEquals("f3-chk3", LocalStore.localResourceName("f3", "chk3"));
+ assertEquals("f3-chk3", LocalStore.localResourceName(rsrc("f3", "chk3")));
// weird files with trailing dots and no file suffix
- assertEquals("f4.-chk4", LocalStore.localResourceName("f4.", "chk4"));
- assertEquals("f4..-chk4", LocalStore.localResourceName("f4..", "chk4"));
- assertEquals("f4...-chk4", LocalStore.localResourceName("f4...", "chk4"));
+ assertEquals("f4.-chk4", LocalStore.localResourceName(rsrc("f4.",
"chk4")));
+ assertEquals("f4..-chk4", LocalStore.localResourceName(rsrc("f4..",
"chk4")));
+ assertEquals("f4...-chk4", LocalStore.localResourceName(rsrc("f4...",
"chk4")));
// weird dotfiles that don't really have a suffix
- assertEquals(".f5-chk5", LocalStore.localResourceName(".f5", "chk5"));
- assertEquals("..f5-chk5", LocalStore.localResourceName("..f5", "chk5"));
+ assertEquals(".f5-chk5", LocalStore.localResourceName(rsrc(".f5",
"chk5")));
+ assertEquals("..f5-chk5", LocalStore.localResourceName(rsrc("..f5",
"chk5")));
// weird files with weird dots, but do have a valid suffix
- assertEquals("f6.-chk6.jar", LocalStore.localResourceName("f6..jar",
"chk6"));
- assertEquals("f6..-chk6.jar", LocalStore.localResourceName("f6...jar",
"chk6"));
- assertEquals(".f6-chk6.jar", LocalStore.localResourceName(".f6.jar",
"chk6"));
- assertEquals("..f6-chk6.jar", LocalStore.localResourceName("..f6.jar",
"chk6"));
- assertEquals(".f6.-chk6.jar", LocalStore.localResourceName(".f6..jar",
"chk6"));
- assertEquals("..f6.-chk6.jar", LocalStore.localResourceName("..f6..jar",
"chk6"));
+ assertEquals("f6.-chk6.jar", LocalStore.localResourceName(rsrc("f6..jar",
"chk6")));
+ assertEquals("f6..-chk6.jar",
LocalStore.localResourceName(rsrc("f6...jar", "chk6")));
+ assertEquals(".f6-chk6.jar", LocalStore.localResourceName(rsrc(".f6.jar",
"chk6")));
+ assertEquals("..f6-chk6.jar",
LocalStore.localResourceName(rsrc("..f6.jar", "chk6")));
+ assertEquals(".f6.-chk6.jar",
LocalStore.localResourceName(rsrc(".f6..jar", "chk6")));
+ assertEquals("..f6.-chk6.jar",
LocalStore.localResourceName(rsrc("..f6..jar", "chk6")));
}
@Test
@@ -191,10 +203,8 @@ public class LocalStoreTest {
assertTrue(Files.exists(baseCacheDir));
assertTrue(Files.exists(baseCacheDir.resolve("contexts").resolve(def.getChecksum()
+ ".json")));
for (Resource r : def.getResources()) {
- String filename = TestUtils.getFileName(r.getLocation());
- String checksum = r.getChecksum();
- assertTrue(Files.exists(baseCacheDir.resolve("resources")
- .resolve(LocalStore.localResourceName(filename, checksum))));
+ assertTrue(
+
Files.exists(baseCacheDir.resolve("resources").resolve(LocalStore.localResourceName(r))));
}
}
@@ -237,17 +247,15 @@ public class LocalStoreTest {
assertTrue(
Files.exists(baseCacheDir.resolve("contexts").resolve(updatedDef.getChecksum()
+ ".json")));
for (Resource r : updatedDef.getResources()) {
- String filename = TestUtils.getFileName(r.getLocation());
- assertFalse(filename.contains("C"));
- String checksum = r.getChecksum();
- assertTrue(Files.exists(baseCacheDir.resolve("resources")
- .resolve(LocalStore.localResourceName(filename, checksum))));
+ assertFalse(r.getFileName().contains("C"));
+ assertTrue(
+
Files.exists(baseCacheDir.resolve("resources").resolve(LocalStore.localResourceName(r))));
}
- String filename = TestUtils.getFileName(removedResource.getLocation());
- assertTrue(filename.contains("C"), "cache location should still contain
'C'");
- assertTrue(Files.exists(baseCacheDir.resolve("resources")
- .resolve(LocalStore.localResourceName(filename,
removedResource.getChecksum()))));
+ assertTrue(removedResource.getFileName().contains("C"),
+ "cache location should still contain 'C'");
+ assertTrue(Files.exists(
+
baseCacheDir.resolve("resources").resolve(LocalStore.localResourceName(removedResource))));
final var updatedContextClassLoader = LccUtils.createClassLoader("url",
urls);
diff --git a/modules/vfs-class-loader/pom.xml b/modules/vfs-class-loader/pom.xml
index b827e4d..e4ed0af 100644
--- a/modules/vfs-class-loader/pom.xml
+++ b/modules/vfs-class-loader/pom.xml
@@ -156,6 +156,25 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>analyze</id>
+ <goals>
+ <goal>analyze-only</goal>
+ </goals>
+ <configuration>
+ <ignoredUnusedDeclaredDependencies combine.children="append">
+ <!-- ignore false positive runtime dependencies -->
+ <unused>org.apache.commons:commons-vfs2-hdfs:*</unused>
+
<unused>org.apache.httpcomponents.client5:httpclient5:*</unused>
+ </ignoredUnusedDeclaredDependencies>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
</plugins>
</build>
</project>
diff --git a/pom.xml b/pom.xml
index 0335e9d..16766a5 100644
--- a/pom.xml
+++ b/pom.xml
@@ -77,6 +77,7 @@
<module>modules/example-iterators-b</module>
<module>modules/vfs-class-loader</module>
<module>modules/local-caching-classloader</module>
+ <module>modules/hdfs-urlstreamhandler-provider</module>
</modules>
<scm>
<connection>scm:git:https://gitbox.apache.org/repos/asf/accumulo-classloaders.git</connection>
@@ -147,6 +148,11 @@ under the License.
<type>pom</type>
<scope>import</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.accumulo</groupId>
+ <artifactId>hdfs-urlstreamhandler-provider</artifactId>
+ <version>${project.version}</version>
+ </dependency>
<dependency>
<groupId>org.apache.httpcomponents.client5</groupId>
<artifactId>httpclient5</artifactId>
@@ -367,16 +373,14 @@ under the License.
</goals>
<configuration>
<failOnWarning>true</failOnWarning>
- <ignoredUsedUndeclaredDependencies>
+ <ignoredUsedUndeclaredDependencies combine.children="append">
<!-- auto-service-annotations is transitive via auto-service
-->
<undeclared>com.google.auto.service:auto-service-annotations:jar:*</undeclared>
</ignoredUsedUndeclaredDependencies>
- <ignoredUnusedDeclaredDependencies>
+ <ignoredUnusedDeclaredDependencies combine.children="append">
<!-- auto-service used by the compiler for annotation
processing, not by code -->
<unused>com.google.auto.service:auto-service:jar:*</unused>
<!-- ignore false positive runtime dependencies -->
- <unused>org.apache.commons:commons-vfs2-hdfs:*</unused>
-
<unused>org.apache.httpcomponents.client5:httpclient5:*</unused>
<unused>org.apache.logging.log4j:log4j-slf4j2-impl:*</unused>
<!-- spotbugs annotations may or may not be used in each
module -->
<unused>com.github.spotbugs:spotbugs-annotations:jar:*</unused>
