gianm commented on code in PR #12816:
URL: https://github.com/apache/druid/pull/12816#discussion_r933881959
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/HadoopTask.java:
##########
@@ -48,13 +49,11 @@
public abstract class HadoopTask extends AbstractBatchIndexTask
{
private static final Logger log = new Logger(HadoopTask.class);
- private static final ExtensionsConfig EXTENSIONS_CONFIG;
-
- static final Injector INJECTOR = GuiceInjectors.makeStartupInjector();
- static {
- EXTENSIONS_CONFIG = INJECTOR.getInstance(ExtensionsConfig.class);
- }
+ // Note: static variables mean that this task cannot run in a shared JVM,
Review Comment:
Is this because ExtensionsLoader isn't concurrency-safe?
As far as I can tell, the prior code was OK to use in a shared JVM: it was
an Injector (which I think is concurrency-safe?) and and ExtensionsConfig
(immutable; definitely fine).
##########
processing/src/main/java/org/apache/druid/guice/GuiceInjectors.java:
##########
@@ -19,51 +19,27 @@
package org.apache.druid.guice;
-import com.google.common.collect.ImmutableList;
-import com.google.inject.Guice;
import com.google.inject.Injector;
import com.google.inject.Module;
-import org.apache.druid.jackson.JacksonModule;
-import org.apache.druid.math.expr.ExpressionProcessingModule;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.List;
+import java.util.Collections;
/**
+ * Creates the startup injector. Regained for backward compatibility.
Review Comment:
Retained?
##########
server/src/main/java/org/apache/druid/initialization/ServerInjectorBuilder.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.initialization;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+import com.google.inject.Injector;
+import com.google.inject.Module;
+import com.google.inject.TypeLiteral;
+import com.google.inject.multibindings.MapBinder;
+import com.google.inject.multibindings.Multibinder;
+import org.apache.druid.discovery.DruidService;
+import org.apache.druid.discovery.NodeRole;
+import org.apache.druid.guice.annotations.Self;
+
+import java.util.Set;
+
+/**
+ * Initialize Guice for a server. Clients and tests should use
+ * the individual builders to create a non-server environment.
+ */
+public class ServerInjectorBuilder
+{
+ private Injector baseInjector;
+ private Set<NodeRole> nodeRoles;
+ private Iterable<? extends Module> modules;
+
+ /**
+ * Create a server injector. Located here for testing. Should only be
+ * used by {@code GuiceRunnable} (and tests).
+ *
+ * @param nodeRoles the roles which this server provides
+ * @param baseInjector the startup injector
+ * @param modules modules for this server
+ * @return the injector for the server
+ */
+ // TODO: Move to the CLI package, perhaps GuiceRunnable, once
Review Comment:
Please don't commit TODOs. (I know there are some in the codebase, but we
generally try to avoid them.) Either do it now, or rewrite it as a regular
comment, or raise a follow-up github issue, or simply delete it if it's not a
big deal and not going to be done any time soon.
##########
indexing-service/src/test/java/org/apache/druid/indexing/common/task/batch/parallel/MultiPhaseParallelIndexingRowStatsTest.java:
##########
@@ -107,6 +108,7 @@ public void testHashPartitionRowStats()
}
@Test
+ @Ignore("assumes record rates, to be fixed in separate PR")
Review Comment:
What's the issue here? I don't understand the comment.
##########
core/src/main/java/org/apache/druid/initialization/DruidModule.java:
##########
@@ -25,6 +25,13 @@
import java.util.List;
/**
+ * A Guice module which also provides Jackson modules.
Review Comment:
Text reads funny to me: the second sentence says "you only need this if you
are registering Jackson modules" and the next sentence says "well, OK, also you
need it for extensions". Could be clearer. Maybe:
> A Guice module that is discoverable by ExtensionsLoader and can optionally
provide Jackson modules.
>
> Built-in implementations that do not provide Jackson modules can implement
the simpler Module interface instead.
Or, we could add a default impl for getJacksonModules, and then I don't
think we need to bother with the guidance. (Since then implementing Module or
DruidModule would be equal amounts of code, so there's no real reason to point
people at Module.)
##########
processing/src/main/java/org/apache/druid/guice/ExtensionsLoader.java:
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.guice;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Injector;
+import org.apache.commons.io.FileUtils;
+import org.apache.druid.initialization.DruidModule;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import javax.inject.Inject;
+
+import java.io.File;
+import java.io.FilenameFilter;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+public class ExtensionsLoader
+{
+ private static final Logger log = new Logger(ExtensionsLoader.class);
+
+ private final ExtensionsConfig extensionsConfig;
+ private final ConcurrentHashMap<File, URLClassLoader> loaders = new
ConcurrentHashMap<>();
+
+ /**
+ * Map of loaded extensions, keyed by class (or interface).
+ */
+ private final ConcurrentHashMap<Class<?>, Collection<?>> extensions = new
ConcurrentHashMap<>();
+
+ @Inject
+ public ExtensionsLoader(ExtensionsConfig config)
+ {
+ this.extensionsConfig = config;
+ }
+
+ public static ExtensionsLoader instance(Injector injector)
+ {
+ return injector.getInstance(ExtensionsLoader.class);
+ }
+
+ public ExtensionsConfig config()
+ {
+ return extensionsConfig;
+ }
+
+ /**
+ * @param clazz service class
+ * @param <T> the service type
+ *
+ * @return Returns a collection of implementations loaded.
+ */
+ public <T> Collection<T> getLoadedImplementations(Class<T> clazz)
+ {
+ @SuppressWarnings("unchecked")
+ Collection<T> retVal = (Collection<T>) extensions.get(clazz);
+ if (retVal == null) {
+ return new HashSet<>();
+ }
+ return retVal;
+ }
+
+ /**
+ * @return Returns a collection of implementations loaded.
+ */
+ public Collection<DruidModule> getLoadedModules()
+ {
+ return getLoadedImplementations(DruidModule.class);
+ }
+
+ @VisibleForTesting
+ public Map<File, URLClassLoader> getLoadersMap()
+ {
+ return loaders;
+ }
+
+ /**
+ * Look for implementations for the given class from both classpath and
extensions directory, using {@link
+ * ServiceLoader}. A user should never put the same two extensions in
classpath and extensions directory, if he/she
+ * does that, the one that is in the classpath will be loaded, the other
will be ignored.
+ *
+ * @param serviceClass The class to look the implementations of (e.g.,
DruidModule)
+ *
+ * @return A collection that contains implementations (of distinct concrete
classes) of the given class. The order of
+ * elements in the returned collection is not specified and not guaranteed
to be the same for different calls to
+ * getFromExtensions().
+ */
+ @SuppressWarnings("unchecked")
+ public <T> Collection<T> getFromExtensions(Class<T> serviceClass)
+ {
+ // Classes classes are loaded once upon first request. Since the class
path does
Review Comment:
One too many "classes".
##########
processing/src/main/java/org/apache/druid/guice/ExtensionsLoader.java:
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.guice;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Injector;
+import org.apache.commons.io.FileUtils;
+import org.apache.druid.initialization.DruidModule;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import javax.inject.Inject;
+
+import java.io.File;
+import java.io.FilenameFilter;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+public class ExtensionsLoader
+{
+ private static final Logger log = new Logger(ExtensionsLoader.class);
+
+ private final ExtensionsConfig extensionsConfig;
+ private final ConcurrentHashMap<File, URLClassLoader> loaders = new
ConcurrentHashMap<>();
+
+ /**
+ * Map of loaded extensions, keyed by class (or interface).
+ */
+ private final ConcurrentHashMap<Class<?>, Collection<?>> extensions = new
ConcurrentHashMap<>();
+
+ @Inject
+ public ExtensionsLoader(ExtensionsConfig config)
+ {
+ this.extensionsConfig = config;
+ }
+
+ public static ExtensionsLoader instance(Injector injector)
+ {
+ return injector.getInstance(ExtensionsLoader.class);
+ }
+
+ public ExtensionsConfig config()
+ {
+ return extensionsConfig;
+ }
+
+ /**
+ * @param clazz service class
+ * @param <T> the service type
+ *
+ * @return Returns a collection of implementations loaded.
+ */
+ public <T> Collection<T> getLoadedImplementations(Class<T> clazz)
+ {
+ @SuppressWarnings("unchecked")
+ Collection<T> retVal = (Collection<T>) extensions.get(clazz);
+ if (retVal == null) {
+ return new HashSet<>();
Review Comment:
Or `Collections.emptySet()`?
##########
server/src/test/java/org/apache/druid/initialization/ServerInjectorBuilderTest.java:
##########
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.initialization;
+
+import com.fasterxml.jackson.databind.Module;
+import com.google.common.base.Function;
+import com.google.common.collect.Collections2;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.inject.Binder;
+import com.google.inject.ConfigurationException;
+import com.google.inject.Inject;
+import com.google.inject.Injector;
+import com.google.inject.Key;
+import com.google.inject.name.Names;
+import org.apache.druid.discovery.NodeRole;
+import org.apache.druid.guice.ExtensionsLoader;
+import org.apache.druid.guice.JsonConfigProvider;
+import org.apache.druid.guice.StartupInjectorBuilder;
+import org.apache.druid.guice.annotations.LoadScope;
+import org.apache.druid.guice.annotations.Self;
+import org.apache.druid.server.DruidNode;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+
+// See ExtensionsLoaderTest for tests formerly in this file related
Review Comment:
Should this be something like "Tests formerly in InitializationTest are
split between this class and ExtensionsLoaderTest"? (Because this file didn't
previously exist, so nothing was formerly in it.)
##########
processing/src/main/java/org/apache/druid/guice/ExtensionsLoader.java:
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.guice;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Injector;
+import org.apache.commons.io.FileUtils;
+import org.apache.druid.initialization.DruidModule;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import javax.inject.Inject;
+
+import java.io.File;
+import java.io.FilenameFilter;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+public class ExtensionsLoader
+{
+ private static final Logger log = new Logger(ExtensionsLoader.class);
+
+ private final ExtensionsConfig extensionsConfig;
+ private final ConcurrentHashMap<File, URLClassLoader> loaders = new
ConcurrentHashMap<>();
+
+ /**
+ * Map of loaded extensions, keyed by class (or interface).
+ */
+ private final ConcurrentHashMap<Class<?>, Collection<?>> extensions = new
ConcurrentHashMap<>();
+
+ @Inject
+ public ExtensionsLoader(ExtensionsConfig config)
+ {
+ this.extensionsConfig = config;
+ }
+
+ public static ExtensionsLoader instance(Injector injector)
+ {
+ return injector.getInstance(ExtensionsLoader.class);
+ }
+
+ public ExtensionsConfig config()
+ {
+ return extensionsConfig;
+ }
+
+ /**
+ * @param clazz service class
+ * @param <T> the service type
+ *
+ * @return Returns a collection of implementations loaded.
+ */
+ public <T> Collection<T> getLoadedImplementations(Class<T> clazz)
+ {
+ @SuppressWarnings("unchecked")
+ Collection<T> retVal = (Collection<T>) extensions.get(clazz);
+ if (retVal == null) {
+ return new HashSet<>();
+ }
+ return retVal;
+ }
+
+ /**
+ * @return Returns a collection of implementations loaded.
+ */
+ public Collection<DruidModule> getLoadedModules()
+ {
+ return getLoadedImplementations(DruidModule.class);
+ }
+
+ @VisibleForTesting
+ public Map<File, URLClassLoader> getLoadersMap()
+ {
+ return loaders;
+ }
+
+ /**
+ * Look for implementations for the given class from both classpath and
extensions directory, using {@link
+ * ServiceLoader}. A user should never put the same two extensions in
classpath and extensions directory, if he/she
+ * does that, the one that is in the classpath will be loaded, the other
will be ignored.
+ *
+ * @param serviceClass The class to look the implementations of (e.g.,
DruidModule)
+ *
+ * @return A collection that contains implementations (of distinct concrete
classes) of the given class. The order of
+ * elements in the returned collection is not specified and not guaranteed
to be the same for different calls to
+ * getFromExtensions().
+ */
+ @SuppressWarnings("unchecked")
+ public <T> Collection<T> getFromExtensions(Class<T> serviceClass)
+ {
+ // Classes classes are loaded once upon first request. Since the class
path does
+ // not change during a run, the set of extension classes cannot change once
+ // computed.
+ //
+ // In practice, it appears the only place this matters is with DruidModule:
+ // initialization gets the list of extensions, and two REST API calls later
+ // ask for the same list.
+ Collection<?> modules = extensions.computeIfAbsent(
+ serviceClass,
+ serviceC -> new ServiceLoadingFromExtensions<>(serviceC).implsToLoad
+ );
+ //noinspection unchecked
+ return (Collection<T>) modules;
+ }
+
+ public Collection<DruidModule> getModules()
+ {
+ return getFromExtensions(DruidModule.class);
+ }
+
+ /**
+ * Find all the extension files that should be loaded by druid.
+ * <p/>
+ * If user explicitly specifies druid.extensions.loadList, then it will look
for those extensions under root
+ * extensions directory. If one of them is not found, druid will fail loudly.
+ * <p/>
+ * If user doesn't specify druid.extension.toLoad (or its value is empty),
druid will load all the extensions
+ * under the root extensions directory.
+ *
+ * @return an array of druid extension files that will be loaded by druid
process
+ */
+ public File[] getExtensionFilesToLoad()
+ {
+ final File rootExtensionsDir = new File(extensionsConfig.getDirectory());
+ if (rootExtensionsDir.exists() && !rootExtensionsDir.isDirectory()) {
+ throw new ISE("Root extensions directory [%s] is not a directory!?",
rootExtensionsDir);
+ }
+ File[] extensionsToLoad;
+ final LinkedHashSet<String> toLoad = extensionsConfig.getLoadList();
+ if (toLoad == null) {
+ extensionsToLoad = rootExtensionsDir.listFiles();
+ } else {
+ int i = 0;
+ extensionsToLoad = new File[toLoad.size()];
+ for (final String extensionName : toLoad) {
+ File extensionDir = new File(extensionName);
+ if (!extensionDir.isAbsolute()) {
+ extensionDir = new File(rootExtensionsDir, extensionName);
+ }
+
+ if (!extensionDir.isDirectory()) {
+ throw new ISE(
+ "Extension [%s] specified in \"druid.extensions.loadList\"
didn't exist!?",
+ extensionDir.getAbsolutePath()
+ );
+ }
+ extensionsToLoad[i++] = extensionDir;
+ }
+ }
+ return extensionsToLoad == null ? new File[]{} : extensionsToLoad;
+ }
+
+ /**
+ * @param extension The File instance of the extension we want to load
+ *
+ * @return a URLClassLoader that loads all the jars on which the extension
is dependent
+ */
+ public URLClassLoader getClassLoaderForExtension(File extension, boolean
useExtensionClassloaderFirst)
+ {
+ return loaders.computeIfAbsent(
Review Comment:
Is it OK that `useExtensionClassloaderFirst` is not part of the `loaders`
map key? Seems like that's only valid if callers always pass in the same value.
Will it ever be anything other than
`extensionsConfig.isUseExtensionClassloaderFirst()`? If not: perhaps it'd be
better to have this method only accept the `File`, and to use
`extensionsConfig.isUseExtensionClassloaderFirst()` directly.
##########
server/src/main/java/org/apache/druid/initialization/Initialization.java:
##########
@@ -19,536 +19,24 @@
package org.apache.druid.initialization;
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableSet;
-import com.google.inject.Guice;
import com.google.inject.Injector;
-import com.google.inject.Key;
import com.google.inject.Module;
-import com.google.inject.util.Modules;
-import org.apache.commons.io.FileUtils;
-import org.apache.druid.curator.CuratorModule;
-import org.apache.druid.curator.discovery.DiscoveryModule;
-import org.apache.druid.discovery.NodeRole;
-import org.apache.druid.guice.AnnouncerModule;
-import org.apache.druid.guice.CoordinatorDiscoveryModule;
-import org.apache.druid.guice.DruidProcessingConfigModule;
-import org.apache.druid.guice.DruidSecondaryModule;
-import org.apache.druid.guice.ExpressionModule;
-import org.apache.druid.guice.ExtensionsConfig;
-import org.apache.druid.guice.FirehoseModule;
-import org.apache.druid.guice.IndexingServiceDiscoveryModule;
-import org.apache.druid.guice.JacksonConfigManagerModule;
-import org.apache.druid.guice.JavaScriptModule;
-import org.apache.druid.guice.LifecycleModule;
-import org.apache.druid.guice.LocalDataStorageDruidModule;
-import org.apache.druid.guice.MetadataConfigModule;
-import org.apache.druid.guice.ModulesConfig;
-import org.apache.druid.guice.NestedDataModule;
-import org.apache.druid.guice.ServerModule;
-import org.apache.druid.guice.ServerViewModule;
-import org.apache.druid.guice.StartupLoggingModule;
-import org.apache.druid.guice.StorageNodeModule;
-import org.apache.druid.guice.annotations.Client;
-import org.apache.druid.guice.annotations.EscalatedClient;
-import org.apache.druid.guice.annotations.Json;
-import org.apache.druid.guice.annotations.LoadScope;
-import org.apache.druid.guice.annotations.Smile;
-import org.apache.druid.guice.http.HttpClientModule;
-import org.apache.druid.guice.security.AuthenticatorModule;
-import org.apache.druid.guice.security.AuthorizerModule;
-import org.apache.druid.guice.security.DruidAuthModule;
-import org.apache.druid.guice.security.EscalatorModule;
-import org.apache.druid.java.util.common.ISE;
-import org.apache.druid.java.util.common.logger.Logger;
-import org.apache.druid.metadata.storage.derby.DerbyMetadataStorageDruidModule;
-import org.apache.druid.rpc.guice.ServiceClientModule;
-import org.apache.druid.segment.writeout.SegmentWriteOutMediumModule;
-import org.apache.druid.server.emitter.EmitterModule;
-import org.apache.druid.server.initialization.AuthenticatorMapperModule;
-import org.apache.druid.server.initialization.AuthorizerMapperModule;
-import
org.apache.druid.server.initialization.ExternalStorageAccessSecurityModule;
-import org.apache.druid.server.initialization.jetty.JettyServerModule;
-import org.apache.druid.server.metrics.MetricsModule;
-import org.apache.druid.server.security.TLSCertificateCheckerModule;
-import org.eclipse.aether.artifact.DefaultArtifact;
-
-import java.io.File;
-import java.io.FilenameFilter;
-import java.io.IOException;
-import java.net.MalformedURLException;
-import java.net.URL;
-import java.net.URLClassLoader;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashSet;
-import java.util.LinkedHashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.ServiceLoader;
-import java.util.Set;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.stream.Collectors;
/**
- *
+ * Initialize Guice for a server. Clients and tests should use
+ * the individual builders to create a non-server environment.
*/
public class Initialization
{
- private static final Logger log = new Logger(Initialization.class);
- private static final ConcurrentHashMap<File, URLClassLoader> LOADERS_MAP =
new ConcurrentHashMap<>();
-
- private static final ConcurrentHashMap<Class<?>, Collection<?>>
EXTENSIONS_MAP = new ConcurrentHashMap<>();
-
- /**
- * @param clazz service class
- * @param <T> the service type
- *
- * @return Returns a collection of implementations loaded.
- */
- public static <T> Collection<T> getLoadedImplementations(Class<T> clazz)
- {
- @SuppressWarnings("unchecked")
- Collection<T> retVal = (Collection<T>) EXTENSIONS_MAP.get(clazz);
- if (retVal == null) {
- return new HashSet<>();
- }
- return retVal;
- }
-
- @VisibleForTesting
- static void clearLoadedImplementations()
- {
- EXTENSIONS_MAP.clear();
- }
-
- @VisibleForTesting
- static Map<File, URLClassLoader> getLoadersMap()
- {
- return LOADERS_MAP;
- }
-
- /**
- * Look for implementations for the given class from both classpath and
extensions directory, using {@link
- * ServiceLoader}. A user should never put the same two extensions in
classpath and extensions directory, if he/she
- * does that, the one that is in the classpath will be loaded, the other
will be ignored.
- *
- * @param config Extensions configuration
- * @param serviceClass The class to look the implementations of (e.g.,
DruidModule)
- *
- * @return A collection that contains implementations (of distinct concrete
classes) of the given class. The order of
- * elements in the returned collection is not specified and not guaranteed
to be the same for different calls to
- * getFromExtensions().
- */
- public static <T> Collection<T> getFromExtensions(ExtensionsConfig config,
Class<T> serviceClass)
- {
- // It's not clear whether we should recompute modules even if they have
been computed already for the serviceClass,
- // but that's how it used to be an preserving the old behaviour here.
- Collection<?> modules = EXTENSIONS_MAP.compute(
- serviceClass,
- (serviceC, ignored) -> new ServiceLoadingFromExtensions<>(config,
serviceC).implsToLoad
- );
- //noinspection unchecked
- return (Collection<T>) modules;
- }
-
- private static class ServiceLoadingFromExtensions<T>
- {
- private final ExtensionsConfig extensionsConfig;
- private final Class<T> serviceClass;
- private final List<T> implsToLoad = new ArrayList<>();
- private final Set<String> implClassNamesToLoad = new HashSet<>();
-
- private ServiceLoadingFromExtensions(ExtensionsConfig extensionsConfig,
Class<T> serviceClass)
- {
- this.extensionsConfig = extensionsConfig;
- this.serviceClass = serviceClass;
- if (extensionsConfig.searchCurrentClassloader()) {
- addAllFromCurrentClassLoader();
- }
- addAllFromFileSystem();
- }
-
- private void addAllFromCurrentClassLoader()
- {
- ServiceLoader
- .load(serviceClass, Thread.currentThread().getContextClassLoader())
- .forEach(impl -> tryAdd(impl, "classpath"));
- }
-
- private void addAllFromFileSystem()
- {
- for (File extension : getExtensionFilesToLoad(extensionsConfig)) {
- log.debug("Loading extension [%s] for class [%s]",
extension.getName(), serviceClass);
- try {
- final URLClassLoader loader = getClassLoaderForExtension(
- extension,
- extensionsConfig.isUseExtensionClassloaderFirst()
- );
-
- log.info(
- "Loading extension [%s], jars: %s",
- extension.getName(),
- Arrays.stream(loader.getURLs())
- .map(u -> new File(u.getPath()).getName())
- .collect(Collectors.joining(", "))
- );
-
- ServiceLoader.load(serviceClass, loader).forEach(impl ->
tryAdd(impl, "local file system"));
- }
- catch (Exception e) {
- throw new RuntimeException(e);
- }
- }
- }
-
- private void tryAdd(T serviceImpl, String extensionType)
- {
- final String serviceImplName = serviceImpl.getClass().getName();
- if (serviceImplName == null) {
- log.warn(
- "Implementation [%s] was ignored because it doesn't have a
canonical name, "
- + "is it a local or anonymous class?",
- serviceImpl.getClass().getName()
- );
- } else if (!implClassNamesToLoad.contains(serviceImplName)) {
- log.debug(
- "Adding implementation [%s] for class [%s] from %s extension",
- serviceImplName,
- serviceClass,
- extensionType
- );
- implClassNamesToLoad.add(serviceImplName);
- implsToLoad.add(serviceImpl);
- }
- }
- }
-
- /**
- * Find all the extension files that should be loaded by druid.
- * <p/>
- * If user explicitly specifies druid.extensions.loadList, then it will look
for those extensions under root
- * extensions directory. If one of them is not found, druid will fail loudly.
- * <p/>
- * If user doesn't specify druid.extension.toLoad (or its value is empty),
druid will load all the extensions
- * under the root extensions directory.
- *
- * @param config ExtensionsConfig configured by druid.extensions.xxx
- *
- * @return an array of druid extension files that will be loaded by druid
process
- */
- public static File[] getExtensionFilesToLoad(ExtensionsConfig config)
- {
- final File rootExtensionsDir = new File(config.getDirectory());
- if (rootExtensionsDir.exists() && !rootExtensionsDir.isDirectory()) {
- throw new ISE("Root extensions directory [%s] is not a directory!?",
rootExtensionsDir);
- }
- File[] extensionsToLoad;
- final LinkedHashSet<String> toLoad = config.getLoadList();
- if (toLoad == null) {
- extensionsToLoad = rootExtensionsDir.listFiles();
- } else {
- int i = 0;
- extensionsToLoad = new File[toLoad.size()];
- for (final String extensionName : toLoad) {
- File extensionDir = new File(extensionName);
- if (!extensionDir.isAbsolute()) {
- extensionDir = new File(rootExtensionsDir, extensionName);
- }
-
- if (!extensionDir.isDirectory()) {
- throw new ISE(
- "Extension [%s] specified in \"druid.extensions.loadList\"
didn't exist!?",
- extensionDir.getAbsolutePath()
- );
- }
- extensionsToLoad[i++] = extensionDir;
- }
- }
- return extensionsToLoad == null ? new File[]{} : extensionsToLoad;
- }
-
- /**
- * Find all the hadoop dependencies that should be loaded by druid
- *
- * @param hadoopDependencyCoordinates
e.g.["org.apache.hadoop:hadoop-client:2.3.0"]
- * @param extensionsConfig ExtensionsConfig configured by
druid.extensions.xxx
- *
- * @return an array of hadoop dependency files that will be loaded by druid
process
- */
- public static File[] getHadoopDependencyFilesToLoad(
- List<String> hadoopDependencyCoordinates,
- ExtensionsConfig extensionsConfig
- )
- {
- final File rootHadoopDependenciesDir = new
File(extensionsConfig.getHadoopDependenciesDir());
- if (rootHadoopDependenciesDir.exists() &&
!rootHadoopDependenciesDir.isDirectory()) {
- throw new ISE("Root Hadoop dependencies directory [%s] is not a
directory!?", rootHadoopDependenciesDir);
- }
- final File[] hadoopDependenciesToLoad = new
File[hadoopDependencyCoordinates.size()];
- int i = 0;
- for (final String coordinate : hadoopDependencyCoordinates) {
- final DefaultArtifact artifact = new DefaultArtifact(coordinate);
- final File hadoopDependencyDir = new File(rootHadoopDependenciesDir,
artifact.getArtifactId());
- final File versionDir = new File(hadoopDependencyDir,
artifact.getVersion());
- // find the hadoop dependency with the version specified in coordinate
- if (!hadoopDependencyDir.isDirectory() || !versionDir.isDirectory()) {
- throw new ISE("Hadoop dependency [%s] didn't exist!?",
versionDir.getAbsolutePath());
- }
- hadoopDependenciesToLoad[i++] = versionDir;
- }
- return hadoopDependenciesToLoad;
- }
-
- /**
- * @param extension The File instance of the extension we want to load
- *
- * @return a URLClassLoader that loads all the jars on which the extension
is dependent
- */
- public static URLClassLoader getClassLoaderForExtension(File extension,
boolean useExtensionClassloaderFirst)
- {
- return LOADERS_MAP.computeIfAbsent(
- extension,
- theExtension -> makeClassLoaderForExtension(theExtension,
useExtensionClassloaderFirst)
- );
- }
-
- private static URLClassLoader makeClassLoaderForExtension(
- final File extension,
- final boolean useExtensionClassloaderFirst
- )
- {
- final Collection<File> jars = FileUtils.listFiles(extension, new
String[]{"jar"}, false);
- final URL[] urls = new URL[jars.size()];
-
- try {
- int i = 0;
- for (File jar : jars) {
- final URL url = jar.toURI().toURL();
- log.debug("added URL[%s] for extension[%s]", url, extension.getName());
- urls[i++] = url;
- }
- }
- catch (MalformedURLException e) {
- throw new RuntimeException(e);
- }
-
- if (useExtensionClassloaderFirst) {
- return new ExtensionFirstClassLoader(urls,
Initialization.class.getClassLoader());
- } else {
- return new URLClassLoader(urls, Initialization.class.getClassLoader());
- }
- }
-
- public static List<URL> getURLsForClasspath(String cp)
- {
- try {
- String[] paths = cp.split(File.pathSeparator);
-
- List<URL> urls = new ArrayList<>();
- for (String path : paths) {
- File f = new File(path);
- if ("*".equals(f.getName())) {
- File parentDir = f.getParentFile();
- if (parentDir.isDirectory()) {
- File[] jars = parentDir.listFiles(
- new FilenameFilter()
- {
- @Override
- public boolean accept(File dir, String name)
- {
- return name != null && (name.endsWith(".jar") ||
name.endsWith(".JAR"));
- }
- }
- );
- for (File jar : jars) {
- urls.add(jar.toURI().toURL());
- }
- }
- } else {
- urls.add(new File(path).toURI().toURL());
- }
- }
- return urls;
- }
- catch (IOException ex) {
- throw new RuntimeException(ex);
- }
- }
-
- public static Injector makeInjectorWithModules(
- final Injector baseInjector,
- final Iterable<? extends Module> modules
- )
- {
- return makeInjectorWithModules(ImmutableSet.of(), baseInjector, modules);
- }
-
+ // Use individual builders for testing: this method brings in
+ // server-only dependencies, which is generally not desired.
+ @Deprecated
public static Injector makeInjectorWithModules(
Review Comment:
Include a link to the preferred mechanism?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
