This is an automated email from the ASF dual-hosted git repository.
anton-vinogradov pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new a6dd87a095f IGNITE-29211 Hot-reload TLS certificates automatically
before expiry (#13786)
a6dd87a095f is described below
commit a6dd87a095f90f4055d58494ae2c67903ce9a098
Author: Anton Vinogradov <[email protected]>
AuthorDate: Tue Nov 17 16:28:32 2026 +0300
IGNITE-29211 Hot-reload TLS certificates automatically before expiry
(#13786)
---
docs/_docs/monitoring-metrics/new-metrics.adoc | 14 +
docs/_docs/security/ssl-tls.adoc | 94 +++
.../ignite/ssl/AbstractSslContextFactory.java | 2 +-
.../org/apache/ignite/ssl/SslContextFactory.java | 118 +++
.../util/GridCommandHandlerSslReloadTest.java | 2 +
modules/core/pom.xml | 7 +
.../org/apache/ignite/internal/IgniteKernal.java | 4 +
.../internal/management/ssl/SslReloadTask.java | 25 +-
.../processors/ssl/SslFactoryRegistry.java | 249 +++++-
.../processors/ssl/SslFactoryRegistryTest.java | 861 ++++++++++++++++++++-
.../ignite/ssl/SslCertificateHotReloadTest.java | 208 +++++
.../ignite/ssl/SslContextFactoryHotReloadTest.java | 527 +++++++++++++
.../ignite/ssl/TestCertificateAuthority.java | 154 ++++
.../ignite/testsuites/SecurityTestSuite.java | 4 +
14 files changed, 2237 insertions(+), 32 deletions(-)
diff --git a/docs/_docs/monitoring-metrics/new-metrics.adoc
b/docs/_docs/monitoring-metrics/new-metrics.adoc
index 545c0fa8af7..3d6c6085234 100644
--- a/docs/_docs/monitoring-metrics/new-metrics.adoc
+++ b/docs/_docs/monitoring-metrics/new-metrics.adoc
@@ -433,6 +433,20 @@ Register name: `communication.tcp.{node_consistent_id}`
|sentMessagesToNode | long | Total number of messages sent by current node to
the given node.
|===
+== Certificate Hot Reload
+
+Register name: `ssl`
+
+[cols="2,1,3",opts="header"]
+|===
+|Name | Type | Description
+|ConsecutiveHotReloadFailures | long | Number of consecutive automatic hot
reload failures.
+|ManualReloadAttempts | long | Number of manual reload attempts.
+|AutomaticHotReloadAttempts | long | Number of automatic hot reload attempts.
+|===
+
+See link:security/ssl-tls#automatic-certificate-hot-reload[Automatic
Certificate Hot Reload] for configuration and recovery.
+
== Communication Connection Pool
Register name: `communication.tcp.connectionPool`
diff --git a/docs/_docs/security/ssl-tls.adoc b/docs/_docs/security/ssl-tls.adoc
index 9ede647e3ed..56531d4cecc 100644
--- a/docs/_docs/security/ssl-tls.adoc
+++ b/docs/_docs/security/ssl-tls.adoc
@@ -188,6 +188,92 @@ If any factory on a node has another type, the command
reports it as unsupported
For these factories, follow the same trust-first sequence and restart affected
nodes one at a time instead of running reload.
If you configured `trustManagers` directly, the command reuses them instead of
reloading a trust store.
+== Automatic Certificate Hot Reload
+
+Automatic hot reload lets nodes load replacement certificates without a reload
command.
+You must supply the replacements: Ignite reads them from the configured source
and does not issue certificates.
+Hot reload covers the same TCP transports as <<Upgrading Certificates,manual
reload>>.
+It does not update the HTTP REST endpoint or external client applications.
+
+=== Enable Hot Reload
+
+Enable hot reload on each configured `SslContextFactory` before starting the
node:
+
+[source,java]
+----
+factory.setHotReloadEnabled(true);
+----
+
+Changing this setting requires a node restart.
+The key store must contain a private key with an X.509 certificate chain.
+A chain contains the key's certificate and the certificates of its issuers.
+Ignite ignores keys without such a chain, as TLS does.
+If the key store has no such key, the node logs a warning and skips hot reload.
+After a manual reload of a key store with such a key, hot reload works again.
+
+Before replacing key stores, prepare peers' trust stores as described in
<<Upgrading Certificates>>.
+Once every affected peer trusts the replacement certificates, put the
replacement key stores in the configured source.
+Put only certificates whose validity period has already started.
+Ignite installs the replacements without checking their dates, and peers may
reject certificates that are not yet valid.
+Keep the configured paths, store types and passwords unchanged.
+Ignite loads the replacements automatically when hot reload is due; no reload
command is needed for this step.
+
+=== When Hot Reload Runs
+
+At startup, Ignite checks each factory with hot reload enabled.
+After it finishes checking all factories, it waits one hour before checking
them again.
+A slow certificate source can therefore delay a factory's next check by more
than an hour.
+
+For each factory, hot reload becomes due `hotReloadBeforeExpiryMs` before the
earliest certificate expiry in its private key chains.
+This includes issuing certificates. The default lead time is 7 days.
+The first check at or after that date attempts a reload, including the startup
check.
+Allow enough time before expiry to supply the replacements, wait through the
one-hour interval and load the stores.
+Keep the lead time shorter than the lifetime of the replacement certificates,
or they will already be due when loaded.
+See
link:{javadoc_base_url}/org/apache/ignite/ssl/SslContextFactory.html#getHotReloadTime()[getHotReloadTime(),
window=_blank] for the calculation.
+
+If the loaded certificates are still due for hot reload, Ignite records a
failed hot reload attempt even though it installed them.
+This can happen when the certificates have not changed, only the trust store
changed, or the replacements expire within `hotReloadBeforeExpiryMs`.
+The loaded context remains in use until a later reload replaces it.
+
+After a manual reload, the next hourly check uses the loaded certificates.
+
+New connections use the loaded certificates; established connections remain
open.
+To confirm a replacement, open a new TLS connection to each affected endpoint
and compare its peer certificate with the replacement.
+
+=== Check Failures and Recover
+
+If loading the stores or creating the TLS context fails, Ignite keeps the
current certificates and retries on the next check, even after they expire.
+Keeping expired certificates does not extend their validity, so new
connections may fail until a valid replacement is available.
+If the replacements appear after hot reload is due, the next retry loads them.
+
+The node log shows the affected transports.
+At `INFO` level it shows when the loaded certificates are due for hot reload,
when an attempt starts and when it succeeds.
+At `WARN` level it shows why an attempt failed and when the node retries
automatically.
+At `DEBUG` level it also shows the start of each check.
+The `ssl.ConsecutiveHotReloadFailures` metric sums consecutive failed
automatic attempts across the node's factories.
+Each factory's count resets after a successful manual reload or when a check
finds that the loaded certificates are no longer due for hot reload.
+
+To recover without waiting for the next retry:
+
+. Correct the replacement stores and run
link:tools/control-script#reloading-tls-certificates[`control.sh --ssl reload`].
+. Check the command result and verify a new connection with the replacement
certificates.
+
+*What happens to automatic hot reload.*
+If enabled, automatic hot reload continues after manual reload, as described
in <<When Hot Reload Runs>>.
+A failed automatic attempt does not undo the manual reload.
+
+*If the command cannot connect.*
+First check the command's certificates and trust settings.
+If the problem is on the node, put a valid replacement in its source.
+If the node log shows automatic retries, wait for the next attempt.
+Otherwise, or to load the corrected stores without waiting, restart the
affected nodes.
+
+=== Custom Certificate Sources
+
+For automatic hot reload, custom sources require a subclass of
javadoc:org.apache.ignite.ssl.SslContextFactory[].
+Factories extending only `AbstractSslContextFactory` support manual reload
only.
+Limit request duration in custom sources: a blocked load delays other hot
reloads and manual reload.
+
== SslContextFactory Properties
`SslContextFactory` supports the following properties:
@@ -216,6 +302,14 @@ If you configured `trustManagers` directly, the command
reuses them instead of r
|The protocol for secure transport.
https://docs.oracle.com/en/java/javase/11/docs/specs/security/standard-names.html#sslcontext-algorithms[Supported
algorithms,window=_blank].
|`TLS`
+|`hotReloadEnabled`
+|Enables automatic certificate hot reload. See <<Automatic Certificate Hot
Reload>>.
+|`false`
+
+|`hotReloadBeforeExpiryMs`
+|How long before the certificates expire hot reload becomes due, in
milliseconds. See <<When Hot Reload Runs>>.
+|`604800000` (7 days)
+
|`trustStoreFilePath`
|The path to the trust store file.
|`N/A`
diff --git
a/modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java
b/modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java
index 15306f1d2b9..6babf0267f3 100644
---
a/modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java
+++
b/modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java
@@ -113,7 +113,7 @@ public abstract class AbstractSslContextFactory implements
Factory<SSLContext> {
* @return Initialized SSL context.
* @throws SSLException If SSL context could not be created.
*/
- private SSLContext createSslContext() throws SSLException {
+ SSLContext createSslContext() throws SSLException {
checkParameters();
KeyManager[] keyMgrs = createKeyManagers();
diff --git
a/modules/commons/src/main/java/org/apache/ignite/ssl/SslContextFactory.java
b/modules/commons/src/main/java/org/apache/ignite/ssl/SslContextFactory.java
index 5db5c511c7b..f3116d2e658 100644
--- a/modules/commons/src/main/java/org/apache/ignite/ssl/SslContextFactory.java
+++ b/modules/commons/src/main/java/org/apache/ignite/ssl/SslContextFactory.java
@@ -24,11 +24,15 @@ import java.io.InputStream;
import java.security.GeneralSecurityException;
import java.security.KeyStore;
import java.security.NoSuchAlgorithmException;
+import java.security.cert.Certificate;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.Arrays;
+import java.util.Enumeration;
+import java.util.concurrent.TimeUnit;
import javax.net.ssl.KeyManager;
import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLException;
import javax.net.ssl.TrustManager;
import javax.net.ssl.TrustManagerFactory;
@@ -61,6 +65,9 @@ public class SslContextFactory extends
AbstractSslContextFactory {
/** Default key manager / trust manager algorithm. Specifying different
trust manager algorithm is not supported. */
public static final String DFLT_KEY_ALGORITHM =
System.getProperty("ssl.KeyManagerFactory.algorithm", "SunX509");
+ /** Default lead time for certificate hot reload: 7 days. */
+ public static final long DFLT_HOT_RELOAD_BEFORE_EXPIRY_MS =
TimeUnit.DAYS.toMillis(7);
+
/** Key manager algorithm. */
protected String keyAlgorithm = DFLT_KEY_ALGORITHM;
@@ -85,6 +92,73 @@ public class SslContextFactory extends
AbstractSslContextFactory {
/** Trust managers. */
protected TrustManager[] trustMgrs;
+ /** */
+ private boolean hotReloadEnabled;
+
+ /** */
+ private volatile long hotReloadBeforeExpiryMs =
DFLT_HOT_RELOAD_BEFORE_EXPIRY_MS;
+
+ /** */
+ private transient volatile long earliestExpiry;
+
+ /** */
+ private transient long pendingExpiry;
+
+ /**
+ * @return Whether automatic certificate hot reload is enabled. Defaults
to {@code false}.
+ */
+ public boolean isHotReloadEnabled() {
+ return hotReloadEnabled;
+ }
+
+ /**
+ * @param enabled Whether automatic certificate hot reload is enabled.
Changes require a node restart.
+ */
+ public void setHotReloadEnabled(boolean enabled) {
+ hotReloadEnabled = enabled;
+ }
+
+ /**
+ * @return How long before the certificates expire hot reload becomes due,
in milliseconds.
+ * Defaults to {@link #DFLT_HOT_RELOAD_BEFORE_EXPIRY_MS}.
+ */
+ public long getHotReloadBeforeExpiryMs() {
+ return hotReloadBeforeExpiryMs;
+ }
+
+ /**
+ * @param millis How long before the certificates expire hot reload
becomes due, in milliseconds. Must be positive.
+ */
+ public void setHotReloadBeforeExpiryMs(long millis) {
+ A.ensure(millis > 0, "millis > 0");
+
+ hotReloadBeforeExpiryMs = millis;
+ }
+
+ /**
+ * The hot reload time is calculated from the loaded certificates.
+ * It comes {@link #getHotReloadBeforeExpiryMs()} before the earliest
expiry among the chains of the private keys.
+ * A chain expires when any certificate in it expires.
+ * Keys without an X.509 certificate chain are ignored, as TLS cannot use
them.
+ *
+ * @return Hot reload time in milliseconds since the epoch, or {@code 0}
if the certificates have not been loaded yet
+ * or have no private key with an X.509 chain.
+ */
+ public long getHotReloadTime() {
+ long expiry = earliestExpiry;
+
+ return expiry == 0 ? 0 : expiry - hotReloadBeforeExpiryMs;
+ }
+
+ /** {@inheritDoc} */
+ @Override SSLContext createSslContext() throws SSLException {
+ SSLContext ctx = super.createSslContext();
+
+ earliestExpiry = pendingExpiry;
+
+ return ctx;
+ }
+
/**
* Gets key store type used for context creation.
*
@@ -266,6 +340,8 @@ public class SslContextFactory extends
AbstractSslContextFactory {
keyMgrFactory.init(keyStore, keyStorePwd);
+ pendingExpiry = keyChainsExpiry(keyStore);
+
return keyMgrFactory.getKeyManagers();
}
catch (NoSuchAlgorithmException e) {
@@ -277,6 +353,48 @@ public class SslContextFactory extends
AbstractSslContextFactory {
}
}
+ /**
+ * @param store Loaded key store.
+ * @return Earliest certificate expiry in the private key chains, or
{@code 0} if the store has no private key
+ * with an X.509 certificate chain.
+ * @throws GeneralSecurityException If the key store cannot be read.
+ */
+ private static long keyChainsExpiry(KeyStore store) throws
GeneralSecurityException {
+ long earliest = Long.MAX_VALUE;
+
+ for (Enumeration<String> aliases = store.aliases();
aliases.hasMoreElements(); ) {
+ String alias = aliases.nextElement();
+
+ if (!store.isKeyEntry(alias))
+ continue;
+
+ earliest = Math.min(earliest,
chainExpiry(store.getCertificateChain(alias)));
+ }
+
+ return earliest == Long.MAX_VALUE ? 0 : earliest;
+ }
+
+ /**
+ * @param chain Certificate chain of a key entry, or {@code null} if the
entry has none.
+ * @return Earliest expiry in the chain, or {@link Long#MAX_VALUE} if
there is no chain or it holds a certificate other
+ * than X.509, as TLS ignores such keys.
+ */
+ private static long chainExpiry(Certificate[] chain) {
+ long expiry = Long.MAX_VALUE;
+
+ if (chain == null)
+ return expiry;
+
+ for (Certificate cert : chain) {
+ if (!(cert instanceof X509Certificate x509))
+ return Long.MAX_VALUE;
+
+ expiry = Math.min(expiry, x509.getNotAfter().getTime());
+ }
+
+ return expiry;
+ }
+
/** {@inheritDoc} */
@Override protected final TrustManager[] createTrustManagers() throws
SSLException {
if (trustMgrs != null)
diff --git
a/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerSslReloadTest.java
b/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerSslReloadTest.java
index 81809a7fc91..254194524ec 100644
---
a/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerSslReloadTest.java
+++
b/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerSslReloadTest.java
@@ -36,6 +36,7 @@ import org.apache.ignite.internal.IgniteEx;
import org.apache.ignite.internal.IgniteInternalFuture;
import org.apache.ignite.internal.TestRecordingCommunicationSpi;
import
org.apache.ignite.internal.processors.rest.client.message.GridClientPingPacket;
+import org.apache.ignite.spi.metric.LongMetric;
import org.apache.ignite.ssl.SslContextFactory;
import org.apache.ignite.testframework.ListeningTestLogger;
import org.apache.ignite.testframework.LogListener;
@@ -97,6 +98,7 @@ public class GridCommandHandlerSslReloadTest extends
GridCommandHandlerClusterPe
: "communication, client connector, binary REST, discovery";
assertContains(log, testOut.toString(),
nodes.get(i).localNode().id() + ": " + transports + ": reloaded");
+ assertEquals(1,
nodes.get(i).context().metric().registry("ssl").<LongMetric>findMetric("ManualReloadAttempts").value());
}
}
diff --git a/modules/core/pom.xml b/modules/core/pom.xml
index bbdf058dea1..47fd6cc45de 100644
--- a/modules/core/pom.xml
+++ b/modules/core/pom.xml
@@ -158,6 +158,13 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.bouncycastle</groupId>
+ <artifactId>bcpkix-jdk18on</artifactId>
+ <version>1.86</version>
+ <scope>test</scope>
+ </dependency>
+
<dependency>
<groupId>org.eclipse.jetty.ee11</groupId>
<artifactId>jetty-ee11-servlets</artifactId>
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java
b/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java
index 58063af1115..2fdad26a97f 100644
--- a/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java
+++ b/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java
@@ -1243,6 +1243,8 @@ public class IgniteKernal implements IgniteEx,
Externalizable {
}
}
+ ctx.ssl().start(ctx);
+
// Start plugins.
for (PluginProvider<?> provider : ctx.plugins().allProviders())
provider.onIgniteStart();
@@ -1722,6 +1724,8 @@ public class IgniteKernal implements IgniteEx,
Externalizable {
GridKernalGateway gw = this.gw.get();
if (stopGuard.compareAndSet(false, true)) {
+ ctx.ssl().stop();
+
// Only one thread is allowed to perform stop sequence.
boolean firstStop = false;
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/management/ssl/SslReloadTask.java
b/modules/core/src/main/java/org/apache/ignite/internal/management/ssl/SslReloadTask.java
index 17315635271..116a8ade169 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/management/ssl/SslReloadTask.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/management/ssl/SslReloadTask.java
@@ -19,12 +19,10 @@ package org.apache.ignite.internal.management.ssl;
import java.util.ArrayList;
import java.util.Collection;
-import java.util.Collections;
import java.util.HashMap;
-import java.util.IdentityHashMap;
import java.util.List;
import java.util.Map;
-import java.util.Set;
+import java.util.Map.Entry;
import java.util.UUID;
import javax.cache.configuration.Factory;
import javax.net.ssl.SSLContext;
@@ -120,7 +118,7 @@ public class SslReloadTask extends
VisorMultiNodeTask<NoArg, String, String> {
/** {@inheritDoc} */
@Override protected String run(NoArg arg) {
- Map<String, Factory<SSLContext>> factories =
ignite.context().ssl().factories();
+ List<Entry<Factory<SSLContext>, String>> factories =
ignite.context().ssl().factories();
UUID nodeId = ignite.localNode().id();
String initiator = "";
@@ -138,9 +136,9 @@ public class SslReloadTask extends
VisorMultiNodeTask<NoArg, String, String> {
return report;
}
- for (Map.Entry<String, Factory<SSLContext>> entry :
factories.entrySet()) {
- if (!(entry.getValue() instanceof AbstractSslContextFactory)) {
- String report = entry.getKey() + ": unsupported: " +
entry.getValue().getClass().getName();
+ for (Entry<Factory<SSLContext>, String> entry : factories) {
+ if (!(entry.getKey() instanceof AbstractSslContextFactory)) {
+ String report = entry.getValue() + ": unsupported: " +
entry.getKey().getClass().getName();
ignite.log().warning("TLS reload: " + nodeId + ": " +
report + initiator);
@@ -148,20 +146,13 @@ public class SslReloadTask extends
VisorMultiNodeTask<NoArg, String, String> {
}
}
- Set<Factory<SSLContext>> processed = Collections.newSetFromMap(new
IdentityHashMap<>());
List<String> reports = new ArrayList<>();
- for (Factory<SSLContext> factory : factories.values()) {
- if (!processed.add(factory))
- continue;
-
- String transports = factories.entrySet().stream()
- .filter(entry -> entry.getValue() == factory)
- .map(Map.Entry::getKey)
- .collect(joining(", "));
+ for (Entry<Factory<SSLContext>, String> entry : factories) {
+ String transports = entry.getValue();
try {
- ((AbstractSslContextFactory)factory).init();
+
ignite.context().ssl().reload((AbstractSslContextFactory)entry.getKey());
}
catch (Exception e) {
ignite.log().warning("TLS reload: " + nodeId + ": " +
transports + ": failed" + initiator, e);
diff --git
a/modules/core/src/main/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistry.java
b/modules/core/src/main/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistry.java
index ae3db72a664..9f1b5b2acbf 100644
---
a/modules/core/src/main/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistry.java
+++
b/modules/core/src/main/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistry.java
@@ -17,24 +17,259 @@
package org.apache.ignite.internal.processors.ssl;
-import java.util.LinkedHashMap;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
import java.util.Map;
+import java.util.Map.Entry;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
import javax.cache.configuration.Factory;
import javax.net.ssl.SSLContext;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.internal.GridKernalContext;
+import org.apache.ignite.internal.processors.metric.MetricRegistryImpl;
+import
org.apache.ignite.internal.thread.context.function.OperationContextAwareRunnable;
+import org.apache.ignite.logger.NullLogger;
+import org.apache.ignite.ssl.AbstractSslContextFactory;
+import org.apache.ignite.ssl.SslContextFactory;
+import org.jetbrains.annotations.Nullable;
+
+import static java.util.concurrent.TimeUnit.HOURS;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static
org.apache.ignite.internal.thread.pool.IgniteScheduledThreadPoolExecutor.newSingleThreadScheduledExecutor;
/** SSL factories used by running transports. */
public class SslFactoryRegistry {
/** */
- private final Map<String, Factory<SSLContext>> factories = new
LinkedHashMap<>();
+ private static final long CHECK_PERIOD = HOURS.toMillis(1);
+
+ /** */
+ private final List<RegisteredFactory> factories = new
CopyOnWriteArrayList<>();
+
+ /** */
+ private volatile ScheduledExecutorService executor;
+
+ /** Logs nothing until the registry starts. */
+ private volatile IgniteLogger log = new NullLogger();
+
+ /** */
+ private final AtomicLong manualReloadAttempts = new AtomicLong();
+
+ /** */
+ private final AtomicLong automaticHotReloadAttempts = new AtomicLong();
/** */
public synchronized void register(String transport, Factory<SSLContext>
factory) {
- if (factories.putIfAbsent(transport, factory) != null)
- throw new IllegalStateException("SSL factory is already registered
[transport=" + transport + ']');
+ for (int i = 0; i < factories.size(); i++) {
+ RegisteredFactory registered = factories.get(i);
+
+ if (registered.factory == factory) {
+ factories.set(i, new RegisteredFactory(factory,
registered.transports + ", " + transport));
+
+ return;
+ }
+ }
+
+ factories.add(new RegisteredFactory(factory, transport));
+ }
+
+ /** @return Registered factories in registration order, each with its
transports joined by commas. */
+ public List<Entry<Factory<SSLContext>, String>> factories() {
+ List<Entry<Factory<SSLContext>, String>> res = new
ArrayList<>(factories.size());
+
+ for (RegisteredFactory registered : factories)
+ res.add(Map.entry(registered.factory, registered.transports));
+
+ return res;
+ }
+
+ /** */
+ public void start(GridKernalContext ctx) {
+ MetricRegistryImpl metrics = ctx.metric().registry("ssl");
+
+ metrics.register("ConsecutiveHotReloadFailures",
this::consecutiveHotReloadFailures,
+ "Number of consecutive automatic hot reload failures.");
+ metrics.register("ManualReloadAttempts", manualReloadAttempts::get,
"Number of manual reload attempts.");
+ metrics.register("AutomaticHotReloadAttempts",
automaticHotReloadAttempts::get,
+ "Number of automatic hot reload attempts.");
+
+ log = ctx.log(getClass());
+
+ boolean hotReload = false;
+
+ for (RegisteredFactory registered : factories) {
+ SslContextFactory sslFactory = registered.hotReloadFactory();
+
+ if (sslFactory != null) {
+ registered.logHotReloadTime(sslFactory.getHotReloadTime());
+
+ hotReload = true;
+ }
+ }
+
+ if (!hotReload)
+ return;
+
+ executor = newSingleThreadScheduledExecutor("ssl-hot-reload",
ctx.igniteInstanceName());
+
+ executor.scheduleWithFixedDelay(new
OperationContextAwareRunnable(this::checkFactories, null),
+ 0, CHECK_PERIOD, MILLISECONDS);
+ }
+
+ /** @return Sum of consecutive automatic hot reload failures across
factories. */
+ private long consecutiveHotReloadFailures() {
+ long sum = 0;
+
+ for (RegisteredFactory registered : factories)
+ sum += registered.failures.get();
+
+ return sum;
}
- /** @return Snapshot of active transport factories. */
- public synchronized Map<String, Factory<SSLContext>> factories() {
- return new LinkedHashMap<>(factories);
+ /** Runs the hourly check of the factories with hot reload enabled. */
+ private void checkFactories() {
+ if (log.isDebugEnabled())
+ log.debug("TLS hot reload check started");
+
+ for (RegisteredFactory registered : factories) {
+ if (registered.hotReloadFactory() != null)
+ registered.check();
+ }
+ }
+
+ /** */
+ public void stop() {
+ if (executor != null)
+ executor.shutdownNow();
+ }
+
+ /**
+ * Reloads a registered factory. For a factory with hot reload enabled,
waits for its running check and resets its failure
+ * count.
+ *
+ * @param factory Registered factory to reload.
+ */
+ public void reload(AbstractSslContextFactory factory) {
+ for (RegisteredFactory registered : factories) {
+ if (registered.factory == factory)
+ registered.reload();
+ }
+ }
+
+ /** Registered factory with its transports and hot reload state. The
hourly check and manual reload hold its monitor. */
+ private class RegisteredFactory {
+ /** */
+ private final Factory<SSLContext> factory;
+
+ /** */
+ private final String transports;
+
+ /** */
+ private final AtomicInteger failures = new AtomicInteger();
+
+ /** */
+ private RegisteredFactory(Factory<SSLContext> factory, String
transports) {
+ this.factory = factory;
+ this.transports = transports;
+ }
+
+ /** @return The factory if hot reload is enabled for it, {@code null}
otherwise. */
+ @Nullable private SslContextFactory hotReloadFactory() {
+ return factory instanceof SslContextFactory sslFactory &&
sslFactory.isHotReloadEnabled() ? sslFactory : null;
+ }
+
+ /**
+ * Checks the factory unless the registry has stopped. Counts and logs
a failed check, so it does not stop
+ * the other factories or the schedule. A check that fails while the
registry stops is logged at debug level
+ * and not counted.
+ */
+ private synchronized void check() {
+ if (executor.isShutdown())
+ return;
+
+ try {
+ reloadIfDue();
+ }
+ catch (Throwable e) {
+ if (!executor.isShutdown()) {
+ failures.incrementAndGet();
+
+ log.warning("TLS hot reload failed [transports=" +
transports + ", nextAttempt="
+ + new Date(System.currentTimeMillis() + CHECK_PERIOD)
+ ']', e);
+ }
+ else if (log.isDebugEnabled())
+ log.debug("TLS hot reload failed during node stop
[transports=" + transports + ", err=" + e + ']');
+ }
+ }
+
+ /**
+ * Reloads the factory if its hot reload time has come, and ends the
failure streak if it has not.
+ *
+ * @throws IgniteException If the reload fails or leaves certificates
that are still due.
+ */
+ private void reloadIfDue() {
+ SslContextFactory sslFactory = (SslContextFactory)factory;
+
+ if (!due(sslFactory.getHotReloadTime())) {
+ failures.set(0);
+
+ return;
+ }
+
+ automaticHotReloadAttempts.incrementAndGet();
+
+ if (log.isInfoEnabled())
+ log.info("TLS hot reload started [transports=" + transports +
']');
+
+ sslFactory.init();
+
+ long hotReloadTime = sslFactory.getHotReloadTime();
+
+ if (due(hotReloadTime)) {
+ throw new IgniteException("Loaded certificates are still due
for hot reload [hotReloadTime="
+ + new Date(hotReloadTime) + ']');
+ }
+
+ failures.set(0);
+
+ if (log.isInfoEnabled())
+ log.info("TLS hot reload succeeded [transports=" + transports
+ ']');
+
+ logHotReloadTime(hotReloadTime);
+ }
+
+ /** @return Whether the loaded certificates have a hot reload time,
and it has come. */
+ private static boolean due(long hotReloadTime) {
+ return hotReloadTime != 0 && hotReloadTime <=
System.currentTimeMillis();
+ }
+
+ /** */
+ private synchronized void reload() {
+ manualReloadAttempts.incrementAndGet();
+
+ ((AbstractSslContextFactory)factory).init();
+
+ SslContextFactory sslFactory = hotReloadFactory();
+
+ if (sslFactory != null) {
+ failures.set(0);
+
+ logHotReloadTime(sslFactory.getHotReloadTime());
+ }
+ }
+
+ /** */
+ private void logHotReloadTime(long hotReloadTime) {
+ if (hotReloadTime == 0) {
+ log.warning("TLS hot reload will not run: no private key with
an X.509 certificate chain was found [transports="
+ + transports + ']');
+ }
+ else if (log.isInfoEnabled())
+ log.info("TLS hot reload due [transports=" + transports + ",
hotReloadTime=" + new Date(hotReloadTime) + ']');
+ }
}
}
diff --git
a/modules/core/src/test/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistryTest.java
b/modules/core/src/test/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistryTest.java
index fe517fe19f1..4cd7b66ddd8 100644
---
a/modules/core/src/test/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistryTest.java
+++
b/modules/core/src/test/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistryTest.java
@@ -17,23 +17,870 @@
package org.apache.ignite.internal.processors.ssl;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.atomic.AtomicInteger;
+import javax.net.ssl.SSLException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.internal.GridKernalContext;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.processors.metric.GridMetricManager;
+import org.apache.ignite.internal.processors.metric.MetricRegistryImpl;
+import org.apache.ignite.internal.thread.context.OperationContext;
+import org.apache.ignite.internal.thread.context.OperationContextAttribute;
+import org.apache.ignite.internal.thread.context.Scope;
+import
org.apache.ignite.internal.thread.pool.IgniteScheduledThreadPoolExecutor;
+import org.apache.ignite.internal.util.typedef.X;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.logger.NullLogger;
+import org.apache.ignite.spi.metric.LongMetric;
+import org.apache.ignite.ssl.AbstractSslContextFactory;
import org.apache.ignite.ssl.SslContextFactory;
+import org.apache.ignite.ssl.TestCertificateAuthority;
+import org.apache.ignite.testframework.GridTestThread;
+import org.apache.ignite.testframework.ListeningTestLogger;
+import org.junit.After;
+import org.junit.Before;
import org.junit.Test;
+import org.mockito.MockedStatic;
-import static org.junit.Assert.assertSame;
+import static java.util.concurrent.TimeUnit.DAYS;
+import static java.util.concurrent.TimeUnit.HOURS;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static org.apache.ignite.ssl.TestCertificateAuthority.PASSWORD;
+import static org.apache.ignite.ssl.TestCertificateAuthority.save;
+import static org.apache.ignite.testframework.GridTestUtils.runAsync;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.mockStatic;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
-/** */
+/**
+ * A manual reload is {@link SslFactoryRegistry#reload}, which the TLS reload
command calls.
+ * An attempt is an automatic hot reload attempt that the hourly check makes.
+ */
public class SslFactoryRegistryTest {
+ /** Upper bound for waiting for an event that must happen. */
+ private static final long TIMEOUT_MS = 30_000;
+
+ /** */
+ private final SslFactoryRegistry registry = new SslFactoryRegistry();
+
+ /** */
+ private final SslContextFactory factory = mock(SslContextFactory.class);
+
+ /** */
+ private final GridKernalContext ctx = mock(GridKernalContext.class);
+
+ /** */
+ private final ScheduledExecutorService executor =
mock(ScheduledExecutorService.class);
+
+ /** */
+ private final MetricRegistryImpl metrics = new MetricRegistryImpl("ssl",
null, null, null, new NullLogger());
+
+ /** */
+ private MockedStatic<IgniteScheduledThreadPoolExecutor> executorFactory;
+
+ /** Hourly hot reload check that the registry scheduled. */
+ private Runnable check;
+
/** */
+ private Path dir;
+
+ /** Key store file of the factory under test. */
+ private Path keys;
+
+ /** */
+ private Path trust;
+
+ /** */
+ private TestCertificateAuthority authority;
+
+ /**
+ * Each test gets a registry whose hot reload executor records the hourly
check instead of running it.
+ * Tests run the check themselves.
+ * The setup also writes the trust store of a test CA to a temporary
directory and stubs the node metrics and logger.
+ */
+ @Before
+ public void setUp() throws Exception {
+ dir = Files.createTempDirectory("ssl-factory-registry");
+ keys = dir.resolve("keys.jks");
+ trust = dir.resolve("trust.jks");
+ authority = new TestCertificateAuthority("issuer",
at(-DAYS.toHours(365)), at(DAYS.toHours(365)));
+
+ save(authority.trustStore(), trust);
+
+ GridMetricManager metricMgr = mock(GridMetricManager.class);
+
+ when(ctx.metric()).thenReturn(metricMgr);
+ when(metricMgr.registry("ssl")).thenReturn(metrics);
+ when(ctx.log(SslFactoryRegistry.class)).thenReturn(new NullLogger());
+
+ when(executor.scheduleWithFixedDelay(any(Runnable.class), anyLong(),
anyLong(), eq(MILLISECONDS))).thenAnswer(call -> {
+ check = call.getArgument(0);
+
+ return mock(ScheduledFuture.class);
+ });
+
+ executorFactory = mockStatic(IgniteScheduledThreadPoolExecutor.class);
+ executorFactory.when(() ->
IgniteScheduledThreadPoolExecutor.newSingleThreadScheduledExecutor(any(),
any()))
+ .thenReturn(executor);
+ }
+
+ /** After each test, the registry stops, and the static executor mock and
the temporary directory are released. */
+ @After
+ public void tearDown() throws Exception {
+ try {
+ registry.stop();
+ }
+ finally {
+ executorFactory.close();
+ U.delete(dir);
+ }
+ }
+
+ /**
+ * Given: communication and discovery register the same factory, with a
different factory registered in between.
+ * When: the registry lists its factories.
+ * Then: each factory appears once under its transports, in the order of
first registration.
+ */
+ @Test
+ public void testFactoriesAreGroupedInRegistrationOrder() {
+ SslContextFactory clientConnector = new SslContextFactory();
+
+ registry.register("communication", factory);
+ registry.register("client connector", clientConnector);
+ registry.register("discovery", factory);
+
+ assertEquals(List.of(Map.entry(factory, "communication, discovery"),
+ Map.entry(clientConnector, "client connector")),
registry.factories());
+ }
+
+ /**
+ * Given: a factory with hot reload disabled and valid certificates.
+ * When: the registry starts.
+ * Then: no hot reload thread starts, and the failure metric reads zero.
+ */
+ @Test
+ public void testDisabledFactoryDoesNotStartHotReload() throws Exception {
+ writeFreshCertificates(keys);
+ registerTransport("communication", factory(keys));
+ registry.start(ctx);
+
+ executorFactory.verifyNoInteractions();
+ assertEquals(0, failures());
+ }
+
+ /**
+ * Given: communication and discovery share a factory, and the client
connector has an equal factory for the same file.
+ * When: the key store is replaced and the hourly check runs.
+ * Then: each factory instance loads the new certificates once.
+ */
+ @Test
+ public void testEachFactoryInstanceReloadsOnce() throws Exception {
+ SslContextFactory communication = factory(keys, new
KeyStorePathFactory());
+ SslContextFactory clientConnector = factory(keys, new
KeyStorePathFactory());
+
+ writeDueCertificates(keys);
+ communication.setHotReloadEnabled(true);
+ clientConnector.setHotReloadEnabled(true);
+ registerTransport("communication", communication);
+ registerTransport("discovery", communication);
+ registerTransport("client connector", clientConnector);
+ registry.start(ctx);
+
+ long prevTime = communication.getHotReloadTime();
+
+ writeFreshCertificates(keys);
+ check.run();
+
+ assertEquals(2, metric("AutomaticHotReloadAttempts"));
+ assertTrue(communication.getHotReloadTime() > prevTime);
+ assertEquals(communication.getHotReloadTime(),
clientConnector.getHotReloadTime());
+ }
+
+ /**
+ * Given: communication and the client connector use equal but separate
factories for the same key store file.
+ * When: the key store is replaced, and the client connector's factory is
reloaded manually.
+ * Then: the client connector's factory loads the new certificates, and
the communication factory keeps its own.
+ */
+ @Test
+ public void testManualReloadLoadsOnlyItsFactoryInstance() throws Exception
{
+ SslContextFactory communication = factory(keys, new
KeyStorePathFactory());
+ SslContextFactory clientConnector = factory(keys, new
KeyStorePathFactory());
+
+ writeDueCertificates(keys);
+ communication.setHotReloadEnabled(true);
+ clientConnector.setHotReloadEnabled(true);
+ registerTransport("communication", communication);
+ registerTransport("client connector", clientConnector);
+ registry.start(ctx);
+
+ long prevTime = communication.getHotReloadTime();
+
+ writeFreshCertificates(keys);
+ registry.reload(clientConnector);
+
+ assertTrue(clientConnector.getHotReloadTime() > prevTime);
+ assertEquals(prevTime, communication.getHotReloadTime());
+ }
+
+ /**
+ * Given: hot reload is enabled, and the key store holds only a CA
certificate.
+ * When: the registry starts and the hourly check runs, then the operator
puts a key store with a private key in place
+ * and reloads it.
+ * Then: the registry logs that hot reload will not run, and the check
makes no attempt.
+ * After the reload, the log shows when hot reload is due.
+ */
+ @Test
+ public void testStoreWithoutPrivateKeyIsReportedUntilKeysAreReloaded()
throws Exception {
+ List<String> logged = captureLog();
+ SslContextFactory communication = factory(keys);
+
+ save(authority.trustStore(), keys);
+ communication.setHotReloadEnabled(true);
+ registerTransport("communication", communication);
+ registry.start(ctx);
+ check.run();
+
+ assertEquals(1, logged.size());
+ assertTrue(logged.get(0), logged.get(0).startsWith("TLS hot reload
will not run"));
+ assertEquals(0, metric("AutomaticHotReloadAttempts"));
+
+ writeFreshCertificates(keys);
+ registry.reload(communication);
+
+ assertEquals("TLS hot reload due [transports=communication,
hotReloadTime=" + new Date(communication.getHotReloadTime()) + ']',
+ logged.get(logged.size() - 1));
+ }
+
+ /**
+ * Given: hot reload is enabled for due certificates, and the key store
file is deleted after start.
+ * When: the hourly check runs three times.
+ * Then: the check runs at start and then every hour, and each run counts
one more failed attempt.
+ */
+ @Test
+ public void testHotReloadIsCheckedEveryHourFromStart() throws Exception {
+ SslContextFactory communication = factory(keys);
+
+ writeDueCertificates(keys);
+ communication.setHotReloadEnabled(true);
+ registerTransport("communication", communication);
+ registry.start(ctx);
+ Files.delete(keys);
+
+ verify(executor).scheduleWithFixedDelay(any(Runnable.class), eq(0L),
eq(HOURS.toMillis(1)), eq(MILLISECONDS));
+
+ for (int failure = 1; failure <= 3; failure++) {
+ check.run();
+
+ assertEquals(failure, failures());
+ }
+ }
+
+ /**
+ * Given: the registry starts during an operation with a request ID.
+ * When: its worker reloads a factory with due certificates.
+ * Then: the factory sees no request ID from the operation that started
the registry.
+ */
+ @Test
+ public void testHotReloadDoesNotInheritStartupContext() throws Exception {
+ executorFactory.when(() ->
IgniteScheduledThreadPoolExecutor.newSingleThreadScheduledExecutor(any(),
any())).thenCallRealMethod();
+
+ OperationContextAttribute<String> reqId =
OperationContextAttribute.newInstance();
+ CompletableFuture<String> reloadReqId = new CompletableFuture<>();
+
+ enableHotReload(factory, due());
+ doAnswer(call -> {
+ reloadReqId.complete(OperationContext.get(reqId));
+
+ return null;
+ }).when(factory).init();
+ registry.register("communication", factory);
+
+ try (Scope ignored = OperationContext.set(reqId, "startup")) {
+ registry.start(ctx);
+ }
+
+ assertNull(reloadReqId.get(TIMEOUT_MS, MILLISECONDS));
+ }
+
+ /**
+ * Given: two factories with due certificates, and loading the first one
throws an error,
+ * as a missing class of a key store provider does.
+ * When: the hourly check runs.
+ * Then: the check completes normally, so the executor keeps the schedule.
It counts and logs the error
+ * and still reloads the second factory.
+ */
+ @Test
+ public void testErrorFromOneFactoryDoesNotStopCheck() throws Exception {
+ List<String> logged = captureLog();
+ SslContextFactory clientConnector = factory(keys);
+
+ enableHotReload(factory, due());
+ doThrow(new NoClassDefFoundError("Key store
provider")).when(factory).init();
+ writeDueCertificates(keys);
+ clientConnector.setHotReloadEnabled(true);
+ registry.register("communication", factory);
+ registerTransport("client connector", clientConnector);
+ registry.start(ctx);
+ writeFreshCertificates(keys);
+ check.run();
+
+ assertEquals(1, failures());
+ assertEquals(2, metric("AutomaticHotReloadAttempts"));
+ assertTrue(logged.toString(), logged.contains("TLS hot reload
succeeded [transports=client connector]"));
+ assertTrue(logged.toString(), logged.toString().contains("TLS hot
reload failed [transports=communication,"));
+ }
+
+ /**
+ * Given: hot reload is enabled for due certificates, and the key store
file is deleted after start.
+ * When: one automatic attempt fails, and the next one loads fresh
certificates.
+ * Then: the log shows when the loaded certificates are due for hot
reload, and the start and result of each attempt.
+ */
+ @Test
+ public void testLogShowsDueTimeStartAndResultOfAttempts() throws Exception
{
+ List<String> logged = captureLog();
+ SslContextFactory communication = factory(keys);
+
+ writeDueCertificates(keys);
+ communication.setHotReloadEnabled(true);
+ registerTransport("communication", communication);
+ registry.start(ctx);
+
+ long due = communication.getHotReloadTime();
+
+ Files.delete(keys);
+ check.run();
+ writeFreshCertificates(keys);
+ check.run();
+
+ List<String> evts = new ArrayList<>();
+
+ for (String line : logged) {
+ assertTrue(line, line.contains("[transports=communication"));
+
+ evts.add(line.substring("TLS hot reload ".length(), line.indexOf("
[")));
+ }
+
+ assertEquals(List.of("due", "started", "failed", "started",
"succeeded", "due"), evts);
+ assertTrue(logged.get(0), logged.get(0).endsWith("hotReloadTime=" +
new Date(due) + ']'));
+ assertTrue(logged.get(2), logged.get(2).contains("nextAttempt="));
+ assertTrue(logged.get(5), logged.get(5).endsWith("hotReloadTime=" +
new Date(communication.getHotReloadTime()) + ']'));
+ }
+
+ /**
+ * Given: hot reload is enabled for due certificates, and the key store
file is deleted after start.
+ * When: one check fails, the source then gets certificates due for hot
reload in 23 days, the next check loads them,
+ * and the check runs again.
+ * Then: both attempts are counted, the success resets the failure count,
and the last check makes no attempt.
+ */
+ @Test
+ public void testSuccessResetsFailuresAndLaterChecksWait() throws Exception
{
+ SslContextFactory communication = factory(keys);
+
+ writeDueCertificates(keys);
+ communication.setHotReloadEnabled(true);
+ registerTransport("communication", communication);
+ registry.start(ctx);
+ Files.delete(keys);
+ check.run();
+
+ assertEquals(1, failures());
+ assertEquals(1, metric("AutomaticHotReloadAttempts"));
+
+ writeFreshCertificates(keys);
+ check.run();
+
+ assertEquals(0, failures());
+ assertEquals(2, metric("AutomaticHotReloadAttempts"));
+
+ check.run();
+
+ assertEquals(2, metric("AutomaticHotReloadAttempts"));
+ }
+
+ /**
+ * Given: an automatic attempt has failed because the loaded certificates
are still due.
+ * When: a load outside this registry installs fresh certificates, as
another node sharing the factory does,
+ * and the next check runs.
+ * Then: the check resets the failure count without loading the factory.
+ */
+ @Test
+ public void testCheckResetsFailuresWhenFactoryIsNoLongerDue() throws
Exception {
+ SslContextFactory communication = factory(keys);
+
+ writeDueCertificates(keys);
+ communication.setHotReloadEnabled(true);
+ registerTransport("communication", communication);
+ registry.start(ctx);
+ check.run();
+
+ assertEquals(1, failures());
+
+ writeFreshCertificates(keys);
+ communication.init();
+ check.run();
+
+ assertEquals(0, failures());
+ assertEquals(1, metric("AutomaticHotReloadAttempts"));
+ }
+
+ /**
+ * Given: a factory with hot reload enabled is registered, and the
registry has not started yet.
+ * When: the key store is replaced, and a manual reload runs.
+ * Then: the factory loads the new certificates.
+ */
+ @Test
+ public void testManualReloadBeforeStartLoadsFactory() throws Exception {
+ SslContextFactory communication = factory(keys);
+
+ writeDueCertificates(keys);
+ communication.setHotReloadEnabled(true);
+ registerTransport("communication", communication);
+
+ long prevTime = communication.getHotReloadTime();
+
+ writeFreshCertificates(keys);
+ registry.reload(communication);
+
+ assertTrue(communication.getHotReloadTime() > prevTime);
+ }
+
+ /**
+ * Given: a factory with hot reload disabled.
+ * When: a manual reload succeeds, and a second one fails because the key
store file is deleted.
+ * Then: both count as manual attempts, and none counts as automatic.
+ */
@Test
- public void testDuplicateRegistrationKeepsOriginalFactory() {
- SslFactoryRegistry registry = new SslFactoryRegistry();
- SslContextFactory factory = new SslContextFactory();
+ public void testManualReloadWithoutHotReloadCountsAppliedAndFailedLoads()
throws Exception {
+ SslContextFactory communication = factory(keys);
+ writeFreshCertificates(keys);
+ registerTransport("communication", communication);
+ registry.start(ctx);
+ registry.reload(communication);
+
+ assertEquals(1, metric("ManualReloadAttempts"));
+
+ Files.delete(keys);
+
+ assertThrows(IgniteException.class, () ->
registry.reload(communication));
+ assertEquals(2, metric("ManualReloadAttempts"));
+ assertEquals(0, metric("AutomaticHotReloadAttempts"));
+ }
+
+ /**
+ * Given: a registered factory that is not an {@link SslContextFactory}.
+ * When: a manual reload runs.
+ * Then: the factory is loaded.
+ */
+ @Test
+ public void testManualReloadSupportsAbstractFactory() throws Exception {
+ AbstractSslContextFactory factory =
mock(AbstractSslContextFactory.class);
+
+ registry.register("client connector", factory);
+ registry.start(ctx);
+ registry.reload(factory);
+
+ verify(factory).init();
+ }
+
+ /**
+ * Given: an automatic attempt has failed on a missing key store file.
+ * When: a manual reload runs while the file is still missing.
+ * Then: the reload fails, and the failure count stays.
+ */
+ @Test
+ public void testFailedManualReloadKeepsFailures() throws Exception {
+ SslContextFactory communication = factory(keys);
+
+ writeDueCertificates(keys);
+ communication.setHotReloadEnabled(true);
+ registerTransport("communication", communication);
+ registry.start(ctx);
+ Files.delete(keys);
+ check.run();
+
+ IgniteException err = assertThrows(IgniteException.class, () ->
registry.reload(communication));
+
+ assertTrue(X.hasCause(err, "key store file was not found",
SSLException.class));
+ assertEquals(1, failures());
+ }
+
+ /**
+ * Given: communication only reloads manually, and the client connector is
due for hot reload.
+ * When: the hourly check runs while communication is waiting for its
certificate source.
+ * Then: the client connector loads its replacement before the manual
reload finishes.
+ */
+ @Test
+ public void testManualOnlyFactoryDoesNotDelayHotReload() throws Exception {
+ SslContextFactory clientConnector = factory(keys);
+ CountDownLatch manualStarted = new CountDownLatch(1);
+ CountDownLatch finishManual = new CountDownLatch(1);
+
+ doAnswer(call -> {
+ manualStarted.countDown();
+ finishManual.await();
+
+ return null;
+ }).when(factory).init();
+
+ writeDueCertificates(keys);
+ clientConnector.setHotReloadEnabled(true);
registry.register("communication", factory);
+ registerTransport("client connector", clientConnector);
+ registry.start(ctx);
+
+ long prevTime = clientConnector.getHotReloadTime();
+
+ writeFreshCertificates(keys);
+
+ IgniteInternalFuture<?> manual = runAsync(() ->
registry.reload(factory));
+
+ try {
+ assertTrue(manualStarted.await(TIMEOUT_MS, MILLISECONDS));
+ runAsync(check).get(TIMEOUT_MS);
+
+ assertTrue(clientConnector.getHotReloadTime() > prevTime);
+ }
+ finally {
+ finishManual.countDown();
+ manual.get(TIMEOUT_MS);
+ }
+ }
+
+ /**
+ * Given: an automatic attempt has started, failed to load a missing key
store file, and pauses at its stop check
+ * before it counts the failure.
+ * When: the operator puts fresh certificates in place and reloads them
during the pause.
+ * Then: the manual reload waits until the attempt finishes, so the
failure count ends at zero.
+ */
+ @Test
+ public void testManualReloadWaitsForRunningAttempt() throws Exception {
+ SslContextFactory communication = factory(keys);
+ CountDownLatch attemptFailed = new CountDownLatch(1);
+ CountDownLatch resumeAttempt = new CountDownLatch(1);
+
+ writeDueCertificates(keys);
+ communication.setHotReloadEnabled(true);
+ registerTransport("communication", communication);
+ registry.start(ctx);
+ Files.delete(keys);
+
+ when(executor.isShutdown()).thenAnswer(call -> {
+ if (metric("AutomaticHotReloadAttempts") > 0) {
+ attemptFailed.countDown();
+
+ assertTrue(resumeAttempt.await(TIMEOUT_MS, MILLISECONDS));
+ }
+
+ return false;
+ });
+
+ GridTestThread attempt = new GridTestThread(check);
+ GridTestThread manualReload = new GridTestThread(() ->
registry.reload(communication));
+
+ attempt.start();
+
+ try {
+ assertTrue(attemptFailed.await(TIMEOUT_MS, MILLISECONDS));
+
+ writeFreshCertificates(keys);
+ manualReload.start();
+
+ long deadline = System.nanoTime() +
MILLISECONDS.toNanos(TIMEOUT_MS);
+
+ while (manualReload.getState() != Thread.State.BLOCKED &&
manualReload.isAlive()) {
+ assertTrue(System.nanoTime() < deadline);
+
+ Thread.sleep(10);
+ }
+ }
+ finally {
+ resumeAttempt.countDown();
+ attempt.join(TIMEOUT_MS);
+ manualReload.join(TIMEOUT_MS);
+ }
+
+ attempt.checkError();
+ manualReload.checkError();
+ assertEquals(0, failures());
+ }
+
+ /**
+ * Given: communication has fresh certificates, and the client connector
has due ones, so its attempt fails.
+ * When: communication loads due certificates manually, the next check
fails for both factories,
+ * and the client connector then loads fresh certificates manually.
+ * Then: the metric sums the failures of both factories, and each manual
reload clears only its own factory's count.
+ */
+ @Test
+ public void testManualReloadResetsOnlyItsFactoryFailures() throws
Exception {
+ Path clientKeys = dir.resolve("client-keys.jks");
+ SslContextFactory communication = factory(keys);
+ SslContextFactory clientConnector = factory(clientKeys);
+
+ writeFreshCertificates(keys);
+ writeDueCertificates(clientKeys);
+ communication.setHotReloadEnabled(true);
+ clientConnector.setHotReloadEnabled(true);
+ registerTransport("communication", communication);
+ registerTransport("client connector", clientConnector);
+ registry.start(ctx);
+ check.run();
+
+ assertEquals(1, failures());
+
+ writeDueCertificates(keys);
+ registry.reload(communication);
+
+ assertEquals(1, failures());
+
+ check.run();
+
+ assertEquals(3, failures());
+
+ writeFreshCertificates(clientKeys);
+ registry.reload(clientConnector);
+
+ assertEquals(1, failures());
+ }
+
+ /**
+ * Given: an automatic load stays blocked even after it receives an
interrupt.
+ * When: the registry stops.
+ * Then: it interrupts the load and returns while the load is still
blocked.
+ */
+ @Test
+ public void testStopInterruptsActiveAttemptWithoutWaitingForIt() throws
Exception {
+ executorFactory.when(() ->
IgniteScheduledThreadPoolExecutor.newSingleThreadScheduledExecutor(any(),
any()))
+ .thenReturn(new
IgniteScheduledThreadPoolExecutor("ssl-hot-reload", null, 1));
+
+ CountDownLatch attemptStarted = new CountDownLatch(1);
+ CountDownLatch attemptInterrupted = new CountDownLatch(1);
+ CountDownLatch allowCompletion = new CountDownLatch(1);
+
+ enableHotReload(factory, due());
+ doAnswer(call -> {
+ attemptStarted.countDown();
+
+ try {
+ allowCompletion.await();
+ }
+ catch (InterruptedException e) {
+ attemptInterrupted.countDown();
+
+ U.awaitQuiet(allowCompletion);
+ }
+
+ return null;
+ }).when(factory).init();
+
+ registry.register("communication", factory);
+
+ try {
+ registry.start(ctx);
+
+ assertTrue(attemptStarted.await(TIMEOUT_MS, MILLISECONDS));
+
+ runAsync(registry::stop).get(TIMEOUT_MS);
+
+ assertTrue(attemptInterrupted.await(TIMEOUT_MS, MILLISECONDS));
+ }
+ finally {
+ allowCompletion.countDown();
+ }
+ }
+
+ /**
+ * Given: the node starts stopping during an automatic attempt.
+ * When: the attempt fails.
+ * Then: the failure is not counted, and the debug log shows the check and
the failed attempt.
+ */
+ @Test
+ public void testAttemptFailingDuringStopIsNotCounted() {
+ List<String> logged = captureLog(true);
+
+ enableHotReload(factory, due());
+ doAnswer(call -> {
+ when(executor.isShutdown()).thenReturn(true);
+
+ throw new IgniteException("Load interrupted by node stop");
+ }).when(factory).init();
+
+ registry.register("communication", factory);
+ registry.start(ctx);
+ check.run();
+
+ assertEquals(0, failures());
+ assertTrue(logged.toString(), logged.contains("TLS hot reload check
started"));
+ assertTrue(logged.toString(),
+ logged.toString().contains("TLS hot reload failed during node stop
[transports=communication, err="));
+ }
+
+ /**
+ * Given: two factories with due certificates.
+ * When: the node stops while the check loads one of them.
+ * Then: the check does not start loading the other one.
+ */
+ @Test
+ public void testStopDuringCheckSkipsRemainingFactories() {
+ SslContextFactory clientConnector = mock(SslContextFactory.class);
+ AtomicInteger loads = new AtomicInteger();
+
+ enableHotReload(factory, due());
+ enableHotReload(clientConnector, due());
+
+ for (SslContextFactory sslFactory : List.of(factory, clientConnector))
{
+ doAnswer(call -> {
+ loads.incrementAndGet();
+ when(executor.isShutdown()).thenReturn(true);
+
+ return null;
+ }).when(sslFactory).init();
+ }
+
+ registry.register("communication", factory);
+ registry.register("client connector", clientConnector);
+ registry.start(ctx);
+ check.run();
+
+ assertEquals(1, loads.get());
+ }
+
+ /** Makes the registry log to a listener and returns the hot reload lines
it logs at info level and above, in order. */
+ private List<String> captureLog() {
+ return captureLog(false);
+ }
+
+ /**
+ * Makes the registry log to a listener and returns the hot reload lines
it logs, in order.
+ *
+ * @param debug Whether the registry logs debug lines too.
+ */
+ private List<String> captureLog(boolean debug) {
+ ListeningTestLogger log = new ListeningTestLogger(debug ? new
DebugLogger() : null);
+ List<String> logged = new ArrayList<>();
+
+ log.registerListener(line -> {
+ if (line.startsWith("TLS hot reload "))
+ logged.add(line);
+ });
+ when(ctx.log(SslFactoryRegistry.class)).thenReturn(log);
+
+ return logged;
+ }
+
+ /** Loads a file-backed factory before registration, as the TCP transports
do during startup. */
+ private void registerTransport(String transport, SslContextFactory
factory) {
+ factory.create();
+ registry.register(transport, factory);
+ }
+
+ /** */
+ private long failures() {
+ return metric("ConsecutiveHotReloadFailures");
+ }
+
+ /** */
+ private long metric(String name) {
+ return metrics.<LongMetric>findMetric(name).value();
+ }
+
+ /**
+ * Makes the factory mock look like a factory with hot reload enabled and
loaded certificates with the given date.
+ * The mock returns {@code true} from {@code isHotReloadEnabled()} and the
date from {@code getHotReloadTime()}.
+ *
+ * @param factory Factory mock.
+ * @param hotReloadTime Hot reload time of the loaded certificates.
+ */
+ private static void enableHotReload(SslContextFactory factory, long
hotReloadTime) {
+ when(factory.isHotReloadEnabled()).thenReturn(true);
+ when(factory.getHotReloadTime()).thenReturn(hotReloadTime);
+ }
+
+ /** @return Hot reload time that has already come. */
+ private static long due() {
+ return System.currentTimeMillis();
+ }
+
+ /**
+ * Creates a factory with hot reload disabled that reads its key store
from the given file.
+ * The factory trusts the test CA through the trust store file from the
setup.
+ */
+ private SslContextFactory factory(Path keyStore) {
+ return factory(keyStore, new SslContextFactory());
+ }
+
+ /** Points the factory at the supplied key store and the test trust store.
*/
+ private SslContextFactory factory(Path keyStore, SslContextFactory
sslFactory) {
+ sslFactory.setKeyStoreFilePath(keyStore.toString());
+ sslFactory.setKeyStorePassword(PASSWORD);
+ sslFactory.setTrustStoreFilePath(trust.toString());
+ sslFactory.setTrustStorePassword(PASSWORD);
+
+ return sslFactory;
+ }
+
+ /**
+ * Writes a key store that is already due for hot reload with the default
lead time.
+ * Its certificate expires in 10 hours.
+ */
+ private void writeDueCertificates(Path keyStore) throws Exception {
+ save(authority.keyStore("node", at(-90), at(10)), keyStore);
+ }
+
+ /**
+ * Writes a key store that is due for hot reload in 23 days with the
default lead time.
+ * Its certificate became valid an hour ago and expires in 30 days.
+ */
+ private void writeFreshCertificates(Path keyStore) throws Exception {
+ save(authority.keyStore("node", at(-1), at(DAYS.toHours(30))),
keyStore);
+ }
+
+ /** */
+ private static long at(long hours) {
+ return System.currentTimeMillis() + HOURS.toMillis(hours);
+ }
+
+ /** Logger with debug level enabled that drops all lines. */
+ private static class DebugLogger extends NullLogger {
+ /** {@inheritDoc} */
+ @Override public boolean isDebugEnabled() {
+ return true;
+ }
+ }
+
+ /** Factories with the same key store path compare equal and cache their
own SSL contexts. */
+ private static class KeyStorePathFactory extends SslContextFactory {
+ /** */
+ private static final long serialVersionUID = 0;
+
+ /** {@inheritDoc} */
+ @Override public boolean equals(Object obj) {
+ return obj instanceof KeyStorePathFactory other &&
getKeyStoreFilePath().equals(other.getKeyStoreFilePath());
+ }
- assertThrows(IllegalStateException.class, () ->
registry.register("communication", new SslContextFactory()));
- assertSame(factory, registry.factories().get("communication"));
+ /** {@inheritDoc} */
+ @Override public int hashCode() {
+ return getKeyStoreFilePath().hashCode();
+ }
}
}
diff --git
a/modules/core/src/test/java/org/apache/ignite/ssl/SslCertificateHotReloadTest.java
b/modules/core/src/test/java/org/apache/ignite/ssl/SslCertificateHotReloadTest.java
new file mode 100644
index 00000000000..1b369d56303
--- /dev/null
+++
b/modules/core/src/test/java/org/apache/ignite/ssl/SslCertificateHotReloadTest.java
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ssl;
+
+import java.nio.file.Files;
+import java.nio.file.Path;
+import javax.net.ssl.SSLContext;
+import org.apache.ignite.Ignition;
+import org.apache.ignite.client.IgniteClient;
+import org.apache.ignite.client.SslMode;
+import org.apache.ignite.configuration.ClientConfiguration;
+import org.apache.ignite.configuration.ClientConnectorConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.spi.metric.LongMetric;
+import org.apache.ignite.testframework.ListeningTestLogger;
+import org.apache.ignite.testframework.LogListener;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+
+import static java.util.concurrent.TimeUnit.DAYS;
+import static java.util.concurrent.TimeUnit.MINUTES;
+import static org.apache.ignite.ssl.TestCertificateAuthority.PASSWORD;
+import static org.apache.ignite.ssl.TestCertificateAuthority.save;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** */
+public class SslCertificateHotReloadTest extends GridCommonAbstractTest {
+ /** Upper bound for waiting for an event that must happen. */
+ private static final long TIMEOUT_MS = 30_000;
+
+ /**
+ * Given: the factory has loaded due certificates of one CA, and the files
then get certificates of another CA.
+ * When: the node starts, and its hot reload attempt runs.
+ * Then: the node installs the new certificates, and a client that trusts
and presents only them connects.
+ */
+ @Test
+ public void testNodeHotReloadsCertificatesForNewConnections() throws
Exception {
+ long now = System.currentTimeMillis();
+ TestCertificateAuthority ca1 = new TestCertificateAuthority("ca1", now
- DAYS.toMillis(1), now + DAYS.toMillis(10));
+ TestCertificateAuthority ca2 = new TestCertificateAuthority("ca2", now
- DAYS.toMillis(1), now + DAYS.toMillis(10));
+ Path dir = Files.createTempDirectory("ssl-hot-reload");
+ Path keys = dir.resolve("keys.jks");
+ Path trust = dir.resolve("trust.jks");
+
+ try {
+ save(ca1.keyStore("node", now - MINUTES.toMillis(110), now +
MINUTES.toMillis(10)), keys);
+ save(ca1.trustStore(), trust);
+
+ SslContextFactory serverFactory = configure(new
SslContextFactory(), keys, trust);
+
+ serverFactory.setHotReloadEnabled(true);
+
+ SSLContext initCtx = serverFactory.create();
+
+ assertTrue("Hot reload must be due at startup",
serverFactory.getHotReloadTime() < now);
+
+ save(ca2.keyStore("node", now - MINUTES.toMillis(1), now +
DAYS.toMillis(10)), keys);
+ save(ca2.trustStore(), trust);
+
+ IgniteEx node = startGrid(0, cfg -> {
+
cfg.setClientConnectorConfiguration(sslClientConnector(serverFactory));
+ });
+
+ assertTrue("The reloaded context was not installed",
waitForCondition(() -> serverFactory.create() != initCtx, TIMEOUT_MS));
+
+ ClientConfiguration clientCfg = new ClientConfiguration()
+ .setAddresses("127.0.0.1:" +
node.context().clientListener().port())
+ .setSslMode(SslMode.REQUIRED)
+ .setSslContextFactory(configure(new SslContextFactory(), keys,
trust));
+
+ try (IgniteClient client = Ignition.startClient(clientCfg)) {
+ client.cacheNames();
+ }
+ }
+ finally {
+ stopAllGrids();
+ U.delete(dir);
+ }
+ }
+
+ /**
+ * Given: a node with hot reload enabled starts with expired certificates,
so hot reload stays due.
+ * When: the automatic attempt reloads the same certificates.
+ * Then: the node counts the failure and logs the transport and the reason.
+ */
+ @Test
+ public void testNodeReportsHotReloadFailureForExpiredCertificates() throws
Exception {
+ long now = System.currentTimeMillis();
+ TestCertificateAuthority ca = new TestCertificateAuthority("ca", now -
DAYS.toMillis(1), now + DAYS.toMillis(10));
+ Path dir = Files.createTempDirectory("ssl-hot-reload");
+ Path keys = dir.resolve("keys.jks");
+ Path trust = dir.resolve("trust.jks");
+ ListeningTestLogger nodeLog = new ListeningTestLogger(log);
+ LogListener failure = LogListener.matches("TLS hot reload failed
[transports=client connector").build();
+ LogListener reason = LogListener.matches("Loaded certificates are
still due for hot reload").build();
+
+ nodeLog.registerListener(failure);
+ nodeLog.registerListener(reason);
+
+ try {
+ save(ca.keyStore("node", now - MINUTES.toMillis(120), now -
MINUTES.toMillis(1)), keys);
+ save(ca.trustStore(), trust);
+
+ SslContextFactory serverFactory = configure(new
SslContextFactory(), keys, trust);
+
+ serverFactory.setHotReloadEnabled(true);
+
+ IgniteEx node = startGrid(0, cfg -> {
+
cfg.setGridLogger(nodeLog).setClientConnectorConfiguration(sslClientConnector(serverFactory));
+ });
+ LongMetric failures =
node.context().metric().registry("ssl").findMetric("ConsecutiveHotReloadFailures");
+
+ assertTrue("The automatic attempt did not fail",
waitForCondition(() -> failures.value() > 0, TIMEOUT_MS));
+ assertTrue(failure.check(TIMEOUT_MS));
+ assertTrue(reason.check(TIMEOUT_MS));
+ }
+ finally {
+ stopAllGrids();
+ U.delete(dir);
+ }
+ }
+
+ /**
+ * Given: a node with hot reload enabled runs its hot reload thread.
+ * When: the node stops.
+ * Then: the hot reload thread ends.
+ */
+ @Test
+ public void testNodeStopEndsHotReloadThread() throws Exception {
+ long now = System.currentTimeMillis();
+ TestCertificateAuthority ca = new TestCertificateAuthority("ca", now -
DAYS.toMillis(1), now + DAYS.toMillis(10));
+ Path dir = Files.createTempDirectory("ssl-hot-reload");
+ Path keys = dir.resolve("keys.jks");
+ Path trust = dir.resolve("trust.jks");
+
+ try {
+ save(ca.keyStore("node", now - DAYS.toMillis(1), now +
DAYS.toMillis(10)), keys);
+ save(ca.trustStore(), trust);
+
+ SslContextFactory serverFactory = configure(new
SslContextFactory(), keys, trust);
+
+ serverFactory.setHotReloadEnabled(true);
+
+ IgniteEx node = startGrid(0, cfg -> {
+
cfg.setClientConnectorConfiguration(sslClientConnector(serverFactory));
+ });
+
+ Thread hotReload = null;
+
+ for (Thread thread : Thread.getAllStackTraces().keySet()) {
+ if (thread.getName().startsWith("ssl-hot-reload") &&
thread.getName().contains(node.name()))
+ hotReload = thread;
+ }
+
+ assertNotNull("The hot reload thread is not running", hotReload);
+
+ stopAllGrids();
+ hotReload.join(TIMEOUT_MS);
+
+ assertFalse("The hot reload thread is still running",
hotReload.isAlive());
+ }
+ finally {
+ stopAllGrids();
+ U.delete(dir);
+ }
+ }
+
+ /**
+ * Creates a client connector configuration that uses the given factory
and requires client certificates.
+ * The connector uses this factory instead of the node factory.
+ */
+ private static ClientConnectorConfiguration
sslClientConnector(SslContextFactory factory) {
+ return new ClientConnectorConfiguration()
+ .setSslEnabled(true)
+ .setSslClientAuth(true)
+ .setUseIgniteSslContextFactory(false)
+ .setSslContextFactory(factory);
+ }
+
+ /**
+ * Points the factory to the given key and trust store files with the test
password.
+ * It returns the same factory, so a test can configure it inline.
+ */
+ private static SslContextFactory configure(SslContextFactory factory, Path
keys, Path trust) {
+ factory.setKeyStoreFilePath(keys.toString());
+ factory.setKeyStorePassword(PASSWORD);
+ factory.setTrustStoreFilePath(trust.toString());
+ factory.setTrustStorePassword(PASSWORD);
+
+ return factory;
+ }
+}
diff --git
a/modules/core/src/test/java/org/apache/ignite/ssl/SslContextFactoryHotReloadTest.java
b/modules/core/src/test/java/org/apache/ignite/ssl/SslContextFactoryHotReloadTest.java
new file mode 100644
index 00000000000..ee4120e4100
--- /dev/null
+++
b/modules/core/src/test/java/org/apache/ignite/ssl/SslContextFactoryHotReloadTest.java
@@ -0,0 +1,527 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ssl;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.security.KeyStore;
+import java.security.PublicKey;
+import java.security.cert.Certificate;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.FutureTask;
+import javax.crypto.spec.SecretKeySpec;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLException;
+import javax.net.ssl.SSLHandshakeException;
+import javax.net.ssl.SSLServerSocket;
+import javax.net.ssl.SSLSocket;
+import javax.net.ssl.TrustManager;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.testframework.GridTestThread;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import static java.util.concurrent.TimeUnit.DAYS;
+import static java.util.concurrent.TimeUnit.HOURS;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static
org.apache.ignite.ssl.SslContextFactory.DFLT_HOT_RELOAD_BEFORE_EXPIRY_MS;
+import static org.apache.ignite.ssl.TestCertificateAuthority.PASSWORD;
+import static org.apache.ignite.ssl.TestCertificateAuthority.merge;
+import static org.apache.ignite.ssl.TestCertificateAuthority.save;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNotSame;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.spy;
+
+/** */
+public class SslContextFactoryHotReloadTest {
+ /** */
+ @Rule public final TemporaryFolder tmp = new TemporaryFolder();
+
+ /** Upper bound for waiting for an event that must happen. */
+ private static final long TIMEOUT_MS = 30_000;
+
+ /** */
+ private final long now = System.currentTimeMillis() / 1_000 * 1_000;
+
+ /** */
+ private TestCertificateAuthority authority;
+
+ /**
+ * Each test gets a test CA whose certificate is valid from a year before
to a year after {@code now}.
+ * The tests issue their certificates from this CA with dates counted from
{@code now}.
+ */
+ @Before
+ public void setUp() throws Exception {
+ authority = new TestCertificateAuthority("issuer", now -
DAYS.toMillis(365), now + DAYS.toMillis(365));
+ }
+
+ /**
+ * Given: a key store with only a CA certificate and the default hot
reload setting.
+ * When: the factory loads the store.
+ * Then: the load succeeds with hot reload disabled and no scheduled
reload date.
+ */
+ @Test
+ public void testDefaultSettingsAcceptStoreWithoutPrivateKeys() throws
Exception {
+ InMemorySslContextFactory factory = new
InMemorySslContextFactory(authority.trustStore());
+
+ assertFalse(factory.isHotReloadEnabled());
+ assertNotNull(factory.create());
+ assertEquals(0, factory.getHotReloadTime());
+ }
+
+ /**
+ * Given: a factory has loaded a certificate that expires in 10 days.
+ * When: the application increases the lead time from the default of 7
days to 14 days before starting the node.
+ * Then: the same certificate is already due for hot reload.
+ */
+ @Test
+ public void testHotReloadTimeUsesCurrentLeadTime() throws Exception {
+ Path keys = tmp.newFile("keys.jks").toPath();
+
+ save(authority.keyStore("node", at(-1), at(DAYS.toHours(10))), keys);
+
+ SslContextFactory factory = new SslContextFactory();
+
+ factory.setKeyStoreFilePath(keys.toString());
+ factory.setKeyStorePassword(PASSWORD);
+ factory.setTrustManagers(SslContextFactory.getDisabledTrustManager());
+ factory.create();
+
+ assertEquals(at(DAYS.toHours(3)), factory.getHotReloadTime());
+
+ factory.setHotReloadBeforeExpiryMs(DAYS.toMillis(14));
+
+ assertEquals(at(-DAYS.toHours(4)), factory.getHotReloadTime());
+ }
+
+ /**
+ * Given: a factory with the default lead time.
+ * When: the application sets a zero or negative lead time.
+ * Then: the setter rejects it and keeps the default.
+ */
+ @Test
+ public void testNonPositiveHotReloadBeforeExpiryMsIsRejected() {
+ SslContextFactory factory = new SslContextFactory();
+
+ assertThrows(IllegalArgumentException.class, () ->
factory.setHotReloadBeforeExpiryMs(0));
+ assertThrows(IllegalArgumentException.class, () ->
factory.setHotReloadBeforeExpiryMs(-1));
+ assertEquals(DFLT_HOT_RELOAD_BEFORE_EXPIRY_MS,
factory.getHotReloadBeforeExpiryMs());
+ }
+
+ /**
+ * Given: key stores in which the chain that expires first has 15 hours
left.
+ * In one of them, that chain gets its expiry from an issuer that
expires before the key certificate.
+ * Other entries are a chain that expires later, a secret key and a
key with a certificate chain that is not X.509.
+ * When: the factory loads each key store with hot reload enabled.
+ * Then: hot reload is due now, 15 hours before the chain that expires
first.
+ */
+ @Test
+ public void testHotReloadTimeComesFromEarliestExpiringChain() throws
Exception {
+ TestCertificateAuthority shortIssuer = new
TestCertificateAuthority("short-issuer", now - DAYS.toMillis(365), at(15));
+ KeyStore node = authority.keyStore("node", at(-85), at(15));
+ KeyStore withUnusableKeys = KeyStore.getInstance("JCEKS");
+
+ withUnusableKeys.load(null, PASSWORD);
+ withUnusableKeys.setKeyEntry("node", node.getKey("node", PASSWORD),
PASSWORD, node.getCertificateChain("node"));
+ withUnusableKeys.setKeyEntry("secret", new SecretKeySpec(new byte[16],
"AES"), PASSWORD, null);
+ withUnusableKeys.setKeyEntry("other", node.getKey("node", PASSWORD),
PASSWORD,
+ new Certificate[] {new
OtherCertificate(node.getCertificate("node").getPublicKey())});
+
+ assertHotReloadIsDueNow(node);
+ assertHotReloadIsDueNow(shortIssuer.keyStore("node", at(-85),
at(100)));
+ assertHotReloadIsDueNow(merge(authority.keyStore("first", at(-85),
at(15)), authority.keyStore("second", at(-200), at(100))));
+ assertHotReloadIsDueNow(merge(authority.keyStore("first", at(-200),
at(100)), authority.keyStore("second", at(-85), at(15))));
+ assertHotReloadIsDueNow(withUnusableKeys);
+ }
+
+ /**
+ * Given: a factory with hot reload enabled and an in-memory key store
that counts its reads.
+ * When: the factory loads its certificates.
+ * Then: it reads the key store once, so the hot reload date matches the
loaded key managers.
+ */
+ @Test
+ public void testLoadReadsKeyStoreOnce() throws Exception {
+ InMemorySslContextFactory factory = new
InMemorySslContextFactory(authority.keyStore("node", at(-85), at(15)));
+
+ factory.setHotReloadEnabled(true);
+ factory.create();
+
+ assertEquals(1, factory.keyStoreLoads);
+ }
+
+ /**
+ * Given: the issuer expired an hour ago, and the key certificate becomes
valid in 99 hours.
+ * When: the factory loads the chain for the first time with hot reload
enabled.
+ * Then: the load succeeds, and hot reload has been due since 15 hours
before the issuer expired.
+ */
+ @Test
+ public void testFirstLoadAcceptsChainOutsideValidityPeriod() throws
Exception {
+ TestCertificateAuthority expiredIssuer = new
TestCertificateAuthority("expired-issuer", at(-100), at(-1));
+ InMemorySslContextFactory factory = new
InMemorySslContextFactory(expiredIssuer.keyStore("node", at(99), at(199)));
+
+ factory.setHotReloadEnabled(true);
+ factory.create();
+
+ assertEquals(at(-16), factory.getHotReloadTime());
+ }
+
+ /**
+ * Given: the key store returns a certificate chain for an entry without a
private key, as Windows-MY does.
+ * When: the factory loads the key store.
+ * Then: the load succeeds, and the factory has no hot reload date.
+ */
+ @Test
+ public void testStoreWithoutPrivateKeyLoadsWithoutHotReloadDate() throws
Exception {
+ KeyStore store = spy(authority.trustStore());
+ String alias = store.aliases().nextElement();
+
+ doReturn(new Certificate[]
{store.getCertificate(alias)}).when(store).getCertificateChain(alias);
+
+ InMemorySslContextFactory factory = new
InMemorySslContextFactory(store);
+
+ factory.setHotReloadEnabled(true);
+
+ assertNotNull(factory.create());
+ assertEquals(0, factory.getHotReloadTime());
+ }
+
+ /**
+ * Given: the server certificate expires in 10 hours, and the client
trusts only the issuer of its replacement.
+ * When: the server reloads a certificate that expires in 10 or 5 hours.
+ * Then: the client accepts the server's replacement certificate and
completes the TLS handshake.
+ */
+ @Test
+ public void testReloadAcceptsSameOrEarlierExpiry() throws Exception {
+ TestCertificateAuthority replacementIssuer = new
TestCertificateAuthority("replacement-issuer", at(-24), at(24));
+ InMemorySslContextFactory client = new
InMemorySslContextFactory(authority.keyStore("client", at(-24), at(24)));
+
+ client.trustStore = replacementIssuer.trustStore();
+ client.useTrustStore("memory-trust");
+
+ SSLContext clientCtx = client.create();
+
+ for (long expiry : new long[] {at(10), at(5)}) {
+ InMemorySslContextFactory factory = dueFactory();
+
+ factory.keyStore = replacementIssuer.keyStore("node", at(-1),
expiry);
+ factory.init();
+
+ assertTrue(serverAccepts(factory.create(), clientCtx));
+ }
+ }
+
+ /**
+ * Given: a server keeps its keys, and a client presents a certificate
from a CA that the server does not trust yet.
+ * When: only the trust store gets that CA, and the factory reloads.
+ * Then: the context loaded before the change rejects the client, and the
reloaded context accepts it.
+ */
+ @Test
+ public void testReloadTrustsCaAddedToTrustStore() throws Exception {
+ InMemorySslContextFactory factory = dueFactory();
+ SSLContext ctx = factory.create();
+ TestCertificateAuthority otherIssuer = new
TestCertificateAuthority("other-issuer", now - DAYS.toMillis(1), now +
DAYS.toMillis(1));
+ InMemorySslContextFactory client = new
InMemorySslContextFactory(otherIssuer.keyStore("client", at(-1), at(23)));
+
+ client.trustStore = authority.trustStore();
+ client.useTrustStore("memory-trust");
+
+ SSLContext clientCtx = client.create();
+
+ assertFalse(serverAccepts(ctx, clientCtx));
+
+ factory.trustStore.setCertificateEntry("other-issuer",
otherIssuer.trustStore().getCertificate("CN=other-issuer"));
+ factory.init();
+
+ assertTrue(serverAccepts(factory.create(), clientCtx));
+ }
+
+ /**
+ * Given: due certificates, and a replacement that has expired, is not
valid yet, or comes from an expired issuer.
+ * When: the factory reloads.
+ * Then: the factory installs a new context without checking the validity
dates.
+ */
+ @Test
+ public void testReloadAcceptsCertificatesOutsideValidityPeriod() throws
Exception {
+ TestCertificateAuthority expiredIssuer = new
TestCertificateAuthority("expired-issuer", at(-100), at(-1));
+
+ KeyStore[] replacements = {
+ authority.keyStore("node", at(-101), at(-1)),
+ authority.keyStore("node", at(1), at(101)),
+ expiredIssuer.keyStore("node", at(-1), at(99))
+ };
+
+ for (KeyStore replacement : replacements) {
+ InMemorySslContextFactory factory = dueFactory();
+ SSLContext ctx = factory.create();
+
+ factory.keyStore = replacement;
+ factory.init();
+
+ assertNotSame(ctx, factory.create());
+ }
+ }
+
+ /**
+ * Given: the factory has loaded due keys and a trust store file.
+ * When: the trust store disappears during key rotation, then returns at
its configured path.
+ * Then: the failed reload keeps the previous context and date. The retry
installs the replacement.
+ */
+ @Test
+ public void testReloadRecoversAfterFailedTrustStoreLoad() throws Exception
{
+ Path trust = tmp.newFile("trust.jks").toPath();
+
+ save(authority.trustStore(), trust);
+
+ InMemorySslContextFactory factory = new
InMemorySslContextFactory(authority.keyStore("node", at(-90), at(10)));
+
+ factory.useTrustStore(trust.toString());
+ factory.setHotReloadEnabled(true);
+
+ SSLContext ctx = factory.create();
+ long hotReloadTime = factory.getHotReloadTime();
+
+ factory.keyStore = authority.keyStore("node", at(-1), at(99));
+ Files.delete(trust);
+
+ IgniteException err = assertThrows(IgniteException.class,
factory::init);
+
+ assertTrue(err.getCause().getMessage(),
err.getCause().getMessage().contains("key store file was not found"));
+ assertSame(ctx, factory.create());
+ assertEquals(hotReloadTime, factory.getHotReloadTime());
+
+ save(authority.trustStore(), trust);
+ factory.init();
+
+ assertNotSame(ctx, factory.create());
+ assertEquals(at(84), factory.getHotReloadTime());
+ }
+
+ /**
+ * Given: a reload of certificates with a new date blocks on the trust
store load.
+ * When: another thread reads {@link SslContextFactory#getHotReloadTime()}
during the reload.
+ * Then: the read returns the current date without waiting for the reload.
+ */
+ @Test
+ public void testHotReloadTimeRemainsReadableDuringLoad() throws Exception {
+ InMemorySslContextFactory factory = spy(dueFactory());
+ long hotReloadTime = factory.getHotReloadTime();
+
+ factory.keyStore = authority.keyStore("node", at(-1), at(99));
+
+ CountDownLatch trustLoadStarted = new CountDownLatch(1);
+ CountDownLatch finishTrustLoad = new CountDownLatch(1);
+
+ doAnswer(call -> {
+ trustLoadStarted.countDown();
+
+ assertTrue(finishTrustLoad.await(TIMEOUT_MS, MILLISECONDS));
+
+ return call.callRealMethod();
+ }).when(factory).loadKeyStore(any(), eq("memory-trust"), any());
+
+ GridTestThread load = new GridTestThread(factory::init);
+
+ load.start();
+
+ try {
+ assertTrue(trustLoadStarted.await(TIMEOUT_MS, MILLISECONDS));
+
+ assertEquals(hotReloadTime, factory.getHotReloadTime());
+ }
+ finally {
+ finishTrustLoad.countDown();
+ load.join(TIMEOUT_MS);
+ }
+
+ assertFalse(load.isAlive());
+ load.checkError();
+ }
+
+ /**
+ * Creates a factory with hot reload enabled that has loaded certificates
due for hot reload 5 hours ago.
+ * It reads both stores from memory. The certificates expire in 10 hours.
+ */
+ private InMemorySslContextFactory dueFactory() throws Exception {
+ InMemorySslContextFactory factory = new
InMemorySslContextFactory(authority.keyStore("node", at(-90), at(10)));
+
+ factory.trustStore = authority.trustStore();
+ factory.useTrustStore("memory-trust");
+ factory.setHotReloadEnabled(true);
+ factory.create();
+
+ assertEquals(at(-5), factory.getHotReloadTime());
+
+ return factory;
+ }
+
+ /** */
+ private void assertHotReloadIsDueNow(KeyStore keys) {
+ InMemorySslContextFactory factory = new
InMemorySslContextFactory(keys);
+
+ factory.setHotReloadEnabled(true);
+ factory.create();
+
+ assertEquals(now, factory.getHotReloadTime());
+ }
+
+ /**
+ * Connects a client to a server that requires a client certificate and
reports whether the server accepted it.
+ * The server side decides, because with TLS 1.3 the client can finish its
part of the handshake first.
+ */
+ private static boolean serverAccepts(SSLContext server, SSLContext client)
throws Exception {
+ InetAddress loopback = InetAddress.getLoopbackAddress();
+
+ try (SSLServerSocket srvSock =
(SSLServerSocket)server.getServerSocketFactory().createServerSocket(0, 1,
loopback)) {
+ srvSock.setNeedClientAuth(true);
+ srvSock.setSoTimeout((int)TIMEOUT_MS);
+
+ FutureTask<Boolean> accept = new FutureTask<>(() -> {
+ try (SSLSocket sock = (SSLSocket)srvSock.accept()) {
+ sock.setSoTimeout((int)TIMEOUT_MS);
+ sock.startHandshake();
+
+ return true;
+ }
+ catch (SSLHandshakeException e) {
+ return false;
+ }
+ });
+
+ new GridTestThread(accept).start();
+
+ try (SSLSocket sock =
(SSLSocket)client.getSocketFactory().createSocket(loopback,
srvSock.getLocalPort())) {
+ sock.setSoTimeout((int)TIMEOUT_MS);
+ sock.startHandshake();
+ }
+ catch (IOException ignored) {
+ // No-op.
+ }
+
+ return accept.get(TIMEOUT_MS, MILLISECONDS);
+ }
+ }
+
+ /** */
+ private long at(long hours) {
+ return now + HOURS.toMillis(hours);
+ }
+
+ /**
+ * A factory that reads its key store from a field instead of a file and
counts the reads.
+ * Tests change the field to put new certificates into the source.
+ * By default the factory trusts all peers. Tests can configure a trust
store from memory or a file.
+ * Hot reload is due 15 hours before expiry, so the tests count
certificate dates in hours.
+ */
+ private static class InMemorySslContextFactory extends SslContextFactory {
+ /** */
+ private static final long serialVersionUID = 0L;
+
+ /** */
+ private KeyStore keyStore;
+
+ /** */
+ private KeyStore trustStore;
+
+ /** */
+ private int keyStoreLoads;
+
+ /** */
+ private InMemorySslContextFactory(KeyStore keyStore) {
+ this.keyStore = keyStore;
+
+ setKeyStoreFilePath("memory");
+ setKeyStorePassword(PASSWORD);
+ setTrustManagers(getDisabledTrustManager());
+ setHotReloadBeforeExpiryMs(HOURS.toMillis(15));
+ }
+
+ /** Selects the trust store that supplies the factory's trust
managers. */
+ private void useTrustStore(String path) {
+ setTrustManagers((TrustManager[])null);
+ setTrustStoreFilePath(path);
+ setTrustStorePassword(PASSWORD);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected KeyStore loadKeyStore(String type, String path,
char[] pwd) throws SSLException {
+ if ("memory-trust".equals(path))
+ return trustStore;
+
+ if (!"memory".equals(path))
+ return super.loadKeyStore(type, path, pwd);
+
+ keyStoreLoads++;
+
+ return keyStore;
+ }
+ }
+
+ /** A certificate of a type other than X.509, which TLS key managers skip.
*/
+ private static class OtherCertificate extends Certificate {
+ /** */
+ private static final long serialVersionUID = 0L;
+
+ /** */
+ private final PublicKey key;
+
+ /** */
+ private OtherCertificate(PublicKey key) {
+ super("Other");
+
+ this.key = key;
+ }
+
+ /** {@inheritDoc} */
+ @Override public byte[] getEncoded() {
+ return new byte[0];
+ }
+
+ /** {@inheritDoc} */
+ @Override public void verify(PublicKey key) {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override public void verify(PublicKey key, String sigProvider) {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override public String toString() {
+ return "OtherCertificate";
+ }
+
+ /** {@inheritDoc} */
+ @Override public PublicKey getPublicKey() {
+ return key;
+ }
+ }
+}
diff --git
a/modules/core/src/test/java/org/apache/ignite/ssl/TestCertificateAuthority.java
b/modules/core/src/test/java/org/apache/ignite/ssl/TestCertificateAuthority.java
new file mode 100644
index 00000000000..a8d5205edc0
--- /dev/null
+++
b/modules/core/src/test/java/org/apache/ignite/ssl/TestCertificateAuthority.java
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ssl;
+
+import java.io.OutputStream;
+import java.math.BigInteger;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.security.KeyPair;
+import java.security.KeyPairGenerator;
+import java.security.KeyStore;
+import java.security.cert.X509Certificate;
+import java.security.spec.ECGenParameterSpec;
+import java.util.Collections;
+import java.util.Date;
+import javax.security.auth.x500.X500Principal;
+import org.bouncycastle.asn1.x509.BasicConstraints;
+import org.bouncycastle.asn1.x509.Extension;
+import org.bouncycastle.cert.jcajce.JcaX509CertificateConverter;
+import org.bouncycastle.cert.jcajce.JcaX509v3CertificateBuilder;
+import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder;
+
+/**
+ * Test CA that issues certificates with explicit validity dates.
+ * Each instance has its own EC key and a self-signed certificate.
+ * Its key stores hold a leaf certificate signed by the CA, followed by the CA
certificate.
+ */
+public class TestCertificateAuthority {
+ /** */
+ public static final char[] PASSWORD = "123456".toCharArray();
+
+ /** */
+ private final X500Principal name;
+
+ /** */
+ private final KeyPair keys = keyPair();
+
+ /** */
+ private final X509Certificate cert;
+
+ /** */
+ private long serial;
+
+ /**
+ * Creates a CA with a self-signed certificate valid between the given
dates.
+ * The name becomes the common name of the CA certificate.
+ */
+ public TestCertificateAuthority(String name, long notBefore, long
notAfter) throws Exception {
+ this.name = new X500Principal("CN=" + name);
+ cert = certificate(this.name, keys, notBefore, notAfter, true);
+ }
+
+ /**
+ * Creates a key store with one private key and its chain under the given
alias.
+ * The chain holds a leaf certificate with the given dates, signed by this
CA, followed by the CA certificate.
+ */
+ public KeyStore keyStore(String alias, long notBefore, long notAfter)
throws Exception {
+ KeyPair leafKeys = keyPair();
+ X509Certificate leaf = certificate(new X500Principal("CN=" + alias),
leafKeys, notBefore, notAfter, false);
+ KeyStore store = emptyStore();
+
+ store.setKeyEntry(alias, leafKeys.getPrivate(), PASSWORD, new
X509Certificate[] {leaf, cert});
+
+ return store;
+ }
+
+ /**
+ * Creates a trust store that trusts only this CA.
+ * It holds the CA certificate under the CA name.
+ */
+ public KeyStore trustStore() throws Exception {
+ KeyStore store = emptyStore();
+
+ store.setCertificateEntry(name.getName(), cert);
+
+ return store;
+ }
+
+ /**
+ * Combines the key entries of several key stores into one key store.
+ * Aliases must differ, because an entry with the same alias replaces the
earlier one.
+ */
+ static KeyStore merge(KeyStore... stores) throws Exception {
+ KeyStore merged = emptyStore();
+
+ for (KeyStore store : stores) {
+ for (String alias : Collections.list(store.aliases()))
+ merged.setKeyEntry(alias, store.getKey(alias, PASSWORD),
PASSWORD, store.getCertificateChain(alias));
+ }
+
+ return merged;
+ }
+
+ /**
+ * Writes the store to the file with the test password.
+ * It replaces an existing file, so a test can put new certificates in
place of old ones.
+ */
+ public static void save(KeyStore store, Path path) throws Exception {
+ try (OutputStream out = Files.newOutputStream(path)) {
+ store.store(out, PASSWORD);
+ }
+ }
+
+ /** */
+ private X509Certificate certificate(X500Principal subject, KeyPair
subjectKeys, long notBefore, long notAfter, boolean ca)
+ throws Exception {
+ JcaX509v3CertificateBuilder builder = new JcaX509v3CertificateBuilder(
+ name,
+ BigInteger.valueOf(++serial),
+ new Date(notBefore),
+ new Date(notAfter),
+ subject,
+ subjectKeys.getPublic());
+
+ if (ca)
+ builder.addExtension(Extension.basicConstraints, true, new
BasicConstraints(true));
+
+ return new JcaX509CertificateConverter().getCertificate(
+ builder.build(new
JcaContentSignerBuilder("SHA256withECDSA").build(keys.getPrivate())));
+ }
+
+ /** */
+ private static KeyPair keyPair() throws Exception {
+ KeyPairGenerator generator = KeyPairGenerator.getInstance("EC");
+
+ generator.initialize(new ECGenParameterSpec("secp256r1"));
+
+ return generator.generateKeyPair();
+ }
+
+ /** */
+ private static KeyStore emptyStore() throws Exception {
+ KeyStore store = KeyStore.getInstance("JKS");
+
+ store.load(null, PASSWORD);
+
+ return store;
+ }
+}
diff --git
a/modules/core/src/test/java/org/apache/ignite/testsuites/SecurityTestSuite.java
b/modules/core/src/test/java/org/apache/ignite/testsuites/SecurityTestSuite.java
index 7b207d1226a..0dcd548b857 100644
---
a/modules/core/src/test/java/org/apache/ignite/testsuites/SecurityTestSuite.java
+++
b/modules/core/src/test/java/org/apache/ignite/testsuites/SecurityTestSuite.java
@@ -83,6 +83,8 @@ import
org.apache.ignite.internal.thread.context.OperationContextAttributesTest;
import
org.apache.ignite.plugin.security.SecurityBasicPermissionSetSerializationTest;
import org.apache.ignite.ssl.MultipleSSLContextsTest;
import org.apache.ignite.ssl.SSLContextWrapperTest;
+import org.apache.ignite.ssl.SslCertificateHotReloadTest;
+import org.apache.ignite.ssl.SslContextFactoryHotReloadTest;
import org.apache.ignite.ssl.SslContextFactoryInitTest;
import org.apache.ignite.ssl.SslContextReloadTest;
import org.apache.ignite.tools.junit.JUnitTeamcityReporter;
@@ -149,6 +151,8 @@ import org.junit.runners.Suite;
MultipleSSLContextsTest.class,
SSLContextWrapperTest.class,
SslContextFactoryInitTest.class,
+ SslContextFactoryHotReloadTest.class,
+ SslCertificateHotReloadTest.class,
SslFactoryRegistryTest.class,
SslReloadTaskTest.class,
SslContextReloadTest.class,