[
https://issues.apache.org/jira/browse/IGNITE-28867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Vinogradov updated IGNITE-28867:
--------------------------------------
Release Note: Added control.sh --ssl reload and control.sh --ssl status to
replace the TLS certificates of running nodes without a restart, and optional
automatic certificate renewal.
Description:
Currently, Ignite node TLS certificates are loaded only once during node
startup and then used to establish secure connections between `node to node`,
`node to thin client`. As a result, updating a certificate requires restarting
the Ignite node, causing unnecessary downtime and requiring additional
administrative effort.
We need to implement hot redeployment of TLS certificates, allowing the Ignite
node to reload the certificates at runtime without requiring a node restart.
Existing TLS sessions should remain unaffected, while all new TLS handshakes
should use the updated credentials.
Basic Steps
1. Introduce a mechanism to trigger certificate reloading (for example, through
`control.sh`).
2. Create a new `SslContext` instance by invoking `SslContextFactory`.
3. Replace the active `SslContext` used for TLS connection establishment. All
newly established connections must use the updated certificates, while existing
connections will continue to operate without interruption.
h2. Design
The pull request describes the design in full:
https://github.com/apache/ignite/pull/13335. In short:
*Commands.* {{control.sh --ssl reload}} is run against any node and reaches
every node of the cluster, servers and clients. Each node asks its own SSL
context factories for new contexts, so the command carries no certificate and
no path. {{control.sh --ssl status}} reports the certificate every node serves
and the result of its last reload.
*Discovery.* The listening socket is now a plain {{ServerSocket}}, and every
accepted connection is wrapped with the context current at that moment. An
{{SSLServerSocket}} captures the context at bind time, which would have left
discovery reloadable for outbound connections only.
*Automatic renewal.* Optional and off by default: with {{renewalEnabled}} set
on the SSL context factory, a node reads its stores again when
{{renewBeforeFraction}} of the certificate lifetime is left, with the same
checks.
*Check before applying.* A rebuilt context goes in use only if every
certificate of its chain is valid now, and it presents a certificate whenever
the context in use does. If the factory serves discovery or communication, the
node's own trust store must also accept the new certificate; the node checks
this with a TLS handshake in memory and asks no other node. Otherwise the node
keeps its current certificates, and the command reports the reason for that
node.
h2. Why the context is replaced rather than the key managers
The alternative is to keep one {{SSLContext}} and swap the delegates inside
{{X509ExtendedKeyManager}}/{{X509ExtendedTrustManager}}. Elasticsearch had
exactly that and removed it in elastic/elasticsearch#30509 for FIPS 140-2,
where only SunJSSE managers may be used; Jetty and Kafka also rebuild the
context. Swapping managers additionally keeps the session cache and ticket keys
of the old context, so a resumed session is still served the old certificate
and trust cannot be revoked by a reload.
was:
Currently, Ignite node TLS certificates are loaded only once during node
startup and then used to establish secure connections between `node to node`,
`node to thin client`. As a result, updating a certificate requires restarting
the Ignite node, causing unnecessary downtime and requiring additional
administrative effort.
We need to implement hot redeployment of TLS certificates, allowing the Ignite
node to reload the certificates at runtime without requiring a node restart.
Existing TLS sessions should remain unaffected, while all new TLS handshakes
should use the updated credentials.
Basic Steps
1. Introduce a mechanism to trigger certificate reloading (for example, through
`control.sh`).
2. Create a new `SslContext` instance by invoking `SslContextFactory`.
3. Replace the active `SslContext` used for TLS connection establishment. All
newly established connections must use the updated certificates, while existing
connections will continue to operate without interruption.
h2. Design
*Command.* {{control.sh --ssl reload}} is run against any node and reaches
every node of the cluster, servers and clients. {{--dry-run}} does the same
work but puts nothing in use, so the certificates on disk can be verified
before the cluster is touched.
The command carries no certificate and no path: each node re-opens the key and
trust store files at the paths its own {{SslContextFactory}} is configured
with, and takes whatever is in them at that moment.
*Registry instead of point-to-point calls.* A component registers itself in
{{GridInternalSubscriptionProcessor}} once it has set SSL up, so an empty
registry means the node does not use SSL. The command talks only to the
registry; a future SSL transport becomes reloadable by registering.
{code:java}
public interface SslContextReloadable {
boolean reloadSslContext() throws IgniteCheckedException; // rebuild and
apply
boolean checkSslContext() throws IgniteCheckedException; // rebuild only
default X509Certificate servedCertificate() { return null; }
}
{code}
Registered today: communication, discovery, client connector, binary (TCP)
REST, HTTP REST.
*Applying a new context.* {{AbstractSslContextFactory}} gains {{reload()}}
(rebuild and replace what {{create()}} hands out) and {{build()}} (rebuild
only, used by the dry run). Publishing into the factory is load-bearing:
outbound communication asks the factory per connection, and that is how it
picks the new certificates up. Inbound NIO transports get the context through
{{GridNioSslFilter}}, which creates an {{SSLEngine}} per session, so
established sessions are untouched.
A factory that hands back the context already in use cannot be reloaded. This
is detected by identity rather than by type, and reported as {{NOT reloaded}}
instead of a false success.
*Discovery.* The listening socket is now a plain {{ServerSocket}}, and every
accepted connection is wrapped with the context current at that moment. An
{{SSLServerSocket}} captures the context at bind time, which would have left
discovery reloadable for outbound connections only.
*Check before applying.* For communication and discovery, a rebuilt context
must complete a mutually authenticated TLS handshake in memory before it
replaces the one in use; otherwise the node keeps its current certificates and
the command fails. Client-facing transports are not checked, because a client
legitimately holds a different key and trust store. Only a refused handshake
counts as a failure, so the check cannot become a new way of blocking an
emergency rotation. That handshake also yields the certificate the node now
serves, which the report prints.
*HTTP REST (Jetty).* Its TLS comes from the user's Jetty XML rather than from
Ignite, so it goes through Jetty's own {{SslContextFactory}} instead of
{{Factory<SSLContext>}}. Jetty rebuilds in place and has no rollback: once its
{{reload()}} has failed, the connector serves no TLS at all until the next
successful one. The stores are therefore loaded on a throw-away factory first,
the context in use is pinned back if the rebuild fails regardless, and the next
attempt unpins it so that Jetty reads the files again. A configuration that
hands Jetty a ready-made context has nothing to re-read and is reported as not
reloaded.
*Clients.* The command reaches nodes, not the applications connecting to them,
and those need no restart either. The JDBC and ODBC drivers build their SSL
context per physical connection, so the next reconnect uses whatever is on
disk. A thin client rebuilds its context from disk once a TLS handshake has
been refused: the trigger is the refusal itself rather than a change on disk,
so switching certificates can only happen where the client already cannot
connect, and never moves it ahead of the cluster.
h2. Why the context is replaced rather than the key managers
The alternative is to keep one {{SSLContext}} and swap the delegates inside
{{X509ExtendedKeyManager}}/{{X509ExtendedTrustManager}}. Elasticsearch had
exactly that and removed it in elastic/elasticsearch#30509 for FIPS 140-2,
where only SunJSSE managers may be used; Jetty and Kafka also rebuild the
context. Swapping managers additionally keeps the session cache and ticket keys
of the old context, so a resumed session is still served the old certificate
and trust cannot be revoked by a reload.
> Add control.sh --ssl commands for TLS certificate hot redeployment
> ------------------------------------------------------------------
>
> Key: IGNITE-28867
> URL: https://issues.apache.org/jira/browse/IGNITE-28867
> Project: Ignite
> Issue Type: Task
> Reporter: Mikhail Petrov
> Assignee: Anton Vinogradov
> Priority: Major
> Labels: ise
> Fix For: 2.19
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Currently, Ignite node TLS certificates are loaded only once during node
> startup and then used to establish secure connections between `node to node`,
> `node to thin client`. As a result, updating a certificate requires
> restarting the Ignite node, causing unnecessary downtime and requiring
> additional administrative effort.
> We need to implement hot redeployment of TLS certificates, allowing the
> Ignite node to reload the certificates at runtime without requiring a node
> restart. Existing TLS sessions should remain unaffected, while all new TLS
> handshakes should use the updated credentials.
> Basic Steps
> 1. Introduce a mechanism to trigger certificate reloading (for example,
> through `control.sh`).
> 2. Create a new `SslContext` instance by invoking `SslContextFactory`.
> 3. Replace the active `SslContext` used for TLS connection establishment. All
> newly established connections must use the updated certificates, while
> existing connections will continue to operate without interruption.
> h2. Design
> The pull request describes the design in full:
> https://github.com/apache/ignite/pull/13335. In short:
> *Commands.* {{control.sh --ssl reload}} is run against any node and reaches
> every node of the cluster, servers and clients. Each node asks its own SSL
> context factories for new contexts, so the command carries no certificate and
> no path. {{control.sh --ssl status}} reports the certificate every node
> serves and the result of its last reload.
> *Discovery.* The listening socket is now a plain {{ServerSocket}}, and every
> accepted connection is wrapped with the context current at that moment. An
> {{SSLServerSocket}} captures the context at bind time, which would have left
> discovery reloadable for outbound connections only.
> *Automatic renewal.* Optional and off by default: with {{renewalEnabled}} set
> on the SSL context factory, a node reads its stores again when
> {{renewBeforeFraction}} of the certificate lifetime is left, with the same
> checks.
> *Check before applying.* A rebuilt context goes in use only if every
> certificate of its chain is valid now, and it presents a certificate whenever
> the context in use does. If the factory serves discovery or communication,
> the node's own trust store must also accept the new certificate; the node
> checks this with a TLS handshake in memory and asks no other node. Otherwise
> the node keeps its current certificates, and the command reports the reason
> for that node.
> h2. Why the context is replaced rather than the key managers
> The alternative is to keep one {{SSLContext}} and swap the delegates inside
> {{X509ExtendedKeyManager}}/{{X509ExtendedTrustManager}}. Elasticsearch had
> exactly that and removed it in elastic/elasticsearch#30509 for FIPS 140-2,
> where only SunJSSE managers may be used; Jetty and Kafka also rebuild the
> context. Swapping managers additionally keeps the session cache and ticket
> keys of the old context, so a resumed session is still served the old
> certificate and trust cannot be revoked by a reload.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
