[
https://issues.apache.org/jira/browse/IGNITE-28867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Vinogradov updated IGNITE-28867:
--------------------------------------
Description:
Currently, Ignite node TLS certificates are loaded only once during node
startup and then used to establish secure connections between `node to node`,
`node to thin client`. As a result, updating a certificate requires restarting
the Ignite node, causing unnecessary downtime and requiring additional
administrative effort.
We need to implement hot redeployment of TLS certificates, allowing the Ignite
node to reload the certificates at runtime without requiring a node restart.
Existing TLS sessions should remain unaffected, while all new TLS handshakes
should use the updated credentials.
Basic Steps
1. Introduce a mechanism to trigger certificate reloading (for example, through
`control.sh`).
2. Create a new `SslContext` instance by invoking `SslContextFactory`.
3. Replace the active `SslContext` used for TLS connection establishment. All
newly established connections must use the updated certificates, while existing
connections will continue to operate without interruption.
h2. Design
*Command.* {{control.sh --ssl reload}} is run against any node and reaches
every node of the cluster, servers and clients. {{--dry-run}} does the same
work but puts nothing in use, so the certificates on disk can be verified
before the cluster is touched.
The command carries no certificate and no path: each node re-opens the key and
trust store files at the paths its own {{SslContextFactory}} is configured
with, and takes whatever is in them at that moment.
*Registry instead of point-to-point calls.* A component registers itself in
{{GridInternalSubscriptionProcessor}} once it has set SSL up, so an empty
registry means the node does not use SSL. The command talks only to the
registry; a future SSL transport becomes reloadable by registering.
{code:java}
public interface SslContextReloadable {
boolean reloadSslContext() throws IgniteCheckedException; // rebuild and
apply
boolean checkSslContext() throws IgniteCheckedException; // rebuild only
default X509Certificate servedCertificate() { return null; }
}
{code}
Registered today: communication, discovery, client connector, binary (TCP)
REST, HTTP REST.
*Applying a new context.* {{AbstractSslContextFactory}} gains {{reload()}}
(rebuild and replace what {{create()}} hands out) and {{build()}} (rebuild
only, used by the dry run). Publishing into the factory is load-bearing:
outbound communication asks the factory per connection, and that is how it
picks the new certificates up. Inbound NIO transports get the context through
{{GridNioSslFilter}}, which creates an {{SSLEngine}} per session, so
established sessions are untouched.
A factory that hands back the context already in use cannot be reloaded. This
is detected by identity rather than by type, and reported as {{NOT reloaded}}
instead of a false success.
*Discovery.* The listening socket is now a plain {{ServerSocket}}, and every
accepted connection is wrapped with the context current at that moment. An
{{SSLServerSocket}} captures the context at bind time, which would have left
discovery reloadable for outbound connections only.
*Check before applying.* For communication and discovery, a rebuilt context
must complete a mutually authenticated TLS handshake in memory before it
replaces the one in use; otherwise the node keeps its current certificates and
the command fails. Client-facing transports are not checked, because a client
legitimately holds a different key and trust store. Only a refused handshake
counts as a failure, so the check cannot become a new way of blocking an
emergency rotation. That handshake also yields the certificate the node now
serves, which the report prints.
*HTTP REST (Jetty).* Its TLS comes from the user's Jetty XML rather than from
Ignite, so it goes through Jetty's own {{SslContextFactory}} instead of
{{Factory<SSLContext>}}. Jetty rebuilds in place and has no rollback: once its
{{reload()}} has failed, the connector serves no TLS at all until the next
successful one. The stores are therefore loaded on a throw-away factory first,
the context in use is pinned back if the rebuild fails regardless, and the next
attempt unpins it so that Jetty reads the files again. A configuration that
hands Jetty a ready-made context has nothing to re-read and is reported as not
reloaded.
*Clients.* The command reaches nodes, not the applications connecting to them,
and those need no restart either. The JDBC and ODBC drivers build their SSL
context per physical connection, so the next reconnect uses whatever is on
disk. A thin client rebuilds its context from disk once a TLS handshake has
been refused: the trigger is the refusal itself rather than a change on disk,
so switching certificates can only happen where the client already cannot
connect, and never moves it ahead of the cluster.
h2. Why the context is replaced rather than the key managers
The alternative is to keep one {{SSLContext}} and swap the delegates inside
{{X509ExtendedKeyManager}}/{{X509ExtendedTrustManager}}. Elasticsearch had
exactly that and removed it in elastic/elasticsearch#30509 for FIPS 140-2,
where only SunJSSE managers may be used; Jetty and Kafka also rebuild the
context. Swapping managers additionally keeps the session cache and ticket keys
of the old context, so a resumed session is still served the old certificate
and trust cannot be revoked by a reload.
was:
Currently, Ignite node TLS certificates are loaded only once during node
startup and then used to establish secure connections between `node to node`,
`node to thin client`. As a result, updating a certificate requires restarting
the Ignite node, causing unnecessary downtime and requiring additional
administrative effort.
We need to implement hot redeployment of TLS certificates, allowing the Ignite
node to reload the certificates at runtime without requiring a node restart.
Existing TLS sessions should remain unaffected, while all new TLS handshakes
should use the updated credentials.
Basic Steps
1. Introduce a mechanism to trigger certificate reloading (for example, through
`control.sh`).
2. Create a new `SslContext` instance by invoking `SslContextFactory`.
3. Replace the active `SslContext` used for TLS connection establishment. All
newly established connections must use the updated certificates, while existing
connections will continue to operate without interruption.
h2. Design
*Command.* {{control.sh --ssl reload}} is run against any node and reaches
every node of the cluster, servers and clients. {{--dry-run}} does the same
work but puts nothing in use, so the certificates on disk can be verified
before the cluster is touched.
The command carries no certificate and no path: each node re-opens the key and
trust store files at the paths its own {{SslContextFactory}} is configured
with, and takes whatever is in them at that moment.
*Registry instead of point-to-point calls.* A component registers itself in
{{GridInternalSubscriptionProcessor}} once it has set SSL up, so an empty
registry means the node does not use SSL. The command talks only to the
registry; a future SSL transport becomes reloadable by registering.
{code:java}
public interface SslContextReloadable {
boolean reloadSslContext() throws IgniteCheckedException; // rebuild and
apply
boolean checkSslContext() throws IgniteCheckedException; // rebuild only
default X509Certificate servedCertificate() { return null; }
}
{code}
Registered today: communication, discovery, client connector, binary (TCP) REST.
*Applying a new context.* {{AbstractSslContextFactory}} gains {{reload()}}
(rebuild and replace what {{create()}} hands out) and {{build()}} (rebuild
only, used by the dry run). Publishing into the factory is load-bearing:
outbound communication asks the factory per connection, and that is how it
picks the new certificates up. Inbound NIO transports get the context through
{{GridNioSslFilter}}, which creates an {{SSLEngine}} per session, so
established sessions are untouched.
A factory that hands back the context already in use cannot be reloaded. This
is detected by identity rather than by type, and reported as {{NOT reloaded}}
instead of a false success.
*Discovery.* The listening socket is now a plain {{ServerSocket}}, and every
accepted connection is wrapped with the context current at that moment. An
{{SSLServerSocket}} captures the context at bind time, which would have left
discovery reloadable for outbound connections only.
*Check before applying.* For communication and discovery, a rebuilt context
must complete a mutually authenticated TLS handshake in memory before it
replaces the one in use; otherwise the node keeps its current certificates and
the command fails. Client-facing transports are not checked, because a client
legitimately holds a different key and trust store. Only a refused handshake
counts as a failure, so the check cannot become a new way of blocking an
emergency rotation. That handshake also yields the certificate the node now
serves, which the report prints.
*Not covered.* The HTTP REST (Jetty) connector: Ignite does not configure its
TLS at all, it exists only when the user supplies a Jetty XML config, and
Jetty's own {{reload()}} applies unconditionally, so the dry run cannot be
expressed for it. Thin clients and the JDBC/ODBC drivers hold their own
configuration.
h2. Why the context is replaced rather than the key managers
The alternative is to keep one {{SSLContext}} and swap the delegates inside
{{X509ExtendedKeyManager}}/{{X509ExtendedTrustManager}}. Elasticsearch had
exactly that and removed it in elastic/elasticsearch#30509 for FIPS 140-2,
where only SunJSSE managers may be used; Jetty and Kafka also rebuild the
context. Swapping managers additionally keeps the session cache and ticket keys
of the old context, so a resumed session is still served the old certificate
and trust cannot be revoked by a reload.
> Add control.sh --ssl commands for TLS certificate hot redeployment
> ------------------------------------------------------------------
>
> Key: IGNITE-28867
> URL: https://issues.apache.org/jira/browse/IGNITE-28867
> Project: Ignite
> Issue Type: Task
> Reporter: Mikhail Petrov
> Assignee: Anton Vinogradov
> Priority: Major
> Labels: ise
> Fix For: 2.19
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Currently, Ignite node TLS certificates are loaded only once during node
> startup and then used to establish secure connections between `node to node`,
> `node to thin client`. As a result, updating a certificate requires
> restarting the Ignite node, causing unnecessary downtime and requiring
> additional administrative effort.
> We need to implement hot redeployment of TLS certificates, allowing the
> Ignite node to reload the certificates at runtime without requiring a node
> restart. Existing TLS sessions should remain unaffected, while all new TLS
> handshakes should use the updated credentials.
> Basic Steps
> 1. Introduce a mechanism to trigger certificate reloading (for example,
> through `control.sh`).
> 2. Create a new `SslContext` instance by invoking `SslContextFactory`.
> 3. Replace the active `SslContext` used for TLS connection establishment. All
> newly established connections must use the updated certificates, while
> existing connections will continue to operate without interruption.
> h2. Design
> *Command.* {{control.sh --ssl reload}} is run against any node and reaches
> every node of the cluster, servers and clients. {{--dry-run}} does the same
> work but puts nothing in use, so the certificates on disk can be verified
> before the cluster is touched.
> The command carries no certificate and no path: each node re-opens the key
> and trust store files at the paths its own {{SslContextFactory}} is
> configured with, and takes whatever is in them at that moment.
> *Registry instead of point-to-point calls.* A component registers itself in
> {{GridInternalSubscriptionProcessor}} once it has set SSL up, so an empty
> registry means the node does not use SSL. The command talks only to the
> registry; a future SSL transport becomes reloadable by registering.
> {code:java}
> public interface SslContextReloadable {
> boolean reloadSslContext() throws IgniteCheckedException; // rebuild and
> apply
> boolean checkSslContext() throws IgniteCheckedException; // rebuild only
> default X509Certificate servedCertificate() { return null; }
> }
> {code}
> Registered today: communication, discovery, client connector, binary (TCP)
> REST, HTTP REST.
> *Applying a new context.* {{AbstractSslContextFactory}} gains {{reload()}}
> (rebuild and replace what {{create()}} hands out) and {{build()}} (rebuild
> only, used by the dry run). Publishing into the factory is load-bearing:
> outbound communication asks the factory per connection, and that is how it
> picks the new certificates up. Inbound NIO transports get the context through
> {{GridNioSslFilter}}, which creates an {{SSLEngine}} per session, so
> established sessions are untouched.
> A factory that hands back the context already in use cannot be reloaded. This
> is detected by identity rather than by type, and reported as {{NOT reloaded}}
> instead of a false success.
> *Discovery.* The listening socket is now a plain {{ServerSocket}}, and every
> accepted connection is wrapped with the context current at that moment. An
> {{SSLServerSocket}} captures the context at bind time, which would have left
> discovery reloadable for outbound connections only.
> *Check before applying.* For communication and discovery, a rebuilt context
> must complete a mutually authenticated TLS handshake in memory before it
> replaces the one in use; otherwise the node keeps its current certificates
> and the command fails. Client-facing transports are not checked, because a
> client legitimately holds a different key and trust store. Only a refused
> handshake counts as a failure, so the check cannot become a new way of
> blocking an emergency rotation. That handshake also yields the certificate
> the node now serves, which the report prints.
> *HTTP REST (Jetty).* Its TLS comes from the user's Jetty XML rather than from
> Ignite, so it goes through Jetty's own {{SslContextFactory}} instead of
> {{Factory<SSLContext>}}. Jetty rebuilds in place and has no rollback: once
> its {{reload()}} has failed, the connector serves no TLS at all until the
> next successful one. The stores are therefore loaded on a throw-away factory
> first, the context in use is pinned back if the rebuild fails regardless, and
> the next attempt unpins it so that Jetty reads the files again. A
> configuration that hands Jetty a ready-made context has nothing to re-read
> and is reported as not reloaded.
> *Clients.* The command reaches nodes, not the applications connecting to
> them, and those need no restart either. The JDBC and ODBC drivers build their
> SSL context per physical connection, so the next reconnect uses whatever is
> on disk. A thin client rebuilds its context from disk once a TLS handshake
> has been refused: the trigger is the refusal itself rather than a change on
> disk, so switching certificates can only happen where the client already
> cannot connect, and never moves it ahead of the cluster.
> h2. Why the context is replaced rather than the key managers
> The alternative is to keep one {{SSLContext}} and swap the delegates inside
> {{X509ExtendedKeyManager}}/{{X509ExtendedTrustManager}}. Elasticsearch had
> exactly that and removed it in elastic/elasticsearch#30509 for FIPS 140-2,
> where only SunJSSE managers may be used; Jetty and Kafka also rebuild the
> context. Swapping managers additionally keeps the session cache and ticket
> keys of the old context, so a resumed session is still served the old
> certificate and trust cannot be revoked by a reload.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
