[
https://issues.apache.org/jira/browse/NIFI-16319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-16319:
----------------------------------
Fix Version/s: 2.12.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> nifi.security.autoreload.enabled does not reload keystore/truststore when
> files are Kubernetes Secret mounts (..data atomic symlink swap)
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16319
> URL: https://issues.apache.org/jira/browse/NIFI-16319
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.7.1, 2.7.2, 2.9.0, 2.10.0, 2.11.0
> Reporter: Bryan Bende
> Assignee: David Handermann
> Priority: Major
> Fix For: 2.12.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> With {{{}nifi.security.autoreload.enabled=true{}}}, NiFi is supposed to pick
> up keystore and truststore changes without a restart. That fails when the
> stores are Kubernetes Secret or projected-volume mounts, which is the normal
> way to inject certs on Kubernetes.
> Kubelet does not overwrite {{keystore.jks}} / {{truststore.jks}} in place. It
> writes a new timestamped directory, atomically retargets the {{..data}}
> symlink, and deletes the old directory:
> {{/nifi/certs/truststore.jks -> ..data/truststore.jks ->
> ..2026_07_21_12_00_00.123/truststore.jks}}
> after Secret update:
> {{/nifi/certs/truststore.jks -> ..data/truststore.jks ->
> ..2026_08_30_12_00_00.456/truststore.jks}}
> The stable names never change. Only the directory behind {{..data}} changes.
> h3. What 2.x does
> NIFI-12125 replaced the Jetty {{StoreScanner}} with
> {{java.nio.file.WatchService}} in {{{}SslContextConfiguration{}}}. On startup
> it:
> * registers {{ENTRY_CREATE}} / {{ENTRY_MODIFY}} on the store parent
> directory _and_ on {{storePath.toRealPath()}} (the timestamped kubelet dir)
> * reloads only when an event filename is {{keystore.jks}} or
> {{truststore.jks}} ({{{}SecurityStoreChangedPathListener{}}})
> * polls one {{WatchKey}} per {{nifi.security.autoreload.interval}}
> ({{{}WatchServiceMonitorCommand{}}})
> The timestamped directory is never modified; it is replaced. The watch key on
> that directory becomes invalid ({{{}Watch Key reset failed: Watch Service no
> longer valid{}}}). Events on the parent are for {{..data}} or
> {{{}..<timestamp>{}}}, which do not match the filename filter. After the
> first Secret update, autoreload never fires again for the life of the JVM.
> The wrapping {{StandardX509ExtendedKeyManager}} /
> {{StandardX509ExtendedTrustManager}} from NIFI-12125 are fine: new sockets
> (including cluster-protocol heartbeats) would use reloaded material _if_ a
> reload happened. The bug is that the watcher never re-reads the files.
> h3. Regression
> This is NIFI-10425 again. That ticket described the same kubelet layout. The
> 1.x fix was Jetty {{reportRealPaths=false}} so the scanner tracked the
> {_}symlink{_}, not the real file. NIFI-12125 dropped that scanner.
> {{toRealPath()}} in {{SslContextConfiguration.addStorePath}} /
> {{addStoreFileName}} is the opposite of that fix.
> h3. Expected
> Updating the Secret that backs {{nifi.security.keystore}} /
> {{nifi.security.truststore}} causes NiFi, within about one autoreload
> interval, to log that the Key Manager and Trust Manager were reloaded and to
> use the new material on the next TLS handshake (Jetty and cluster protocol).
> h3. Actual
> The process keeps the Key Manager / Trust Manager loaded at startup. Cluster
> protocol and HTTPS keep working on the old in-memory trust anchors. A later
> pod restart loads the on-disk stores and can fail (for example PKIX path
> validation if the CA rotated and the leaf did not).
> h3. How to reproduce
> * Run NiFi 2.x in Kubernetes with {{nifi.security.autoreload.enabled=true}}
> and keystore/truststore coming from a Secret or projected volume (not a copy
> into a regular directory).
> * Confirm startup log: {{{}Scheduled Security Store Monitor with Duration
> [...]{}}}.
> * Rotate the Secret (new {{truststore.jks}} / {{keystore.jks}} bytes)
> * Wait several autoreload intervals.
> Expected: {{{}Key Manager and Trust Manager Reloaded from Changed Path{}}}.
> Actual: no reload; often {{Watch Key reset failed: Watch Service no longer
> valid}} when the old timestamped dir is removed. {{openssl s_client}} / a new
> cluster handshake still presents or trusts the original material until the
> pod restarts.
> A local stand-in without a cluster: point the stores at symlinks whose
> targets are replaced by swapping a {{{}..data{}}}-style link the way kubelet
> does, rather than overwriting the file in place.
> h3. Suggested fix
> Do not pin {{WatchService}} to {{toRealPath()}} of a kubelet timestamped
> directory.
> Reload from the _configured_ paths in {{nifi.properties}} (follow the symlink
> at read time). The most reliable approach is to poll those paths every
> {{{}nifi.security.autoreload.interval{}}}: compare digest or
> {{{}lastModified{}}}, then rebuild with {{FrameworkKeyManagerBuilder}} /
> {{FrameworkTrustManagerBuilder}} and {{setKeyManager}} / {{setTrustManager}}
> as today.
> Watching the parent for {{..data}} changes can be an extra trigger, but
> polling is what actually works with projected Secrets. In-place file writes
> on a normal filesystem should keep working.
> h3. Related
> * NIFI-10425 — same kubelet symlink layout; Jetty scanner
> * NIFI-11536 — autoreload broken after 1.17.0
> * NIFI-12125 — WatchService + shared SSLContext / cluster protocol;
> introduced this watcher
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
