[
https://issues.apache.org/jira/browse/ZOOKEEPER-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dávid Paksy updated ZOOKEEPER-5047:
-----------------------------------
Description:
When we have a BCFKS KeyStore and we do NOT explicitly set KeyStore type,
PrometheusMetricsProvider fails to start:
{noformat}
2026-05-11 05:20:49,512 ERROR org.apache.zookeeper.server.ZooKeeperServerMain:
Unexpected exception, exiting abnormally
java.io.IOException: Cannot boot MetricsProvider
org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:131)
at
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:113)
at
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:68)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:141)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
Caused by: org.apache.zookeeper.metrics.MetricsProviderLifeCycleException:
Failed to start Prometheus Jetty server
at
org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:246)
at
org.apache.zookeeper.metrics.impl.MetricsProviderBootstrap.startMetricsProvider(MetricsProviderBootstrap.java:45)
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:127)
... 4 more
Caused by: java.io.IOException: DerValue.getBigIntegerInternal, not expected 48
at
java.base/sun.security.util.DerValue.getBigIntegerInternal(DerValue.java:633)
at
java.base/sun.security.util.DerValue.getIntegerInternal(DerValue.java:594)
at java.base/sun.security.util.DerValue.getInteger(DerValue.java:590)
at
java.base/sun.security.util.DerInputStream.getInteger(DerInputStream.java:126)
at
java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2014)
at
java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
at java.base/java.security.KeyStore.load(KeyStore.java:1473)
at
org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
at
org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1203)
at
org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:322)
at
org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:244)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.apache.zookeeper.server.admin.UnifiedConnectionFactory.doStart(UnifiedConnectionFactory.java:60)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:323)
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.server.Server.doStart(Server.java:401)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:237)
... 6 more
{noformat}
PrometheusMetricsProvider can’t load the configured KeyStore because it tries
to load it as PKCS12 while the keystore is BCFKS - *NOK*.
Explicitly setting KeyStore type to BCFKS resolves the issue:
{noformat}
metricsProvider.ssl.keyStore.type=BCFKS
metricsProvider.ssl.trustStore.type=BCFKS
{noformat}
However we don't need to set KeyStore type explicitly for the other keystores
in ZooKeeper, e.g.:
- ssl.keyStore.type
- ssl.quorum.keyStore.type
JettyAdminServer also seem to detect BCFKS correctly because it uses
X509Util.loadKeyStore() and X509Util.loadTrustStore().
The reason is that PrometheusMetricsProvider has hard coded type for KeyStore,
TrustStore as PKCS12 and no other detection:
https://github.com/apache/zookeeper/blob/master/zookeeper-metrics-providers/zookeeper-prometheus-metrics/src/main/java/org/apache/zookeeper/metrics/prometheus/PrometheusMetricsProvider.java#L147
Propose that PrometheusMetricsProvider should be consistent with the rest of
the codebase with keytore type detection.
was:
When we have a BCFKS KeyStore and we do NOT explicitly set KeyStore type,
PrometheusMetricsProvider fails to start:
{noformat}
2026-05-11 05:20:49,512 ERROR org.apache.zookeeper.server.ZooKeeperServerMain:
Unexpected exception, exiting abnormally
java.io.IOException: Cannot boot MetricsProvider
org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:131)
at
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:113)
at
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:68)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:141)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
Caused by: org.apache.zookeeper.metrics.MetricsProviderLifeCycleException:
Failed to start Prometheus Jetty server
at
org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:246)
at
org.apache.zookeeper.metrics.impl.MetricsProviderBootstrap.startMetricsProvider(MetricsProviderBootstrap.java:45)
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:127)
... 4 more
Caused by: java.io.IOException: DerValue.getBigIntegerInternal, not expected 48
at
java.base/sun.security.util.DerValue.getBigIntegerInternal(DerValue.java:633)
at
java.base/sun.security.util.DerValue.getIntegerInternal(DerValue.java:594)
at java.base/sun.security.util.DerValue.getInteger(DerValue.java:590)
at
java.base/sun.security.util.DerInputStream.getInteger(DerInputStream.java:126)
at
java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2014)
at
java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
at java.base/java.security.KeyStore.load(KeyStore.java:1473)
at
org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
at
org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1203)
at
org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:322)
at
org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:244)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.apache.zookeeper.server.admin.UnifiedConnectionFactory.doStart(UnifiedConnectionFactory.java:60)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:323)
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.server.Server.doStart(Server.java:401)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:237)
... 6 more
{noformat}
PrometheusMetricsProvider can’t load the configured KeyStore because it tries
to load it as PKCS12 while the keystore is BCFKS - *NOK*.
Explicitly setting KeyStore type to BCFKS resolves the issue:
{noformat}
metricsProvider.ssl.keyStore.type=BCFKS
metricsProvider.ssl.trustStore.type=BCFKS
{noformat}
However we don't need to set KeyStore type explicitly for the other keystores
in ZooKeeper, e.g.:
- ssl.keyStore.type
- ssl.quorum.keyStore.type
JettyAdminServer also seem to detect BCFKS correctly because it uses
X509Util.loadKeyStore() and X509Util.loadTrustStore().
The reason is that PrometheusMetricsProvider has hard coded type for KeyStore,
TrustStore as PKCS12 and no other detecion:
https://github.com/apache/zookeeper/blob/master/zookeeper-metrics-providers/zookeeper-prometheus-metrics/src/main/java/org/apache/zookeeper/metrics/prometheus/PrometheusMetricsProvider.java#L147
> Make PrometheusMetricsProvider KeyStore type detection consistent
> -----------------------------------------------------------------
>
> Key: ZOOKEEPER-5047
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5047
> Project: ZooKeeper
> Issue Type: Task
> Components: metric system
> Reporter: Dávid Paksy
> Priority: Major
>
> When we have a BCFKS KeyStore and we do NOT explicitly set KeyStore type,
> PrometheusMetricsProvider fails to start:
> {noformat}
> 2026-05-11 05:20:49,512 ERROR
> org.apache.zookeeper.server.ZooKeeperServerMain: Unexpected exception,
> exiting abnormally
> java.io.IOException: Cannot boot MetricsProvider
> org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
> at
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:131)
> at
> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:113)
> at
> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:68)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:141)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
> Caused by: org.apache.zookeeper.metrics.MetricsProviderLifeCycleException:
> Failed to start Prometheus Jetty server
> at
> org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:246)
> at
> org.apache.zookeeper.metrics.impl.MetricsProviderBootstrap.startMetricsProvider(MetricsProviderBootstrap.java:45)
> at
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:127)
> ... 4 more
> Caused by: java.io.IOException: DerValue.getBigIntegerInternal, not expected
> 48
> at
> java.base/sun.security.util.DerValue.getBigIntegerInternal(DerValue.java:633)
> at
> java.base/sun.security.util.DerValue.getIntegerInternal(DerValue.java:594)
> at java.base/sun.security.util.DerValue.getInteger(DerValue.java:590)
> at
> java.base/sun.security.util.DerInputStream.getInteger(DerInputStream.java:126)
> at
> java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2014)
> at
> java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
> at java.base/java.security.KeyStore.load(KeyStore.java:1473)
> at
> org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
> at
> org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1203)
> at
> org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:322)
> at
> org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:244)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
> org.apache.zookeeper.server.admin.UnifiedConnectionFactory.doStart(UnifiedConnectionFactory.java:60)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
> org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:323)
> at
> org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
> at
> org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at org.eclipse.jetty.server.Server.doStart(Server.java:401)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:237)
> ... 6 more
> {noformat}
> PrometheusMetricsProvider can’t load the configured KeyStore because it tries
> to load it as PKCS12 while the keystore is BCFKS - *NOK*.
> Explicitly setting KeyStore type to BCFKS resolves the issue:
> {noformat}
> metricsProvider.ssl.keyStore.type=BCFKS
> metricsProvider.ssl.trustStore.type=BCFKS
> {noformat}
> However we don't need to set KeyStore type explicitly for the other keystores
> in ZooKeeper, e.g.:
> - ssl.keyStore.type
> - ssl.quorum.keyStore.type
> JettyAdminServer also seem to detect BCFKS correctly because it uses
> X509Util.loadKeyStore() and X509Util.loadTrustStore().
> The reason is that PrometheusMetricsProvider has hard coded type for
> KeyStore, TrustStore as PKCS12 and no other detection:
> https://github.com/apache/zookeeper/blob/master/zookeeper-metrics-providers/zookeeper-prometheus-metrics/src/main/java/org/apache/zookeeper/metrics/prometheus/PrometheusMetricsProvider.java#L147
> Propose that PrometheusMetricsProvider should be consistent with the rest of
> the codebase with keytore type detection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
