Dávid Paksy created ZOOKEEPER-5047:
--------------------------------------
Summary: Make PrometheusMetricsProvider KeyStore type detection
consistent
Key: ZOOKEEPER-5047
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5047
Project: ZooKeeper
Issue Type: Task
Components: metric system
Reporter: Dávid Paksy
When we have a BCFKS KeyStore and we do NOT explicitly set KeyStore type,
PrometheusMetricsProvider fails to start:
{noformat}
2026-05-11 05:20:49,512 ERROR org.apache.zookeeper.server.ZooKeeperServerMain:
Unexpected exception, exiting abnormally
java.io.IOException: Cannot boot MetricsProvider
org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:131)
at
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:113)
at
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:68)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:141)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
Caused by: org.apache.zookeeper.metrics.MetricsProviderLifeCycleException:
Failed to start Prometheus Jetty server
at
org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:246)
at
org.apache.zookeeper.metrics.impl.MetricsProviderBootstrap.startMetricsProvider(MetricsProviderBootstrap.java:45)
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:127)
... 4 more
Caused by: java.io.IOException: DerValue.getBigIntegerInternal, not expected 48
at
java.base/sun.security.util.DerValue.getBigIntegerInternal(DerValue.java:633)
at
java.base/sun.security.util.DerValue.getIntegerInternal(DerValue.java:594)
at java.base/sun.security.util.DerValue.getInteger(DerValue.java:590)
at
java.base/sun.security.util.DerInputStream.getInteger(DerInputStream.java:126)
at
java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2014)
at
java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
at java.base/java.security.KeyStore.load(KeyStore.java:1473)
at
org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
at
org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1203)
at
org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:322)
at
org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:244)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.apache.zookeeper.server.admin.UnifiedConnectionFactory.doStart(UnifiedConnectionFactory.java:60)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:323)
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.server.Server.doStart(Server.java:401)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider.start(PrometheusMetricsProvider.java:237)
... 6 more
{noformat}
PrometheusMetricsProvider can’t load the configured KeyStore because it tries
to load it as PKCS12 while the keystore is BCFKS - *NOK*.
Explicitly setting KeyStore type to BCFKS resolves the issue:
{noformat}
metricsProvider.ssl.keyStore.type=BCFKS
metricsProvider.ssl.trustStore.type=BCFKS
{noformat}
However we don't need to set KeyStore type explicitly for the other keystores
in ZooKeeper, e.g.:
- ssl.keyStore.type
- ssl.quorum.keyStore.type
JettyAdminServer also seem to detect BCFKS correctly because it uses
X509Util.loadKeyStore() and X509Util.loadTrustStore().
The reason is that PrometheusMetricsProvider has hard coded type for KeyStore,
TrustStore as PKCS12 and no other detecion:
https://github.com/apache/zookeeper/blob/master/zookeeper-metrics-providers/zookeeper-prometheus-metrics/src/main/java/org/apache/zookeeper/metrics/prometheus/PrometheusMetricsProvider.java#L147
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
