[
https://issues.apache.org/jira/browse/SPARK-47133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17862836#comment-17862836
]
John Bateman commented on SPARK-47133:
--------------------------------------
Hi Filippo, I am experiencing the same issue. Did you manage to get to the
bottom of this and whether it is a bug or not?
> java.lang.NullPointerException: Missing SslContextFactory when accessing
> Worker WebUI from Master as reverse proxy with SSL enabled
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-47133
> URL: https://issues.apache.org/jira/browse/SPARK-47133
> Project: Spark
> Issue Type: Question
> Components: Web UI
> Affects Versions: 3.5.0
> Environment: * We are running Spark in stand-alone mode, on
> Kubernetes.
> * The containers are based on Debian 11 (minideb)
> * The Spark version is 3.5
> Reporter: Filippo Monari
> Priority: Major
>
> Hi,
>
> We are encountering the error described here below.
> If SSL/TLS is enabled on both, Master and Worker, it is not possible to
> access the WebUI of the latter from the former configured as reverse proxy.
> The returned error is the the following.
> {code:java}
> HTTP ERROR 500 java.lang.NullPointerException: Missing SslContextFactory
> URI:/proxy/worker-20240222171308-10.113.3.1-34959
> STATUS:500
> MESSAGE:java.lang.NullPointerException: Missing SslContextFactory
> SERVLET:org.apache.spark.ui.JettyUtils$$anon$3-7d068d54
> CAUSED BY:java.lang.NullPointerException: Missing SslContextFactory
> Caused by:java.lang.NullPointerException: Missing SslContextFactory
> at java.base/java.util.Objects.requireNonNull(Objects.java:235)
> at
> org.sparkproject.jetty.io.ssl.SslClientConnectionFactory.<init>(SslClientConnectionFactory.java:57)
> at
> org.sparkproject.jetty.client.HttpClient.newSslClientConnectionFactory(HttpClient.java:1273)
> at
> org.sparkproject.jetty.client.HttpClient.newSslClientConnectionFactory(HttpClient.java:1279)
> at
> org.sparkproject.jetty.client.HttpDestination.newSslClientConnectionFactory(HttpDestination.java:209)
> at
> org.sparkproject.jetty.client.HttpDestination.newSslClientConnectionFactory(HttpDestination.java:215)
> at
> org.sparkproject.jetty.client.HttpDestination.<init>(HttpDestination.java:100)
> at
> org.sparkproject.jetty.client.PoolingHttpDestination.<init>(PoolingHttpDestination.java:25)
> at
> org.sparkproject.jetty.client.http.HttpDestinationOverHTTP.<init>(HttpDestinationOverHTTP.java:32)
> at
> org.sparkproject.jetty.client.http.HttpClientTransportOverHTTP.newHttpDestination(HttpClientTransportOverHTTP.java:54)
> at
> org.sparkproject.jetty.client.HttpClient.lambda$resolveDestination$0(HttpClient.java:597)
> at
> java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916)
> at
> org.sparkproject.jetty.client.HttpClient.resolveDestination(HttpClient.java:593)
> at
> org.sparkproject.jetty.client.HttpClient.resolveDestination(HttpClient.java:571)
> at org.sparkproject.jetty.client.HttpClient.send(HttpClient.java:626)
> at
> org.sparkproject.jetty.client.HttpRequest.sendAsync(HttpRequest.java:780)
> at org.sparkproject.jetty.client.HttpRequest.send(HttpRequest.java:767)
> at
> org.sparkproject.jetty.proxy.AbstractProxyServlet.sendProxyRequest(AbstractProxyServlet.java:618)
> at
> org.sparkproject.jetty.proxy.ProxyServlet.service(ProxyServlet.java:114)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
> at
> org.sparkproject.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
> at
> org.sparkproject.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
> at
> org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:95)
> at
> org.sparkproject.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
> at
> org.sparkproject.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
> at
> org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
> at
> org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
> at
> org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.sparkproject.jetty.server.Server.handle(Server.java:516)
> at
> org.sparkproject.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
> at
> org.sparkproject.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
> at
> org.sparkproject.jetty.server.HttpChannel.handle(HttpChannel.java:479)
> at
> org.sparkproject.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
> at
> org.sparkproject.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at
> org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at
> org.sparkproject.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
> at
> org.sparkproject.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
> at
> org.sparkproject.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
> at
> org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at
> org.sparkproject.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> at
> org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
> at
> org.sparkproject.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
> at
> org.sparkproject.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> at
> org.sparkproject.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> at java.base/java.lang.Thread.run(Thread.java:840)
> {code}
> The master has the following configuration file.
> {code:java}
> spark.network.crypto.keyLength 128
> spark.metrics.conf.*.sink.prometheusServlet.class
> org.apache.spark.metrics.sink.PrometheusServlet
> spark.metrics.conf.*.sink.prometheusServlet.path /metrics
> spark.metrics.conf.applications.sink.prometheusServlet.path /metrics
> spark.ssl.needClientAuth false
> spark.ssl.keyStoreType JKS
> spark.network.crypto.enabled true
> spark.metrics.namespace ""
> spark.ssl.trustStore /opt/spark/conf/certs/truststore.jks
> spark.ssl.protocol TLSv1.2
> spark.ssl.keyStore /opt/spark/conf/certs/keystore.jks
> spark.ssl.trustStoreType JKS
> spark.ssl.standalone.port 8480
> spark.ssl.keyPassword the-password
> spark.authenticate.secret the-password
> spark.ui.reverseProxy true
> spark.metrics.conf.master.sink.prometheusServlet.path /metrics
> spark.ssl.keyStorePassword the-password
> spark.ssl.trustStorePassword the-password
> spark.ssl.enabled true
> spark.authenticate true
> spark.ui.reverseProxyUrl https://sdwbgn.poc:8080 {code}
> The worker has the following configuration file.
> {code:java}
> spark.metrics.conf.*.sink.prometheusServlet.path /metrics
> spark.ssl.trustStorePassword the-password
> spark.authenticate.secret the-password
> spark.ui.reverseProxy true
> spark.ui.reverseProxyUrl https://sdwbgn.poc:8080
> spark.metrics.conf.master.sink.prometheusServlet.path /metrics
> spark.ssl.trustStore /opt/spark/conf/certs/truststore.jks
> spark.ssl.trustStoreType JKS
> spark.ssl.enabled true
> spark.ssl.keyStoreType JKS
> spark.network.crypto.keyLength 128
> spark.metrics.namespace ""
> spark.metrics.conf.*.sink.prometheusServlet.class
> org.apache.spark.metrics.sink.PrometheusServlet
> spark.metrics.conf.applications.sink.prometheusServlet.path /metrics
> spark.ssl.protocol TLSv1.2
> spark.authenticate true
> spark.network.crypto.enabled true
> spark.ssl.keyStore /opt/spark/conf/certs/keystore.jks
> spark.ssl.keyPassword the-password
> spark.ssl.keyStorePassword the-password
> spark.ssl.needClientAuth false
> spark.ssl.standalone.port 8480 {code}
> If we disable SSL/TSL on the Worker, than all works fine, and we can use the
> Master configured as reverse proxy as expected.
> We are running Spark in stand-alone mode on a Kubernetes. The containers we
> are using are based on Debian 11 (minideb). The Spark version is 3.5. Do not
> hesitate to ask further information if required.
> It would look like a bug to us, but before opening a bug ticket, I would like
> to have some feedback about possible misconfiguration, or other cause for the
> error.
> Many thanks in advance!
>
> Regards,
> Filippo
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
