Re: Cannot access Spark web UI
My Hadoop version is Hadoop 2.5.0-cdh5.3.0 >From the Driver logs [3] I can see that SparkUI started on a specified port, also my YARN app tracking URL[1] points to that port which is in turn getting redirected to the proxy URL[2] which gives me java.net.BindException: Cannot assign requested address. If there was a port conflict issue the sparkUI stark will have issues but that id not the case. [1] YARN: application_1424814313649_0006 spark-realtime-MessageStoreWriter SPARK ciuser root.ciuser RUNNING UNDEFINED 10% http://host21.cloud.com:44648 [2] ProxyURL: http://host28.cloud.com:8088/proxy/application_1424814313649_0006/ [3] LOGS: 15/02/25 04:25:02 INFO util.Utils: Successfully started service 'SparkUI' on port 44648. 15/02/25 04:25:02 INFO ui.SparkUI: Started SparkUI at http://host21.cloud.com:44648 15/02/25 04:25:02 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler 15/02/25 04:25:02 INFO netty.NettyBlockTransferService: Server created on 41518 On Wed, Feb 18, 2015 at 3:15 PM, Arush Kharbanda wrote: > It seems like that its not able to get a port it needs are you sure that > the required port is available. In what logs did you find this error? > > On Wed, Feb 18, 2015 at 2:21 PM, Akhil Das > wrote: > >> The error says Cannot assign requested address. This means that you need >> to use the correct address for one of your network interfaces or 0.0.0.0 to >> accept connections from all interfaces. Can you paste your spark-env.sh >> file and /etc/hosts file. >> >> Thanks >> Best Regards >> >> On Wed, Feb 18, 2015 at 2:06 PM, Mukesh Jha >> wrote: >> >>> Hello Experts, >>> >>> I am running a spark-streaming app inside YARN. I have Spark History >>> server running as well (Do we need it running to access UI?). >>> >>> The app is running fine as expected but the Spark's web UI is not >>> accessible. >>> >>> When I try to access the ApplicationMaster of the Yarn application I get >>> the below error. >>> >>> This looks very similar to >>> https://issues.apache.org/jira/browse/SPARK-5837 but instead of >>> "java.net.ConnectException: >>> Connection refused" I am getting "java.net.BindException: Cannot assign >>> requested address" as shown below. >>> >>> Please let me know if you have faced / fixed this issue, any help is >>> greatly appreciated. >>> >>> >>> *Exception* >>> >>> HTTP ERROR 500 >>> >>> Problem accessing /proxy/application_1424161379156_0001/. Reason: >>> >>> Cannot assign requested address >>> >>> Caused by: >>> >>> java.net.BindException: Cannot assign requested address >>> at java.net.PlainSocketImpl.socketBind(Native Method) >>> at >>> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) >>> at java.net.Socket.bind(Socket.java:631) >>> at java.net.Socket.(Socket.java:423) >>> at java.net.Socket.(Socket.java:280) >>> at >>> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) >>> at >>> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) >>> at >>> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) >>> at >>> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) >>> at >>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) >>> at >>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) >>> at >>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) >>> at >>> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:188) >>> at >>> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:345) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >>> at >>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >>> at >>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66) >>> at >>> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) >>> at >>> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) >>> at >>> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:84) >>> at >>> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) >>> at >>> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) >>> at >>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >>> at >>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) >>> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) >>> at >>> org.mortbay.jetty.servlet.ServletHand
Re: Cannot access Spark web UI
It seems like that its not able to get a port it needs are you sure that the required port is available. In what logs did you find this error? On Wed, Feb 18, 2015 at 2:21 PM, Akhil Das wrote: > The error says Cannot assign requested address. This means that you need > to use the correct address for one of your network interfaces or 0.0.0.0 to > accept connections from all interfaces. Can you paste your spark-env.sh > file and /etc/hosts file. > > Thanks > Best Regards > > On Wed, Feb 18, 2015 at 2:06 PM, Mukesh Jha > wrote: > >> Hello Experts, >> >> I am running a spark-streaming app inside YARN. I have Spark History >> server running as well (Do we need it running to access UI?). >> >> The app is running fine as expected but the Spark's web UI is not >> accessible. >> >> When I try to access the ApplicationMaster of the Yarn application I get >> the below error. >> >> This looks very similar to >> https://issues.apache.org/jira/browse/SPARK-5837 but instead of >> "java.net.ConnectException: >> Connection refused" I am getting "java.net.BindException: Cannot assign >> requested address" as shown below. >> >> Please let me know if you have faced / fixed this issue, any help is >> greatly appreciated. >> >> >> *Exception* >> >> HTTP ERROR 500 >> >> Problem accessing /proxy/application_1424161379156_0001/. Reason: >> >> Cannot assign requested address >> >> Caused by: >> >> java.net.BindException: Cannot assign requested address >> at java.net.PlainSocketImpl.socketBind(Native Method) >> at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) >> at java.net.Socket.bind(Socket.java:631) >> at java.net.Socket.(Socket.java:423) >> at java.net.Socket.(Socket.java:280) >> at >> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) >> at >> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) >> at >> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) >> at >> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) >> at >> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) >> at >> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) >> at >> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) >> at >> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:188) >> at >> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:345) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >> at >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66) >> at >> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) >> at >> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) >> at >> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:84) >> at >> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) >> at >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) >> at >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> at >> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) >> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> at >> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> at >> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) >> at >> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:555) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> at >> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1223) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> at >>
Re: Cannot access Spark web UI
The error says Cannot assign requested address. This means that you need to use the correct address for one of your network interfaces or 0.0.0.0 to accept connections from all interfaces. Can you paste your spark-env.sh file and /etc/hosts file. Thanks Best Regards On Wed, Feb 18, 2015 at 2:06 PM, Mukesh Jha wrote: > Hello Experts, > > I am running a spark-streaming app inside YARN. I have Spark History > server running as well (Do we need it running to access UI?). > > The app is running fine as expected but the Spark's web UI is not > accessible. > > When I try to access the ApplicationMaster of the Yarn application I get > the below error. > > This looks very similar to > https://issues.apache.org/jira/browse/SPARK-5837 but instead of > "java.net.ConnectException: > Connection refused" I am getting "java.net.BindException: Cannot assign > requested address" as shown below. > > Please let me know if you have faced / fixed this issue, any help is > greatly appreciated. > > > *Exception* > > HTTP ERROR 500 > > Problem accessing /proxy/application_1424161379156_0001/. Reason: > > Cannot assign requested address > > Caused by: > > java.net.BindException: Cannot assign requested address > at java.net.PlainSocketImpl.socketBind(Native Method) > at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) > at java.net.Socket.bind(Socket.java:631) > at java.net.Socket.(Socket.java:423) > at java.net.Socket.(Socket.java:280) > at > org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) > at > org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) > at > org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at > org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at > org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:188) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:345) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) > at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) > at > org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:84) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) > at > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) > at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) > at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:555) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1223) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) > at o
Cannot access Spark web UI
Hello Experts, I am running a spark-streaming app inside YARN. I have Spark History server running as well (Do we need it running to access UI?). The app is running fine as expected but the Spark's web UI is not accessible. When I try to access the ApplicationMaster of the Yarn application I get the below error. This looks very similar to https://issues.apache.org/jira/browse/SPARK-5837 but instead of "java.net.ConnectException: Connection refused" I am getting "java.net.BindException: Cannot assign requested address" as shown below. Please let me know if you have faced / fixed this issue, any help is greatly appreciated. *Exception* HTTP ERROR 500 Problem accessing /proxy/application_1424161379156_0001/. Reason: Cannot assign requested address Caused by: java.net.BindException: Cannot assign requested address at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) at java.net.Socket.bind(Socket.java:631) at java.net.Socket.(Socket.java:423) at java.net.Socket.(Socket.java:280) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:188) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:345) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:84) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:555) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1223) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
