Kaveen Raajan created SPARK-9587: ------------------------------------ Summary: Spark Web UI not displaying while changing another network Key: SPARK-9587 URL: https://issues.apache.org/jira/browse/SPARK-9587 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 1.4.1 Environment: Windows, Hadoop-2.5.2, Reporter: Kaveen Raajan
I want to start my spark-shell with localhost instead of IP. I'm running spark-shell in yarn-client mode. My Hadoop are running as singlenode cluster connecting with localhost. I changed following property in spark-default.conf {panel:title=spark-default.conf} spark.driver.host localhost spark.driver.hosts localhost {panel} Initially while starting spark-shell I'm connecting with some public network (172.16.xxx.yyy) If I disconnect network mean Spark jobs are working without any problem. But Spark web UI are not working. ApplicationMaster always connecting with current IP instead of localhost. My log are here {code} 15/08/04 10:17:10 INFO spark.SecurityManager: Changing view acls to: SYSTEM 15/08/04 10:17:10 INFO spark.SecurityManager: Changing modify acls to: SYSTEM 15/08/04 10:17:10 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(SYSTEM); users with modify permissions: Set(SYSTEM) 15/08/04 10:17:10 INFO spark.HttpServer: Starting HTTP Server 15/08/04 10:17:10 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/08/04 10:17:10 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:58416 15/08/04 10:17:10 INFO util.Utils: Successfully started service 'HTTP class server' on port 58416. 15/08/04 10:17:15 INFO spark.SparkContext: Running Spark version 1.4.0 15/08/04 10:17:15 INFO spark.SecurityManager: Changing view acls to: SYSTEM 15/08/04 10:17:15 INFO spark.SecurityManager: Changing modify acls to: SYSTEM 15/08/04 10:17:15 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(SYSTEM); users with modify permissions: Set(SYSTEM) Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.4.0 /_/ Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_51) Type in expressions to have them evaluated. Type :help for more information. 15/08/04 10:17:15 INFO slf4j.Slf4jLogger: Slf4jLogger started 15/08/04 10:17:15 INFO Remoting: Starting remoting 15/08/04 10:17:16 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@localhost:58439] 15/08/04 10:17:16 INFO util.Utils: Successfully started service 'sparkDriver' on port 58439. 15/08/04 10:17:16 INFO spark.SparkEnv: Registering MapOutputTracker 15/08/04 10:17:16 INFO spark.SparkEnv: Registering BlockManagerMaster 15/08/04 10:17:16 INFO storage.DiskBlockManager: Created local directory at C:\Windows\Temp\spark-86221988-7e8b-4340-be80-a2be283845e3\blockmgr-2c1b95de-936b-44f3-b98d-263c45e310ca 15/08/04 10:17:16 INFO storage.MemoryStore: MemoryStore started with capacity 265.4 MB 15/08/04 10:17:16 INFO spark.HttpFileServer: HTTP File server directory is C:\Windows\Temp\spark-86221988-7e8b-4340-be80-a2be283845e3\httpd-da7b686d-deb0-446d-af20-42ded6d6d035 15/08/04 10:17:16 INFO spark.HttpServer: Starting HTTP Server 15/08/04 10:17:16 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/08/04 10:17:16 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:58440 15/08/04 10:17:16 INFO util.Utils: Successfully started service 'HTTP file server' on port 58440. 15/08/04 10:17:16 INFO spark.SparkEnv: Registering OutputCommitCoordinator 15/08/04 10:17:16 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/08/04 10:17:16 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 15/08/04 10:17:16 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 15/08/04 10:17:16 INFO ui.SparkUI: Started SparkUI at http://172.16.123.123:4040 15/08/04 10:17:16 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 15/08/04 10:17:17 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers 15/08/04 10:17:17 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (2048 MB per container) 15/08/04 10:17:17 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 15/08/04 10:17:17 INFO yarn.Client: Setting up container launch context for our AM 15/08/04 10:17:17 INFO yarn.Client: Preparing resources for our AM container 15/08/04 10:17:17 INFO yarn.Client: Uploading resource file:/C://Spark/lib/spark-assembly-1.4.0-hadoop2.5.2.jar -> hdfs://localhost:9000/user/SYSTEM/.sparkStaging/application_1438662854479_0001/spark-assembly-1.4.0-hadoop2.5.2.jar 15/08/04 10:17:20 INFO yarn.Client: Uploading resource file:/C:/Windows/Temp/spark-86221988-7e8b-4340-be80-a2be283845e3/__hadoop_conf__3573844093591295334.zip -> hdfs://localhost:9000/user/SYSTEM/.sparkStaging/application_1438662854479_0001/__hadoop_conf__3573844093591295334.zip 15/08/04 10:17:21 INFO yarn.Client: Setting up the launch environment for our AM container 15/08/04 10:17:21 INFO spark.SecurityManager: Changing view acls to: SYSTEM 15/08/04 10:17:21 INFO spark.SecurityManager: Changing modify acls to: SYSTEM 15/08/04 10:17:21 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(SYSTEM); users with modify permissions: Set(SYSTEM) 15/08/04 10:17:21 INFO yarn.Client: Submitting application 1 to ResourceManager 15/08/04 10:17:21 INFO impl.YarnClientImpl: Submitted application application_1438662854479_0001 15/08/04 10:17:22 INFO yarn.Client: Application report for application_1438662854479_0001 (state: ACCEPTED) 15/08/04 10:17:22 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1438663641312 final status: UNDEFINED tracking URL: http://MASTER:8088/proxy/application_1438662854479_0001/ user: SYSTEM 15/08/04 10:17:23 INFO yarn.Client: Application report for application_1438662854479_0001 (state: ACCEPTED) 15/08/04 10:17:24 INFO yarn.Client: Application report for application_1438662854479_0001 (state: ACCEPTED) 15/08/04 10:17:25 INFO yarn.Client: Application report for application_1438662854479_0001 (state: ACCEPTED) 15/08/04 10:17:26 INFO yarn.Client: Application report for application_1438662854479_0001 (state: ACCEPTED) 15/08/04 10:17:27 INFO yarn.Client: Application report for application_1438662854479_0001 (state: ACCEPTED) 15/08/04 10:17:28 INFO yarn.Client: Application report for application_1438662854479_0001 (state: ACCEPTED) 15/08/04 10:17:29 INFO yarn.Client: Application report for application_1438662854479_0001 (state: ACCEPTED) 15/08/04 10:17:29 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka.tcp://sparkYarnAM@172.16.123.123:58480/user/YarnAM#-1256718400]) 15/08/04 10:17:29 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> MASTER, PROXY_URI_BASES -> http://mASTER:8088/proxy/application_1438662854479_0001), /proxy/application_1438662854479_0001 15/08/04 10:17:29 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 15/08/04 10:17:30 INFO yarn.Client: Application report for application_1438662854479_0001 (state: RUNNING) 15/08/04 10:17:30 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: 172.16.123.123 ApplicationMaster RPC port: 0 queue: default start time: 1438663641312 final status: UNDEFINED tracking URL: http://MASTER:8088/proxy/application_1438662854479_0001/ user: SYSTEM 15/08/04 10:17:30 INFO cluster.YarnClientSchedulerBackend: Application application_1438662854479_0001 has started running. 15/08/04 10:17:31 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 58502. 15/08/04 10:17:31 INFO netty.NettyBlockTransferService: Server created on 58502 15/08/04 10:17:31 INFO storage.BlockManagerMaster: Trying to register BlockManager 15/08/04 10:17:31 INFO storage.BlockManagerMasterEndpoint: Registering block manager 172.16.123.123:58502 with 265.4 MB RAM, BlockManagerId(driver, 172.16.123.123, 58502) 15/08/04 10:17:31 INFO storage.BlockManagerMaster: Registered BlockManager 15/08/04 10:17:41 INFO cluster.YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@127.0.0.1:58564/user/Executor#1372491846]) with ID 2 15/08/04 10:17:41 INFO cluster.YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@127.0.0.1:58561/user/Executor#-306444795]) with ID 1 15/08/04 10:17:41 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 15/08/04 10:17:41 INFO repl.SparkILoop: Created spark context.. 15/08/04 10:17:42 INFO hive.HiveContext: Initializing execution hive, version 0.13.1 15/08/04 10:17:42 INFO storage.BlockManagerMasterEndpoint: Registering block manager 127.0.0.1:58575 with 530.3 MB RAM, BlockManagerId(2, 127.0.0.1, 58575) 15/08/04 10:17:42 INFO storage.BlockManagerMasterEndpoint: Registering block manager 127.0.0.1:58576 with 530.3 MB RAM, BlockManagerId(1, 127.0.0.1, 58576) 15/08/04 10:17:42 INFO hive.metastore: Trying to connect to metastore with URI thrift://localhost:9083 Spark context available as sc. 15/08/04 10:17:42 INFO hive.metastore: Connected to metastore. 15/08/04 10:17:42 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr. 15/08/04 10:17:42 INFO repl.SparkILoop: Created sql context (with Hive support).. SQL context available as sqlContext. scala> {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org