[
https://issues.apache.org/jira/browse/SPARK-9587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaveen Raajan closed SPARK-9587.
--------------------------------
Resolution: Not A Problem
This was working if I *set SPARK_LOCAL_HOSTNAME={COMPUTERNAME}*. This spark
drivers and web address are refferring to hostname instead of IP.
> Spark Web UI not displaying while changing another network
> ----------------------------------------------------------
>
> Key: SPARK-9587
> URL: https://issues.apache.org/jira/browse/SPARK-9587
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.4.1
> Environment: Windows,
> Hadoop-2.5.2,
> Reporter: Kaveen Raajan
>
> I want to start my spark-shell with localhost instead of IP. I'm running
> spark-shell in yarn-client mode. My Hadoop are running as singlenode cluster
> connecting with localhost.
> I changed following property in spark-default.conf
> {panel:title=spark-default.conf}
> spark.driver.host localhost
> spark.driver.hosts localhost
> {panel}
> Initially while starting spark-shell I'm connecting with some public network
> (172.16.xxx.yyy) If I disconnect network mean Spark jobs are working without
> any problem. But Spark web UI are not working.
> ApplicationMaster always connecting with current IP instead of localhost.
> My log are here
> {code}
> 15/08/04 10:17:10 INFO spark.SecurityManager: Changing view acls to: SYSTEM
> 15/08/04 10:17:10 INFO spark.SecurityManager: Changing modify acls to: SYSTEM
> 15/08/04 10:17:10 INFO spark.SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(SYSTEM); users
> with modify permissions: Set(SYSTEM)
> 15/08/04 10:17:10 INFO spark.HttpServer: Starting HTTP Server
> 15/08/04 10:17:10 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/08/04 10:17:10 INFO server.AbstractConnector: Started
> [email protected]:58416
> 15/08/04 10:17:10 INFO util.Utils: Successfully started service 'HTTP class
> server' on port 58416.
> 15/08/04 10:17:15 INFO spark.SparkContext: Running Spark version 1.4.0
> 15/08/04 10:17:15 INFO spark.SecurityManager: Changing view acls to: SYSTEM
> 15/08/04 10:17:15 INFO spark.SecurityManager: Changing modify acls to: SYSTEM
> 15/08/04 10:17:15 INFO spark.SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(SYSTEM); users
> with modify permissions: Set(SYSTEM)
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 1.4.0
> /_/
> Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_51)
> Type in expressions to have them evaluated.
> Type :help for more information.
> 15/08/04 10:17:15 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 15/08/04 10:17:15 INFO Remoting: Starting remoting
> 15/08/04 10:17:16 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@localhost:58439]
> 15/08/04 10:17:16 INFO util.Utils: Successfully started service 'sparkDriver'
> on port 58439.
> 15/08/04 10:17:16 INFO spark.SparkEnv: Registering MapOutputTracker
> 15/08/04 10:17:16 INFO spark.SparkEnv: Registering BlockManagerMaster
> 15/08/04 10:17:16 INFO storage.DiskBlockManager: Created local directory at
> C:\Windows\Temp\spark-86221988-7e8b-4340-be80-a2be283845e3\blockmgr-2c1b95de-936b-44f3-b98d-263c45e310ca
> 15/08/04 10:17:16 INFO storage.MemoryStore: MemoryStore started with capacity
> 265.4 MB
> 15/08/04 10:17:16 INFO spark.HttpFileServer: HTTP File server directory is
> C:\Windows\Temp\spark-86221988-7e8b-4340-be80-a2be283845e3\httpd-da7b686d-deb0-446d-af20-42ded6d6d035
> 15/08/04 10:17:16 INFO spark.HttpServer: Starting HTTP Server
> 15/08/04 10:17:16 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/08/04 10:17:16 INFO server.AbstractConnector: Started
> [email protected]:58440
> 15/08/04 10:17:16 INFO util.Utils: Successfully started service 'HTTP file
> server' on port 58440.
> 15/08/04 10:17:16 INFO spark.SparkEnv: Registering OutputCommitCoordinator
> 15/08/04 10:17:16 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/08/04 10:17:16 INFO server.AbstractConnector: Started
> [email protected]:4040
> 15/08/04 10:17:16 INFO util.Utils: Successfully started service 'SparkUI' on
> port 4040.
> 15/08/04 10:17:16 INFO ui.SparkUI: Started SparkUI at
> http://172.16.123.123:4040
> 15/08/04 10:17:16 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 15/08/04 10:17:17 INFO yarn.Client: Requesting a new application from cluster
> with 1 NodeManagers
> 15/08/04 10:17:17 INFO yarn.Client: Verifying our application has not
> requested more than the maximum memory capability of the cluster (2048 MB per
> container)
> 15/08/04 10:17:17 INFO yarn.Client: Will allocate AM container, with 896 MB
> memory including 384 MB overhead
> 15/08/04 10:17:17 INFO yarn.Client: Setting up container launch context for
> our AM
> 15/08/04 10:17:17 INFO yarn.Client: Preparing resources for our AM container
> 15/08/04 10:17:17 INFO yarn.Client: Uploading resource
> file:/C://Spark/lib/spark-assembly-1.4.0-hadoop2.5.2.jar ->
> hdfs://localhost:9000/user/SYSTEM/.sparkStaging/application_1438662854479_0001/spark-assembly-1.4.0-hadoop2.5.2.jar
> 15/08/04 10:17:20 INFO yarn.Client: Uploading resource
> file:/C:/Windows/Temp/spark-86221988-7e8b-4340-be80-a2be283845e3/__hadoop_conf__3573844093591295334.zip
> ->
> hdfs://localhost:9000/user/SYSTEM/.sparkStaging/application_1438662854479_0001/__hadoop_conf__3573844093591295334.zip
> 15/08/04 10:17:21 INFO yarn.Client: Setting up the launch environment for our
> AM container
> 15/08/04 10:17:21 INFO spark.SecurityManager: Changing view acls to: SYSTEM
> 15/08/04 10:17:21 INFO spark.SecurityManager: Changing modify acls to: SYSTEM
> 15/08/04 10:17:21 INFO spark.SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(SYSTEM); users
> with modify permissions: Set(SYSTEM)
> 15/08/04 10:17:21 INFO yarn.Client: Submitting application 1 to
> ResourceManager
> 15/08/04 10:17:21 INFO impl.YarnClientImpl: Submitted application
> application_1438662854479_0001
> 15/08/04 10:17:22 INFO yarn.Client: Application report for
> application_1438662854479_0001 (state: ACCEPTED)
> 15/08/04 10:17:22 INFO yarn.Client:
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: N/A
> ApplicationMaster RPC port: -1
> queue: default
> start time: 1438663641312
> final status: UNDEFINED
> tracking URL: http://MASTER:8088/proxy/application_1438662854479_0001/
> user: SYSTEM
> 15/08/04 10:17:23 INFO yarn.Client: Application report for
> application_1438662854479_0001 (state: ACCEPTED)
> 15/08/04 10:17:24 INFO yarn.Client: Application report for
> application_1438662854479_0001 (state: ACCEPTED)
> 15/08/04 10:17:25 INFO yarn.Client: Application report for
> application_1438662854479_0001 (state: ACCEPTED)
> 15/08/04 10:17:26 INFO yarn.Client: Application report for
> application_1438662854479_0001 (state: ACCEPTED)
> 15/08/04 10:17:27 INFO yarn.Client: Application report for
> application_1438662854479_0001 (state: ACCEPTED)
> 15/08/04 10:17:28 INFO yarn.Client: Application report for
> application_1438662854479_0001 (state: ACCEPTED)
> 15/08/04 10:17:29 INFO yarn.Client: Application report for
> application_1438662854479_0001 (state: ACCEPTED)
> 15/08/04 10:17:29 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
> ApplicationMaster registered as
> AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:58480/user/YarnAM#-1256718400])
> 15/08/04 10:17:29 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter.
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS
> -> MASTER, PROXY_URI_BASES ->
> http://mASTER:8088/proxy/application_1438662854479_0001),
> /proxy/application_1438662854479_0001
> 15/08/04 10:17:29 INFO ui.JettyUtils: Adding filter:
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
> 15/08/04 10:17:30 INFO yarn.Client: Application report for
> application_1438662854479_0001 (state: RUNNING)
> 15/08/04 10:17:30 INFO yarn.Client:
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: 172.16.123.123
> ApplicationMaster RPC port: 0
> queue: default
> start time: 1438663641312
> final status: UNDEFINED
> tracking URL: http://MASTER:8088/proxy/application_1438662854479_0001/
> user: SYSTEM
> 15/08/04 10:17:30 INFO cluster.YarnClientSchedulerBackend: Application
> application_1438662854479_0001 has started running.
> 15/08/04 10:17:31 INFO util.Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 58502.
> 15/08/04 10:17:31 INFO netty.NettyBlockTransferService: Server created on
> 58502
> 15/08/04 10:17:31 INFO storage.BlockManagerMaster: Trying to register
> BlockManager
> 15/08/04 10:17:31 INFO storage.BlockManagerMasterEndpoint: Registering block
> manager 172.16.123.123:58502 with 265.4 MB RAM, BlockManagerId(driver,
> 172.16.123.123, 58502)
> 15/08/04 10:17:31 INFO storage.BlockManagerMaster: Registered BlockManager
> 15/08/04 10:17:41 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:58564/user/Executor#1372491846])
> with ID 2
> 15/08/04 10:17:41 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:58561/user/Executor#-306444795])
> with ID 1
> 15/08/04 10:17:41 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend
> is ready for scheduling beginning after reached minRegisteredResourcesRatio:
> 0.8
> 15/08/04 10:17:41 INFO repl.SparkILoop: Created spark context..
> 15/08/04 10:17:42 INFO hive.HiveContext: Initializing execution hive, version
> 0.13.1
> 15/08/04 10:17:42 INFO storage.BlockManagerMasterEndpoint: Registering block
> manager 127.0.0.1:58575 with 530.3 MB RAM, BlockManagerId(2, 127.0.0.1, 58575)
> 15/08/04 10:17:42 INFO storage.BlockManagerMasterEndpoint: Registering block
> manager 127.0.0.1:58576 with 530.3 MB RAM, BlockManagerId(1, 127.0.0.1, 58576)
> 15/08/04 10:17:42 INFO hive.metastore: Trying to connect to metastore with
> URI thrift://localhost:9083
> Spark context available as sc.
> 15/08/04 10:17:42 INFO hive.metastore: Connected to metastore.
> 15/08/04 10:17:42 INFO session.SessionState: No Tez session required at this
> point. hive.execution.engine=mr.
> 15/08/04 10:17:42 INFO repl.SparkILoop: Created sql context (with Hive
> support)..
> SQL context available as sqlContext.
> scala>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
