Kaveen Raajan created SPARK-9587:
------------------------------------
Summary: Spark Web UI not displaying while changing another network
Key: SPARK-9587
URL: https://issues.apache.org/jira/browse/SPARK-9587
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 1.4.1
Environment: Windows,
Hadoop-2.5.2,
Reporter: Kaveen Raajan
I want to start my spark-shell with localhost instead of IP. I'm running
spark-shell in yarn-client mode. My Hadoop are running as singlenode cluster
connecting with localhost.
I changed following property in spark-default.conf
{panel:title=spark-default.conf}
spark.driver.host localhost
spark.driver.hosts localhost
{panel}
Initially while starting spark-shell I'm connecting with some public network
(172.16.xxx.yyy) If I disconnect network mean Spark jobs are working without
any problem. But Spark web UI are not working.
ApplicationMaster always connecting with current IP instead of localhost.
My log are here
{code}
15/08/04 10:17:10 INFO spark.SecurityManager: Changing view acls to: SYSTEM
15/08/04 10:17:10 INFO spark.SecurityManager: Changing modify acls to: SYSTEM
15/08/04 10:17:10 INFO spark.SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(SYSTEM); users
with modify permissions: Set(SYSTEM)
15/08/04 10:17:10 INFO spark.HttpServer: Starting HTTP Server
15/08/04 10:17:10 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/08/04 10:17:10 INFO server.AbstractConnector: Started
[email protected]:58416
15/08/04 10:17:10 INFO util.Utils: Successfully started service 'HTTP class
server' on port 58416.
15/08/04 10:17:15 INFO spark.SparkContext: Running Spark version 1.4.0
15/08/04 10:17:15 INFO spark.SecurityManager: Changing view acls to: SYSTEM
15/08/04 10:17:15 INFO spark.SecurityManager: Changing modify acls to: SYSTEM
15/08/04 10:17:15 INFO spark.SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(SYSTEM); users
with modify permissions: Set(SYSTEM)
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.4.0
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_51)
Type in expressions to have them evaluated.
Type :help for more information.
15/08/04 10:17:15 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/08/04 10:17:15 INFO Remoting: Starting remoting
15/08/04 10:17:16 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkDriver@localhost:58439]
15/08/04 10:17:16 INFO util.Utils: Successfully started service 'sparkDriver'
on port 58439.
15/08/04 10:17:16 INFO spark.SparkEnv: Registering MapOutputTracker
15/08/04 10:17:16 INFO spark.SparkEnv: Registering BlockManagerMaster
15/08/04 10:17:16 INFO storage.DiskBlockManager: Created local directory at
C:\Windows\Temp\spark-86221988-7e8b-4340-be80-a2be283845e3\blockmgr-2c1b95de-936b-44f3-b98d-263c45e310ca
15/08/04 10:17:16 INFO storage.MemoryStore: MemoryStore started with capacity
265.4 MB
15/08/04 10:17:16 INFO spark.HttpFileServer: HTTP File server directory is
C:\Windows\Temp\spark-86221988-7e8b-4340-be80-a2be283845e3\httpd-da7b686d-deb0-446d-af20-42ded6d6d035
15/08/04 10:17:16 INFO spark.HttpServer: Starting HTTP Server
15/08/04 10:17:16 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/08/04 10:17:16 INFO server.AbstractConnector: Started
[email protected]:58440
15/08/04 10:17:16 INFO util.Utils: Successfully started service 'HTTP file
server' on port 58440.
15/08/04 10:17:16 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/08/04 10:17:16 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/08/04 10:17:16 INFO server.AbstractConnector: Started
[email protected]:4040
15/08/04 10:17:16 INFO util.Utils: Successfully started service 'SparkUI' on
port 4040.
15/08/04 10:17:16 INFO ui.SparkUI: Started SparkUI at http://172.16.123.123:4040
15/08/04 10:17:16 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
15/08/04 10:17:17 INFO yarn.Client: Requesting a new application from cluster
with 1 NodeManagers
15/08/04 10:17:17 INFO yarn.Client: Verifying our application has not requested
more than the maximum memory capability of the cluster (2048 MB per container)
15/08/04 10:17:17 INFO yarn.Client: Will allocate AM container, with 896 MB
memory including 384 MB overhead
15/08/04 10:17:17 INFO yarn.Client: Setting up container launch context for our
AM
15/08/04 10:17:17 INFO yarn.Client: Preparing resources for our AM container
15/08/04 10:17:17 INFO yarn.Client: Uploading resource
file:/C://Spark/lib/spark-assembly-1.4.0-hadoop2.5.2.jar ->
hdfs://localhost:9000/user/SYSTEM/.sparkStaging/application_1438662854479_0001/spark-assembly-1.4.0-hadoop2.5.2.jar
15/08/04 10:17:20 INFO yarn.Client: Uploading resource
file:/C:/Windows/Temp/spark-86221988-7e8b-4340-be80-a2be283845e3/__hadoop_conf__3573844093591295334.zip
->
hdfs://localhost:9000/user/SYSTEM/.sparkStaging/application_1438662854479_0001/__hadoop_conf__3573844093591295334.zip
15/08/04 10:17:21 INFO yarn.Client: Setting up the launch environment for our
AM container
15/08/04 10:17:21 INFO spark.SecurityManager: Changing view acls to: SYSTEM
15/08/04 10:17:21 INFO spark.SecurityManager: Changing modify acls to: SYSTEM
15/08/04 10:17:21 INFO spark.SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(SYSTEM); users
with modify permissions: Set(SYSTEM)
15/08/04 10:17:21 INFO yarn.Client: Submitting application 1 to ResourceManager
15/08/04 10:17:21 INFO impl.YarnClientImpl: Submitted application
application_1438662854479_0001
15/08/04 10:17:22 INFO yarn.Client: Application report for
application_1438662854479_0001 (state: ACCEPTED)
15/08/04 10:17:22 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1438663641312
final status: UNDEFINED
tracking URL: http://MASTER:8088/proxy/application_1438662854479_0001/
user: SYSTEM
15/08/04 10:17:23 INFO yarn.Client: Application report for
application_1438662854479_0001 (state: ACCEPTED)
15/08/04 10:17:24 INFO yarn.Client: Application report for
application_1438662854479_0001 (state: ACCEPTED)
15/08/04 10:17:25 INFO yarn.Client: Application report for
application_1438662854479_0001 (state: ACCEPTED)
15/08/04 10:17:26 INFO yarn.Client: Application report for
application_1438662854479_0001 (state: ACCEPTED)
15/08/04 10:17:27 INFO yarn.Client: Application report for
application_1438662854479_0001 (state: ACCEPTED)
15/08/04 10:17:28 INFO yarn.Client: Application report for
application_1438662854479_0001 (state: ACCEPTED)
15/08/04 10:17:29 INFO yarn.Client: Application report for
application_1438662854479_0001 (state: ACCEPTED)
15/08/04 10:17:29 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
ApplicationMaster registered as
AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:58480/user/YarnAM#-1256718400])
15/08/04 10:17:29 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter.
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS ->
MASTER, PROXY_URI_BASES ->
http://mASTER:8088/proxy/application_1438662854479_0001),
/proxy/application_1438662854479_0001
15/08/04 10:17:29 INFO ui.JettyUtils: Adding filter:
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
15/08/04 10:17:30 INFO yarn.Client: Application report for
application_1438662854479_0001 (state: RUNNING)
15/08/04 10:17:30 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 172.16.123.123
ApplicationMaster RPC port: 0
queue: default
start time: 1438663641312
final status: UNDEFINED
tracking URL: http://MASTER:8088/proxy/application_1438662854479_0001/
user: SYSTEM
15/08/04 10:17:30 INFO cluster.YarnClientSchedulerBackend: Application
application_1438662854479_0001 has started running.
15/08/04 10:17:31 INFO util.Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 58502.
15/08/04 10:17:31 INFO netty.NettyBlockTransferService: Server created on 58502
15/08/04 10:17:31 INFO storage.BlockManagerMaster: Trying to register
BlockManager
15/08/04 10:17:31 INFO storage.BlockManagerMasterEndpoint: Registering block
manager 172.16.123.123:58502 with 265.4 MB RAM, BlockManagerId(driver,
172.16.123.123, 58502)
15/08/04 10:17:31 INFO storage.BlockManagerMaster: Registered BlockManager
15/08/04 10:17:41 INFO cluster.YarnClientSchedulerBackend: Registered executor:
AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:58564/user/Executor#1372491846])
with ID 2
15/08/04 10:17:41 INFO cluster.YarnClientSchedulerBackend: Registered executor:
AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:58561/user/Executor#-306444795])
with ID 1
15/08/04 10:17:41 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is
ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
15/08/04 10:17:41 INFO repl.SparkILoop: Created spark context..
15/08/04 10:17:42 INFO hive.HiveContext: Initializing execution hive, version
0.13.1
15/08/04 10:17:42 INFO storage.BlockManagerMasterEndpoint: Registering block
manager 127.0.0.1:58575 with 530.3 MB RAM, BlockManagerId(2, 127.0.0.1, 58575)
15/08/04 10:17:42 INFO storage.BlockManagerMasterEndpoint: Registering block
manager 127.0.0.1:58576 with 530.3 MB RAM, BlockManagerId(1, 127.0.0.1, 58576)
15/08/04 10:17:42 INFO hive.metastore: Trying to connect to metastore with URI
thrift://localhost:9083
Spark context available as sc.
15/08/04 10:17:42 INFO hive.metastore: Connected to metastore.
15/08/04 10:17:42 INFO session.SessionState: No Tez session required at this
point. hive.execution.engine=mr.
15/08/04 10:17:42 INFO repl.SparkILoop: Created sql context (with Hive
support)..
SQL context available as sqlContext.
scala>
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
