I am trying to get an edge server up and running connecting to our Spark 1.1
cluster. The edge server is in a different DMZ than the rest of the cluster
and we have to specifically open firewall ports between the edge server and the
rest of the cluster. I can log on to any node in the cluster (other than the
edge) and submit code through spark-shell just fine. I have port 7077 from
the edge to the master open (verified), and I have port 7078 open from the edge
to all the workers (also verified). I have tried setting the worker port to
not be dynamic by using SPARK_WORKER_PORT in the spark-env.sh but it does not
seem to stop the dynamic port behavior. I have included the startup output
when running spark-shell from the edge server in a different dmz and then from
a node in the cluster. Any help greatly appreciated.
Paul Magid
Toyota Motor Sales IS Enterprise Architecture (EA)
Architect I R&D
Ph: 310-468-9091 (X69091)
PCN 1C2970, Mail Drop PN12
Running spark-shell from the edge server
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
14/09/23 14:20:38 INFO SecurityManager: Changing view acls to: root,
14/09/23 14:20:38 INFO SecurityManager: Changing modify acls to: root,
14/09/23 14:20:38 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root, ); users
with modify permissions: Set(root, )
14/09/23 14:20:38 INFO HttpServer: Starting HTTP Server
14/09/23 14:20:39 INFO Utils: Successfully started service 'HTTP class server'
on port 22788.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.1.0
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_55)
Type in expressions to have them evaluated.
Type :help for more information.
14/09/23 14:20:42 INFO SecurityManager: Changing view acls to: root,
14/09/23 14:20:42 INFO SecurityManager: Changing modify acls to: root,
14/09/23 14:20:42 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root, ); users
with modify permissions: Set(root, )
14/09/23 14:20:43 INFO Slf4jLogger: Slf4jLogger started
14/09/23 14:20:43 INFO Remoting: Starting remoting
14/09/23 14:20:43 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkdri...@votlbdcd09.tms.toyota.com:32356]
14/09/23 14:20:43 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://sparkdri...@votlbdcd09.tms.toyota.com:32356]
14/09/23 14:20:43 INFO Utils: Successfully started service 'sparkDriver' on
port 32356.
14/09/23 14:20:43 INFO SparkEnv: Registering MapOutputTracker
14/09/23 14:20:43 INFO SparkEnv: Registering BlockManagerMaster
14/09/23 14:20:43 INFO DiskBlockManager: Created local directory at
/tmp/spark-local-20140923142043-4454
14/09/23 14:20:43 INFO Utils: Successfully started service 'Connection manager
for block manager' on port 48469.
14/09/23 14:20:43 INFO ConnectionManager: Bound socket to port 48469 with id =
ConnectionManagerId(votlbdcd09.tms.toyota.com,48469)
14/09/23 14:20:43 INFO MemoryStore: MemoryStore started with capacity 265.9 MB
14/09/23 14:20:43 INFO BlockManagerMaster: Trying to register BlockManager
14/09/23 14:20:43 INFO BlockManagerMasterActor: Registering block manager
votlbdcd09.tms.toyota.com:48469 with 265.9 MB RAM
14/09/23 14:20:43 INFO BlockManagerMaster: Registered BlockManager
14/09/23 14:20:43 INFO HttpFileServer: HTTP File server directory is
/tmp/spark-888c359a-5a2a-4aaa-80e3-8009cdfa25c8
14/09/23 14:20:43 INFO HttpServer: Starting HTTP Server
14/09/23 14:20:43 INFO Utils: Successfully started service 'HTTP file server'
on port 12470.
14/09/23 14:20:43 INFO Utils: Successfully started service 'SparkUI' on port
4040.
14/09/23 14:20:43 INFO SparkUI: Started SparkUI at
http://votlbdcd09.tms.toyota.com:4040
14/09/23 14:20:43 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
14/09/23 14:20:44 INFO EventLoggingListener: Logging events to
file:/user/spark/applicationHistory//spark-shell-1411507243973
14/09/23 14:20:44 INFO AppClient$ClientActor: Connecting to master
spark://votlbdcd01.tms.toyota.com:7077...
14/09/23 14:20:44 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready
for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
14/09/23 14:20:44 INFO SparkILoop: Created spark context..
14/09/23 14:20:44 INFO SparkDeploySchedulerBackend: Connected to Spark cluster
with app ID app-20140923142044-0006
Spark context available as sc.
scala> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/0 on
worker-20140923084845-votlbdcd03.tms.toyota.com-7078
(votlbdcd03.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/0 on hostPort votlbdcd03.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/1 on
worker-20140923141811-votlbdcd08.tms.toyota.com-7078
(votlbdcd08.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/1 on hostPort votlbdcd08.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/2 on
worker-20140923084651-votlbdcd02.tms.toyota.com-7078
(votlbdcd02.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/2 on hostPort votlbdcd02.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/3 on
worker-20140923091151-votlbdcd05.tms.toyota.com-7078
(votlbdcd05.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/3 on hostPort votlbdcd05.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/4 on
worker-20140923103250-votlbdcd06.tms.toyota.com-7078
(votlbdcd06.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/4 on hostPort votlbdcd06.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/5 on
worker-20140923104019-votlbdcd07.tms.toyota.com-7078
(votlbdcd07.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/5 on hostPort votlbdcd07.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/6 on
worker-20140923085411-votlbdcd04.tms.toyota.com-7078
(votlbdcd04.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/6 on hostPort votlbdcd04.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/3 is now RUNNING
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/6 is now RUNNING
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/0 is now RUNNING
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/2 is now RUNNING
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/4 is now RUNNING
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/5 is now RUNNING
14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/1 is now RUNNING
scala> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/3 is now EXITED (Command exited with code 1)
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor
app-20140923142044-0006/3 removed: Command exited with code 1
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/7 on
worker-20140923091151-votlbdcd05.tms.toyota.com-7078
(votlbdcd05.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/7 on hostPort votlbdcd05.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/7 is now RUNNING
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/6 is now EXITED (Command exited with code 1)
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor
app-20140923142044-0006/6 removed: Command exited with code 1
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/8 on
worker-20140923085411-votlbdcd04.tms.toyota.com-7078
(votlbdcd04.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/8 on hostPort votlbdcd04.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/8 is now RUNNING
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/0 is now EXITED (Command exited with code 1)
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor
app-20140923142044-0006/0 removed: Command exited with code 1
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/9 on
worker-20140923084845-votlbdcd03.tms.toyota.com-7078
(votlbdcd03.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/9 on hostPort votlbdcd03.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/9 is now RUNNING
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/2 is now EXITED (Command exited with code 1)
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor
app-20140923142044-0006/2 removed: Command exited with code 1
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/10 on
worker-20140923084651-votlbdcd02.tms.toyota.com-7078
(votlbdcd02.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/10 on hostPort votlbdcd02.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/10 is now RUNNING
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/1 is now EXITED (Command exited with code 1)
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor
app-20140923142044-0006/1 removed: Command exited with code 1
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/11 on
worker-20140923141811-votlbdcd08.tms.toyota.com-7078
(votlbdcd08.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/11 on hostPort votlbdcd08.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/11 is now RUNNING
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/4 is now EXITED (Command exited with code 1)
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor
app-20140923142044-0006/4 removed: Command exited with code 1
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/12 on
worker-20140923103250-votlbdcd06.tms.toyota.com-7078
(votlbdcd06.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/12 on hostPort votlbdcd06.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/12 is now RUNNING
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/5 is now EXITED (Command exited with code 1)
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor
app-20140923142044-0006/5 removed: Command exited with code 1
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/13 on
worker-20140923104019-votlbdcd07.tms.toyota.com-7078
(votlbdcd07.tms.toyota.com:7078) with 8 cores
14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/13 on hostPort votlbdcd07.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/13 is now RUNNING
14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/7 is now EXITED (Command exited with code 1)
14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Executor
app-20140923142044-0006/7 removed: Command exited with code 1
14/09/23 14:22:30 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/14 on
worker-20140923091151-votlbdcd05.tms.toyota.com-7078
(votlbdcd05.tms.toyota.com:7078) with 8 cores
14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/14 on hostPort votlbdcd05.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/14 is now RUNNING
14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/8 is now EXITED (Command exited with code 1)
14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Executor
app-20140923142044-0006/8 removed: Command exited with code 1
14/09/23 14:22:30 INFO AppClient$ClientActor: Executor added:
app-20140923142044-0006/15 on
worker-20140923085411-votlbdcd04.tms.toyota.com-7078
(votlbdcd04.tms.toyota.com:7078) with 8 cores
14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923142044-0006/15 on hostPort votlbdcd04.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/15 is now RUNNING
14/09/23 14:22:31 INFO AppClient$ClientActor: Executor updated:
app-20140923142044-0006/9 is now EXITED (Command exited with code 1)
14/09/23 14:22:31 INFO SparkDeploySchedulerBackend: Executor
app-20140923142044-0006/9 removed: Command exited with code 1
14/09/23 14:22:31 ERROR SparkDeploySchedulerBackend: Application has been
killed. Reason: Master removed our application: FAILED
14/09/23 14:22:31 ERROR TaskSchedulerImpl: Exiting due to error from cluster
scheduler: Master removed our application: FAILED
Running spark-shell in the cluster
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
14/09/23 14:47:26 INFO SecurityManager: Changing view acls to: root,
14/09/23 14:47:26 INFO SecurityManager: Changing modify acls to: root,
14/09/23 14:47:26 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root, ); users
with modify permissions: Set(root, )
14/09/23 14:47:26 INFO HttpServer: Starting HTTP Server
14/09/23 14:47:26 INFO Utils: Successfully started service 'HTTP class server'
on port 23968.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.1.0
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_55)
Type in expressions to have them evaluated.
Type :help for more information.
14/09/23 14:47:30 INFO SecurityManager: Changing view acls to: root,
14/09/23 14:47:30 INFO SecurityManager: Changing modify acls to: root,
14/09/23 14:47:30 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root, ); users
with modify permissions: Set(root, )
14/09/23 14:47:31 INFO Slf4jLogger: Slf4jLogger started
14/09/23 14:47:31 INFO Remoting: Starting remoting
14/09/23 14:47:31 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkdri...@votlbdcd01.tms.toyota.com:44430]
14/09/23 14:47:31 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://sparkdri...@votlbdcd01.tms.toyota.com:44430]
14/09/23 14:47:31 INFO Utils: Successfully started service 'sparkDriver' on
port 44430.
14/09/23 14:47:31 INFO SparkEnv: Registering MapOutputTracker
14/09/23 14:47:31 INFO SparkEnv: Registering BlockManagerMaster
14/09/23 14:47:31 INFO DiskBlockManager: Created local directory at
/tmp/spark-local-20140923144731-48c5
14/09/23 14:47:31 INFO Utils: Successfully started service 'Connection manager
for block manager' on port 30579.
14/09/23 14:47:31 INFO ConnectionManager: Bound socket to port 30579 with id =
ConnectionManagerId(votlbdcd01.tms.toyota.com,30579)
14/09/23 14:47:31 INFO MemoryStore: MemoryStore started with capacity 265.9 MB
14/09/23 14:47:31 INFO BlockManagerMaster: Trying to register BlockManager
14/09/23 14:47:31 INFO BlockManagerMasterActor: Registering block manager
votlbdcd01.tms.toyota.com:30579 with 265.9 MB RAM
14/09/23 14:47:31 INFO BlockManagerMaster: Registered BlockManager
14/09/23 14:47:31 INFO HttpFileServer: HTTP File server directory is
/tmp/spark-1a38cbd9-107b-40ce-bb28-f1574c9fbcda
14/09/23 14:47:31 INFO HttpServer: Starting HTTP Server
14/09/23 14:47:31 INFO Utils: Successfully started service 'HTTP file server'
on port 31028.
14/09/23 14:47:31 INFO Utils: Successfully started service 'SparkUI' on port
4040.
14/09/23 14:47:31 INFO SparkUI: Started SparkUI at
http://votlbdcd01.tms.toyota.com:4040
14/09/23 14:47:31 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
14/09/23 14:47:32 INFO EventLoggingListener: Logging events to
file:/user/spark/applicationHistory//spark-shell-1411508851947
14/09/23 14:47:32 INFO AppClient$ClientActor: Connecting to master
spark://votlbdcd01.tms.toyota.com:7077...
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready
for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
14/09/23 14:47:32 INFO SparkILoop: Created spark context..
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Connected to Spark cluster
with app ID app-20140923144732-0009
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added:
app-20140923144732-0009/0 on
worker-20140923084845-votlbdcd03.tms.toyota.com-7078
(votlbdcd03.tms.toyota.com:7078) with 8 cores
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923144732-0009/0 on hostPort votlbdcd03.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added:
app-20140923144732-0009/1 on
worker-20140923084651-votlbdcd02.tms.toyota.com-7078
(votlbdcd02.tms.toyota.com:7078) with 8 cores
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923144732-0009/1 on hostPort votlbdcd02.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added:
app-20140923144732-0009/2 on
worker-20140923091151-votlbdcd05.tms.toyota.com-7078
(votlbdcd05.tms.toyota.com:7078) with 8 cores
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923144732-0009/2 on hostPort votlbdcd05.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added:
app-20140923144732-0009/3 on
worker-20140923103250-votlbdcd06.tms.toyota.com-7078
(votlbdcd06.tms.toyota.com:7078) with 8 cores
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923144732-0009/3 on hostPort votlbdcd06.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added:
app-20140923144732-0009/4 on
worker-20140923144121-votlbdcd08.tms.toyota.com-7078
(votlbdcd08.tms.toyota.com:7078) with 8 cores
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923144732-0009/4 on hostPort votlbdcd08.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added:
app-20140923144732-0009/5 on
worker-20140923104019-votlbdcd07.tms.toyota.com-7078
(votlbdcd07.tms.toyota.com:7078) with 8 cores
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923144732-0009/5 on hostPort votlbdcd07.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added:
app-20140923144732-0009/6 on
worker-20140923085411-votlbdcd04.tms.toyota.com-7078
(votlbdcd04.tms.toyota.com:7078) with 8 cores
14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140923144732-0009/6 on hostPort votlbdcd04.tms.toyota.com:7078 with 8
cores, 25.0 GB RAM
Spark context available as sc.
scala> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated:
app-20140923144732-0009/2 is now RUNNING
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated:
app-20140923144732-0009/6 is now RUNNING
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated:
app-20140923144732-0009/0 is now RUNNING
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated:
app-20140923144732-0009/1 is now RUNNING
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated:
app-20140923144732-0009/5 is now RUNNING
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated:
app-20140923144732-0009/3 is now RUNNING
14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated:
app-20140923144732-0009/4 is now RUNNING
14/09/23 14:47:34 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkexecu...@votlbdcd05.tms.toyota.com:34924/user/Executor#-195649138]
with ID 2
14/09/23 14:47:34 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkexecu...@votlbdcd04.tms.toyota.com:22963/user/Executor#-2105826271]
with ID 6
14/09/23 14:47:34 INFO BlockManagerMasterActor: Registering block manager
votlbdcd05.tms.toyota.com:11677 with 12.9 GB RAM
14/09/23 14:47:34 INFO BlockManagerMasterActor: Registering block manager
votlbdcd04.tms.toyota.com:14746 with 12.9 GB RAM
14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkexecu...@votlbdcd03.tms.toyota.com:19463/user/Executor#964509447]
with ID 0
14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkexecu...@votlbdcd02.tms.toyota.com:47003/user/Executor#461654156]
with ID 1
14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkexecu...@votlbdcd07.tms.toyota.com:27089/user/Executor#-397798004]
with ID 5
14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkexecu...@votlbdcd08.tms.toyota.com:22617/user/Executor#-449311685]
with ID 4
14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkexecu...@votlbdcd06.tms.toyota.com:43167/user/Executor#-1493582134]
with ID 3
14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager
votlbdcd03.tms.toyota.com:13456 with 12.9 GB RAM
14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager
votlbdcd02.tms.toyota.com:33808 with 12.9 GB RAM
14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager
votlbdcd07.tms.toyota.com:34519 with 12.9 GB RAM
14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager
votlbdcd08.tms.toyota.com:14214 with 12.9 GB RAM
14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager
votlbdcd06.tms.toyota.com:45670 with 12.9 GB RAM
