Hi Paul, There are several ports you need to configure in order to run in a tight network environment. It sounds like you the DMZ that contains the spark cluster is wide open internally, but you have to poke holes between that and the driver.
You should take a look at the port configuration options at the bottom of http://spark.apache.org/docs/latest/security.html for the ones you'll need to set to make this configuration work. The ones that are to or from the driver are the ones you should configure. The other thing to consider is that some connections are from the workers back to the driver (not all connections are initiated by the driver). Does your configuration allow outbound connections from the DMZ -> edge server? Good luck! Andrew On Tue, Sep 23, 2014 at 3:10 PM, Paul Magid <paul_ma...@toyota.com> wrote: > I am trying to get an edge server up and running connecting to our Spark > 1.1 cluster. The edge server is in a different DMZ than the rest of the > cluster and we have to specifically open firewall ports between the edge > server and the rest of the cluster. I can log on to any node in the > cluster (other than the edge) and submit code through spark-shell just > fine. I have port 7077 from the edge to the master open (verified), and I > have port 7078 open from the edge to all the workers (also verified). I > have tried setting the worker port to not be dynamic by using > SPARK_WORKER_PORT in the spark-env.sh but it does not seem to stop the > dynamic port behavior. I have included the startup output when running > spark-shell from the edge server in a different dmz and then from a node in > the cluster. Any help greatly appreciated. > > > > Paul Magid > > Toyota Motor Sales IS Enterprise Architecture (EA) > > Architect I R&D > > Ph: 310-468-9091 (X69091) > > PCN 1C2970, Mail Drop PN12 > > > > > > Running spark-shell from the edge server > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > 14/09/23 14:20:38 INFO SecurityManager: Changing view acls to: root, > > 14/09/23 14:20:38 INFO SecurityManager: Changing modify acls to: root, > > 14/09/23 14:20:38 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(root, ); users > with modify permissions: Set(root, ) > > 14/09/23 14:20:38 INFO HttpServer: Starting HTTP Server > > 14/09/23 14:20:39 INFO Utils: Successfully started service 'HTTP class > server' on port 22788. > > Welcome to > > ____ __ > > / __/__ ___ _____/ /__ > > _\ \/ _ \/ _ `/ __/ '_/ > > /___/ .__/\_,_/_/ /_/\_\ version 1.1.0 > > /_/ > > > > Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java > 1.7.0_55) > > Type in expressions to have them evaluated. > > Type :help for more information. > > 14/09/23 14:20:42 INFO SecurityManager: Changing view acls to: root, > > 14/09/23 14:20:42 INFO SecurityManager: Changing modify acls to: root, > > 14/09/23 14:20:42 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(root, ); users > with modify permissions: Set(root, ) > > 14/09/23 14:20:43 INFO Slf4jLogger: Slf4jLogger started > > 14/09/23 14:20:43 INFO Remoting: Starting remoting > > 14/09/23 14:20:43 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkdri...@votlbdcd09.tms.toyota.com:32356] > > 14/09/23 14:20:43 INFO Remoting: Remoting now listens on addresses: > [akka.tcp://sparkdri...@votlbdcd09.tms.toyota.com:32356] > > 14/09/23 14:20:43 INFO Utils: Successfully started service 'sparkDriver' > on port 32356. > > 14/09/23 14:20:43 INFO SparkEnv: Registering MapOutputTracker > > 14/09/23 14:20:43 INFO SparkEnv: Registering BlockManagerMaster > > 14/09/23 14:20:43 INFO DiskBlockManager: Created local directory at > /tmp/spark-local-20140923142043-4454 > > 14/09/23 14:20:43 INFO Utils: Successfully started service 'Connection > manager for block manager' on port 48469. > > 14/09/23 14:20:43 INFO ConnectionManager: Bound socket to port 48469 with > id = ConnectionManagerId(votlbdcd09.tms.toyota.com,48469) > > 14/09/23 14:20:43 INFO MemoryStore: MemoryStore started with capacity > 265.9 MB > > 14/09/23 14:20:43 INFO BlockManagerMaster: Trying to register BlockManager > > 14/09/23 14:20:43 INFO BlockManagerMasterActor: Registering block manager > votlbdcd09.tms.toyota.com:48469 with 265.9 MB RAM > > 14/09/23 14:20:43 INFO BlockManagerMaster: Registered BlockManager > > 14/09/23 14:20:43 INFO HttpFileServer: HTTP File server directory is > /tmp/spark-888c359a-5a2a-4aaa-80e3-8009cdfa25c8 > > 14/09/23 14:20:43 INFO HttpServer: Starting HTTP Server > > 14/09/23 14:20:43 INFO Utils: Successfully started service 'HTTP file > server' on port 12470. > > 14/09/23 14:20:43 INFO Utils: Successfully started service 'SparkUI' on > port 4040. > > 14/09/23 14:20:43 INFO SparkUI: Started SparkUI at > http://votlbdcd09.tms.toyota.com:4040 > > 14/09/23 14:20:43 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > > 14/09/23 14:20:44 INFO EventLoggingListener: Logging events to > file:/user/spark/applicationHistory//spark-shell-1411507243973 > > 14/09/23 14:20:44 INFO AppClient$ClientActor: Connecting to master > spark://votlbdcd01.tms.toyota.com:7077... > > 14/09/23 14:20:44 INFO SparkDeploySchedulerBackend: SchedulerBackend is > ready for scheduling beginning after reached minRegisteredResourcesRatio: > 0.0 > > 14/09/23 14:20:44 INFO SparkILoop: Created spark context.. > > 14/09/23 14:20:44 INFO SparkDeploySchedulerBackend: Connected to Spark > cluster with app ID app-20140923142044-0006 > > Spark context available as sc. > > > > scala> 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/0 on > worker-20140923084845-votlbdcd03.tms.toyota.com-7078 ( > votlbdcd03.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/0 on hostPort votlbdcd03.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/1 on > worker-20140923141811-votlbdcd08.tms.toyota.com-7078 ( > votlbdcd08.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/1 on hostPort votlbdcd08.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/2 on > worker-20140923084651-votlbdcd02.tms.toyota.com-7078 ( > votlbdcd02.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/2 on hostPort votlbdcd02.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/3 on > worker-20140923091151-votlbdcd05.tms.toyota.com-7078 ( > votlbdcd05.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/3 on hostPort votlbdcd05.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/4 on > worker-20140923103250-votlbdcd06.tms.toyota.com-7078 ( > votlbdcd06.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/4 on hostPort votlbdcd06.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/5 on > worker-20140923104019-votlbdcd07.tms.toyota.com-7078 ( > votlbdcd07.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/5 on hostPort votlbdcd07.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/6 on > worker-20140923085411-votlbdcd04.tms.toyota.com-7078 ( > votlbdcd04.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:21:26 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/6 on hostPort votlbdcd04.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/3 is now RUNNING > > 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/6 is now RUNNING > > 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/0 is now RUNNING > > 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/2 is now RUNNING > > 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/4 is now RUNNING > > 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/5 is now RUNNING > > 14/09/23 14:21:26 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/1 is now RUNNING > > > > > > scala> 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/3 is now EXITED (Command exited with code 1) > > 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor > app-20140923142044-0006/3 removed: Command exited with code 1 > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/7 on > worker-20140923091151-votlbdcd05.tms.toyota.com-7078 ( > votlbdcd05.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/7 on hostPort votlbdcd05.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/7 is now RUNNING > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/6 is now EXITED (Command exited with code 1) > > 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor > app-20140923142044-0006/6 removed: Command exited with code 1 > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/8 on > worker-20140923085411-votlbdcd04.tms.toyota.com-7078 ( > votlbdcd04.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/8 on hostPort votlbdcd04.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/8 is now RUNNING > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/0 is now EXITED (Command exited with code 1) > > 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor > app-20140923142044-0006/0 removed: Command exited with code 1 > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/9 on > worker-20140923084845-votlbdcd03.tms.toyota.com-7078 ( > votlbdcd03.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/9 on hostPort votlbdcd03.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/9 is now RUNNING > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/2 is now EXITED (Command exited with code 1) > > 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor > app-20140923142044-0006/2 removed: Command exited with code 1 > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/10 on > worker-20140923084651-votlbdcd02.tms.toyota.com-7078 ( > votlbdcd02.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/10 on hostPort votlbdcd02.tms.toyota.com:7078 > with 8 cores, 25.0 GB RAM > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/10 is now RUNNING > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/1 is now EXITED (Command exited with code 1) > > 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor > app-20140923142044-0006/1 removed: Command exited with code 1 > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/11 on > worker-20140923141811-votlbdcd08.tms.toyota.com-7078 ( > votlbdcd08.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/11 on hostPort votlbdcd08.tms.toyota.com:7078 > with 8 cores, 25.0 GB RAM > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/11 is now RUNNING > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/4 is now EXITED (Command exited with code 1) > > 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor > app-20140923142044-0006/4 removed: Command exited with code 1 > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/12 on > worker-20140923103250-votlbdcd06.tms.toyota.com-7078 ( > votlbdcd06.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/12 on hostPort votlbdcd06.tms.toyota.com:7078 > with 8 cores, 25.0 GB RAM > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/12 is now RUNNING > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/5 is now EXITED (Command exited with code 1) > > 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Executor > app-20140923142044-0006/5 removed: Command exited with code 1 > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/13 on > worker-20140923104019-votlbdcd07.tms.toyota.com-7078 ( > votlbdcd07.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:21:58 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/13 on hostPort votlbdcd07.tms.toyota.com:7078 > with 8 cores, 25.0 GB RAM > > 14/09/23 14:21:58 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/13 is now RUNNING > > 14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/7 is now EXITED (Command exited with code 1) > > 14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Executor > app-20140923142044-0006/7 removed: Command exited with code 1 > > 14/09/23 14:22:30 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/14 on > worker-20140923091151-votlbdcd05.tms.toyota.com-7078 ( > votlbdcd05.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/14 on hostPort votlbdcd05.tms.toyota.com:7078 > with 8 cores, 25.0 GB RAM > > 14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/14 is now RUNNING > > 14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/8 is now EXITED (Command exited with code 1) > > 14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Executor > app-20140923142044-0006/8 removed: Command exited with code 1 > > 14/09/23 14:22:30 INFO AppClient$ClientActor: Executor added: > app-20140923142044-0006/15 on > worker-20140923085411-votlbdcd04.tms.toyota.com-7078 ( > votlbdcd04.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:22:30 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923142044-0006/15 on hostPort votlbdcd04.tms.toyota.com:7078 > with 8 cores, 25.0 GB RAM > > 14/09/23 14:22:30 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/15 is now RUNNING > > 14/09/23 14:22:31 INFO AppClient$ClientActor: Executor updated: > app-20140923142044-0006/9 is now EXITED (Command exited with code 1) > > 14/09/23 14:22:31 INFO SparkDeploySchedulerBackend: Executor > app-20140923142044-0006/9 removed: Command exited with code 1 > > 14/09/23 14:22:31 ERROR SparkDeploySchedulerBackend: Application has been > killed. Reason: Master removed our application: FAILED > > 14/09/23 14:22:31 ERROR TaskSchedulerImpl: Exiting due to error from > cluster scheduler: Master removed our application: FAILED > > > > > > > > > > > > Running spark-shell in the cluster > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > 14/09/23 14:47:26 INFO SecurityManager: Changing view acls to: root, > > 14/09/23 14:47:26 INFO SecurityManager: Changing modify acls to: root, > > 14/09/23 14:47:26 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(root, ); users > with modify permissions: Set(root, ) > > 14/09/23 14:47:26 INFO HttpServer: Starting HTTP Server > > 14/09/23 14:47:26 INFO Utils: Successfully started service 'HTTP class > server' on port 23968. > > Welcome to > > ____ __ > > / __/__ ___ _____/ /__ > > _\ \/ _ \/ _ `/ __/ '_/ > > /___/ .__/\_,_/_/ /_/\_\ version 1.1.0 > > /_/ > > > > Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java > 1.7.0_55) > > Type in expressions to have them evaluated. > > Type :help for more information. > > 14/09/23 14:47:30 INFO SecurityManager: Changing view acls to: root, > > 14/09/23 14:47:30 INFO SecurityManager: Changing modify acls to: root, > > 14/09/23 14:47:30 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(root, ); users > with modify permissions: Set(root, ) > > 14/09/23 14:47:31 INFO Slf4jLogger: Slf4jLogger started > > 14/09/23 14:47:31 INFO Remoting: Starting remoting > > 14/09/23 14:47:31 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkdri...@votlbdcd01.tms.toyota.com:44430] > > 14/09/23 14:47:31 INFO Remoting: Remoting now listens on addresses: > [akka.tcp://sparkdri...@votlbdcd01.tms.toyota.com:44430] > > 14/09/23 14:47:31 INFO Utils: Successfully started service 'sparkDriver' > on port 44430. > > 14/09/23 14:47:31 INFO SparkEnv: Registering MapOutputTracker > > 14/09/23 14:47:31 INFO SparkEnv: Registering BlockManagerMaster > > 14/09/23 14:47:31 INFO DiskBlockManager: Created local directory at > /tmp/spark-local-20140923144731-48c5 > > 14/09/23 14:47:31 INFO Utils: Successfully started service 'Connection > manager for block manager' on port 30579. > > 14/09/23 14:47:31 INFO ConnectionManager: Bound socket to port 30579 with > id = ConnectionManagerId(votlbdcd01.tms.toyota.com,30579) > > 14/09/23 14:47:31 INFO MemoryStore: MemoryStore started with capacity > 265.9 MB > > 14/09/23 14:47:31 INFO BlockManagerMaster: Trying to register BlockManager > > 14/09/23 14:47:31 INFO BlockManagerMasterActor: Registering block manager > votlbdcd01.tms.toyota.com:30579 with 265.9 MB RAM > > 14/09/23 14:47:31 INFO BlockManagerMaster: Registered BlockManager > > 14/09/23 14:47:31 INFO HttpFileServer: HTTP File server directory is > /tmp/spark-1a38cbd9-107b-40ce-bb28-f1574c9fbcda > > 14/09/23 14:47:31 INFO HttpServer: Starting HTTP Server > > 14/09/23 14:47:31 INFO Utils: Successfully started service 'HTTP file > server' on port 31028. > > 14/09/23 14:47:31 INFO Utils: Successfully started service 'SparkUI' on > port 4040. > > 14/09/23 14:47:31 INFO SparkUI: Started SparkUI at > http://votlbdcd01.tms.toyota.com:4040 > > 14/09/23 14:47:31 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > > 14/09/23 14:47:32 INFO EventLoggingListener: Logging events to > file:/user/spark/applicationHistory//spark-shell-1411508851947 > > 14/09/23 14:47:32 INFO AppClient$ClientActor: Connecting to master > spark://votlbdcd01.tms.toyota.com:7077... > > 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: SchedulerBackend is > ready for scheduling beginning after reached minRegisteredResourcesRatio: > 0.0 > > 14/09/23 14:47:32 INFO SparkILoop: Created spark context.. > > 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Connected to Spark > cluster with app ID app-20140923144732-0009 > > 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added: > app-20140923144732-0009/0 on > worker-20140923084845-votlbdcd03.tms.toyota.com-7078 ( > votlbdcd03.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923144732-0009/0 on hostPort votlbdcd03.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added: > app-20140923144732-0009/1 on > worker-20140923084651-votlbdcd02.tms.toyota.com-7078 ( > votlbdcd02.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923144732-0009/1 on hostPort votlbdcd02.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added: > app-20140923144732-0009/2 on > worker-20140923091151-votlbdcd05.tms.toyota.com-7078 ( > votlbdcd05.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923144732-0009/2 on hostPort votlbdcd05.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added: > app-20140923144732-0009/3 on > worker-20140923103250-votlbdcd06.tms.toyota.com-7078 ( > votlbdcd06.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923144732-0009/3 on hostPort votlbdcd06.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added: > app-20140923144732-0009/4 on > worker-20140923144121-votlbdcd08.tms.toyota.com-7078 ( > votlbdcd08.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923144732-0009/4 on hostPort votlbdcd08.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added: > app-20140923144732-0009/5 on > worker-20140923104019-votlbdcd07.tms.toyota.com-7078 ( > votlbdcd07.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923144732-0009/5 on hostPort votlbdcd07.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor added: > app-20140923144732-0009/6 on > worker-20140923085411-votlbdcd04.tms.toyota.com-7078 ( > votlbdcd04.tms.toyota.com:7078) with 8 cores > > 14/09/23 14:47:32 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140923144732-0009/6 on hostPort votlbdcd04.tms.toyota.com:7078 with > 8 cores, 25.0 GB RAM > > Spark context available as sc. > > > > scala> 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated: > app-20140923144732-0009/2 is now RUNNING > > 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated: > app-20140923144732-0009/6 is now RUNNING > > 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated: > app-20140923144732-0009/0 is now RUNNING > > 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated: > app-20140923144732-0009/1 is now RUNNING > > 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated: > app-20140923144732-0009/5 is now RUNNING > > 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated: > app-20140923144732-0009/3 is now RUNNING > > 14/09/23 14:47:32 INFO AppClient$ClientActor: Executor updated: > app-20140923144732-0009/4 is now RUNNING > > 14/09/23 14:47:34 INFO SparkDeploySchedulerBackend: Registered executor: > Actor[akka.tcp:// > sparkexecu...@votlbdcd05.tms.toyota.com:34924/user/Executor#-195649138] > with ID 2 > > 14/09/23 14:47:34 INFO SparkDeploySchedulerBackend: Registered executor: > Actor[akka.tcp:// > sparkexecu...@votlbdcd04.tms.toyota.com:22963/user/Executor#-2105826271] > with ID 6 > > 14/09/23 14:47:34 INFO BlockManagerMasterActor: Registering block manager > votlbdcd05.tms.toyota.com:11677 with 12.9 GB RAM > > 14/09/23 14:47:34 INFO BlockManagerMasterActor: Registering block manager > votlbdcd04.tms.toyota.com:14746 with 12.9 GB RAM > > 14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor: > Actor[akka.tcp:// > sparkexecu...@votlbdcd03.tms.toyota.com:19463/user/Executor#964509447] > with ID 0 > > 14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor: > Actor[akka.tcp:// > sparkexecu...@votlbdcd02.tms.toyota.com:47003/user/Executor#461654156] > with ID 1 > > 14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor: > Actor[akka.tcp:// > sparkexecu...@votlbdcd07.tms.toyota.com:27089/user/Executor#-397798004] > with ID 5 > > 14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor: > Actor[akka.tcp:// > sparkexecu...@votlbdcd08.tms.toyota.com:22617/user/Executor#-449311685] > with ID 4 > > 14/09/23 14:47:35 INFO SparkDeploySchedulerBackend: Registered executor: > Actor[akka.tcp:// > sparkexecu...@votlbdcd06.tms.toyota.com:43167/user/Executor#-1493582134] > with ID 3 > > 14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager > votlbdcd03.tms.toyota.com:13456 with 12.9 GB RAM > > 14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager > votlbdcd02.tms.toyota.com:33808 with 12.9 GB RAM > > 14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager > votlbdcd07.tms.toyota.com:34519 with 12.9 GB RAM > > 14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager > votlbdcd08.tms.toyota.com:14214 with 12.9 GB RAM > > 14/09/23 14:47:35 INFO BlockManagerMasterActor: Registering block manager > votlbdcd06.tms.toyota.com:45670 with 12.9 GB RAM >
