Thank you all for your fast and extremely helpful replies! Brians question concerning a network segment brought me to the idea to check my hosts-definition. It was of course false, so changing my /etc/hosts – file to use my actual IP instead of 127.0.1.1 instantly solved this problem.
I hope this helps others with the same issue! Thanks again! Joahnes From: Tim St Clair [mailto:[email protected]] Sent: Mittwoch, 15. Oktober 2014 19:17 To: [email protected] Subject: Re: Connecting spark from a different Machine to mesos cluster Just as a general note, especially when dealing with VM's, NTP is *required* otherwise we've found you can get out of snyc'd updates which can result in some pretty strange behavior. ________________________________ From: "Brian Devins" <[email protected]<mailto:[email protected]>> To: [email protected]<mailto:[email protected]> Sent: Wednesday, October 15, 2014 11:57:19 AM Subject: Re: Connecting spark from a different Machine to mesos cluster Also Johannes, is there a network segment between Spark and the Mesos master? This looks like behavior I have seen before when the Master cannot connect back to the framework. The master also needs to be able to reach the Spark machine by IP From: Tim Chen <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Wednesday, October 15, 2014 at 12:52 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Connecting spark from a different Machine to mesos cluster Hi Johannes, When you started your 2nd shell, what log output from the slave do you see for that framework? Master seems to think it's already terminated. Tim On Wed, Oct 15, 2014 at 6:31 AM, Johannes Schillinger (Intern) <[email protected]<mailto:[email protected]>> wrote: Hi Tim, We are running Spark 1.1.0 with Hadoop 2.4. Mesos is in Version 0.20.1 all in binary releases. The Spark console is running in default mode, which is fine grained. The Spark process is started from a physical Machine running Ubuntu, the Mesos nodes are running in VMs also in Ubuntu. This is the output of the Spark Shell: -------------------------------------------------------------------------------------------------------------------------------- Spark assembly has been built with Hive, including Datanucleus jars on classpath Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 14/10/15 15:18:24 INFO SecurityManager: Changing view acls to: USERNAME, 14/10/15 15:18:24 INFO SecurityManager: Changing modify acls to: USERNAME, 14/10/15 15:18:24 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(USERNAME, ); users with modify permissions: Set(USERNAME, ) 14/10/15 15:18:24 INFO HttpServer: Starting HTTP Server 14/10/15 15:18:24 INFO Utils: Successfully started service 'HTTP class server' on port 42469. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.1.0 /_/ Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_65) Type in expressions to have them evaluated. Type :help for more information. 14/10/15 15:18:26 WARN Utils: Your hostname, karwjohannes01 resolves to a loopback address: 127.0.1.1; using CLIENT_IP instead (on interface eth0) 14/10/15 15:18:26 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 14/10/15 15:18:27 INFO SecurityManager: Changing view acls to: USERNAME, 14/10/15 15:18:27 INFO SecurityManager: Changing modify acls to: USERNAME, 14/10/15 15:18:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(USERNAME, ); users with modify permissions: Set(USERNAME, ) 14/10/15 15:18:27 INFO Slf4jLogger: Slf4jLogger started 14/10/15 15:18:27 INFO Remoting: Starting remoting 14/10/15 15:18:27 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@CLIENT_IP:51879] 14/10/15 15:18:27 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@CLIENT_IP:51879] 14/10/15 15:18:27 INFO Utils: Successfully started service 'sparkDriver' on port 51879. 14/10/15 15:18:27 INFO SparkEnv: Registering MapOutputTracker 14/10/15 15:18:27 INFO SparkEnv: Registering BlockManagerMaster 14/10/15 15:18:27 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20141015151827-1a2e 14/10/15 15:18:27 INFO Utils: Successfully started service 'Connection manager for block manager' on port 60963. 14/10/15 15:18:27 INFO ConnectionManager: Bound socket to port 60963 with id = ConnectionManagerId(CLIENT_IP,60963) 14/10/15 15:18:27 INFO MemoryStore: MemoryStore started with capacity 265.4 MB 14/10/15 15:18:27 INFO BlockManagerMaster: Trying to register BlockManager 14/10/15 15:18:27 INFO BlockManagerMasterActor: Registering block manager CLIENT_IP:60963 with 265.4 MB RAM 14/10/15 15:18:27 INFO BlockManagerMaster: Registered BlockManager 14/10/15 15:18:27 INFO HttpFileServer: HTTP File server directory is /tmp/spark-b032c76c-93e1-473e-802c-c55e12e85d41 14/10/15 15:18:27 INFO HttpServer: Starting HTTP Server 14/10/15 15:18:27 INFO Utils: Successfully started service 'HTTP file server' on port 47989. 14/10/15 15:18:27 INFO Utils: Successfully started service 'SparkUI' on port 4040. 14/10/15 15:18:27 INFO SparkUI: Started SparkUI at http://CLIENT_IP:4040 14/10/15 15:18:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable I1015 15:18:28.524736 4748 sched.cpp:139] Version: 0.20.1 I1015 15:18:28.527180 4750 sched.cpp:235] New master detected at master@MESOS_MASTER_IP:5050 I1015 15:18:28.527300 4750 sched.cpp:243] No credentials provided. Attempting to register without authentication -------------------------------------------------------------------------------------------------------------------------------- Mesos master WARNING log: W1015 14:13:00.235213 1118 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3490 because the framework has terminated or is inactive W1015 14:13:35.244055 1121 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3525 because the framework has terminated or is inactive W1015 14:13:50.252436 1121 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3540 because the framework has terminated or is inactive W1015 14:14:05.252708 1117 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3555 because the framework has terminated or is inactive Mesos slave WARNING log : W1015 13:58:19.103196 1211 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3116 W1015 13:58:20.104650 1210 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3117 W1015 13:58:21.119839 1211 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3118 W1015 13:58:22.115965 1210 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3119 W1015 13:58:23.104925 1211 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3120 W1015 13:58:24.104652 1210 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3121 W1015 13:58:59.853744 1212 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3122 W1015 13:59:00.853086 1214 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3123 W1015 13:59:01.853137 1212 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3124 W1015 13:59:03.318259 1214 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3029 I hope this information helps, please ask if you have any more questions and thank you for your help! Johannes From: Tim St Clair [mailto:[email protected]<mailto:[email protected]>] Sent: Mittwoch, 15. Oktober 2014 15:11 To: [email protected]<mailto:[email protected]> Subject: Re: Connecting spark from a different Machine to mesos cluster Details? 1. What versions are you running? 2. Fine grained mode or Course Gained? 3. Are you running in VM's? Logs always help too. Cheers, Tim ________________________________ From: "Johannes Schillinger (Intern)" <[email protected]<mailto:[email protected]>> To: [email protected]<mailto:[email protected]> Sent: Wednesday, October 15, 2014 7:42:36 AM Subject: Connecting spark from a different Machine to mesos cluster Hi, we are currently trying to get a mesos cluster running as a base for Spark. The mesos cluster itself runs and connecting a spark shell from the machine the maser runs on works perfectly. We can see the Framework being started and the slaves working. If we try to connect the exact same shell from a different machine to the exact same cluster the screen stops at … 4013 sched.cpp:243] No credentials provided. Attempting to register without authentication The cluster spins up a framework every two seconds with a new ID and stops it immediately. This continues (we stopped it after a few dozen starts). We can see the frameworks being started in the master- and slave-logs as well as the command of the master to terminate it. Has anyone ever encountered a similar problem or has any advice on solving this problem? Thanks! Johannes -- Cheers, Timothy St. Clair Red Hat Inc. Brian Devins | Java Developer [email protected]<mailto:[email protected]> [Dealer.com] -- Cheers, Timothy St. Clair Red Hat Inc.
