Re: Spark on Mesos - Shut down failed while running spark-shell
Hi Haripriya, Your master has registered it's public ip to be 127.0.0.1:5050 which won't be able to be reached by the slave node. If mesos didn't pick up the right ip you can specifiy one yourself via the --ip flag. Tim On Mon, Jul 27, 2015 at 8:32 PM, Haripriya Ayyalasomayajula aharipriy...@gmail.com wrote: Hi all, I am running Spark 1.4.1 on mesos 0.23.0 While I am able to start spark-shell on the node with mesos-master running, it works fine. But when I try to start spark-shell on mesos-slave nodes, I'm encounter this error. I greatly appreciate any help. 15/07/27 22:14:44 INFO Utils: Successfully started service 'SparkUI' on port 4040. 15/07/27 22:14:44 INFO SparkUI: Started SparkUI at http://10.142.0.140:4040 Warning: MESOS_NATIVE_LIBRARY is deprecated, use MESOS_NATIVE_JAVA_LIBRARY instead. Future releases will not support JNI bindings via MESOS_NATIVE_LIBRARY. Warning: MESOS_NATIVE_LIBRARY is deprecated, use MESOS_NATIVE_JAVA_LIBRARY instead. Future releases will not support JNI bindings via MESOS_NATIVE_LIBRARY. WARNING: Logging before InitGoogleLogging() is written to STDERR W0727 22:14:45.091286 33441 sched.cpp:1326] ** Scheduler driver bound to loopback interface! Cannot communicate with remote master(s). You might want to set 'LIBPROCESS_IP' environment variable to use a routable IP address. ** 2015-07-27 22:14:45,091:33222(0x7fff9e1fc700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2015-07-27 22:14:45,091:33222(0x7fff9e1fc700):ZOO_INFO@log_env@716: Client environment:host.name=nid00011 I0727 22:14:45.091995 33441 sched.cpp:157] Version: 0.23.0 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@724: Client environment:os.arch=2.6.32-431.el6_1..8785-cray_ari_athena_c_cos 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@725: Client environment:os.version=#1 SMP Wed Jun 24 19:34:50 UTC 2015 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@733: Client environment:user.name=root 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@741: Client environment:user.home=/root 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@753: Client environment:user.dir=/opt/spark-1.4.1/spark-source 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=192.168.0.10:2181 sessionTimeout=1 watcher=0x7fffb561a8e0 sessionId=0 sessionPasswd=nullcontext=0x7ffdd930 flags=0 2015-07-27 22:14:45,092:33222(0x7fff6ebfd700):ZOO_INFO@check_events@1703: initiated connection to server [192.168.0.10:2181] 2015-07-27 22:14:45,096:33222(0x7fff6ebfd700):ZOO_INFO@check_events@1750: session establishment complete on server [192.168.0.10:2181], sessionId=0x14ed296a0fd000a, negotiated timeout=1 I0727 22:14:45.096891 33479 group.cpp:313] Group process (group(1)@ 127.0.0.1:45546) connected to ZooKeeper I0727 22:14:45.096914 33479 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0727 22:14:45.096923 33479 group.cpp:385] Trying to create path '/mesos' in ZooKeeper I0727 22:14:45.099181 33471 detector.cpp:138] Detected a new leader: (id='4') I0727 22:14:45.099298 33483 group.cpp:656] Trying to get '/mesos/info_04' in ZooKeeper W0727 22:14:45.100443 33453 detector.cpp:444] Leading master master@127.0.0.1:5050 is using a Protobuf binary format when registering with ZooKeeper (info): this will be deprecated as of Mesos 0.24 (see MESOS-2340) I0727 22:14:45.100544 33453 detector.cpp:481] A new leading master (UPID= master@127.0.0.1:5050) is detected I0727 22:14:45.100739 33478 sched.cpp:254] New master detected at master@127.0.0.1:5050 I0727 22:14:45.101104 33478 sched.cpp:264] No credentials provided. Attempting to register without authentication E0727 22:14:45.101210 33490 socket.hpp:107] Shutdown failed on fd=88: Transport endpoint is not connected [107] E0727 22:14:45.101380 33490 socket.hpp:107] Shutdown failed on fd=89: Transport endpoint is not connected [107] E0727 22:14:46.643348 33490 socket.hpp:107] Shutdown failed on fd=88: Transport endpoint is not connected [107] E0727 22:14:47.111336 33490 socket.hpp:107] Shutdown failed on fd=88: Transport endpoint is not connected [107] 15/07/27 22:14:50 INFO DiskBlockManager: Shutdown hook called 15/07/27 22:14:50 INFO Utils: path = /tmp/spark-3f94442b-7873-463f-91dd-3ee62ed5b263/blockmgr-74a5ed25-025b-4186-b1d8-dc395f287a8f, already present as root for deletion. 15/07/27 22:14:50 INFO Utils: Shutdown hook called 15/07/27 22:14:50 INFO Utils: Deleting directory /tmp/spark-3f94442b-7873-463f-91dd-3ee62ed5b263/httpd-5d2a71e5-1d36-47f7-b122-31f1dd12a0f0
Spark on Mesos - Shut down failed while running spark-shell
Hi all, I am running Spark 1.4.1 on mesos 0.23.0 While I am able to start spark-shell on the node with mesos-master running, it works fine. But when I try to start spark-shell on mesos-slave nodes, I'm encounter this error. I greatly appreciate any help. 15/07/27 22:14:44 INFO Utils: Successfully started service 'SparkUI' on port 4040. 15/07/27 22:14:44 INFO SparkUI: Started SparkUI at http://10.142.0.140:4040 Warning: MESOS_NATIVE_LIBRARY is deprecated, use MESOS_NATIVE_JAVA_LIBRARY instead. Future releases will not support JNI bindings via MESOS_NATIVE_LIBRARY. Warning: MESOS_NATIVE_LIBRARY is deprecated, use MESOS_NATIVE_JAVA_LIBRARY instead. Future releases will not support JNI bindings via MESOS_NATIVE_LIBRARY. WARNING: Logging before InitGoogleLogging() is written to STDERR W0727 22:14:45.091286 33441 sched.cpp:1326] ** Scheduler driver bound to loopback interface! Cannot communicate with remote master(s). You might want to set 'LIBPROCESS_IP' environment variable to use a routable IP address. ** 2015-07-27 22:14:45,091:33222(0x7fff9e1fc700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2015-07-27 22:14:45,091:33222(0x7fff9e1fc700):ZOO_INFO@log_env@716: Client environment:host.name=nid00011 I0727 22:14:45.091995 33441 sched.cpp:157] Version: 0.23.0 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@724: Client environment:os.arch=2.6.32-431.el6_1..8785-cray_ari_athena_c_cos 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@725: Client environment:os.version=#1 SMP Wed Jun 24 19:34:50 UTC 2015 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@733: Client environment:user.name=root 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@741: Client environment:user.home=/root 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@753: Client environment:user.dir=/opt/spark-1.4.1/spark-source 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=192.168.0.10:2181 sessionTimeout=1 watcher=0x7fffb561a8e0 sessionId=0 sessionPasswd=nullcontext=0x7ffdd930 flags=0 2015-07-27 22:14:45,092:33222(0x7fff6ebfd700):ZOO_INFO@check_events@1703: initiated connection to server [192.168.0.10:2181] 2015-07-27 22:14:45,096:33222(0x7fff6ebfd700):ZOO_INFO@check_events@1750: session establishment complete on server [192.168.0.10:2181], sessionId=0x14ed296a0fd000a, negotiated timeout=1 I0727 22:14:45.096891 33479 group.cpp:313] Group process (group(1)@ 127.0.0.1:45546) connected to ZooKeeper I0727 22:14:45.096914 33479 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0727 22:14:45.096923 33479 group.cpp:385] Trying to create path '/mesos' in ZooKeeper I0727 22:14:45.099181 33471 detector.cpp:138] Detected a new leader: (id='4') I0727 22:14:45.099298 33483 group.cpp:656] Trying to get '/mesos/info_04' in ZooKeeper W0727 22:14:45.100443 33453 detector.cpp:444] Leading master master@127.0.0.1:5050 is using a Protobuf binary format when registering with ZooKeeper (info): this will be deprecated as of Mesos 0.24 (see MESOS-2340) I0727 22:14:45.100544 33453 detector.cpp:481] A new leading master (UPID= master@127.0.0.1:5050) is detected I0727 22:14:45.100739 33478 sched.cpp:254] New master detected at master@127.0.0.1:5050 I0727 22:14:45.101104 33478 sched.cpp:264] No credentials provided. Attempting to register without authentication E0727 22:14:45.101210 33490 socket.hpp:107] Shutdown failed on fd=88: Transport endpoint is not connected [107] E0727 22:14:45.101380 33490 socket.hpp:107] Shutdown failed on fd=89: Transport endpoint is not connected [107] E0727 22:14:46.643348 33490 socket.hpp:107] Shutdown failed on fd=88: Transport endpoint is not connected [107] E0727 22:14:47.111336 33490 socket.hpp:107] Shutdown failed on fd=88: Transport endpoint is not connected [107] 15/07/27 22:14:50 INFO DiskBlockManager: Shutdown hook called 15/07/27 22:14:50 INFO Utils: path = /tmp/spark-3f94442b-7873-463f-91dd-3ee62ed5b263/blockmgr-74a5ed25-025b-4186-b1d8-dc395f287a8f, already present as root for deletion. 15/07/27 22:14:50 INFO Utils: Shutdown hook called 15/07/27 22:14:50 INFO Utils: Deleting directory /tmp/spark-3f94442b-7873-463f-91dd-3ee62ed5b263/httpd-5d2a71e5-1d36-47f7-b122-31f1dd12a0f0 15/07/27 22:14:50 INFO Utils: Deleting directory /tmp/spark-3f94442b-7873-463f-91dd-3ee62ed5b263 15/07/27 22:14:50 INFO Utils: Deleting directory /tmp/spark-bfd6c444-5346-4315-9501-1baed4d500de -- Regards, Haripriya Ayyalasomayajula
