Thanks for the reply Akhil I saw the logs in /tmp/mesos and found that my tar.gz was not properly created. I corrected that but now got another error which i can't find an answer for on google.
The error is pretty much the same "org.apache.spark.SparkException: Job aborted: Task 0.0:6 failed 4 times (most recent failure: unknown)" The logs(stderr) on the worker says: Warning: $HADOOP_HOME is deprecated. 14/06/04 19:55:26 INFO MesosExecutorBackend: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 14/06/04 19:55:26 INFO MesosExecutorBackend: Registered with Mesos as executor ID 201406041941-3213994176-5050-21843-0 14/06/04 19:55:26 INFO Executor: Using REPL class URI: http://192.168.145.191:48052 14/06/04 19:55:27 INFO Slf4jLogger: Slf4jLogger started 14/06/04 19:55:27 INFO Remoting: Starting remoting 14/06/04 19:55:27 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sp...@host-dsrv04.host.co.in:34024] 14/06/04 19:55:27 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sp...@host-dsrv04.host.co.in:34024] 14/06/04 19:55:27 INFO SparkEnv: Connecting to BlockManagerMaster: akka.tcp://sp...@host-dsrv01.host.co.in:52146/user/BlockManagerMaster 14/06/04 19:55:27 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140604195527-72b8 14/06/04 19:55:27 INFO MemoryStore: MemoryStore started with capacity 294.9 MB. 14/06/04 19:55:27 INFO ConnectionManager: Bound socket to port 35687 with id = ConnectionManagerId(host-DSRV04.host.co.in,35687) 14/06/04 19:55:27 INFO BlockManagerMaster: Trying to register BlockManager 14/06/04 19:55:27 INFO BlockManagerMaster: Registered BlockManager 14/06/04 19:55:27 INFO SparkEnv: Connecting to MapOutputTracker: akka.tcp://sp...@host-dsrv01.host.co.in:52146/user/MapOutputTracker 14/06/04 19:55:27 INFO HttpFileServer: HTTP File server directory is /tmp/spark-45838575-980e-44dc-8d88-5acee7bb9981 14/06/04 19:55:27 INFO HttpServer: Starting HTTP Server 14/06/04 19:55:27 ERROR Executor: Uncaught exception in thread Thread[Executor task launch worker-7,5,main] java.lang.Error: java.io.IOException: failure to login at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1151) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.IOException: failure to login at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:452) at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:40) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ... 2 more Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class: org/apache/hadoop/security/UserGroupInformation$HadoopLoginModule at javax.security.auth.login.LoginContext.invoke(LoginContext.java:822) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203) at javax.security.auth.login.LoginContext$5.run(LoginContext.java:721) at javax.security.auth.login.LoginContext$5.run(LoginContext.java:719) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:718) at javax.security.auth.login.LoginContext.login(LoginContext.java:590) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:471) ... 6 more 14/06/04 19:55:27 ERROR Executor: Uncaught exception in thread Thread[Executor task launch worker-6,5,main] java.lang.Error: java.io.IOException: failure to login at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1151) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.IOException: failure to login at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:452) at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:40) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ... 2 more Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class: org/apache/hadoop/security/UserGroupInformation$HadoopLoginModule at javax.security.auth.login.LoginContext.invoke(LoginContext.java:822) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203) at javax.security.auth.login.LoginContext$5.run(LoginContext.java:721) at javax.security.auth.login.LoginContext$5.run(LoginContext.java:719) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:718) at javax.security.auth.login.LoginContext.login(LoginContext.java:590) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:471) ... 6 more 14/06/04 19:55:27 ERROR Executor: Uncaught exception in thread Thread[Executor task launch worker-0,5,main] java.lang.Error: java.io.IOException: failure to login at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1151) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.IOException: failure to login at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:452) at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:40) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ... 2 more Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class: org/apache/hadoop/security/UserGroupInformation$HadoopLoginModule at javax.security.auth.login.LoginContext.invoke(LoginContext.java:822) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203) at javax.security.auth.login.LoginContext$5.run(LoginContext.java:721) at javax.security.auth.login.LoginContext$5.run(LoginContext.java:719) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:718) at javax.security.auth.login.LoginContext.login(LoginContext.java:590) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:471) ... 6 more 14/06/04 19:55:27 ERROR Executor: Uncaught exception in thread Thread[Executor task launch worker-2,5,main] java.lang.Error: java.io.IOException: failure to login at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1151) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.IOException: failure to login at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:452) at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:40) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ... 2 more Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class: org/apache/hadoop/security/UserGroupInformation$HadoopLoginModule at javax.security.auth.login.LoginContext.invoke(LoginContext.java:822) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203) at javax.security.auth.login.LoginContext$5.run(LoginContext.java:721) at javax.security.auth.login.LoginContext$5.run(LoginContext.java:719) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:718) at javax.security.auth.login.LoginContext.login(LoginContext.java:590) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:471) ... 6 more If you can suggest what is causing this error that would be great. -- Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-not-working-with-mesos-tp6806p6927.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
