[jira] [Commented] (SPARK-6305) Add support for log4j 2.x to Spark
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326422#comment-16326422 ] Roque Vassal'lo commented on SPARK-6305: Hi, I would like to know current status of this ticket. Log4j 1 reached EOL two years ago, and current log4j 2 version (2.10) has compatibility with log4j 1 properties files (which I have read in this thread that it is a must). I have run some tests using log4j's bridge and slf4j, and it seems to work fine and it does not need properties files to be modified. Will this be a good approach to keep compatibility while getting new log4j 2 functionality and bug-fixing? > Add support for log4j 2.x to Spark > -- > > Key: SPARK-6305 > URL: https://issues.apache.org/jira/browse/SPARK-6305 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Tal Sliwowicz >Priority: Minor > > log4j 2 requires replacing the slf4j binding and adding the log4j jars in the > classpath. Since there are shaded jars, it must be done during the build. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5265) Submitting applications on Standalone cluster controlled by Zookeeper forces to know active master
[ https://issues.apache.org/jira/browse/SPARK-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547963#comment-14547963 ] Roque Vassal'lo commented on SPARK-5265: Yes, SPARK-6443 and this jira are the same. Submitting applications on Standalone cluster controlled by Zookeeper forces to know active master -- Key: SPARK-5265 URL: https://issues.apache.org/jira/browse/SPARK-5265 Project: Spark Issue Type: Bug Components: Deploy Reporter: Roque Vassal'lo Labels: cluster, spark-submit, standalone, zookeeper Hi, this is my first JIRA here, so I hope it is clear enough. I'm using Spark 1.2.0 and trying to submit an application on a Spark Standalone cluster in cluster deploy mode with supervise. Standalone cluster is running in high availability mode, using Zookeeper to provide leader election between three available Masters (named master1, master2 and master3). As read at Spark's documentation, to register a Worker to the Standalone cluster, I provide complete cluster info as the spark route. I mean, spark://master1:7077,master2:7077,master3:7077 and that route is parsed and three attempts are launched, first one to master1:7077, second one to master2:7077 and third one to master3:7077. This works great! But if I try to do the same while submitting applications, it fails. I mean, if I provide complete cluster info as the --master option to spark-submit script, it throws an exception because it tries to connect as it was a single node. Example: spark-submit --class org.apache.spark.examples.SparkPi --master spark://master1:7077,master2:7077,master3:7077 --deploy-mode cluster --supervise examples.jar 100 This is the output I got: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/01/14 17:02:11 INFO SecurityManager: Changing view acls to: mytest 15/01/14 17:02:11 INFO SecurityManager: Changing modify acls to: mytest 15/01/14 17:02:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mytest); users with modify permissions: Set(mytest) 15/01/14 17:02:11 INFO Slf4jLogger: Slf4jLogger started 15/01/14 17:02:11 INFO Utils: Successfully started service 'driverClient' on port 53930. 15/01/14 17:02:11 ERROR OneForOneStrategy: Invalid master URL: spark://master1:7077,master2:7077,master3:7077 akka.actor.ActorInitializationException: exception during creation at akka.actor.ActorInitializationException$.apply(Actor.scala:164) at akka.actor.ActorCell.create(ActorCell.scala:596) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: org.apache.spark.SparkException: Invalid master URL: spark://master1:7077,master2:7077,master3:7077 at org.apache.spark.deploy.master.Master$.toAkkaUrl(Master.scala:830) at org.apache.spark.deploy.ClientActor.preStart(Client.scala:42) at akka.actor.Actor$class.aroundPreStart(Actor.scala:470) at org.apache.spark.deploy.ClientActor.aroundPreStart(Client.scala:35) at akka.actor.ActorCell.create(ActorCell.scala:580) ... 9 more Shouldn't it parse it as on Worker registration? It will not force client to know which is the current active Master of the Standalone cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-5265) Submitting applications on Standalone cluster controlled by Zookeeper forces to know active master
[ https://issues.apache.org/jira/browse/SPARK-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547742#comment-14547742 ] Roque Vassal'lo edited comment on SPARK-5265 at 5/18/15 8:52 AM: - Hi there Sean, Sorry for the comment but they are not the same. SPARK-5497 is about Spark workers and how they act when active master goes down at a Standalone cluster (because Spark's scripts that are not working properly, at least at 1.2). When a Spark's master goes down, workers do not reassign properly to new master (as seen at Spark's WebUI, they don't appear as workers in new master's webUI). This jira, SPARK-5265, is about an external application submitting a job to Spark. That application needs to know which is the current active master of the standalone cluster because, currently, you can't submit a job setting a standalone cluster as master (/--master spark://master1:7077,master2:7077) while setting deploy-mode as cluster (--deploy-mode cluster). So I think it should be reopened. was (Author: zujorv): Hi there Sean, Sorry for the comment but they are not the same. SPARK-5497 is about Spark workers and how they act when active master goes down at a Standalone cluster (because Spark's scripts that are not working properly, at least at 1.2). When a Spark's master goes down, workers do not reassign properly to new master (as seen at Spark's WebUI, they don't appear as workers in new master's webUI). This jira, SPARK-5265, is about an external application submitting a job to Spark. That application needs to know which is the current active master of the standalone cluster because, currently, you can't submit a job setting a standalone cluster as master (--master spark://master1:7077,master2:7077) while setting deploy-mode as cluster (--deploy-mode cluster). So I think it should be reopened. Submitting applications on Standalone cluster controlled by Zookeeper forces to know active master -- Key: SPARK-5265 URL: https://issues.apache.org/jira/browse/SPARK-5265 Project: Spark Issue Type: Bug Components: Deploy Reporter: Roque Vassal'lo Labels: cluster, spark-submit, standalone, zookeeper Hi, this is my first JIRA here, so I hope it is clear enough. I'm using Spark 1.2.0 and trying to submit an application on a Spark Standalone cluster in cluster deploy mode with supervise. Standalone cluster is running in high availability mode, using Zookeeper to provide leader election between three available Masters (named master1, master2 and master3). As read at Spark's documentation, to register a Worker to the Standalone cluster, I provide complete cluster info as the spark route. I mean, spark://master1:7077,master2:7077,master3:7077 and that route is parsed and three attempts are launched, first one to master1:7077, second one to master2:7077 and third one to master3:7077. This works great! But if I try to do the same while submitting applications, it fails. I mean, if I provide complete cluster info as the --master option to spark-submit script, it throws an exception because it tries to connect as it was a single node. Example: spark-submit --class org.apache.spark.examples.SparkPi --master spark://master1:7077,master2:7077,master3:7077 --deploy-mode cluster --supervise examples.jar 100 This is the output I got: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/01/14 17:02:11 INFO SecurityManager: Changing view acls to: mytest 15/01/14 17:02:11 INFO SecurityManager: Changing modify acls to: mytest 15/01/14 17:02:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mytest); users with modify permissions: Set(mytest) 15/01/14 17:02:11 INFO Slf4jLogger: Slf4jLogger started 15/01/14 17:02:11 INFO Utils: Successfully started service 'driverClient' on port 53930. 15/01/14 17:02:11 ERROR OneForOneStrategy: Invalid master URL: spark://master1:7077,master2:7077,master3:7077 akka.actor.ActorInitializationException: exception during creation at akka.actor.ActorInitializationException$.apply(Actor.scala:164) at akka.actor.ActorCell.create(ActorCell.scala:596) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at
[jira] [Commented] (SPARK-5265) Submitting applications on Standalone cluster controlled by Zookeeper forces to know active master
[ https://issues.apache.org/jira/browse/SPARK-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547742#comment-14547742 ] Roque Vassal'lo commented on SPARK-5265: Hi there Sean, Sorry for the comment but they are not the same. SPARK-5497 is about Spark workers and how they act when active master goes down at a Standalone cluster (because Spark's scripts that are not working properly, at least at 1.2). When a Spark's master goes down, workers do not reassign properly to new master (as seen at Spark's WebUI, they don't appear as workers in new master's webUI). This jira, SPARK-5265, is about an external application submitting a job to Spark. That application needs to know which is the current active master of the standalone cluster because, currently, you can't submit a job setting a standalone cluster as master (--master spark://master1:7077,master2:7077) while setting deploy-mode as cluster (--deploy-mode cluster). So I think it should be reopened. Submitting applications on Standalone cluster controlled by Zookeeper forces to know active master -- Key: SPARK-5265 URL: https://issues.apache.org/jira/browse/SPARK-5265 Project: Spark Issue Type: Bug Components: Deploy Reporter: Roque Vassal'lo Labels: cluster, spark-submit, standalone, zookeeper Hi, this is my first JIRA here, so I hope it is clear enough. I'm using Spark 1.2.0 and trying to submit an application on a Spark Standalone cluster in cluster deploy mode with supervise. Standalone cluster is running in high availability mode, using Zookeeper to provide leader election between three available Masters (named master1, master2 and master3). As read at Spark's documentation, to register a Worker to the Standalone cluster, I provide complete cluster info as the spark route. I mean, spark://master1:7077,master2:7077,master3:7077 and that route is parsed and three attempts are launched, first one to master1:7077, second one to master2:7077 and third one to master3:7077. This works great! But if I try to do the same while submitting applications, it fails. I mean, if I provide complete cluster info as the --master option to spark-submit script, it throws an exception because it tries to connect as it was a single node. Example: spark-submit --class org.apache.spark.examples.SparkPi --master spark://master1:7077,master2:7077,master3:7077 --deploy-mode cluster --supervise examples.jar 100 This is the output I got: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/01/14 17:02:11 INFO SecurityManager: Changing view acls to: mytest 15/01/14 17:02:11 INFO SecurityManager: Changing modify acls to: mytest 15/01/14 17:02:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mytest); users with modify permissions: Set(mytest) 15/01/14 17:02:11 INFO Slf4jLogger: Slf4jLogger started 15/01/14 17:02:11 INFO Utils: Successfully started service 'driverClient' on port 53930. 15/01/14 17:02:11 ERROR OneForOneStrategy: Invalid master URL: spark://master1:7077,master2:7077,master3:7077 akka.actor.ActorInitializationException: exception during creation at akka.actor.ActorInitializationException$.apply(Actor.scala:164) at akka.actor.ActorCell.create(ActorCell.scala:596) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: org.apache.spark.SparkException: Invalid master URL: spark://master1:7077,master2:7077,master3:7077 at org.apache.spark.deploy.master.Master$.toAkkaUrl(Master.scala:830) at org.apache.spark.deploy.ClientActor.preStart(Client.scala:42) at akka.actor.Actor$class.aroundPreStart(Actor.scala:470) at org.apache.spark.deploy.ClientActor.aroundPreStart(Client.scala:35) at akka.actor.ActorCell.create(ActorCell.scala:580) ... 9 more Shouldn't it parse it as on Worker registration? It will not force client to know which is the current active Master of the Standalone cluster. -- This message was sent by
[jira] [Comment Edited] (SPARK-5265) Submitting applications on Standalone cluster controlled by Zookeeper forces to know active master
[ https://issues.apache.org/jira/browse/SPARK-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547742#comment-14547742 ] Roque Vassal'lo edited comment on SPARK-5265 at 5/18/15 8:52 AM: - Hi there Sean, Sorry for the comment but they are not the same. SPARK-5497 is about Spark workers and how they act when active master goes down at a Standalone cluster (because Spark's scripts that are not working properly, at least at 1.2). When a Spark's master goes down, workers do not reassign properly to new master (as seen at Spark's WebUI, they don't appear as workers in new master's webUI). This jira, SPARK-5265, is about an external application submitting a job to Spark. That application needs to know which is the current active master of the standalone cluster because, currently, you can't submit a job setting a standalone cluster as master (\--master spark://master1:7077,master2:7077) while setting deploy-mode as cluster (--deploy-mode cluster). So I think it should be reopened. was (Author: zujorv): Hi there Sean, Sorry for the comment but they are not the same. SPARK-5497 is about Spark workers and how they act when active master goes down at a Standalone cluster (because Spark's scripts that are not working properly, at least at 1.2). When a Spark's master goes down, workers do not reassign properly to new master (as seen at Spark's WebUI, they don't appear as workers in new master's webUI). This jira, SPARK-5265, is about an external application submitting a job to Spark. That application needs to know which is the current active master of the standalone cluster because, currently, you can't submit a job setting a standalone cluster as master (/--master spark://master1:7077,master2:7077) while setting deploy-mode as cluster (--deploy-mode cluster). So I think it should be reopened. Submitting applications on Standalone cluster controlled by Zookeeper forces to know active master -- Key: SPARK-5265 URL: https://issues.apache.org/jira/browse/SPARK-5265 Project: Spark Issue Type: Bug Components: Deploy Reporter: Roque Vassal'lo Labels: cluster, spark-submit, standalone, zookeeper Hi, this is my first JIRA here, so I hope it is clear enough. I'm using Spark 1.2.0 and trying to submit an application on a Spark Standalone cluster in cluster deploy mode with supervise. Standalone cluster is running in high availability mode, using Zookeeper to provide leader election between three available Masters (named master1, master2 and master3). As read at Spark's documentation, to register a Worker to the Standalone cluster, I provide complete cluster info as the spark route. I mean, spark://master1:7077,master2:7077,master3:7077 and that route is parsed and three attempts are launched, first one to master1:7077, second one to master2:7077 and third one to master3:7077. This works great! But if I try to do the same while submitting applications, it fails. I mean, if I provide complete cluster info as the --master option to spark-submit script, it throws an exception because it tries to connect as it was a single node. Example: spark-submit --class org.apache.spark.examples.SparkPi --master spark://master1:7077,master2:7077,master3:7077 --deploy-mode cluster --supervise examples.jar 100 This is the output I got: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/01/14 17:02:11 INFO SecurityManager: Changing view acls to: mytest 15/01/14 17:02:11 INFO SecurityManager: Changing modify acls to: mytest 15/01/14 17:02:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mytest); users with modify permissions: Set(mytest) 15/01/14 17:02:11 INFO Slf4jLogger: Slf4jLogger started 15/01/14 17:02:11 INFO Utils: Successfully started service 'driverClient' on port 53930. 15/01/14 17:02:11 ERROR OneForOneStrategy: Invalid master URL: spark://master1:7077,master2:7077,master3:7077 akka.actor.ActorInitializationException: exception during creation at akka.actor.ActorInitializationException$.apply(Actor.scala:164) at akka.actor.ActorCell.create(ActorCell.scala:596) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at
[jira] [Comment Edited] (SPARK-5265) Submitting applications on Standalone cluster controlled by Zookeeper forces to know active master
[ https://issues.apache.org/jira/browse/SPARK-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547742#comment-14547742 ] Roque Vassal'lo edited comment on SPARK-5265 at 5/18/15 8:52 AM: - Hi there Sean, Sorry for the comment but they are not the same. SPARK-5497 is about Spark workers and how they act when active master goes down at a Standalone cluster (because Spark's scripts that are not working properly, at least at 1.2). When a Spark's master goes down, workers do not reassign properly to new master (as seen at Spark's WebUI, they don't appear as workers in new master's webUI). This jira, SPARK-5265, is about an external application submitting a job to Spark. That application needs to know which is the current active master of the standalone cluster because, currently, you can't submit a job setting a standalone cluster as master (\-\-master spark://master1:7077,master2:7077) while setting deploy-mode as cluster (\-\-deploy-mode cluster). So I think it should be reopened. was (Author: zujorv): Hi there Sean, Sorry for the comment but they are not the same. SPARK-5497 is about Spark workers and how they act when active master goes down at a Standalone cluster (because Spark's scripts that are not working properly, at least at 1.2). When a Spark's master goes down, workers do not reassign properly to new master (as seen at Spark's WebUI, they don't appear as workers in new master's webUI). This jira, SPARK-5265, is about an external application submitting a job to Spark. That application needs to know which is the current active master of the standalone cluster because, currently, you can't submit a job setting a standalone cluster as master (\--master spark://master1:7077,master2:7077) while setting deploy-mode as cluster (--deploy-mode cluster). So I think it should be reopened. Submitting applications on Standalone cluster controlled by Zookeeper forces to know active master -- Key: SPARK-5265 URL: https://issues.apache.org/jira/browse/SPARK-5265 Project: Spark Issue Type: Bug Components: Deploy Reporter: Roque Vassal'lo Labels: cluster, spark-submit, standalone, zookeeper Hi, this is my first JIRA here, so I hope it is clear enough. I'm using Spark 1.2.0 and trying to submit an application on a Spark Standalone cluster in cluster deploy mode with supervise. Standalone cluster is running in high availability mode, using Zookeeper to provide leader election between three available Masters (named master1, master2 and master3). As read at Spark's documentation, to register a Worker to the Standalone cluster, I provide complete cluster info as the spark route. I mean, spark://master1:7077,master2:7077,master3:7077 and that route is parsed and three attempts are launched, first one to master1:7077, second one to master2:7077 and third one to master3:7077. This works great! But if I try to do the same while submitting applications, it fails. I mean, if I provide complete cluster info as the --master option to spark-submit script, it throws an exception because it tries to connect as it was a single node. Example: spark-submit --class org.apache.spark.examples.SparkPi --master spark://master1:7077,master2:7077,master3:7077 --deploy-mode cluster --supervise examples.jar 100 This is the output I got: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/01/14 17:02:11 INFO SecurityManager: Changing view acls to: mytest 15/01/14 17:02:11 INFO SecurityManager: Changing modify acls to: mytest 15/01/14 17:02:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mytest); users with modify permissions: Set(mytest) 15/01/14 17:02:11 INFO Slf4jLogger: Slf4jLogger started 15/01/14 17:02:11 INFO Utils: Successfully started service 'driverClient' on port 53930. 15/01/14 17:02:11 ERROR OneForOneStrategy: Invalid master URL: spark://master1:7077,master2:7077,master3:7077 akka.actor.ActorInitializationException: exception during creation at akka.actor.ActorInitializationException$.apply(Actor.scala:164) at akka.actor.ActorCell.create(ActorCell.scala:596) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at
[jira] [Comment Edited] (SPARK-5265) Submitting applications on Standalone cluster controlled by Zookeeper forces to know active master
[ https://issues.apache.org/jira/browse/SPARK-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547742#comment-14547742 ] Roque Vassal'lo edited comment on SPARK-5265 at 5/18/15 9:01 AM: - Hi there Sean, Sorry for the comment but they are not the same. SPARK-5497 is about Spark workers and how they act when active master goes down at a Standalone cluster (because Spark's scripts that are not working properly, at least at 1.2). When a Spark's master goes down, workers do not reassign properly to new master (as seen at Spark's WebUI, they don't appear as workers in new master's webUI). This jira, SPARK-5265, is about an external application submitting a job to Spark. That application needs to know which is the current active master of the standalone cluster because, currently, you can't submit a job setting a standalone cluster as master (\-\-master spark://master1:7077,master2:7077) while setting deploy-mode as cluster (\-\-deploy-mode cluster). As Wojciech said, it can be avoided setting deploy-mode as client, but I think that is a work-around but not the good way. So I think it should be reopened. was (Author: zujorv): Hi there Sean, Sorry for the comment but they are not the same. SPARK-5497 is about Spark workers and how they act when active master goes down at a Standalone cluster (because Spark's scripts that are not working properly, at least at 1.2). When a Spark's master goes down, workers do not reassign properly to new master (as seen at Spark's WebUI, they don't appear as workers in new master's webUI). This jira, SPARK-5265, is about an external application submitting a job to Spark. That application needs to know which is the current active master of the standalone cluster because, currently, you can't submit a job setting a standalone cluster as master (\-\-master spark://master1:7077,master2:7077) while setting deploy-mode as cluster (\-\-deploy-mode cluster). So I think it should be reopened. Submitting applications on Standalone cluster controlled by Zookeeper forces to know active master -- Key: SPARK-5265 URL: https://issues.apache.org/jira/browse/SPARK-5265 Project: Spark Issue Type: Bug Components: Deploy Reporter: Roque Vassal'lo Labels: cluster, spark-submit, standalone, zookeeper Hi, this is my first JIRA here, so I hope it is clear enough. I'm using Spark 1.2.0 and trying to submit an application on a Spark Standalone cluster in cluster deploy mode with supervise. Standalone cluster is running in high availability mode, using Zookeeper to provide leader election between three available Masters (named master1, master2 and master3). As read at Spark's documentation, to register a Worker to the Standalone cluster, I provide complete cluster info as the spark route. I mean, spark://master1:7077,master2:7077,master3:7077 and that route is parsed and three attempts are launched, first one to master1:7077, second one to master2:7077 and third one to master3:7077. This works great! But if I try to do the same while submitting applications, it fails. I mean, if I provide complete cluster info as the --master option to spark-submit script, it throws an exception because it tries to connect as it was a single node. Example: spark-submit --class org.apache.spark.examples.SparkPi --master spark://master1:7077,master2:7077,master3:7077 --deploy-mode cluster --supervise examples.jar 100 This is the output I got: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/01/14 17:02:11 INFO SecurityManager: Changing view acls to: mytest 15/01/14 17:02:11 INFO SecurityManager: Changing modify acls to: mytest 15/01/14 17:02:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mytest); users with modify permissions: Set(mytest) 15/01/14 17:02:11 INFO Slf4jLogger: Slf4jLogger started 15/01/14 17:02:11 INFO Utils: Successfully started service 'driverClient' on port 53930. 15/01/14 17:02:11 ERROR OneForOneStrategy: Invalid master URL: spark://master1:7077,master2:7077,master3:7077 akka.actor.ActorInitializationException: exception during creation at akka.actor.ActorInitializationException$.apply(Actor.scala:164) at akka.actor.ActorCell.create(ActorCell.scala:596) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at
[jira] [Created] (SPARK-5497) start-all script not working properly on Standalone HA cluster (with Zookeeper)
Roque Vassal'lo created SPARK-5497: -- Summary: start-all script not working properly on Standalone HA cluster (with Zookeeper) Key: SPARK-5497 URL: https://issues.apache.org/jira/browse/SPARK-5497 Project: Spark Issue Type: Bug Components: Deploy Affects Versions: 1.2.0 Reporter: Roque Vassal'lo I have configured a Standalone HA cluster with Zookeeper with: - 3 Zookeeper nodes - 2 Spark master nodes (1 alive and 1 in standby mode) - 2 Spark slave nodes While executing start-all.sh on each master, it will start the master and start a worker on each configured slave. If alive master goes down, those worker are supposed to reconfigure themselves to use the new active master automatically. I have noticed that the spark-env property SPARK_MASTER_IP is used in both called scripts, start-master and start-slaves. The problem is that if you configure SPARK_MASTER_IP with the active master ip, when it goes down, workers don't reassign themselves to the new active master. And if you configure SPARK_MASTER_IP with the masters cluster route (well, an approximation, because you have to write master's port in all-but-last ips, that is master1:7077,master2, in order to make it work), slaves start properly but master doesn't. So, the start-master script needs SPARK_MASTER_IP property to contain its ip in order to start master properly; and start-slaves script needs SPARK_MASTER_IP property to contain the masters cluster ips (that is master1:7077,master2) To test that idea, I have modified start-slaves and spark-env scripts on master nodes. On spark-env.sh, I have set SPARK_MASTER_IP property to master's own ip on each master node (that is, on master node 1, SPARK_MASTER_IP=master1; and on master node 2, SPARK_MASTER_IP=master2) On spark-env.sh, I have added a new property SPARK_MASTER_CLUSTER_IP with the pseudo-masters-cluster-ips (SPARK_MASTER_CLUSTER_IP=master1:7077,master2) on both masters. On start-slaves.sh, I have modified all references to SPARK_MASTER_IP to SPARK_MASTER_CLUSTER_IP. I have tried that and it works great! When active master node goes down, all workers reassign themselves to the new active node. Maybe there is a better fix for this issue. Hope this quick-fix idea can help. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
