[
https://issues.apache.org/jira/browse/SPARK-5171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roberto Vaquerizo Rodriguez resolved SPARK-5171.
------------------------------------------------
Resolution: Not a Problem
My fault ... I have found what the problem is.
The env configuration should read:
-Dspark.deploy.recoveryMode=ZOOKEEPER
and not:
-Dspark.deploy.recoverymode=ZOOKEEPER
Thanks.
> Standalone cluster: masters scheduling independently
> ----------------------------------------------------
>
> Key: SPARK-5171
> URL: https://issues.apache.org/jira/browse/SPARK-5171
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 1.2.0
> Reporter: Roberto Vaquerizo Rodriguez
>
> Hi.
> I'm trying to configure a spark standalone cluster (release = spark 1.2),
> with three master nodes (bigdata1, bigdata2 and bigdata3) managed by
> Zookeeper.
> It seems there's a configuration problem, since every master node is saying
> it is the cluster leader:
> .........
> 14/12/30 13:54:59 INFO Master: I have been elected leader! New state:
> ALIVE
> The message above is dumped by every master I start up.
> Zookeeper is configured identically in all of them, as follows:
> ........
> dataDir=/spark
> ........
> The only difference is the "myid" file in the /spark directory, of course.
> The masters are started using the following configuration:
> .........
> export SPARK_DAEMON_JAVA_OPTS=" \
> -Dspark.deploy.recoverymode=ZOOKEEPER \
> -Dspark.deploy.zookeeper.url=bigdata1:2181,bigdata2:2181,bigdata3:2181"
> I have not set the spark.deploy.zookeeper.dir variable, since I'm using the
> default value, /spark, configured in zookeeper, as I mentioned before.
> I would like to know if there is any other thing I have to configure, in
> order to make the masters behave correctly (only one master node active at a
> time, while the others are in stand-by mode).
> With the current situation, I can connect workers and applications to the
> whole cluster. For instance, I can connect a worker to the cluster using:
> spark-class org.apache.spark.deploy.worker.Worker
> spark://bigdata1:2181,bigdata2:2181,bigdata3:2181
> But the worker gets registered to each of the masters independently.
> If I stop one of the masters, it tries to re-register to it.
> The notion of only one active master is completely lost. All masters are
> scheduling independently.
> If I run "zkServer.sh status" in the cluster, only one of the master nodes
> says "leader", and the other ones say "follower". So, from a Zookeeper point
> of view, it seems that the configuration is correct.
> But the standalone cluster is not interacting with zookeeper properly.
> Do you have any idea?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
