I send another time this mail, no response abpit my question.



Envoyé : jeudi 13 juillet 2017 10:43
À : user@spark.apache.org
Objet : how to identify the alive master spark via Zookeeper ?


In our project, we have a Spark cluster with 2 master and 4 workers and 
Zookeeper decides which master is alive.
We have a problem with our reverse proxy to display the Spark Web UI. The RP 
redirect on a master with IP address configured in initial configuration but if 
Zookeeper decides to change the master, our spark Web UI is not accessible 
because the IP address of master changed.
We want to find dynamically which master is elected every time.
We search in Internet a solution to know with Zookeeper which master is alive 
but we don't find anything. It is possible with confd to search if property 
changed but none property is saved in Zookeeper. In folder /spark in Zookeeper, 
nothing is logged.
Why Spark does not send property to Zookeeper to indicate which ip address or 
hostname is elected ? In your class ZooKeeperLeaderElectionAgent.scala, you 
logged which master is elected but perhaps it will be also a good solution to 
send a property to Zookeeper to indicate host.

We already asked to Zookeeper user mailing list and they said that:
"This question may be better suited for the Spark mailing lists as Zookeeper 
doesn't really "decide" which master is alive but rather provides a mechanism 
for the application to make the correct decision."

So, we think that we are not alone with this type of problem but we can't find 
anything on Internet.

Can you help us to solve this problem ?


Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

Reply via email to