Hi Alvin, TAJO-611 will introduce Curator as a service discovery service to Tajo and Curator is based on ZK. Maybe we can work together.
Thanks, Xuhui On Wed, Apr 16, 2014 at 12:17 PM, Min Zhou <[email protected]> wrote: > HI Alvin, > > I think this jira has somewhat overlap with TAJO-611, can you have some > cooperation? > > Thanks, > Min > > > On Tue, Apr 15, 2014 at 7:22 PM, Henry Saputra <[email protected] > >wrote: > > > Jaehwa, I think we should think about pluggable mechanism that would > > allow some kind distributed system like ZK to be used if wanted. > > > > - Henry > > > > On Tue, Apr 15, 2014 at 7:15 PM, Jaehwa Jung <[email protected]> > wrote: > > > Hi, Alvin > > > > > > I'm sorry for late response, and thank you very much for your > > contribution. > > > I agree with your opinion for zookeeper. But, zookeeper requires an > > > additional dependency that someone does not want. > > > > > > I'd like to suggest adding an abstraction layer for handling TajoMaster > > HA. > > > When I had created TAJO-740, I wished that TajoMaster HA would have a > > > generic interface and a basic implementation using HDFS. Next, your > > > proposed zookeeper implementation will be added there. It will allow > > users > > > to choice their desired implementation according to their environments. > > > > > > In addition, I'd like to propose that TajoMaster embeds the HA module, > > and > > > it would be great if HA works well by launching a backup TajoMaster. > > > Deploying additional process besides TajoMaster and TajoWorker > processes > > > may give more burden to users. > > > > > > *Cheers* > > > *Jaehwa* > > > > > > > > > 2014-04-13 14:36 GMT+09:00 Jihoon Son <[email protected]>: > > > > > >> Hi Alvin. > > >> Thanks for your suggestion. > > >> > > >> In overall, your suggestion looks very reasonable to me! > > >> I'll check the POC. > > >> > > >> Many thanks, > > >> Jihoon > > >> Hi All , > > >> After doing lot of research in my opinion we should > utilize > > >> zookeeper for Tajo Master HA.I have created a small POC and shared it > > on my > > >> Github repository ( [email protected]:alvinhenrick/zooKeeper-poc.git). > > >> > > >> Just to make things little bit easier and maintainable I > am > > >> utilizing Apache Curator the Fluent Zookeeper Client API developed at > > >> Netflix and is now part of an apache open source project. > > >> > > >> I have attached the diagram to convey my message to the > team > > >> members.Will upload it to JIRA once everyone agree with the proposed > > >> solution. > > >> > > >> Here is the flow going to look like. > > >> > > >> TajoMasterZkController ==> > > >> > > >> > > >> 1. This component will start and connect to zookeeper quorum and > > fight > > >> ( :) ) to obtain the latch / lock to become the master . > > >> 2. Once the lock is obtained the Apache Curator API will invoke > > >> takeLeadership () method at this time will start the TajoMaster. > > >> 3. As long as the TajoMaster is running the Controller will keep > > the > > >> lock and update the meta data on zookeeper server with the > > >> HOSTNAME and RPC > > >> PORT. > > >> 4. The other participant will keep waiting for the latch/ lock > to > > be > > >> released by zookeeper to obtain the leadership. > > >> 5. The advantage is we can have as many Tajo Master's as we > wan't > > but > > >> only one can be the leader and will consume the resources only > > after > > >> obtaining the latch/lock. > > >> > > >> > > >> TajoWorkerZkController ==> > > >> > > >> 1. This component will start and connect to zookeeper (will create > > >> EPHEMERAL ZNODE) and wait for the events from zookeeper. > > >> 2. The first listener will listener for successful registration. > > >> 3. The second listener on master node will listen for any > > changes to > > >> the master node received from zookeeper server. > > >> 4. If the failover occurs the data on the master ZNODE will be > > >> changed and the new HOSTNAME and RPC PORT can be obtained and > the > > >> TajoWorker can establish the new RPC connection with the > > TajoMaster. > > >> > > >> To demonstrate I have created the small Readme.txt file > > >> on Github on how to run the example. Please read the log statements on > > the > > >> console. > > >> > > >> Similar to TajoWorkerZkController we can also > > >> implement TajoClientZkController. > > >> > > >> Any help or advice is appreciated. > > >> > > >> Thanks! > > >> Warm Regards, > > >> Alvin. > > >> > > > > > > -- > My research interests are distributed systems, parallel computing and > bytecode based virtual machine. > > My profile: > http://www.linkedin.com/in/coderplay > My blog: > http://coderplay.javaeye.com >
