Jaehwa, I think we should think about pluggable mechanism that would allow some kind distributed system like ZK to be used if wanted.
- Henry On Tue, Apr 15, 2014 at 7:15 PM, Jaehwa Jung <[email protected]> wrote: > Hi, Alvin > > I'm sorry for late response, and thank you very much for your contribution. > I agree with your opinion for zookeeper. But, zookeeper requires an > additional dependency that someone does not want. > > I'd like to suggest adding an abstraction layer for handling TajoMaster HA. > When I had created TAJO-740, I wished that TajoMaster HA would have a > generic interface and a basic implementation using HDFS. Next, your > proposed zookeeper implementation will be added there. It will allow users > to choice their desired implementation according to their environments. > > In addition, I'd like to propose that TajoMaster embeds the HA module, and > it would be great if HA works well by launching a backup TajoMaster. > Deploying additional process besides TajoMaster and TajoWorker processes > may give more burden to users. > > *Cheers* > *Jaehwa* > > > 2014-04-13 14:36 GMT+09:00 Jihoon Son <[email protected]>: > >> Hi Alvin. >> Thanks for your suggestion. >> >> In overall, your suggestion looks very reasonable to me! >> I'll check the POC. >> >> Many thanks, >> Jihoon >> Hi All , >> After doing lot of research in my opinion we should utilize >> zookeeper for Tajo Master HA.I have created a small POC and shared it on my >> Github repository ( [email protected]:alvinhenrick/zooKeeper-poc.git). >> >> Just to make things little bit easier and maintainable I am >> utilizing Apache Curator the Fluent Zookeeper Client API developed at >> Netflix and is now part of an apache open source project. >> >> I have attached the diagram to convey my message to the team >> members.Will upload it to JIRA once everyone agree with the proposed >> solution. >> >> Here is the flow going to look like. >> >> TajoMasterZkController ==> >> >> >> 1. This component will start and connect to zookeeper quorum and fight >> ( :) ) to obtain the latch / lock to become the master . >> 2. Once the lock is obtained the Apache Curator API will invoke >> takeLeadership () method at this time will start the TajoMaster. >> 3. As long as the TajoMaster is running the Controller will keep the >> lock and update the meta data on zookeeper server with the >> HOSTNAME and RPC >> PORT. >> 4. The other participant will keep waiting for the latch/ lock to be >> released by zookeeper to obtain the leadership. >> 5. The advantage is we can have as many Tajo Master's as we wan't but >> only one can be the leader and will consume the resources only after >> obtaining the latch/lock. >> >> >> TajoWorkerZkController ==> >> >> 1. This component will start and connect to zookeeper (will create >> EPHEMERAL ZNODE) and wait for the events from zookeeper. >> 2. The first listener will listener for successful registration. >> 3. The second listener on master node will listen for any changes to >> the master node received from zookeeper server. >> 4. If the failover occurs the data on the master ZNODE will be >> changed and the new HOSTNAME and RPC PORT can be obtained and the >> TajoWorker can establish the new RPC connection with the TajoMaster. >> >> To demonstrate I have created the small Readme.txt file >> on Github on how to run the example. Please read the log statements on the >> console. >> >> Similar to TajoWorkerZkController we can also >> implement TajoClientZkController. >> >> Any help or advice is appreciated. >> >> Thanks! >> Warm Regards, >> Alvin. >>
