Hi Kishore, Do you have any documentation/sample regarding zookeeper implementation programmatically?
Thanks, Erwin On Jun 15, 2013 11:28 AM, "Erwin Karbasi" <[email protected]> wrote: > Hi kishore, > > Thank you for preservance and helpful support. > > Do you have any direction how can we implement zookeeper clustering > programmatically in my aforementioned suggestion? > As far as I know we need to use quorum. Do you know any example that can > help us. > > Thanks, > Erwin > On Jun 15, 2013 3:46 AM, "kishore g" <[email protected]> wrote: > >> Hi Erwin, >> >> Yes thats possible. Good idea. >> >> thanks, >> Kishore G >> >> >> On Fri, Jun 14, 2013 at 3:42 PM, Erwin Karbasi <[email protected]>wrote: >> >>> Kishore, >>> >>> Thank you a lot for detailed explanation. >>> >>> I thought to define in configuration of each participant whether it will >>> act as controller or Zookeeper as well. All participants will know those >>> controllers and zookeepers that defined. >>> For example I will define in 5 participants that they also should act as >>> controller and zookeepers as well on the same process of the participants. >>> Is it possible? >>> >>> Cheers, >>> Erwin >>> On Jun 14, 2013 6:21 PM, "kishore g" <[email protected]> wrote: >>> >>>> Hi Erwin, >>>> >>>> Here are the options: >>>> >>>> 1. participant controller and zookeeper on one process on all 50 boxes: >>>> This is not possible, you need only 3 or 5 zookeepers running and all >>>> participant/controllers need to know the machinename: port of the zookeeper >>>> boxes upfront. >>>> >>>> 2. separate zookeeper but participant and controller in one process: >>>> This is possible, controller can be started as part of all participants but >>>> only one controller will be active. This is fine until you have like 100 >>>> nodes, after which its better to have a separate controller. >>>> >>>> 3. zookeeper and controller on same box but separate process, >>>> participants in different: This is possible, note that once you this >>>> configuration all additional clusters you deploy can share the same >>>> zookeeper/controller. So if you plan to have many clusters this is a better >>>> option and much easier to manage the clusters. You can use standard >>>> commands to start zookeeper and helix controller. >>>> >>>> 4. zookeeper and controller on same box and in one process: This also >>>> works and the advantages of 3 holds good here as well. You will have to >>>> start >>>> both of them programmatically. Any time you restart a process, both >>>> zookeeper and controller will be upgraded. >>>> >>>> >>>> My preference is 3) 4) 2), you cannot do 1). >>>> >>>> I agree with you that it requires additional monitoring but actual >>>> experience suggests that co-locating zookeeper with participants causes >>>> un-wanted problems. For example, the participants are heavily loaded and >>>> your application might have large GC pauses/consume al cpu/memory on that >>>> box. This will adversely impact the functionality of zookeeper. >>>> >>>> Thanks, >>>> Kishore G >>>> >>>> >>>> >>>> >>>> >>>> On Fri, Jun 14, 2013 at 5:44 AM, Erwin Karbasi <[email protected]>wrote: >>>> >>>>> My question is, whether is it possible to collocate participant, >>>>> controller and zookeeper on one process, is there any limitations in term >>>>> of Helix or Zookeeper? >>>>> Any direction regarding above approach would be appreciated. >>>>> On Jun 14, 2013 3:23 PM, "Ming Fang" <[email protected]> wrote: >>>>> >>>>>> That's not possible at this time. >>>>>> ZAC currently must run as a standalone process and only allow one >>>>>> instance. >>>>>> >>>>>> Sent from my iPad >>>>>> >>>>>> On Jun 14, 2013, at 7:35 AM, Erwin Karbasi <[email protected]> >>>>>> wrote: >>>>>> >>>>>> If I have 40 app server instances (JVMs) could I run participants on >>>>>> all of them, controller and zookeeper just on 5 of then, but on the same >>>>>> process of the 5 participants? >>>>>> On Jun 14, 2013 2:29 PM, "Ming Fang" <[email protected]> wrote: >>>>>> >>>>>>> ZAC currently runs Zookeeper in standalone mode. >>>>>>> Therefore you can only run one instance. >>>>>>> If you also run the participant in it then that limits you to one >>>>>>> instance of the participant also. >>>>>>> >>>>>>> >>>>>>> Sent from my iPad >>>>>>> >>>>>>> On Jun 14, 2013, at 7:23 AM, Erwin Karbasi <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>> Hi Ming, >>>>>>> >>>>>>> Thank you for fast turnaround response. >>>>>>> >>>>>>> Could I run my app that uses Helix (participant), controller and >>>>>>> zookeeper on the same process? >>>>>>> >>>>>>> Cheers, >>>>>>> Erwin >>>>>>> On Jun 14, 2013 2:16 PM, "Ming Fang" <[email protected]> wrote: >>>>>>> >>>>>>>> Erwin >>>>>>>> >>>>>>>> Here is an example of running Zookeeper, and Controller in the same >>>>>>>> process, >>>>>>>> >>>>>>>> https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java >>>>>>>> We have this in production. >>>>>>>> >>>>>>>> Sent from my iPad >>>>>>>> >>>>>>>> On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>> I know that it's better that participants, controller and zookeeper >>>>>>>> will have separate java processes but we have more then 40 boxes with >>>>>>>> app >>>>>>>> server that we'd like to manage them with Helix Leader election and >>>>>>>> cannot >>>>>>>> add more processes because each additional java process requires >>>>>>>> management >>>>>>>> (monitoring, alarms, KPI, etc...), so we prefer to avoid the additional >>>>>>>> java processes. We'll not manage to approve additional processes >>>>>>>> >>>>>>>> Your insight would appreciated. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Erwin >>>>>>>> On Jun 14, 2013 9:58 AM, "kishore g" <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Erwin, >>>>>>>>> >>>>>>>>> This is how I would setup your cluster. Allocate 3 boxes that only >>>>>>>>> run zookeeper and Helix controller. Its better that they are two >>>>>>>>> separate >>>>>>>>> process. These boxes need not be powerful but isolating zookeeper and >>>>>>>>> helix >>>>>>>>> controller from actual server is a good deployment practice. See >>>>>>>>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetupon >>>>>>>>> how to use zookeeper clustering ensemble. >>>>>>>>> >>>>>>>>> >>>>>>>>> Run the real servers on the rest of the boxes. Do not start >>>>>>>>> zookeeper as part of these processes, instead simply provide the >>>>>>>>> zookeeper >>>>>>>>> address (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Kishore G >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Kishore, >>>>>>>>>> >>>>>>>>>> We have more then 50 nodes. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Erwin >>>>>>>>>> On Jun 14, 2013 1:51 AM, "kishore g" <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Erwin, >>>>>>>>>>> >>>>>>>>>>> How many nodes do you have in your cluster ? >>>>>>>>>>> >>>>>>>>>>> thanks, >>>>>>>>>>> Kishore G >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hello Maters, >>>>>>>>>>>> >>>>>>>>>>>> We don't want to have separate/specific JAVA process for >>>>>>>>>>>> Zookeeper. >>>>>>>>>>>> As I figured out from the Master-Slave sample we can start the >>>>>>>>>>>> Zookeeper programmatically using the Zookeeper API. >>>>>>>>>>>> >>>>>>>>>>>> I have several questions: >>>>>>>>>>>> >>>>>>>>>>>> 1. If we are using the Zookeeper API for initializing the >>>>>>>>>>>> Zookeeper from the Helix app (out app that uses Helix) would we >>>>>>>>>>>> have >>>>>>>>>>>> specific process for Zookeeper? Would we have one process for our >>>>>>>>>>>> app and >>>>>>>>>>>> another for the Zookeeper? >>>>>>>>>>>> What happens if we'll not initialize the Zookeeper from the >>>>>>>>>>>> Helix app programmatically but using the commands, whether then >>>>>>>>>>>> would we >>>>>>>>>>>> have specific process for Zookeeper? >>>>>>>>>>>> >>>>>>>>>>>> 2. If we'll decide to initialize Zookeeper from Helix app, >>>>>>>>>>>> programmatically, How can we use Zookeeper clustering (ensemble) , >>>>>>>>>>>> is there >>>>>>>>>>>> any API for Zookeeper clustering, any direction would highly >>>>>>>>>>>> appreciated. >>>>>>>>>>>> >>>>>>>>>>>> 3. Is it possible to start the Zookeeper by using commands and >>>>>>>>>>>> not initializing specific process for it but using the Helix app >>>>>>>>>>>> (our app >>>>>>>>>>>> that uses Helix API)? >>>>>>>>>>>> >>>>>>>>>>>> I hope my questions are enough clear. >>>>>>>>>>>> >>>>>>>>>>>> Thanks in advance, >>>>>>>>>>>> Erwin Karbasi >>>>>>>>>>>> AT&T, Senior Software Architect >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>> >>
