Great options. 300k is definitely a lot. We have some benchmarking tool that can be modified to see what happens when we have 300k partitions.There is also another option to compress the data in idealstate. I am not sure how to enable this. Jason had implemented this.
thanks, Kishore G On Fri, Aug 1, 2014 at 10:18 AM, Kanak Biscuitwala <[email protected]> wrote: > a) By default, there is one znode per resource, which as you know is a > grouping of partitions. The biggest limitation is that ZK has a 1MB limit > on znode sizing. To get around this, Helix has the concept of bucketizing, > where in your ideal state, you can set a bucket size, which will > effectively create that many znodes to fully represent all your partitions. > I believe that you can have ~2k partitions before you start needing to > bucketize. > > 300k may cause you separate issues, and you may want to consider doing > things like enabling batch message mode in your ideal state so that each > message we send to an instance contains transitions for all partitions > hosted on that instance, rather than creating a znode per partition state > change. However, in theory (we've never played with this many in practice), > Helix should be able to function correctly with that many partitions. > > b) Yes, if you have a hard limit of 1 master per partition, Helix will > transition the first node to OFFLINE before sending the MASTER transition > to the new master. > > Kanak > > ------------------------------ > Date: Fri, 1 Aug 2014 10:09:24 -0700 > > Subject: Re: Questions about custom helix rebalancer/controller/agent > From: [email protected] > To: [email protected] > > Sounds fine to me. I can work without the FINALIZE notification for now, > but I hope its going to come out soon. A few more questions: > > a) How well does Helix scale with partitions - is each partition a > separate znode inside helix ? If I have 300K partitions in Helix would that > be an issue ? > b) If a partition which was assigned as a master on node1 is now assigned > as a master on node2, will node1 get a callback execution for transition > from MASTER-->OFFLINE > > Thanks > Varun > > > On Thu, Jul 31, 2014 at 11:18 PM, Kanak Biscuitwala <[email protected]> > wrote: > > s/run/start/g -- sorry about that, fixed in javadocs for future releases > > You may need to register for a notification type; I believe > HelixCustomCodeRunner complains if you don't. However, you can simply > ignore that notification type, and just check for INIT and FINALIZE > notification types in your callback to to track whether or not you're the > leader. On INIT, you start your 30 minute timer, and on FINALIZE you stop > it. You may need to wait for us to make a 0.6.4 release (we will likely do > this soon) to get the FINALIZE notification. > > Here is an example of a custom code runner usage: > Registration: > https://github.com/kishoreg/fullmatix/blob/master/mysql-cluster/src/main/java/org/apache/fullmatix/mysql/MySQLAgent.java > Callback: > https://github.com/kishoreg/fullmatix/blob/master/mysql-cluster/src/main/java/org/apache/fullmatix/mysql/MasterSlaveRebalancer.java > > Regarding setting up the Helix controller, you actually don't need to > instantiate a GenericHelixController. If you create a HelixManager with > InstanceType.CONTROLLER, then ZKHelixManager automatically creates a > GenericHelixController and sets it up with leader election. We really > should update the documentation to clarify that. > > ------------------------------ > Date: Thu, 31 Jul 2014 23:00:13 -0700 > > Subject: Re: Questions about custom helix rebalancer/controller/agent > From: [email protected] > To: [email protected] > > Thanks for the suggestions.. > > Seems like the HelixCustomCodeRunner could do it. However, it seems like > the CustomCodeRunner only provides hooks for plugging into notifications. > The documentation example in the above link suggests a run() method, which > does not seem to exist. > > However, this maybe sufficient for my case. I essentially hook in an empty > CustomCodeRunner into my helix manager. Then I can instantiate my own > thread which would run above snippet and keep writing ideal states every 30 > minutes. I guess I would still need to attach the GenericHelixController > with the following code snippet to take action whenever the ideal state > changes ?? > > GenericHelixController controller = new GenericHelixController(); > manager.addConfigChangeListener(controller); > manager.addLiveInstanceChangeListener(controller); > manager.addIdealStateChangeListener(controller); > manager.addExternalViewChangeListener(controller); > manager.addControllerListener(controller); > > > > > > On Thu, Jul 31, 2014 at 6:01 PM, kishore g <[email protected]> wrote: > > List resourceList = helixAdmin.getResourceList(); > for each resource: > Compute target ideal state > helixAdmin.setIdealState(resource, targetIdealState); > > Thread.sleep(30minutes); > > This can work right. This code can be as part of CustomCodeRunner. > http://helix.apache.org/javadocs/0.6.3/reference/org/apache/helix/participant/HelixCustomCodeRunner.html. > You can say you are interested in notifications but can ignore that. > > thanks, > Kishore G > > > On Thu, Jul 31, 2014 at 5:45 PM, Kanak Biscuitwala <[email protected]> > wrote: > > i.e. helixAdmin.enableCluster(clusterName, false); > > ------------------------------ > From: [email protected] > To: [email protected] > Subject: RE: Questions about custom helix rebalancer/controller/agent > Date: Thu, 31 Jul 2014 17:44:40 -0700 > > > Unfortunately HelixAdmin#rebalance is a misnomer, and it is a function of > all the configured instances and not the live instances. The closest you > can get to that is to use the third option I listed related to CUSTOMIZED > mode, where you write the mappings yourself based on what is live. > > Another thing you could do is pause the cluster controller and unpause it > for a period every 30 minutes. That will essentially enforce that the > controller will not send transitions (or do anything else, really) during > the time it is paused. This sounds a little like a hack to me, but it may > do what you want. > > Kanak > > ------------------------------ > Date: Thu, 31 Jul 2014 17:39:40 -0700 > Subject: Re: Questions about custom helix rebalancer/controller/agent > From: [email protected] > To: [email protected] > > Thanks Kanak, for your detailed response and this is really very helpful. > I was wondering if its possible for me do something like the following: > > List resourceList = helixAdmin.getResourceList(); > for each resource: > Compute target ideal state > helixAdmin.rebalance(resource); > > Thread.sleep(30minutes); > > So, the above happens inside a while loop thread and this is the only > place where we do the rebalancing ? > > Thanks > Varun > > > On Thu, Jul 31, 2014 at 5:25 PM, Kanak Biscuitwala <[email protected]> > wrote: > > Hi Varun, > > Sorry for the delay. > > 1 and 3) There are a number of ways to do this, with various tradeoffs. > > - You can write a user-defined rebalancer. In helix 0.6.x, it involves > implementing the following interface: > > > https://github.com/apache/helix/blob/helix-0.6.x/helix-core/src/main/java/org/apache/helix/controller/rebalancer/Rebalancer.java > > Essentially what it does is given an existing ideal state, compute a new > ideal state. For 0.6.x, this will read the preference lists in the output > ideal state and compute a state mapping based on them. If you need more > control, you can also implement: > > > https://github.com/apache/helix/blob/helix-0.6.x/helix-core/src/main/java/org/apache/helix/controller/rebalancer/internal/MappingCalculator.java > > which will allow you to create a mapping from partition to map of > participant and state. In 0.7.x, we consolidated these into a single method. > > Here is a tutorial on the user-defined rebalancer: > http://helix.apache.org/0.6.3-docs/tutorial_user_def_rebalancer.html > > Now, running this every 30 minutes is tricky because by default the > controller responds to all cluster events (and really it needs to because > it aggregates all participant current states into the external view -- > unless you don't care about that). > > - Combined with the user-defined rebalancer (or not), you can have a > GenericHelixController that doesn't listen on any events, but calls > startRebalancingTimer(), into which you can pass 30 minutes. The problem > with this is that the instructions at > http://helix.apache.org/0.6.3-docs/tutorial_controller.html won't work as > described because of a known issue. The workaround is to connect > HelixManager as role ADMINISTRATOR instead of CONTROLLER. > > However, if you connect as ADMINISTRATOR, you have to set up leader > election yourself (assuming you want a fault-tolerant controller). See > https://github.com/apache/helix/blob/helix-0.6.x/helix-core/src/main/java/org/apache/helix/manager/zk/DistributedLeaderElection.java > for > a controller change listener that can do leader election, but your version > will have to be different, as you actually don't want to add listeners, but > rather set up a timer. > > This also gives you the benefit of plugging in your own logic into the > controller pipeline. See > https://github.com/apache/helix/blob/helix-0.6.x/helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java > createDefaultRegistry() > for how to create an appropriate PipelineRegistry. > > - You can take a completely different approach and put your ideal state in > CUSTOMIZED rebalance mode. Then you can have a meta-resource where one > participant is a leader and the others are followers (you can create an > ideal state in SEMI_AUTO mode, where the replica count and the replica > count and preference list of resourceName_0 is "ANY_LIVEINSTANCE". When one > participant is told to become leader, you can set a timer for 30 minutes > and update and write the map fields of the ideal state accordingly. > > 2) I'm not sure I understand the question. If you're in the JVM, you > simply need to connect as a PARTICIPANT for your callbacks, but that can > just be something you do at the beginning of your node startup. The rest of > your code is more or less governed by your transitions, but if there are > things you need to do on the side, there is nothing in Helix preventing you > from doing so. See > http://helix.apache.org/0.6.3-docs/tutorial_participant.html for > participant logic. > > 4) The current state is per-instance and is literally called CurrentState. > For a given participant, you can query a current state by doing something > like: > > HelixDataAccessor accessor = helixManager.getHelixDataAccessor(); > CurrentState currentState = > accessor.getProperty(accessor.keyBuilder().currentState(instanceName, > sessionId, resourceName); > > If you implement a user-defined rebalancer as above, we automatically > aggregate all these current states into a CurrentStateOutput object. > > 5) You can use a Helix spectator: > > http://helix.apache.org/0.6.3-docs/tutorial_spectator.html > > This basically gives you a live-updating routing table for the mappings of > the Helix-managed resource. However, it requires the external view to be up > to date, going back to my other point of perhaps separating the concept of > changing mappings every 30 minutes from the frequency at which the > controller runs. > > Hopefully this helps. > > Kanak > > ------------------------------ > Date: Thu, 31 Jul 2014 12:13:27 -0700 > Subject: Questions about custom helix rebalancer/controller/agent > From: [email protected] > To: [email protected] > > > Hi, > > I am trying to write a customized rebalancing algorithm. I would like to > run the rebalancer every 30 minutes inside a single thread. I would also > like to completely disable Helix triggering the rebalancer. > > I have a few questions: > 1) What's the best way to run the custom controller ? Can I simply > instantiate a ZKHelixAdmin object and then keep running my rebalancer > inside a thread or do I need to do something more. > > Apart from rebalancing, I want to do other things inside the the > controller, so it would be nice if I could simply fire up the controller > through code. I could not find this in the documentation. > > 2) Same question for the Helix agent. My Helix Agent is a JVM process > which does other things apart from exposing the callbacks for state > transitions. Is there a code sample for the same ? > > 3) How do I disable Helix triggered rebalancing once I am able to run the > custom controller ? > > 4) During my custom rebalance run, how I can get the current cluster state > - is it through ClusterDataCache.getIdealState() ? > > 5) For clients talking to the cluster, does helix provide an easy > abstraction to find the partition distribution for a helix resource ? > > Thanks > > > > > >
