+helix On Jun 2, 2017 6:50 PM, "Subramanian Raghunathan" < [email protected]> wrote:
> Hi kishore, > > > > Trying to manage a cluster with a resource that has > Dynamic number of partitions please provide me your valuable thoughts on > same. > > > > Version(s) : > > Helix: 0.7.1 > > Zookeeper:3.3.4 > > > > -Zookeeper quorum (3 nodes) > > - State Model: OnlineOffline > > - Controller (leader elected from one of the cluster nodes) > > - Single resources with partitions. > > - Custom rebalancer > > > > Few of the approaches that I could think of ... > > > > Approach 1: *Buffer Partitions* > > Let’s say the current number of partitions are N. > > 1 ) Define N+X (Where x could be the > delta where it can expand maximum , expecting to be of few in number > (1-10)) > > 2 ) Use property store to identify which > are all valid partition(s) > > 3 ) Don’t act upon invalid ones by > looking up property store (or) Disable the unallocated partition(s) > > > > When a new partition to be added - > > 1. Update property store of new partitions (Delta merge) > 2. Enable the partition(s) > > > > --- The balancing algorithm should be able to balance valid & extra > partition(s) differently. Lots of customizations. > > > > Approach 2: *New Resource Addition* > > Let’s say the current number of partitions are N. > > > > Create Resource 1 with N partitions initially. > > When a new partition(s) to be added - > > 1. Drop the resource > 2. Create new resource with N+X (newly added partitions ) > > > > --- Caveat that I could see is Dropping the resource would stop the entire > workflow (It’s a bigger price to pay for us since we are real time). > > > > > > Thanks & Regards, > > Subramanian. > > > > Tel: +1 (650) 424 4655 <(650)%20424-4655> > > Mob:+1(650) 656 6006 <(650)%20656-6006> > > > > 850 Hansen Way, > > Palo Alto, CA 94304 > > www.integral.com > > [image: Logo_signature_block] > <http://www.integral.com/fxcloud_features/risk_management.html#ym> > > NOTICE: This e-mail message and any attachments, which may contain > confidential information, are to be viewed solely by the intended recipient > of Integral Development Corp. For further information, please visit > http://www.integral.com/about/disclaimer.html. > > > > > > >
