Hi,
Trying to follow up if there are any thoughts/suggestions over
the approaches.
Trying to manage a cluster with a resource that has Dynamic
number of partitions please provide me your valuable thoughts on same.
Version(s) :
Helix: 0.7.1
Zookeeper:3.3.4
-Zookeeper quorum (3 nodes)
- State Model: OnlineOffline
- Controller (leader elected from one of the cluster nodes)
- Single resources with partitions.
- Custom rebalancer
Few of the approaches that I could think of ...
Approach 1: Buffer Partitions
Let’s say the current number of partitions are N.
1 ) Define N+X (Where x could be the delta
where it can expand maximum , expecting to be of few in number (1-10))
2 ) Use property store to identify which are
all valid partition(s)
3 ) Don’t act upon invalid ones by looking up
property store (or) Disable the unallocated partition(s)
When a new partition to be added -
1. Update property store of new partitions (Delta merge)
2. Enable the partition(s)
--- The balancing algorithm should be able to balance valid & extra
partition(s) differently. Lots of customizations.
Approach 2: New Resource Addition
Let’s say the current number of partitions are N.
Create Resource 1 with N partitions initially.
When a new partition(s) to be added -
1. Drop the resource
2. Create new resource with N+X (newly added partitions )
--- Caveat that I could see is Dropping the resource would stop the entire
workflow (It’s a bigger price to pay for us since we are real time).
Thanks & Regards,
Subramanian.
From: kishore g [mailto:[email protected]]
Sent: Friday, June 02, 2017 9:52 PM
To: Subramanian Raghunathan <[email protected]>;
[email protected]
Subject: Re: Dynamic partition management
+helix
On Jun 2, 2017 6:50 PM, "Subramanian Raghunathan"
<[email protected]<mailto:[email protected]>>
wrote:
Hi kishore,
Trying to manage a cluster with a resource that has Dynamic
number of partitions please provide me your valuable thoughts on same.
Version(s) :
Helix: 0.7.1
Zookeeper:3.3.4
-Zookeeper quorum (3 nodes)
- State Model: OnlineOffline
- Controller (leader elected from one of the cluster nodes)
- Single resources with partitions.
- Custom rebalancer
Few of the approaches that I could think of ...
Approach 1: Buffer Partitions
Let’s say the current number of partitions are N.
1 ) Define N+X (Where x could be the delta
where it can expand maximum , expecting to be of few in number (1-10))
2 ) Use property store to identify which are
all valid partition(s)
3 ) Don’t act upon invalid ones by looking up
property store (or) Disable the unallocated partition(s)
When a new partition to be added -
3. Update property store of new partitions (Delta merge)
4. Enable the partition(s)
--- The balancing algorithm should be able to balance valid & extra
partition(s) differently. Lots of customizations.
Approach 2: New Resource Addition
Let’s say the current number of partitions are N.
Create Resource 1 with N partitions initially.
When a new partition(s) to be added -
3. Drop the resource
4. Create new resource with N+X (newly added partitions )
--- Caveat that I could see is Dropping the resource would stop the entire
workflow (It’s a bigger price to pay for us since we are real time).
Thanks & Regards,
Subramanian.
Tel: +1 (650) 424 4655<tel:(650)%20424-4655>
Mob:+1(650) 656 6006<tel:(650)%20656-6006>
850 Hansen Way,
Palo Alto, CA 94304
www.integral.com<http://www.integral.com/>
[Logo_signature_block]<http://www.integral.com/fxcloud_features/risk_management.html#ym>
NOTICE: This e-mail message and any attachments, which may contain confidential
information, are to be viewed solely by the intended recipient of Integral
Development Corp. For further information, please visit
http://www.integral.com/about/disclaimer.html.
