Hi Chris, Could you please elaborate the expected behavior a little bit? Helix supports changing the partition list now. But application needs to define how data is transferred between different partitions. For example, when the new partition replica is boot up, the state transition logic can move data accordingly. If you are using CRUSH algorithm, the existing partitions' placement should be stable.
Best Regards, Jiajun On Mon, May 21, 2018 at 11:12 PM, Chris Lu <[email protected]> wrote: > In most distributed systems, the data is over sharded. Helix seems taking > this as an assumption. > > Is there any way to use Helix to manage splitting the shards, for data > stores? > > I am trying to fit the jump consistent hash into Helix model. > > Basically jumping hash can change from N shards to M shards, where N and M > can be just any positive integers. If growing, the (M-N)/M data on N shards > would be moved to the new M-N shards. > > https://arxiv.org/abs/1406.2294 > > Jumping Hash is needed to provide an atomic operation to switch to the new > topology. When growing, the data queries still go to the existing shards, > until the new shards are ready. So the new servers can prepare data as fast > as possible, rather than having to throttle the data preparation. > > Chris > >
