Jiajun, Thanks for answering! I could not find any doc about "Helix supports changing the partition list".
Bootstrap a partition replica can be done by reading off other existing partitions, as long as the new partition replica knows its own partition index. Basically in my design, all machines follow the same Kafka topic, but they may have progressed to different Kafka offsets. When data moves, they need to resume from the original offsets. It's hard to track different offsets for many moving partitions. Is there any way to put some meta data, such as the Kafka offsets, to the moving partition? Chris On Thu, May 24, 2018 at 3:24 PM, Wang Jiajun <[email protected]> wrote: > Hi Chris, > > Could you please elaborate the expected behavior a little bit? > Helix supports changing the partition list now. > But application needs to define how data is transferred between different > partitions. For example, when the new partition replica is boot up, the > state transition logic can move data accordingly. > If you are using CRUSH algorithm, the existing partitions' placement > should be stable. > > > Best Regards, > Jiajun > > On Mon, May 21, 2018 at 11:12 PM, Chris Lu <[email protected]> wrote: > >> In most distributed systems, the data is over sharded. Helix seems taking >> this as an assumption. >> >> Is there any way to use Helix to manage splitting the shards, for data >> stores? >> >> I am trying to fit the jump consistent hash into Helix model. >> >> Basically jumping hash can change from N shards to M shards, where N and >> M can be just any positive integers. If growing, the (M-N)/M data on N >> shards would be moved to the new M-N shards. >> >> https://arxiv.org/abs/1406.2294 >> >> Jumping Hash is needed to provide an atomic operation to switch to the >> new topology. When growing, the data queries still go to the existing >> shards, until the new shards are ready. So the new servers can prepare data >> as fast as possible, rather than having to throttle the data preparation. >> >> Chris >> >> >
