zpinto opened a new issue, #2662:
URL: https://github.com/apache/helix/issues/2662
# Helix Node/Instance Swap
**Implementation for N -> N + 1 -> N instance swap for all replicas on the
instance**
1. Bootstrap new replicas on the SWAP_IN node: New replicas are created for
all the partitions hosted on the node to be replaced. These new replicas are
then assigned and bootstrapped on the SWAP_IN-node.
2. Remove replicas from the SWAP_OUT node: After successfully bootstrapping
**all** the new replicas, the SWAP_OUT node will have its replicas dropped.
Waiting until all replicas bootstrap on the SWAP_IN node will ensure we
maintain all partitions desired replication factor.
3. Configurable option to not populate SWAP_IN replicas in ExternalView
until swap is completed. (Avoid spectator serving traffic)
## Requirements
- Swapping of a node requires “transaction”. When bootstrapping new replicas
on the SWAP_IN host, replica count of all partitions should be N + 1 (and
should be in topState or secondTopState) before SWAP_OUT node can begin
dropping replicas (which will return replica count to N)
## Assumptions
When a node is being swapped, the following criteria must be met for the
SWAP_IN node:
1. The cluster must have `TOPOLOGY` with at least one top level key that
corresponds to a globally unique value:
-
`n-layers/FAULT_ZONE_TYPE/n-layers/key_for_globally_unique_value/n-layers/logical_id_key`
- All instances in the cluster need a unique `DOMAIN` field. It is
possible to have a unique domain field across 2 instances which share the same
logicalId(value for the last key in the `TOPOLOGY` config)
- This is necessary as logicalId will need to be used for deterministic
placement when the SWAP_IN node is introduced as an assignable node.
2. This will be further explained in the topology section
3. The `DOMAIN` field of the SWAP_IN instance must match the SWAP_OUT node
exactly except for the value for `key_for_globally_unique_value`
4. If WAGED is enabled in the cluster, the `INSTANCE_CAPACITY_MAP` should
match exactly.
Without these assumptions, it is not possible to swap a node as this would
not be considered the best possible assignment for the cluster. If these
assumptions cannot be met, this functionality should not be utilized. Instead
the old node should use the Node Evacuation flow. A new instance can be added
in the ENABLED state either before or after to replenish the capacity of
removing the old node.
## Executions
- [ ] HelixAdmin.setInstanceOperation and addInstance sanity checks for
SWAP_IN & SWAP_OUT
- Can only set instance to SWAP_OUT
- Can only add instance with the same logicaId as another if the
already existing has InstanceOperation set to SWAP_OUT and there is not more
than 1 already.
- [ ] Create HelixAdmin.isSwapReadyToComplete and
completeSwapIfReady(returns if swap finished)
- completeSwapIfFinished will check if the swap is ready to complete. If
ready, mark the SWAP_OUT instance as `HELIX_ENABLED: false` and clear the
`INSTANCE_OPERATION` for SWAP_IN node.
- [ ] Change WAGED assignment tie breaker logic to use logicalId when
present
- May want to make this configurable, as it will lead to shuffling when
this change is deployed
- [ ] Refactor DelayedAutoRebalancerUtil and
DelayedAutoRebalancer.computeBestPossibleStateForPartition
to filter out SWAP_IN node because we will add SWAP_IN node to the end of
preference list in computeBestPossibleStateForPartition until the swap is
complete
- Adding it to the end of the preference list will ensure that the
SWAP_IN node only carries secondary states
- When SWAP_IN node has identical Current State to SWAP_OUT,
isSwapFinished completeSwapIfFinished will return true
- [ ] Refactor ExternalViewComputationStage to not include Current State for
SWAP_IN nodes
- Possible make this configurable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
