rdhabalia opened a new issue, #16551:
URL: https://github.com/apache/pulsar/issues/16551
# Motivation
Cluster migration or Blue-Green cluster deployment is one of the proven
solutions to migrate live traffic from one cluster to another. One of the
examples is applications running on Kubernetes sometimes require a Kubernetes
cluster upgrade which can cause downtime for the entire application during a
Kubernetes cluster upgrade. Blue-green deployment is an application release
model that gradually transfers user traffic from a previous version of an app
or microservice to a nearly identical new release—both of which are running in
production.
The old version can be called the blue environment while the new version can
be known as the green environment. Once production traffic is fully transferred
from blue to green, blue can standby in case of rollback or be pulled from
production and updated to become the template upon which the next update is
made.
We need such capability in Apache pulsar to migrate live traffic from the
blue cluster to the green cluster so, eventually, the entire traffic moves from
the blue cluster to the green cluster without causing downtime for the topics.
# Goal
This PIP adds support to migrate and redirect the blue cluster’s traffic to
the green cluster. Therefore, the Broker will support admin-API using which
admin-user can mark migrate cluster along with redirection URLs where traffic
should be redirected. Broker persists migration state and new redirected
cluster’s URL as part of cluster metadata.
Once the cluster is marked as migrating, the broker asynchronously marks
each topic owned by that broker as migrated by calling the new managed-ledger
API `asyncMigrate()`. Once, the topic is marked as migrated, broker notifies
all the producers and consumers (which have drained the backlog for their
subscriptions) with a new client-protocol command called “Migrated-Topic” which
has redirection URLs to the green cluster. Producers and consumers for those
topics cache the redirection URLs and retry to connect to the broker with that
URL which redirects them to the green cluster.
Broker can redirect only those consumers which have reached to end of a
terminated topic or create a new subscription in the blue cluster. Therefore,
the broker can determine the redirection of consumers in the consumer-creation
phase and the pulsar client has to handle redirection after sending
producer/consumer creation requests.
Broker will unsubscribe the subscription once that subscription reaches end
of topic and broker will also not allow creation of any new producer or
subscription for the topics. Therefore, eventually, all the topics in the blue
cluster will not have a subscription or producer attached and eventually, those
topics will be deleted by the garbage collector.
Broker marks the cluster state as migration-completed once all the topics
are deleted and that cluster will not allow any new topic creation.
# Broker/Client changes
## Managed-Ledger Changes
This PIP will add API to managed-ledger to change ManagedLedger state as
migrated. This API will terminate the topic and persist the status of
managed-ledger as migrated. Broker triggers migrate API of managed-ledger once
cluster becomes the blue cluster and traffic should redirect to the green
cluster.
```
1. ManagedLedger.java
CompletableFuture<Position> asyncMigrate();
2. MLDataFormats.proto
message ManagedLedgerInfo {
// Flag to check if topic is terminated and migrated to different cluster
optional bool migrated = 4;
}
```
## Broker changes
1. Broker provides pulsar-admin API to mark the cluster in migration state.
**Pulsar Admin Change**
```
pulsar-admin clusters set-cluster-migrated \
–brokerServiceUrl <> \
–brokerServiceUrlTls <>
```
2. Broker stores cluster migration state and redirection urls in
cluster-metadata.
**Cluster-metadata change**
```
ClusterData.java
boolean migrated;
ClusterUrl migratedClusterUrl;
class ClusterUrl {
String serviceUrl;
String serviceUrlTls;
String brokerServiceUrl;
String brokerServiceUrlTls;
}
```
3. Broker runs periodic task to check cluster migration state and moves
topic into migration state by calling managed-ledger’s migration-api.
```
ServiceConfiguration.java
private long clusterMigrationCheckDurationSeconds = 0; //disable task with
default value=0
```
4. Broker sends topic migration message to client so, producer/consumer at
client side can handle redirection accordingly.
```
PulsarApi.proto
message CommandMigratedTopic {
required uint64 consumer_id = 1;
optional string brokerServiceUrl = 2;
optional string brokerServiceUrlTls = 3;
}
```
## Client Changes
Once producers and consumers receive the Migrated-Topic command with a list
of redirect URLs, they will cache those URLs and try to reconnect with a broker
by using those URLs. The client will add handling of the `CommandMigratedTopic`
protocol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
