rdhabalia opened a new issue #1582: Introduce pulsar replicator
Pulsar already supports geo-replication that persists messages across
multiple clusters of pulsar instances. Therefore, client can set replication
clusters for a topic, and pulsar broker internally takes care of replication to
all the clusters. However, sometimes application may want to replicate the same
published messages to other external systems which is not part of pulsar-eco
system such as AWS-Kinesis, DynamoDB. Therefore, right now, client-application
has to take this extra burden to publish same messages for pulsar and other
Therefore, it will be useful to introduce server side replication that can
replicate pulsar messages to external system without client intervention. Also
server side replication should be extensible which can provide a plugin
mechanism to add various replicators to support message-replication to
different external systems.
- Replicate Pulsar message to external system (eg: Kinesis, Kafka, another
- Easy onboarding: client should be able to add replicator configuration
with CLI/Admin api and it should auto start appropriate replicators.
- Isolation from core message bus
- Pluggable framework and extensible to support multiple external system
- New connectors should be developed with minimal efforts
- Operability and monitoring
- API to control start and stop individual replicator
- API to get replicator stats
- Replicator framework should provide pluggable mechanism to plugin
KeyStore implementation that can store and fetch client’s credentials which
will be required to connect external system.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
Apache Git Services