rdhabalia opened a new issue #1582: Introduce pulsar replicator URL: https://github.com/apache/incubator-pulsar/issues/1582 ## Motivation Pulsar already supports geo-replication that persists messages across multiple clusters of pulsar instances. Therefore, client can set replication clusters for a topic, and pulsar broker internally takes care of replication to all the clusters. However, sometimes application may want to replicate the same published messages to other external systems which is not part of pulsar-eco system such as AWS-Kinesis, DynamoDB. Therefore, right now, client-application has to take this extra burden to publish same messages for pulsar and other external systems. Therefore, it will be useful to introduce server side replication that can replicate pulsar messages to external system without client intervention. Also server side replication should be extensible which can provide a plugin mechanism to add various replicators to support message-replication to different external systems. ## Requirement - Replicate Pulsar message to external system (eg: Kinesis, Kafka, another Pulsar cluster) - Easy onboarding: client should be able to add replicator configuration with CLI/Admin api and it should auto start appropriate replicators. - Isolation from core message bus - Pluggable framework and extensible to support multiple external system - New connectors should be developed with minimal efforts - Operability and monitoring - API to control start and stop individual replicator - API to get replicator stats - Security - Replicator framework should provide pluggable mechanism to plugin KeyStore implementation that can store and fetch client’s credentials which will be required to connect external system. **PIP:** https://github.com/apache/incubator-pulsar/wiki/PIP-18:-Pulsar-Replicator
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
