Hi all, I am starting a proposal about adding change events for a namespace to support topic level policy.
*motivation* Pulsar is a multi-tenant event streaming system. Administrators can manage the tenants and namespaces by setting policies at different levels. However the policies such as retention policy, storage quota policy are only able to be set at a namespace level. In many use cases, users would like to have the ability to override a policy at the topic level. But Pulsar doesn’t support topic level policy. So they ended up mapping topic to namespaces, which result in using one namespace per topic and it is only not efficient in managing many namespaces and topics. I am proposing introducing `__change_events` topic for namespaces. This `__change_events` topic can be used for implementing topic-level policy in an efficient way. In this proposal, it uses Pulsar as an event log for storing namespace change events (the events are topic policy changes in this case). This approach has a few benefits: - It is moving away from using zookeeper to store more metadata and preventing introducing more traffic to zookeeper. - It can scale efficiently by using Pulsar as the event log to propagate policy cache. - Once the policy changes land in the `__change_events` topic, administrators can use Pulsar Functions or Presto SQL to query the changes and audit the whole system. The detail proposal is in https://docs.google.com/document/d/1kBiICo3lUvWfeah1YVNSn2Y6qkI6l5-S3WZv5PZcbwk/edit PIP-39: https://github.com/apache/pulsar/wiki/PIP-39:-Namespace-Change-Events Looking forward to any feedback. Thanks, Sijie
