Thanks for your response, PengHui. I think this feature would be useful to end users for cluster management, which is why I want to contribute a first class feature instead of writing my own plugin that would add little value to the community.
> With the broker interceptor you can intercept all the REST API request and response, Pulsar commands between the broker and clients. Based on looking through the interceptor trait, I don't see a way to trigger events based on auto created/deleted topics. For example, when a producer connects to a broker for a nonexistent topic (assuming auto topic creation is allowed), a managed ledger, and thus a topic, is created without ever interacting with that interceptor trait. The same appears to be true for garbage collected topics. I think we'll need more than this interceptor to properly capture all cases where topics are created or deleted. Regarding my reference to potential further work, it does appear that low level auditing of connections and pulsar commands could be covered by the interceptor. However, it would still be on the end user to implement such functionality. Thanks, Michael On Wed, Apr 21, 2021 at 3:51 AM PengHui Li <codelipeng...@gmail.com> wrote: > Hi Michael, > > Currently, Pulsar supports a pluginable Broker Interceptor, you can find > it here > https://github.com/apache/pulsar/blob/6704f12104219611164aa2bb5bbdfc929613f1bf/pulsar-broker/src/main/java/org/apache/pulsar/broker/intercept/BrokerInterceptor.java > > With the broker interceptor you can intercept all the REST API request and > response, Pulsar commands between the broker and clients. > This can be used to audit the system events. > > Thanks, > Penghui > On Apr 21, 2021, 5:13 AM +0800, Michael Marshall <mikemars...@gmail.com>, > wrote: > > Hello all, > > > > I would like to propose adding a new feature to Pulsar that will require > a > > PIP. In addition to feedback on the proposed feature, I am looking for > > guidance on how to go about creating the PIP. Thanks for any help you can > > provide. > > > > I would like to add an optional system topic where topic creation and > topic > > deletion events are published. This feature will make it easier to > leverage > > the auto topic creation and inactive topic deletion features by > providing a > > way for users to reactively discover changes to topics. The largest > benefit > > is that users won't need to poll for these updates with an admin client. > > Instead, they will get them as messages. > > > > I looked to see if an equivalent feature already exists, but I don't see > > one. For reference, the `PatternMultiTopicsConsumerImpl` currently polls > > for all topics in a namespace and then does set operations to compute the > > "new" topics to which it should subscribe. This client implementation > could > > possibly leverage the new feature. > > > > There are still details I need to work out, like how it will work for > > partitioned vs unpartitioned topics and what kind of guarantees we have > > regarding messaging semantics (I think we'll want at least once message > > delivery here). I plan to include these details in the PIP with > discussions > > about trade offs for different implementations. > > > > Does this feature sound helpful and reasonable to others? If so, is the > > next step to formally write a proposal in a Google Doc or to put > together a > > doc on the Pulsar GitHub Wiki? > > > > Related and/or future work to consider in this design: I can see adding > > different system topics for these types of auditable system events. We > > currently rely on log lines as our primary way for end users to audit > > system events, e.g. a producer connecting to a broker or a subscription > > getting created, but we could instead have topics that represent streams > of > > these different kinds of events. A persistent topic could make these > audit > > events more durable and more structured which should lend themselves to > > being more easily analyzed. Further, users could choose to turn on/off > > these audit events, perhaps at the broker or namespace level, to fit > their > > own needs. > > > > Let me know what you think and how I should proceed. > > > > Regards, > > Michael Marshall >
