Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

2020-10-29 Thread Viktor Somogyi-Vass
Hi Tom. Sorry for the delay. Answering your points: > Why is it necessary to introduce this interface to produce the audit trail > when there is logging that can already record a lot of the same > information, albeit in less structured form? If logging isn't adequate it > would be good to

Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

2020-10-01 Thread Tom Bentley
Hi Viktor, Like Mickael, I can see that there's value in having an audit trail. For me the KIP raises a number of questions in its current form: Why is it necessary to introduce this interface to produce the audit trail when there is logging that can already record a lot of the same information,

Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

2020-10-01 Thread Dániel Urbán
Hi Viktor, I think the current state of the proposal is flexible enough to support use-cases where the response data is of interest to the auditor. This part ensures that: "... doing the auditing before sending the response back ...". Additionally, event classes could be extended with additional

Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

2020-09-30 Thread Viktor Somogyi-Vass
Hi Daniel, I think in this sense we can use the precedence set with the KAfkaAdminClient. It has *Result and *Options classes which in this interpretation are similar in versioning and usage as they transform and convey the responses of the protocol in a minimalistic API. I've modified the KIP a

Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

2020-09-22 Thread Dániel Urbán
An example I had in mind was the ProduceResponse - the auditor might need access to the new end offset of the partitions. The event-based approach sounds good - new events and fields can be added on-demand. Do we need the same versioning strategy we use with the requests/responses? Daniel Viktor

Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

2020-09-21 Thread Viktor Somogyi-Vass
Hi Daniel, > If the auditor needs access to the details of the action, one could argue that even the response should be passed down to the auditor. At this point I don't think we need to include responses into the interface but if you have a use-case we can consider doing that. > Is it feasible

Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

2020-09-18 Thread Dániel Urbán
Hi, Thanks for the KIP. If the auditor needs access to the details of the action, one could argue that even the response should be passed down to the auditor. Is it feasible to convert the Java requests and responses to public API? If not, do we have another option to access this info in the

Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

2020-09-16 Thread Viktor Somogyi-Vass
One more after-thought on your second point (AbstractRequest): the reason I introduced it in the first place was that this way implementers can access request data. A use case can be if they want to audit a change in configuration or client quotas but not just acknowledge the fact that such an

Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

2020-09-16 Thread Viktor Somogyi-Vass
Hi Mickael, Thanks for reviewing the KIP. 1.) I just wanted to follow the conventions used with the Authorizer as it is built in a similar fashion, although it's true that in KafkaServer we call the configure() method and the start() in the next line. This would be the same in Auditor and even

Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

2020-09-14 Thread Mickael Maison
Hi Viktor, Thanks for restarting the discussion on this KIP. Being able to easily audit usage of a Kafka cluster is a very valuable feature. Regarding the API, I have a few of questions: 1) You introduced a start() method. I don't think any other interfaces have such a method. Users can do any

Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

2020-09-14 Thread Viktor Somogyi-Vass
Hi everyone, Changed the interface a little bit to accommodate methods better where authorization happens for multiple operations so the implementer of the audit interface will receive all authorizations together. I'll wait a few more days to allow people to react or give feedback but if there

[DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

2020-09-08 Thread Viktor Somogyi-Vass
Hi Everyone, I'd like to restart the discussion on this. Since the KIP has been revamped I thought I'd start a new discussion thread. Link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-567%3A+Kafka+Cluster+Audit Short summary: - Would like to introduce a new interface similar to the