Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[Don Bosco Durai]

Sounds good to me.

Thanks

Bosco


On 9/6/19, 11:47 AM, "Colin McCabe"  wrote:

Hi Rajini,

I agree that there is probably some value in async createAcls and 
deleteAcls methods.  It's also a smaller change to use a futures-based 
interface there, since it really only affects the CreateAcls and DeleteAcls 
operations.  It's probably fine for the other methods to be sync for now we 
would need a longer discussion about how to manage and monitor the 
authorization purgatory to make this async, and it's not clear that we need it 
(yet?)

best,
Colin


On Fri, Sep 6, 2019, at 11:32, Rajini Sivaram wrote:
> I would suggest that we go with synchronous authorize() and acls() method
> to describe ACLs, but asynchronous createAcls() and deleteAcls() that
> return CompletionStage. With KIP-500, I think async create/delete methods
> will be useful. For now, we can assume that brokers are able to cache all
> ACLs. That keeps the API and implementation simple, while still avoiding
> blocking request threads for the expensive update operations. Even though
> we don't expect large rates of concurrent ACL updates, blocking request
> threads during updates (e.g. to a database) isn't ideal. Since custom
> authorizers can still do synchronous updates by returning completed
> futures, would this be a reasonable change to the API?
> 
> Thanks,
> 
> Rajini
    > 
> 
> On Fri, Sep 6, 2019 at 5:32 PM Don Bosco Durai  wrote:
> 
> > +1
> >
> > I might be wrong here, but here are few of my assumptions..
> >
> > 1. Unlike transactional services, in Kafka most of the users are
> > application users, so once the connection is established, then they are
> > going to be long running sessions. So the caching is more predictable 
and
> > heavy lifting can be done during initial session setup and cache can be
> > frequently updated using background thread. KSQL might have a different
> > flow, but I don't have enough datapoints for it.
> > 2. For publish/consume authorization calls, providing async option will 
be
> > misleading for Kafka like load
> > 3. For admin calls, hopefully no one does hundreds of call in multiple
> > thread within few seconds. This would be a bad design.
> >
> > So I feel, we should be okay with sync calls for now. And consider async
> > in the future if needed. At least in Apache Ranger implementation, we 
would
> > be okay using sync operation. I am not sure if any other plugin
> > implementation would benefit from async implementation.
> >
> > Thanks
> >
> > Bosco
> >
> > On 9/6/19, 8:45 AM, "Ismael Juma"  wrote:
> >
> > One more thing: if people have strong opinions that authorize has 
to be
> > sync, we could leave it like that for now. If needed, we can later 
add
> > an
> > authorizeAsync with another method returning a Boolean indicating 
that
> > it's
> > supported. That is, there is a path to evolve the interface (even 
if a
> > bit
> > ugly).
> >
> > What about the other methods, is there consensus that they should be
> > updated to return CompletionStage?
> >
> > Ismael
> >
> > On Fri, Sep 6, 2019, 8:32 AM Ismael Juma  wrote:
> >
> > > I'm on the move, but what Rajini said seems reasonable. I don't 
think
> > > using SSDs solves the issue. They can still hang for seconds when
> > they
> > > fail. Also, many people may not have local SSDs available (remote
> > SSDs like
> > > EBS hang for tens of seconds when there are issues).
> > >
> > > We are currently vulnerable to all of these in the normal 
read/write
> > path,
> > > but it seems suboptimal to continue assuming non blocking behavior
> > for
> > > things that actually do block.
> > >
> > > Ismael
> > >
> > > On Fri, Sep 6, 2019, 8:24 AM Rajini Sivaram 
 > >
> > > wrote:
> > >
> > >> Hi Jun,
> > >>
> > >> For ACLs, the size is also dependent on the number of users. So 
in
> > >> deployments with large number of users where some users are not
> > active, an
> > >> LRU cache may be useful. Agree that there may be other solutions
&g

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[Don Bosco Durai]

t;> ZK-based
>> > > > > implementation as well as any custom implementations that don't
>> want
>> > to
>> > > > be
>> > > > > async will just need to return a sync'ed value. So instead of
>> > > returning `
>> > > > > value`, the code would just return
>> > > > > `CompletableFuture.completedFuture(value)
>> > > > > `. So it would be just a single line change in the implementation
>> > with
>> > > > the
>> > > > > new API. The caller would treat completedFuture exactly as it 
does
>> > > today,
>> > > > > processing the request synchronously without using a purgatory.
>> > > > >
>> > > > > 3) For implementations that return a completedFuture as described
>> in
>> > > 2),
>> > > > > the behaviour would remain exactly the same. No additional
    >> threads or
>> > > > > purgatory will be used for this case. So there would be no
>> > performance
>> > > > > penalty. For implementations that return a future that is not
>> > complete,
>> > > > we
>> > > > > prioritise running more requests concurrently. So in a deployment
>> > with
>> > > a
>> > > > > large number of clients, we would improve performance by allowing
>> > other
>> > > > > requests to be processed on the request threads while some are
>> > waiting
>> > > > for
>> > > > > authorization metadata.
>> > > > >
>> > > > > 4) I was concerned about this too. The goal is to make the API
>> > flexible
>> > > > > enough to handle large scale deployments in future when caching
>> all
>> > > > > authorization metadata in each broker is not viable. Using an
>> async
>> > API
>> > > > > that returns CompletionStage, the caller has the option to handle
>> the
>> > > > > result synchronously or asynchronously, so we don't necessarily
>> need
>> > to
>> > > > > update the calling code right away. Custom authorizers using the
>> > async
>> > > > API
>> > > > > have full control over whether authorization is performed in-line
>> > since
>> > > > > completedFuture will always be handled synchronously. We do need
>> to
>> > > > update
>> > > > > KafkaApis to take advantage of the asynchronous API to improve
>> scale.
>> > > > Even
>> > > > > though this is a big change, since we will be doing the same for
>> all
>> > > > > requests, it shouldn't be too hard to maintain since the same
>> pattern
>> > > > will
>> > > > > be used for all requests.
>> > > > >
>> > > > > Regards,
>> > > > >
>> > > > > Rajini
>> > > > >
>> > > > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai > >
>> > > > wrote:
>> > > > >
>> > > > > > Hi Rajini
>> > > > > >
>> > > > > > Help me understand this a bit more.
>> > > > > >
>> > > > > > 1. For all practical purpose, without authorization you can't
>> go to
>> > > the
>> > > > > > next step. The calling code needs to block anyway. So should we
>> > just
>> > > > let
>> > > > > > the implementation code do the async part?
>> > > > > > 2. If you feel management calls need to be async, then we 
should
>> > > > consider
>> > > > > > another set of async APIs. I don't feel we should complicate it
>> > > > further (
>> > > > > > 3. Another concern I have is wrt performance. Kafka has been
>> built
>> > to
>> > > > > > handle 1000s per second requests. Not sure whether making it
>> async
>> > > will
>> > > > > add
>> > > > &

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[Don Bosco Durai]

Hi Rajini

Overall, it looks good. 

Thanks

Bosco


On 9/4/19, 3:39 AM, "Rajini Sivaram"  wrote:

Hi Colin & Don,

I have submitted a PR to help with reviewing the new API:
https://github.com/apache/kafka/pull/7293/files.

   - Authorizer.java contains the updated API and includes javadoc
   update including the threading model.
   - AclAuthorizer.scala changes in the PR should give an idea of the
   implementation changes for custom authorizers that want to continue to be
   synchronous.
   - KafkaApis.scala continues to use the API synchronously for now. It can
   be updated later.


Thank you,

Rajini


On Wed, Sep 4, 2019 at 9:38 AM Rajini Sivaram 
wrote:

> Hi Don,
>
> Thanks for your note.
>
> 1) The intention is to avoid blocking in the calling thread. We already
> have several requests that are put into a purgatory when waiting for 
remote
> communication, for example produce request waiting for replication. Since
> we have a limited number of request threads in the broker, we want to make
> progress with other requests, while one is waiting on any form of remote
> communication.
>
> 2) Async management calls will be useful with the default authorizer when
> KIP-500 removes ZK and we rely on Kafka instead. Our current ZK-based
> implementation as well as any custom implementations that don't want to be
> async will just need to return a sync'ed value. So instead of returning `
> value`, the code would just return `CompletableFuture.completedFuture
> (value)`. So it would be just a single line change in the implementation
> with the new API. The caller would treat completedFuture exactly as it
> does today, processing the request synchronously without using a 
purgatory.
>
> 3) For implementations that return a completedFuture as described in 2),
> the behaviour would remain exactly the same. No additional threads or
> purgatory will be used for this case. So there would be no performance
> penalty. For implementations that return a future that is not complete, we
> prioritise running more requests concurrently. So in a deployment with a
> large number of clients, we would improve performance by allowing other
> requests to be processed on the request threads while some are waiting for
> authorization metadata.
>
> 4) I was concerned about this too. The goal is to make the API flexible
> enough to handle large scale deployments in future when caching all
> authorization metadata in each broker is not viable. Using an async API
> that returns CompletionStage, the caller has the option to handle the
> result synchronously or asynchronously, so we don't necessarily need to
> update the calling code right away. Custom authorizers using the async API
> have full control over whether authorization is performed in-line since
> completedFuture will always be handled synchronously. We do need to
> update KafkaApis to take advantage of the asynchronous API to improve
> scale. Even though this is a big change, since we will be doing the same
> for all requests, it shouldn't be too hard to maintain since the same
> pattern will be used for all requests.
>
> Regards,
>
> Rajini
>
> On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai  wrote:
>
>> Hi Rajini
>>
>> Help me understand this a bit more.
>>
>> 1. For all practical purpose, without authorization you can't go to the
>> next step. The calling code needs to block anyway. So should we just let
>> the implementation code do the async part?
>> 2. If you feel management calls need to be async, then we should consider
>> another set of async APIs. I don't feel we should complicate it further (
>> 3. Another concern I have is wrt performance. Kafka has been built to
>> handle 1000s per second requests. Not sure whether making it async will 
add
>> any unnecessary overhead.
>> 4. How much complication would this add on the calling side? And is it
>> worth it?
>>
>> Thanks
>>
>> Bosco
>>
>>
>> On 9/3/19, 8:50 AM, "Rajini Sivaram"  wrote:
>>
>> Hi all,
>>
>> Ismael brought up a point that it will be good to make the Authorizer
>> interface asynchronous to avoid blocking request threads during 
remote
>> operations.
>>
>> 1) Since we want to support different backends for authorization
>> metadata,
   

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[Don Bosco Durai]

Hi Rajini

Help me understand this a bit more.

1. For all practical purpose, without authorization you can't go to the next 
step. The calling code needs to block anyway. So should we just let the 
implementation code do the async part?
2. If you feel management calls need to be async, then we should consider 
another set of async APIs. I don't feel we should complicate it further ( 
3. Another concern I have is wrt performance. Kafka has been built to handle 
1000s per second requests. Not sure whether making it async will add any 
unnecessary overhead.
4. How much complication would this add on the calling side? And is it worth it?

Thanks

Bosco


On 9/3/19, 8:50 AM, "Rajini Sivaram"  wrote:

Hi all,

Ismael brought up a point that it will be good to make the Authorizer
interface asynchronous to avoid blocking request threads during remote
operations.

1) Since we want to support different backends for authorization metadata,
making createAcls() and deleteAcls() asynchronous makes sense since these
always involve remote operations. When KIP-500 removes ZooKeeper, we would
want to move ACLs to Kafka and async updates will avoid unnecessary
blocking.
2) For authorize() method, we currently use cached ACLs in the built-in
authorizers, so synchronous authorise operations work well now. But async
authorize() would support this scenario as well as authorizers in large
organisations where an LRU cache would enable a smaller cache even when the
backend holds a large amount of ACLs for infrequently used resources or
users who don't use the system frequently.

For both cases, the built-in authorizer will continue to be synchronous,
using CompletableFuture.completedFuture() to return the actual result. But
async API will make custom authorizer implementations more flexible. I
would like to know if there are any concerns with these changes before
updating the KIP.

*Proposed API:*
public interface Authorizer extends Configurable, Closeable {

Map> start(AuthorizerServerInfo 
serverInfo);
List>
authorize(AuthorizableRequestContext requestContext, List
actions);
List>
createAcls(AuthorizableRequestContext requestContext, List
aclBindings);
List>
deleteAcls(AuthorizableRequestContext requestContext,
List aclBindingFilters);
CompletionStage> acls(AclBindingFilter filter);
}


Thank you,

Rajini

On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai  wrote:

> Hi Rajini
>
> Thanks for clarifying. I am good for now.
>
> Regards
>
> Bosco
>
>
> On 8/16/19, 11:30 AM, "Rajini Sivaram"  wrote:
>
> Hi Don,
>
> That should be fine. I guess Ranger loads policies from the database
> synchronously when the authorizer is configured, similar to
> SimpleAclAuthorizer loading from ZooKeeper. Ranger can continue to 
load
> synchronously from `configure()` or `start()` and return an empty map
> from
> `start()`. That would retain the existing behaviour.. When the same
> database stores policies for all listeners and the policies are not
> stored
> in Kafka, there is no value in making the load asynchronous.
    >
    > Regards,
>
> Rajini
>
>
> On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai 
> wrote:
>
> > Hi Rajini
> >
> > Assuming this doesn't affect custom plugins, I don't have any
> concerns
> > regarding this.
> >
> > I do have one question regarding:
> >
> > "For authorizers that don’t store metadata in ZooKeeper, ensure that
> > authorizer metadata for each listener is available before starting
> up the
> > listener. This enables different authorization metadata stores for
> > different listeners."
> >
> > Since Ranger uses its own database for storing policies, do you
> anticipate
> > any issues?
> >
> > Thanks
> >
> > Bosco
> >
> >
> > On 8/16/19, 6:49 AM, "Rajini Sivaram" 
> wrote:
> >
> > Hi all,
> >
> > I made another change to the KIP. The KIP was originally
> proposing to
> > extend SimpleAclAuthorizer to also implement the new API (in
> addition
> > to
> > the existing API). But since we use the new API when available,
> this
> > can
>

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[Don Bosco Durai]

Hi Rajini

Thanks for clarifying. I am good for now.

Regards

Bosco


On 8/16/19, 11:30 AM, "Rajini Sivaram"  wrote:

Hi Don,

That should be fine. I guess Ranger loads policies from the database
synchronously when the authorizer is configured, similar to
SimpleAclAuthorizer loading from ZooKeeper. Ranger can continue to load
synchronously from `configure()` or `start()` and return an empty map from
`start()`. That would retain the existing behaviour.. When the same
database stores policies for all listeners and the policies are not stored
in Kafka, there is no value in making the load asynchronous.

Regards,

Rajini


On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai  wrote:

> Hi Rajini
>
> Assuming this doesn't affect custom plugins, I don't have any concerns
> regarding this.
>
> I do have one question regarding:
>
> "For authorizers that don’t store metadata in ZooKeeper, ensure that
> authorizer metadata for each listener is available before starting up the
> listener. This enables different authorization metadata stores for
> different listeners."
>
> Since Ranger uses its own database for storing policies, do you anticipate
> any issues?
>
> Thanks
>
> Bosco
>
>
> On 8/16/19, 6:49 AM, "Rajini Sivaram"  wrote:
>
> Hi all,
>
> I made another change to the KIP. The KIP was originally proposing to
> extend SimpleAclAuthorizer to also implement the new API (in addition
> to
> the existing API). But since we use the new API when available, this
> can
> break custom authorizers that extend this class and override specific
> methods of the old API. To avoid breaking any existing custom
> implementations that extend this class, particularly because it is in
> the
> public package kafka.security.auth, the KIP now proposes to retain the
> old
> authorizer as-is.  SimpleAclAuthorizer will continue to implement the
> old
> API, but will be deprecated. A new authorizer implementation
> kafka.security.authorizer.AclAuthorizer will be added for the new API
> (this
> will not be in the public package).
>
> Please let me know if you have any concerns.
>
> Regards,
>
> Rajini
>
>
> On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
> rajinisiva...@gmail.com>
> wrote:
>
> > Thanks Colin.
> >
> > If there are no other concerns, I will start vote later today. Many
> thanks
> > to every one for the feedback.
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe 
> wrote:
> >
> >> Thanks, Rajini.  It looks good to me.
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram wrote:
> >> > Hi Colin,
> >> >
> >> > Thanks for the review. I have updated the KIP to move the
> interfaces for
> >> > request context and server info to the authorizer package. These
> are now
> >> > called AuthorizableRequestContext and AuthorizerServerInfo.
> Endpoint is
> >> now
> >> > a class in org.apache.kafka.common to make it reusable since we
> already
> >> > have multiple implementations of it. I have removed requestName
> from the
> >> > request context interface since authorizers can distinguish
> follower
> >> fetch
> >> > and consumer fetch from the operation being authorized. So 16-bit
> >> request
> >> > type should be sufficient for audit logging.  Also replaced
> AuditFlag
> >> with
> >> > two booleans as you suggested.
> >> >
> >> > Can you take another look and see if the KIP is ready for voting?
> >> >
> >> > Thanks for all your help!
> >> >
> >> > Regards,
> >> >
> >> > Rajini
> >> >
> >> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe 
> >> wrote:
> >> >
>

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[Don Bosco Durai]

ifically for the authorizer.
>> > >
>> > > I agree that we probably don't have enough information preserved for
>> > > requests currently to always know what entity made them.  So we can
>> leave
>> > > that out for now (except in the special case of Fetch).  Perhaps we
>> can add
>> > > this later if it's needed.
>> > >
>> > > I understand the intention behind AuthorizationMode (which is now
>> called
>> > > AuditFlag in the latest revision).  But it still feels complex.  What
>> if we
>> > > just had two booleans in Action: logSuccesses and logFailures?  That
>> seems
>> > > to cover all the cases here.  MANDATORY_AUTHORIZE = true, true.
>> > > OPTIONAL_AUTHORIZE = true, false.  FILTER = true, false.
>> LIST_AUTHORIZED =
>> > > false, false.  Would there be anything lost versus having the enum?
>> > >
>> > > best,
>> > > Colin
>> > >
>> > >
>> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison wrote:
>> > > > Hi Rajini,
>> > > >
>> > > > Thanks for the KIP!
>> > > > I really like that authorize() will be able to take a batch of
>> > > > requests, this will speed up many implementations!
>> > > >
>> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <
>> rajinisiva...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > Thanks David! I have fixed the typo.
>> > > > >
>> > > > > Also made a couple of changes to make the context interfaces more
>> > > generic.
>> > > > > KafkaRequestContext now returns the 16-bit API key as Colin
>> suggested
>> > > as
>> > > > > well as the friendly name used in metrics which are useful in
>> audit
>> > > logs.
>> > > > > `Authorizer#start` is now provided a generic `BrokerInfo`
>> interface
>> > > that
>> > > > > gives cluster id, broker id and endpoint information. The generic
>> > > interface
>> > > > > can potentially be used in other broker plugins in future and
>> provides
>> > > > > dynamically generated configs like broker id and ports which are
>> > > currently
>> > > > > not available to plugins unless these configs are statically
>> > > configured.
>> > > > > Please let me know if there are any concerns.
>> > > > >
>> > > > > Regards,
>> > > > >
>> > > > > Rajini
>> > > > >
>> > > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot 
>> > > wrote:
>> > > > >
>> > > > > > Hi Rajini,
>> > > > > >
>> > > > > > Thank you for the update! It looks good to me. There is a typo
>> in the
>> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` ->
>> `MANDATORY_AUTHORIZE`.
>> > > > > >
>> > > > > > Regards,
>> > > > > > David
>> > > > > >
>> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <
>> > > rajinisiva...@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi David,
>> > > > > > >
>> > > > > > > Thanks for reviewing the KIP! Since questions about
>> `authorization
>> > > mode`
>> > > > > > > and `count` have come up multiple times, I have renamed both.
>> > > > > > >
>> > > > > > > 1) Renamed `count` to `resourceReferenceCount`. It is the
>> number
>> > > of times
>> > > > > > > the resource being authorized is referenced within the
>> request.
>> > > > > > >
>> > > > > > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided
>> to
>> > > improve
>> > > > > > > audit logging in the authorizer. The enum va

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[Don Bosco Durai]

Hi Rajini

3.2 - This makes sense. Thanks for clarifying. 

Rest looks fine. Once the implementations are done, it will be more clear on 
the different values RequestType and Mode.

Thanks

Bosco


On 8/9/19, 5:08 AM, "Rajini Sivaram"  wrote:

Hi Don,

Thanks for the suggestions. A few responses below:

3.1 Can rename and improve docs if we keep this. Let's finish the
discussion on Colin's suggestions regarding this first.
3.2 No, I was thinking of some requests that have an old way of authorizing
and a new way where we have retained the old way for backward
compatibility. One example is Cluster:Create permission to create topics.
We have replaced this with fine-grained topic create access using 
Topic:Create
for topic patterns. But we still check if user has Cluster:Create first. If
Denied, the deny is ignored and we check Topic:Create. We dont want to log
DENY for Cluster:Create at INFO level for this, since this is not a
mandatory ACL for creating topics. We will get appropriate logs from the
subsequent Topic:Create in this case.
3.3 They are not quite the same. FILTER implies that user actually
requested access to and performed some operation on the filtered resources.
LIST_AUTHORZED did not result in any actual access. User simply wanted to
know what they are allowed to access.
3.4 Request types are Produce, JoinGroup, OffsetCommit etc. So that is
different from authorization mode, operation etc.


On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai  wrote:

> Hi Rajini
>
> Thanks for clarifying. This is very helpful...
>
> #1 - On the Ranger side, we should be able to handle multiple requests at
> the same time. I was just not sure how much processing overhead will be
> there on the Broker side to split and then consolidate the results. If it
> is negligible,  then this is the better way. It will make it future proof.
> #2 -  I agree, having a single "start" call makes it cleaner. The
> Authorization implementation will only have to do the initialization only
> once.
> #3.1 - Thanks for the clarification. I think it makes sense to have this.
> The term "Mode" might not be explicit enough. Essentially it seems you 
want
> the Authorizer to know the Intent/Purpose of the authorize call and let 
the
> Authorizer decide what to log as Audit event. Changing the class/field 
name
> or giving more documentation will do.
> #3.2 - Regarding the option "OPTIONAL". Are you thinking from chaining
> multiple Authorizer? If so,  I am not sure whether the Broker would have
> enough information to make that decision. I feel the Authorizer will be 
the
> one who would have that knowledge. E.g. in Ranger we have explicit deny,
> which means no matter what, the request should be denied for the 
user/group
> or condition. So if you are thinking of chaining Authorizers, then
> responses should have the third state, e.g. "DENIED_FINAL", in which case
> if there is an Authorization chain, it will be stop and the request will 
be
> denied and if it is just denied, then you might fall back to next
> authorizer. If we don't have chaining of Authorizing in mind, then we
> should reconsider OPTIONAL for now. Or clarify under which scenario
> OPTIONAL will be called.
> #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't LIST_AUTHORIZED a
> special case for "FILTER"?
> #3.4 KafkaRequestContext. requestType() v/s Action. authorizationMode. I
> am not sure about the overlap or ambiguity.
> #4 - Cool. This is good, it will be less stress on the Authorizer. Ranger
> already supports the "count" concept and also has batching capability to
> aggregate similar requests to reduce the number of audit logs to write. We
> should be able to pass this through.
> #5 - Assuming if the object instance is going out of scope, we should be
> fine. Not a super important ask. Ranger is already catching KILL signal 
for
> clean up.
>
> Thanks again. These are good enhancements. Appreciate your efforts here.
>
> Bosco
>
>
>
> On 8/8/19, 2:03 AM, "Rajini Sivaram"  wrote:
>
> Hi Don,
>
> Thanks for reviewing the KIP.
>
> 1. I had this originally as a single Action, but thought it may be
> useful
> to support batched authorize calls as well and keep it consistent with
> other methods. Single requests can contain multiple topics. For
> example a
> produce request can contain records for several partitions of 
different
> topics. Broker could po

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[Don Bosco Durai]

ose() to flush
audits?

On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai  wrote:

> Rajini
>
> Thanks for putting this together. It is looking good. I have few
> questions...
>
> 1. List authorize(..., List actions).  Do you
> see a scenario where the broker will call authorize for multiple topics at
> the same time? I can understand that during creating/deleting ACLS,
> multiple permissions for multiple resources might be done. For authorize
> call, would this be a case? And does the Authorize implementation will be
> able to do performance optimization because of this? Or should we just 
keep
> it simple? I don't see it as an issue from Apache Ranger side, but just
> checking to see whether we need to be aware of something.
> 2. Should I assume that the SecurityProtocol passed during start and the
> one return by KafkaRequestContext.securityProtocol() will be the same?
> CompletableFuture start(String listenerName, SecurityProtocol
> securityProtocol);
> KafkaRequestContext.securityProtocol()
> 3. What is the purpose of AuthorizationMode? How does the broker decide
> what mode to use when the authorize() method is called?
> 4. Can we clarify "count" in Action a bit more? How is it used?
> 5. Do you feel having "stop" along with "start" be helpful? E.g. In Ranger
> we try to optimize the Audit writing by caching the logs for a fixed
> interval. But when the Broker terminates, we do a forced flush. Having an
> explicit "stop" might give us a formal way to flush our audits.
>
> Thanks
>
> Bosco
>
> On 8/7/19, 3:59 PM, "Rajini Sivaram"  wrote:
>
> Hi Ron/Harsha/Satish,
>
> Thanks for reviewing the KIP!
>
> We should perhaps have a wider discussion outside this KIP for
> refactoring
> clients so that others who are not following this KIP also notice the
> discussion. Satish, would you like to start a discussion thread on 
dev?
>
> Regards,
>
> Rajini
>
>
> On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
> satish.dugg...@gmail.com>
> wrote:
>
> > I felt the same need when we want to add a pluggable API for core
> > server functionality. This does not need to be part of this KIP, it
> > can be a separate KIP. I can contribute those refactoring changes if
> > others are OK with that.
> >
> > It is better to have a structure like below.
> >
> > kafka-common:
> > common classes which can be used in any of the other modules in 
Kafka
> > like client, Kafka-server-common and server etc.
> >
> > kafka-client-common:
> > common classes which can be used in the client module. This can be
> > part of client module itself.
> >
> > kafka-server-common:
> > classes required only for kafka-server.
> >
> > Thanks.
> > Satish.
> >
> > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani 
> > wrote:
> > >
> > > Thanks for the KIP Rajini.
> > > Quick thought, it would be good to have a common module outside of
> > clients
> > > that only applies to server side interfaces & changes. It looks
> like we
> > are
> > > increasingly in favor of using Java interface for pluggable
> modules  on
> > the
> > > broker side.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have created a KIP to replace the Scala Authorizer API with a
> new
> > Java
> > > > API:
> > > >
> > > > -
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > >
> > > > This is replacement for KIP-50 which was accepted but never
> merged.
> > Apart
> > > > from moving to a Java API consistent with other pluggable
> interfaces
> > in the
> > > > broker, KIP-504 also attempts to address known limitations in 
the
> > > > authorizer. If you have come across other limitations that you
> would
> > like
> > > > to see addressed in the new API, please raise these on the
> discussion
> > > > thread so that we can consider those too. All suggestions and
> feedback
> > are
> > > > welcome.
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> >
>
>
>
>

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[Don Bosco Durai]

Rajini

Thanks for putting this together. It is looking good. I have few questions...

1. List authorize(..., List actions).  Do you see 
a scenario where the broker will call authorize for multiple topics at the same 
time? I can understand that during creating/deleting ACLS, multiple permissions 
for multiple resources might be done. For authorize call, would this be a case? 
And does the Authorize implementation will be able to do performance 
optimization because of this? Or should we just keep it simple? I don't see it 
as an issue from Apache Ranger side, but just checking to see whether we need 
to be aware of something.
2. Should I assume that the SecurityProtocol passed during start and the one 
return by KafkaRequestContext.securityProtocol() will be the same?
CompletableFuture start(String listenerName, SecurityProtocol 
securityProtocol);
KafkaRequestContext.securityProtocol()
3. What is the purpose of AuthorizationMode? How does the broker decide what 
mode to use when the authorize() method is called?
4. Can we clarify "count" in Action a bit more? How is it used?
5. Do you feel having "stop" along with "start" be helpful? E.g. In Ranger we 
try to optimize the Audit writing by caching the logs for a fixed interval. But 
when the Broker terminates, we do a forced flush. Having an explicit "stop" 
might give us a formal way to flush our audits.

Thanks

Bosco

On 8/7/19, 3:59 PM, "Rajini Sivaram"  wrote:

Hi Ron/Harsha/Satish,

Thanks for reviewing the KIP!

We should perhaps have a wider discussion outside this KIP for refactoring
clients so that others who are not following this KIP also notice the
discussion. Satish, would you like to start a discussion thread on dev?

Regards,

Rajini


On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana 
wrote:

> I felt the same need when we want to add a pluggable API for core
> server functionality. This does not need to be part of this KIP, it
> can be a separate KIP. I can contribute those refactoring changes if
> others are OK with that.
>
> It is better to have a structure like below.
>
> kafka-common:
> common classes which can be used in any of the other modules in Kafka
> like client, Kafka-server-common and server etc.
>
> kafka-client-common:
> common classes which can be used in the client module. This can be
> part of client module itself.
>
> kafka-server-common:
> classes required only for kafka-server.
>
> Thanks.
> Satish.
>
> On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani 
> wrote:
> >
> > Thanks for the KIP Rajini.
> > Quick thought, it would be good to have a common module outside of
> clients
> > that only applies to server side interfaces & changes. It looks like we
> are
> > increasingly in favor of using Java interface for pluggable modules  on
> the
> > broker side.
> >
> > Thanks,
> > Harsha
> >
> >
> > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram  >
> > wrote:
> >
> > > Hi all,
> > >
> > > I have created a KIP to replace the Scala Authorizer API with a new
> Java
> > > API:
> > >
> > > -
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > >
> > > This is replacement for KIP-50 which was accepted but never merged.
> Apart
> > > from moving to a Java API consistent with other pluggable interfaces
> in the
> > > broker, KIP-504 also attempts to address known limitations in the
> > > authorizer. If you have come across other limitations that you would
> like
> > > to see addressed in the new API, please raise these on the discussion
> > > thread so that we can consider those too. All suggestions and feedback
> are
> > > welcome.
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
>

Re: [DISCUSS] KIP-433: Provide client API version to authorizer

[Don Bosco Durai]

In general, making authorization decision based on client version might not be 
a good idea. It will just provide a loop hole for someone to downgrade their 
client version to exploit or workaround any restrictions.

> > propose a single broker dynamic configuration: client.min.api.version, to
> On the other hand, most users are not likely to write a custom authorizer to 
> try to take advantage of version information being passed to the authorizer.  
These could be Authorizer implementation detail. If needed, native 
implementation could use this property to make the decision, while the custom 
authorizer can use this property or come up with their own set of properties to 
support or not support certain checks or features. Regardless, it would be good 
to have checks consistent across the version.

> Another thing to consider is that if we add a new broker configuration that 
> lets us set a minimum client version which is allowed, that could be useful 
> to other users as well. 
I am not sure whether the client version is available during authorization API 
call. It might be good if we can send these information in a generic name/value 
list in session context. In that way,  the authorization implementation can use 
it the way the way want to or ignore it.

Bosco


On 2/26/19, 10:04 AM, "Colin McCabe"  wrote:

Hi Harsha,

I think Ismael and Gwen here bring up a good point.  The version of the 
request is a technical detail that isn't really related to authorization.  
There are a lot of other technical details like this like the size of the 
request, the protocol it came in on, etc.  None of them are passed to the 
authorizer-- they all have configuration knobs to control how we handle them.  
If we add this technical detail, logically we'll have to start adding all the 
others, and the authorizer API will get really bloated.  It's better to keep it 
focused on authorization, I think.

Another thing to consider is that if we add a new broker configuration that 
lets us set a minimum client version which is allowed, that could be useful to 
other users as well.  On the other hand, most users are not likely to write a 
custom authorizer to try to take advantage of version information being passed 
to the authorizer.  So, I think  using a configuration is clearly the better 
way to go here.  Perhaps it can be a KIP-226 dynamic configuration to make this 
easier to deploy?

cheers,
Colin


On Mon, Feb 25, 2019, at 15:43, Harsha wrote:
> Hi Ying,
> I think the question is can we add a module in the core which 
> can take up the dynamic config and does a block certain APIs.  This 
> module will be called in each of the APIs like the authorizer does 
> today to check if the API is supported for the client. 
> Instead of throwing AuthorizationException like the authorizer does 
> today it can throw UnsupportedException.
> Benefits are,  we are keeping the authorizer interface as is and adding 
> the flexibility based on dynamic configs without the need for 
> categorizing broker APIs and it will be easy to extend to do additional 
> options,  like turning off certain features which might be in interest 
> to the service providers.
> One drawback,  It will introduce another call to check instead of 
> centralizing everything around Authorizer.
> 
> Thanks,
> Harsha
> 
> On Mon, Feb 25, 2019, at 2:43 PM, Ying Zheng wrote:
> > If you guys don't like the extension of authorizer interface, I will 
just
> > propose a single broker dynamic configuration: client.min.api.version, 
to
> > keep things simple.
> > 
> > What do you think?
> > 
> > On Mon, Feb 25, 2019 at 2:23 PM Ying Zheng  wrote:
> > 
> > > @Viktor Somogyi-Vass, @Harsha, It seems the biggest concern is the
> > > backward-compatibility to the existing authorizers. We can put the new
> > > method into a new trait / interface:
> > > trait AuthorizerEx extends Authorizer {
> > >def authorize(session: Session, operation: Operation, resource: 
Resource,
> > > apiVersion: Short): Boolean
> > > }
> > >
> > > When loading an authorizer class, broker will check if the class
> > > implemented AuthorizerEx interface. If not, broker will wrapper the
> > > Authorizer object with an Adapter class, in which authorizer(...
> > > apiVersion) call is translated to the old authorizer() call. So that, 
both
> > > old and new Authorizer is supported and can be treated as 
AuthorizerEx in
> > > the new broker code.
> > >
> > > As for the broker dynamic configuration approach, I'm not sure how to
> > > correctly categorize the 40+ broker APIs into a few categories.
> > > For example, describe is used by producer, consumer, and admin. 
Should it
> > > be controlled by producer.min.api.version or consumer.min.api.version?
> > > Should 

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

[Don Bosco Durai]

Jason, thanks for the clarification.

Bosco


On 8/25/17, 4:59 PM, "Jason Gustafson" <ja...@confluent.io> wrote:

Hey Don,

That is not actually part of the KIP. It was a (somewhat pedantic) example
used to illustrate how the kafka principal semantics could be applied to
authorizers which understood group-level ACLs. The key point is this:
although a principal is identified only by its type and name, the
KafkaPrincipal can be used to represent relations to other principals. In
this case, we have a user principal which is related to a group principal
through the UserPrincipalAndGroup object. A GroupAuthorizer could then
leverage this relation. As you suggest, a true implementation would allow
multiple groups.

I will add a note to the KIP to emphasize that this is just an example.

Thanks,
Jason

On Fri, Aug 25, 2017 at 4:37 PM, Don Bosco Durai <bo...@apache.org> wrote:

> Jason, thanks for confirming that. Since there are existing custom
> plugins, we might have to give enough time for them to start using the
> newer interface.
>
> I quickly glanced over the KIP, it looks good. Here is one comment:
>
> ---
> In the future, we may add support for groups to Kafka. This was brought up
> in the KIP-111 discussion. To support this, we can provide a groupId()
> method in KafkaPrincipal which defaults to a null value or an empty 
string.
> Extensions can override this just as before. Also note that it is still
> possible for the Authorizer implementation to derive its own group
> information for enforcement.
> class UserPrincipalAndGroup extends KafkaPrincipal {
>   private final String userId;
>   private final String groupId;
> ---
>
> We should assume that users might belong to multiple groups.
>
> Also, not sure what the below method is really doing?
> ---
> class UserPrincipalAndGroup extends KafkaPrincipal {
>
> public KafkaPrincipal group() {
> return new KafkaPrincipal(KafkaPrincipal.GROUP_TYPE, groupId);
>   }
> ---
> Thanks
>
> Bosco
>
>
>
> On 8/25/17, 4:11 PM, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> Hi Don,
>
> I don't think so. We are not making any changes to the Authorizer
> interface
> itself. The KafkaPrincipal object does not change either, though we 
now
> explicitly allow it to be extended. That means you have to exercise a
> little caution when combining a custom PrincipalBuilder with a custom
> Authorizer. For the default principal builder shipped with Kafka, it
> will
> work the same as it currently does. Old implementations of
> PrincipalBuilder
> will also continue to work exactly as they do now, but please note
> that I
> am proposing to deprecate this interface. It will still be supported 
in
> 1.0.0, but we may remove it in a future major release.
>
> -Jason
>
> On Fri, Aug 25, 2017 at 3:51 PM, Don Bosco Durai <bo...@apache.org>
> wrote:
>
> > Jason
> >
> > Do you anticipate any backward compatibility issues with existing
> custom
> > implementation of the authorization interface/plugins?
> >
> > Thanks
> >
> > Bosco
> >
> >
> > On 8/25/17, 3:22 PM, "Jason Gustafson" <ja...@confluent.io> wrote:
> >
> > No problem. I'll add a note to the KIP to emphasize that we will
> use
> > the
> > same object built by the KafkaPrincipalBuilder in the Session
> object
> > passed
> > to the authorizer.
> >
> > -Jason
> >
> > On Fri, Aug 25, 2017 at 3:17 PM, Mayuresh Gharat <
> > gharatmayures...@gmail.com
> > > wrote:
> >
> > > Perfect.
> > > As long as there is a way we can access the originally created
> > Principal in
> > > the Authorizer, it would solve the KIP-111 issue.
> > >
> > > This is really helpful, thanks again.
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Fri, Aug 25, 2017 at 3:13 PM, Jason Gustafson <
> ja...@confluent.io
> > >

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

[Don Bosco Durai]

Jason, thanks for confirming that. Since there are existing custom plugins, we 
might have to give enough time for them to start using the newer interface.

I quickly glanced over the KIP, it looks good. Here is one comment:

---
In the future, we may add support for groups to Kafka. This was brought up in 
the KIP-111 discussion. To support this, we can provide a groupId() method in 
KafkaPrincipal which defaults to a null value or an empty string. Extensions 
can override this just as before. Also note that it is still possible for the 
Authorizer implementation to derive its own group information for enforcement.
class UserPrincipalAndGroup extends KafkaPrincipal {
  private final String userId;
  private final String groupId;
---

We should assume that users might belong to multiple groups.

Also, not sure what the below method is really doing?
---
class UserPrincipalAndGroup extends KafkaPrincipal {

public KafkaPrincipal group() {
return new KafkaPrincipal(KafkaPrincipal.GROUP_TYPE, groupId);
  }
---
Thanks

Bosco



On 8/25/17, 4:11 PM, "Jason Gustafson" <ja...@confluent.io> wrote:

Hi Don,

I don't think so. We are not making any changes to the Authorizer interface
itself. The KafkaPrincipal object does not change either, though we now
explicitly allow it to be extended. That means you have to exercise a
little caution when combining a custom PrincipalBuilder with a custom
Authorizer. For the default principal builder shipped with Kafka, it will
work the same as it currently does. Old implementations of PrincipalBuilder
will also continue to work exactly as they do now, but please note that I
am proposing to deprecate this interface. It will still be supported in
1.0.0, but we may remove it in a future major release.

-Jason

On Fri, Aug 25, 2017 at 3:51 PM, Don Bosco Durai <bo...@apache.org> wrote:

> Jason
>
> Do you anticipate any backward compatibility issues with existing custom
> implementation of the authorization interface/plugins?
>
> Thanks
>
> Bosco
>
>
> On 8/25/17, 3:22 PM, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> No problem. I'll add a note to the KIP to emphasize that we will use
> the
> same object built by the KafkaPrincipalBuilder in the Session object
> passed
> to the authorizer.
>
> -Jason
>
> On Fri, Aug 25, 2017 at 3:17 PM, Mayuresh Gharat <
> gharatmayures...@gmail.com
> > wrote:
>
> > Perfect.
> > As long as there is a way we can access the originally created
> Principal in
> > the Authorizer, it would solve the KIP-111 issue.
> >
> > This is really helpful, thanks again.
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Fri, Aug 25, 2017 at 3:13 PM, Jason Gustafson <ja...@confluent.io
> >
> > wrote:
> >
> > > Hi Mayuresh,
> > >
> > > To clarify, the intention is to use the KafkaPrincipal object
> built by
> > the
> > > KafkaPrincipalBuilder inside the Session. So we would remove the
> logic to
> > > construct a new KafkaPrincipal using only the name from the
> Principal.
> > Then
> > > it should be possible to pass the `AuthzPrincipal` to the
> underlying
> > > library through the `Extended_Plugged_In_Class` as you've 
suggested
> > above.
> > > Is that reasonable for this use case?
> > >
> > > Thanks,
> > > Jason
> > >
> > >
> > > On Fri, Aug 25, 2017 at 2:44 PM, Mayuresh Gharat <
> > > gharatmayures...@gmail.com
> > > > wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > Thanks for the replies.
> > > >
> > > > I think it would be better to discuss with an example that we
> were
> > trying
> > > > to address with KIP-111 and see if the current mentioned
> solution would
> > > > address it.
> > > >
> > > > Let's consider a third party library called authz_lib that is
> provided
> > by
> > > > some Security team at  some company.
> > > >
> > > >- When we call authz_lib.createPrincipal(X509_cert), it would
   

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

[Don Bosco Durai]

Jason

Do you anticipate any backward compatibility issues with existing custom 
implementation of the authorization interface/plugins?

Thanks

Bosco


On 8/25/17, 3:22 PM, "Jason Gustafson"  wrote:

No problem. I'll add a note to the KIP to emphasize that we will use the
same object built by the KafkaPrincipalBuilder in the Session object passed
to the authorizer.

-Jason

On Fri, Aug 25, 2017 at 3:17 PM, Mayuresh Gharat  wrote:

> Perfect.
> As long as there is a way we can access the originally created Principal 
in
> the Authorizer, it would solve the KIP-111 issue.
>
> This is really helpful, thanks again.
>
> Thanks,
>
> Mayuresh
>
> On Fri, Aug 25, 2017 at 3:13 PM, Jason Gustafson 
> wrote:
>
> > Hi Mayuresh,
> >
> > To clarify, the intention is to use the KafkaPrincipal object built by
> the
> > KafkaPrincipalBuilder inside the Session. So we would remove the logic 
to
> > construct a new KafkaPrincipal using only the name from the Principal.
> Then
> > it should be possible to pass the `AuthzPrincipal` to the underlying
> > library through the `Extended_Plugged_In_Class` as you've suggested
> above.
> > Is that reasonable for this use case?
> >
> > Thanks,
> > Jason
> >
> >
> > On Fri, Aug 25, 2017 at 2:44 PM, Mayuresh Gharat <
> > gharatmayures...@gmail.com
> > > wrote:
> >
> > > Hi Jason,
> > >
> > > Thanks for the replies.
> > >
> > > I think it would be better to discuss with an example that we were
> trying
> > > to address with KIP-111 and see if the current mentioned solution 
would
> > > address it.
> > >
> > > Let's consider a third party library called authz_lib that is provided
> by
> > > some Security team at  some company.
> > >
> > >- When we call authz_lib.createPrincipal(X509_cert), it would
> return
> > an
> > >AuthzPrincipal that implements Java.Security.Principal.
> > >
> > >
> > >- The authz_lib also provides an checkAccess() call that takes
> in
> > 3
> > >parameters :
> > >   - authz_principal
> > >   - operation type ("Read", "Write"...)
> > >   - resource (for simplicity lets consider it as a TopicName)
> > >
> > >
> > >- The AuthzPrincipal looks like this :
> > >
> > > class AuthzPrincipal implements java.security.Principal
> > > {
> > > String name;
> > > String field1;
> > > Object field2;
> > > Object field3;
> > > .//Some third party logic..
> > > }
> > >
> > >
> > >- In PrincipalBuilder.buildPrincipal() would return AuthzPrincipal
> as
> > >follows :
> > >
> > > public Principal buildPrincipal(...)
> > > {
> > > ..
> > > X509Certificate x509Cert = session.getCert(..);
> > > return authz_lib.createPrincipal(x509Cert);
> > > }
> > >
> > >
> > >- The custom Authorizer (lets call it CustomAuthzAuthorizer), we
> would
> > >use the checkAccess() function provided by the authz_lib as follows
> :
> > >
> > > public class CustomAuthzAuthorizer implements Authorizer
> > > {
> > > .
> > > public boolean authorize(.)
> > > {
> > >AuthzPrincipal authz_principal = (AuthzPrincipal)
> > > session.getPrincipal();
> > > return authz_lib.checkAccess(authz_principal, "Read", "topicX");
> > > }
> > > ..
> > > }
> > >
> > >
> > >- The issue with current implementation is that in
> > >processCompletedReceives() in SocketServer we create a
> KafkaPrincipal
> > >that just extracts the name from AuthzPrincipal as follows :
> > >
> > > session = RequestChannel.Session(new
> > > KafkaPrincipal(KafkaPrincipal.USER_TYPE,
> > > *openOrClosingChannel.principal.getName*),
> > > openOrClosingChannel.socketAddress)
> > >
> > > So the "AuthzPrincipal authz_principal = (AuthzPrincipal)
> > > session.getPrincipal()" call in the CustomAuthzAuthorizer would error
> > > out because we are trying to cast a KafkaPrincipal to AuthzPrincipal.
> > >
> > >
> > >
> > > In your reply when you said that :
> > >
> > > The KIP says that a user can have a class that extends KafkaPrincipal.
> > > Would this extended class be used when constructing the Session object
> > > in the SocketServer instead of constructing a new KafkaPrincipal?
> > >
> > > Yes, that's correct. We want to allow the authorizer to be able to
> > leverage
> > > > additional information from the authentication layer.
> > >
> > >
> > > Would it make sense to make this extended class pluggable and when
> > > 

Re: [DISCUSS] Client Side Auto Topic Creation

[Don Bosco Durai]

Jun, thanks. I will check again.

Regards

Bosco


On 7/17/16, 9:36 PM, "Jun Rao" <j...@confluent.io> wrote:

Bosco,

Currently, if acl is enabled, auto topic creation will succeed if the
client has the CREATE permission.

Thanks,

Jun

On Mon, Jul 11, 2016 at 6:26 PM, Don Bosco Durai <bo...@apache.org> wrote:

> Jun, my understanding is, currently if ACLs are enabled, then auto topic
> creation is disabled. Is this going to change with this requirement?
>
> Thanks
>
> Bosco
>
>
> On 7/11/16, 1:14 PM, "Jun Rao" <j...@confluent.io> wrote:
>
> Ismael,
>
> We could change the existing behavior if it's bad for most of the users. 
In
> the case of auto topic creation in the producer, it seems that it's at
> least convenient in a testing environment. So, I am not sure if that
> behavior is universally bad.
>
> Also, I am not sure if we can rely on the client to set the configuration
> properly to disable auto topic creation. It seems that a safer way is to 
do
> that on the broker side (e.g., only allow the admin to create topics
> through ACL). Once you do that, I am not sure if we need a configuration 
to
> enable topic creation in the producer. A producer will just error out if
> the topic doesn't exist and the topic creation is disabled on the broker.
>
> Thanks,
>
> Jun
>
> On Fri, Jul 8, 2016 at 6:06 AM, Ismael Juma <ism...@juma.me.uk> wrote:
>
> > Hi Jun,
> >
> > I agree that it's closer to the existing behaviour, which some people 
may
> > be used to by now. However, I am not sure that it won't surprise people.
> As
> > Grant said, auto-topic creation is a common source of confusion and it
> > interacts badly with topic deletion.
> >
> > If we need to provide auto-topic creation in the client as a migration
> path
> > for people who rely on it and so that we can remove the server based one
> > (after a suitable deprecation period), then can we at least have it 
false
> > by default? This way it's more likely that people who enable it would be
> > aware of the pitfalls and it would reduce the number of confused users.
> >
> > Ismael
> >
> > On Thu, Jul 7, 2016 at 9:47 PM, Jun Rao <j...@confluent.io> wrote:
> >
> > > It seems that it makes sense for the writer to trigger auto topic
> > creation,
> > > but not the reader. So, my preference is Jay's option #1: add a new
> > > configuration to enable topic creation on the producer side and
> defaults
> > to
> > > true. If the topic doesn't exist, the producer will send a
> > > createTopicRequest and pick up the broker side defaults for 
#partitions
> > and
> > > replication factor. This matches the current behavior and won't
> surprise
> > > people. People who want to enforce manual topic creation can disable
> auto
> > > topic creation on the producer.
> > >
> > > On the consumer side, throwing an exception to the client when a topic
> > > doesn't exist probably works for most cases. I am wondering if there
> is a
> > > case where a user really wants to start the consumer before the topic
> is
> > > created.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Fri, Jul 1, 2016 at 4:09 AM, Ismael Juma <ism...@juma.me.uk> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I think there are a few things being discussed and it would be good
> to
> > > make
> > > > that explicit:
> > > >
> > > > 1. If and how we expose auto-topic creation in the client (under the
> > > > assumption that the server auto-topic creation will be deprecated 
and
> > > > eventually removed)
> > > > 2. The ability to create topics with the cluster defaults for
> > replication
> > > > factor and partition counts
> > > > 3. Support for topic "specs"
> > > > 4. The fact that some exceptions are retriable in some cases, but 
not
> > > > others
> > > >
> > > > My thoughts on each:
> > > >
> > > > 1. I prefer the approach where we throw an exception and let the
> &

Re: [DISCUSS] Client Side Auto Topic Creation

[Don Bosco Durai]

Jun, my understanding is, currently if ACLs are enabled, then auto topic 
creation is disabled. Is this going to change with this requirement? 

Thanks

Bosco


On 7/11/16, 1:14 PM, "Jun Rao"  wrote:

Ismael,

We could change the existing behavior if it's bad for most of the users. In
the case of auto topic creation in the producer, it seems that it's at
least convenient in a testing environment. So, I am not sure if that
behavior is universally bad.

Also, I am not sure if we can rely on the client to set the configuration
properly to disable auto topic creation. It seems that a safer way is to do
that on the broker side (e.g., only allow the admin to create topics
through ACL). Once you do that, I am not sure if we need a configuration to
enable topic creation in the producer. A producer will just error out if
the topic doesn't exist and the topic creation is disabled on the broker.

Thanks,

Jun

On Fri, Jul 8, 2016 at 6:06 AM, Ismael Juma  wrote:

> Hi Jun,
>
> I agree that it's closer to the existing behaviour, which some people may
> be used to by now. However, I am not sure that it won't surprise people. As
> Grant said, auto-topic creation is a common source of confusion and it
> interacts badly with topic deletion.
>
> If we need to provide auto-topic creation in the client as a migration path
> for people who rely on it and so that we can remove the server based one
> (after a suitable deprecation period), then can we at least have it false
> by default? This way it's more likely that people who enable it would be
> aware of the pitfalls and it would reduce the number of confused users.
>
> Ismael
>
> On Thu, Jul 7, 2016 at 9:47 PM, Jun Rao  wrote:
>
> > It seems that it makes sense for the writer to trigger auto topic
> creation,
> > but not the reader. So, my preference is Jay's option #1: add a new
> > configuration to enable topic creation on the producer side and defaults
> to
> > true. If the topic doesn't exist, the producer will send a
> > createTopicRequest and pick up the broker side defaults for #partitions
> and
> > replication factor. This matches the current behavior and won't surprise
> > people. People who want to enforce manual topic creation can disable auto
> > topic creation on the producer.
> >
> > On the consumer side, throwing an exception to the client when a topic
> > doesn't exist probably works for most cases. I am wondering if there is a
> > case where a user really wants to start the consumer before the topic is
> > created.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Jul 1, 2016 at 4:09 AM, Ismael Juma  wrote:
> >
> > > Hi all,
> > >
> > > I think there are a few things being discussed and it would be good to
> > make
> > > that explicit:
> > >
> > > 1. If and how we expose auto-topic creation in the client (under the
> > > assumption that the server auto-topic creation will be deprecated and
> > > eventually removed)
> > > 2. The ability to create topics with the cluster defaults for
> replication
> > > factor and partition counts
> > > 3. Support for topic "specs"
> > > 4. The fact that some exceptions are retriable in some cases, but not
> > > others
> > >
> > > My thoughts on each:
> > >
> > > 1. I prefer the approach where we throw an exception and let the
> clients
> > > create the topic via `AdminClient` if that's what they need.
> > > 2. Like Grant, I'm unsure that will generally be used in a positive
> way.
> > > However, if this is what we need to be able to deprecate server
> > auto-topic
> > > creation, the benefits outweigh the costs in my opinion.
> > > 3. Something like this would be good to have and could potentially
> > provide
> > > a better solution than 2. However, it needs a separate KIP and may
> take a
> > > while for the final design to be agreed. So, it should not prevent
> > progress
> > > from being made in my opinion.
> > > 4. This has come up before. Encoding whether an exception is retriable
> or
> > > not via inheritance is a bit restrictive. Also, something that should
> be
> > > discussed separately, probably.
> > >
> > > Ismael
> > >
> > > On Wed, Jun 29, 2016 at 10:37 PM, Grant Henke 
> > wrote:
> > >
> > > > Hi Roger and Constantine,
> > > >
> > > > Thanks for the feedback.
> > > >
> > > > I agree that configuration to maintain guarantees is commonly spread
> > > across
> > > > enterprise teams, making it difficult to get right. That said its
> also
> > > hard
> > > > to solve for every company structure too. I think there is room for
> an
> > > open
> > > > discussion about what configs should be able to be
> > > > validated/enforced/overridden and where configurations should live. I
> > > think
> > > > thats big enough for a whole new KIP and would like to push that
> > > discussion
> > > > out until that KIP is opened. These discussions would also make sense
> > in
> > > > KIP-37
> > > > - Add Namespaces to Kafka
> > > > <
> > > 

Re: [VOTE] KIP-50: Move Authorizer to o.a.k.common package - Round 2

[Don Bosco Durai]

+1 (non binding)




On 4/28/16, 7:47 AM, "Jun Rao"  wrote:

>Ashish,
>
>Thanks for the proposal. +1
>
>Jun
>
>On Mon, Apr 25, 2016 at 10:04 AM, Ashish Singh  wrote:
>
>> Hey Guys,
>>
>> I would like to re-initiate the voting process for KIP-50: Move Authorizer
>> to o.a.k.common package. KIP-50 is available here
>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-50+-+Move+Authorizer+to+o.a.k.common+package
>> >.
>> Associated PR is available here > >.
>>
>> The vote will run for 72 hours and I would like to start it with my +1
>> (non-binding).
>> ​
>> --
>>
>> Regards,
>> Ashish
>>

Re: [DISCUSS] KIP-50 - Enhance Authorizer interface to be aware of supported Principal Types

[Don Bosco Durai]

Yes, if we have Java interface, then we won’t need Scala interface.

Thanks

Bosco





On 4/8/16, 9:04 AM, "Ashish Singh" <asi...@cloudera.com> wrote:

>Thanks for the input Don. One of the possible paths for Option 2 is to
>completely drop Scala interface, would that be Ok with you folks?
>
>On Thursday, April 7, 2016, Don Bosco Durai <bo...@apache.org> wrote:
>
>> Ranger team would prefer option #2. Right now, we have to access some of
>> the nested constants using constructs like Group$.MODULE$, which is not
>> intuitive in Java.
>>
>> Thanks
>>
>> Bosco
>>
>>
>>
>>
>> On 4/7/16, 4:30 PM, "Ashish Singh" <asi...@cloudera.com <javascript:;>>
>> wrote:
>>
>> >Harsha/ Don,
>> >
>> >Are you guys OK with option 2? I am not aware of all the existing
>> >authorizer implementations, however ranger has one for sure. Getting
>> direct
>> >feedback from you guys will be really valuable.
>> >
>> >On Thu, Apr 7, 2016 at 3:52 PM, Ismael Juma <ism...@juma.me.uk
>> <javascript:;>> wrote:
>> >
>> >> Hi Don,
>> >>
>> >> This is true in Java 7, but Java 8 introduces default methods and this
>> >> workaround is no longer required. During the Interceptor KIP
>> discussion, it
>> >> was decided that it was fine to stick to interfaces given that we are
>> >> likely to move to Java 8 in the nearish future (probably no later than
>> the
>> >> Java 9 release).
>> >>
>> >> Ismael
>> >>
>> >> On Thu, Apr 7, 2016 at 11:36 PM, Don Bosco Durai <bo...@apache.org
>> <javascript:;>> wrote:
>> >>
>> >> > Hi Ashish
>> >> >
>> >> > If we are going by option #2, then I can suggest we give an abstract
>> >> > implementation of the Interface and recommend anyone implementing
>> their
>> >> own
>> >> > plugin to extend from the abstract class, rather than implement the
>> >> > interface?
>> >> >
>> >> > The advantage is, in the future if we add add any new methods in the
>> >> > Interface (e.g. Similar to getDescription()), then we can give a dummy
>> >> > implementation of the new method and this won’t break the compilation
>> of
>> >> > any external implementation. Else over the time it will be challenging
>> >> for
>> >> > anyone customizing the implementation to keep track of changes to the
>> >> > Interface.
>> >> >
>> >> > Thanks
>> >> >
>> >> > Bosco
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On 4/7/16, 11:21 AM, "Ashish Singh" <asi...@cloudera.com
>> <javascript:;>> wrote:
>> >> >
>> >> > >Hello Harsha,
>> >> > >
>> >> > >On Thu, Apr 7, 2016 at 11:03 AM, Harsha <m...@harsha.io
>> <javascript:;>> wrote:
>> >> > >
>> >> > >"My only ask is to have this in 0.10. As Jay pointed out, right now
>> >> > >> there
>> >> > >> are not many implementations out there, we might want to fix it
>> ASAP."
>> >> > >>
>> >> > >> Probably there aren't many implementations but there are lot of
>> users
>> >> > >> using these implementations in production clusters.
>> >> > >> Isn't this going to break the rolling upgrade?
>> >> > >
>> >> > >It will and it is a concern, in my previous mail I have mentioned
>> this
>> >> as
>> >> > >an issue if we choose to go this route. However, if we actually
>> decide
>> >> to
>> >> > >do this, I would say it is better to do it sooner than later, as
>> fewer
>> >> > >implementations will be affected. Below is excerpt from my previous
>> >> mail.
>> >> > >
>> >> > >Increase scope of KIP-50 to move authorizer and related classes to a
>> >> > >separate package. The new package will have java interface. This will
>> >> > allow
>> >> > >implementations to not depend on kafka core and just on authorizer
>> >> > package,
>> >> > >make authorization interface follow kafka’s coding standards and will
>>

Re: [DISCUSS] KIP-50 - Enhance Authorizer interface to be aware of supported Principal Types

[Don Bosco Durai]

Ranger team would prefer option #2. Right now, we have to access some of the 
nested constants using constructs like Group$.MODULE$, which is not intuitive 
in Java.

Thanks

Bosco




On 4/7/16, 4:30 PM, "Ashish Singh" <asi...@cloudera.com> wrote:

>Harsha/ Don,
>
>Are you guys OK with option 2? I am not aware of all the existing
>authorizer implementations, however ranger has one for sure. Getting direct
>feedback from you guys will be really valuable.
>
>On Thu, Apr 7, 2016 at 3:52 PM, Ismael Juma <ism...@juma.me.uk> wrote:
>
>> Hi Don,
>>
>> This is true in Java 7, but Java 8 introduces default methods and this
>> workaround is no longer required. During the Interceptor KIP discussion, it
>> was decided that it was fine to stick to interfaces given that we are
>> likely to move to Java 8 in the nearish future (probably no later than the
>> Java 9 release).
>>
>> Ismael
>>
>> On Thu, Apr 7, 2016 at 11:36 PM, Don Bosco Durai <bo...@apache.org> wrote:
>>
>> > Hi Ashish
>> >
>> > If we are going by option #2, then I can suggest we give an abstract
>> > implementation of the Interface and recommend anyone implementing their
>> own
>> > plugin to extend from the abstract class, rather than implement the
>> > interface?
>> >
>> > The advantage is, in the future if we add add any new methods in the
>> > Interface (e.g. Similar to getDescription()), then we can give a dummy
>> > implementation of the new method and this won’t break the compilation of
>> > any external implementation. Else over the time it will be challenging
>> for
>> > anyone customizing the implementation to keep track of changes to the
>> > Interface.
>> >
>> > Thanks
>> >
>> > Bosco
>> >
>> >
>> >
>> >
>> > On 4/7/16, 11:21 AM, "Ashish Singh" <asi...@cloudera.com> wrote:
>> >
>> > >Hello Harsha,
>> > >
>> > >On Thu, Apr 7, 2016 at 11:03 AM, Harsha <m...@harsha.io> wrote:
>> > >
>> > >"My only ask is to have this in 0.10. As Jay pointed out, right now
>> > >> there
>> > >> are not many implementations out there, we might want to fix it ASAP."
>> > >>
>> > >> Probably there aren't many implementations but there are lot of users
>> > >> using these implementations in production clusters.
>> > >> Isn't this going to break the rolling upgrade?
>> > >
>> > >It will and it is a concern, in my previous mail I have mentioned this
>> as
>> > >an issue if we choose to go this route. However, if we actually decide
>> to
>> > >do this, I would say it is better to do it sooner than later, as fewer
>> > >implementations will be affected. Below is excerpt from my previous
>> mail.
>> > >
>> > >Increase scope of KIP-50 to move authorizer and related classes to a
>> > >separate package. The new package will have java interface. This will
>> > allow
>> > >implementations to not depend on kafka core and just on authorizer
>> > package,
>> > >make authorization interface follow kafka’s coding standards and will
>> > allow
>> > >java implementations to be cleaner. We can either completely drop scala
>> > >interface, which might be a pain for existing implementations, or we can
>> > >have scala interface wrap java interface. Later allows a cleaner
>> > >deprecation path for existing scala authorizer interface, however it may
>> > or
>> > >may not be feasible as Kafka server will have to somehow decide which
>> > >interface it should be looking for while loading authorizer
>> > implementation,
>> > >this can probably be solved with a config or some reflection. If we
>> choose
>> > >to go this route, I can dig deeper.
>> > >
>> > >If we go with option 2 and commit on getting this in ASAP, preferably in
>> > >0.10, there will be fewer implementations that will be affected.
>> > >
>> > >and also moving to Java ,
>> > >> a authorizer implementation going to run inside a KafkaBroker and I
>> > >> don't see why this is necessary to move to clients package.
>> > >> Are we planning on introducing common module to have it independent of
>> > >> broker and client code?
>> > >>
>> > >Yes, I think that would take away

Re: [DISCUSS] KIP-50 - Enhance Authorizer interface to be aware of supported Principal Types

[Don Bosco Durai]

Hi Ashish

If we are going by option #2, then I can suggest we give an abstract 
implementation of the Interface and recommend anyone implementing their own 
plugin to extend from the abstract class, rather than implement the interface?

The advantage is, in the future if we add add any new methods in the Interface 
(e.g. Similar to getDescription()), then we can give a dummy implementation of 
the new method and this won’t break the compilation of any external 
implementation. Else over the time it will be challenging for anyone 
customizing the implementation to keep track of changes to the Interface.

Thanks

Bosco




On 4/7/16, 11:21 AM, "Ashish Singh"  wrote:

>Hello Harsha,
>
>On Thu, Apr 7, 2016 at 11:03 AM, Harsha  wrote:
>
>"My only ask is to have this in 0.10. As Jay pointed out, right now
>> there
>> are not many implementations out there, we might want to fix it ASAP."
>>
>> Probably there aren't many implementations but there are lot of users
>> using these implementations in production clusters.
>> Isn't this going to break the rolling upgrade?
>
>It will and it is a concern, in my previous mail I have mentioned this as
>an issue if we choose to go this route. However, if we actually decide to
>do this, I would say it is better to do it sooner than later, as fewer
>implementations will be affected. Below is excerpt from my previous mail.
>
>Increase scope of KIP-50 to move authorizer and related classes to a
>separate package. The new package will have java interface. This will allow
>implementations to not depend on kafka core and just on authorizer package,
>make authorization interface follow kafka’s coding standards and will allow
>java implementations to be cleaner. We can either completely drop scala
>interface, which might be a pain for existing implementations, or we can
>have scala interface wrap java interface. Later allows a cleaner
>deprecation path for existing scala authorizer interface, however it may or
>may not be feasible as Kafka server will have to somehow decide which
>interface it should be looking for while loading authorizer implementation,
>this can probably be solved with a config or some reflection. If we choose
>to go this route, I can dig deeper.
>
>If we go with option 2 and commit on getting this in ASAP, preferably in
>0.10, there will be fewer implementations that will be affected.
>
>and also moving to Java ,
>> a authorizer implementation going to run inside a KafkaBroker and I
>> don't see why this is necessary to move to clients package.
>> Are we planning on introducing common module to have it independent of
>> broker and client code?
>>
>Yes, I think that would take away the requirement of depending on Kafka
>core from authorizer implementations. Do you suggest otherwise?
>
>
>> -Harsha
>>
>> On Thu, Apr 7, 2016, at 10:52 AM, Ashish Singh wrote:
>> > We might want to take a call here. Following are the options.
>> >
>> >1. Let KIP-50 be the way it is, i.e., just add getDescription() to
>> >existing scala authorizer interface. It will break binary
>> >compatibility
>> >(only when using CLI and/or AdminCommand from >= 0.10 against
>> >authorizer
>> >implementations based on 0.9.). If we go this route, it is a simpler
>> >change
>> >and existing implementations won’t have to change anything on their
>> >end.
>> >2. Increase scope of KIP-50 to move authorizer and related classes to
>> >a
>> >separate package. The new package will have java interface. This will
>> >allow
>> >implementations to not depend on kafka core and just on authorizer
>> >package,
>> >make authorization interface follow kafka’s coding standards and will
>> >allow
>> >java implementations to be cleaner. We can either completely drop
>> >scala
>> >interface, which might be a pain for existing implementations, or we
>> >can
>> >have scala interface wrap java interface. Later allows a cleaner
>> >deprecation path for existing scala authorizer interface, however it
>> >may or
>> >may not be feasible as Kafka server will have to somehow decide which
>> >interface it should be looking for while loading authorizer
>> >implementation,
>> >this can probably be solved with a config or some reflection. If we
>> >choose
>> >to go this route, I can dig deeper.
>> >
>> > If we decide to go with option 1, I think it would be fair to say that
>> > scala authorizer interface will be around for some time, as there will be
>> > more implementations relying on it. If we go with option 2 and commit on
>> > getting this in ASAP, preferably in 0.10, there will be fewer
>> > implementations that will be affected.
>> >
>> > *Another thing to notice is that scala authorizer interface is not
>> > annotated as unstable.*
>> > ​
>> >
>> > On Wed, Apr 6, 2016 at 9:41 AM, Ashish Singh 
>> wrote:
>> >
>> > > I see value in minimizing 

Re: Authorization Engine For Kafka Related to KPI-11

[Don Bosco Durai]

Bhavesh

I am from the Apache Ranger team, so let me answer the questions pertaining to 
Ranger…

> 1) Is there any performance impact with Brokers/Producer/Consumer while using 
> Apache Ranger ?
As such Ranger overhead is negligible. Specifically for Kafka, we did some more 
optimization for the unique usage pattern of Kafka consumers and publishers. 
Considering Kafka itself is optimized for super throughput, expected some 
overhead penalty. We don’t have the exact numbers yet. Would you be willing to 
help us some run some real-world scenarios to determine the overhead? If so, we 
can discuss it in the Ranger mailing list. Thanks.

> 2) Is Audit log really useful out-of-box ? or let me know what sort of 
> reports you run on audit logs (e.g pumping Apache Ranger audit log into any 
> other system for reporting purpose).
If you are referring to Ranger audit logs, then it is configurable. OOTB, it 
has options to go to HDFS, SOLR and/or DB. We have extensions to send it to 
Kafka and any Log4J appender. Audit logs has its own purpose, from compliance 
requirements to operation monitoring usage to anomaly detection. Ranger portal 
has its own UI for querying. Since the audits are stored in normalized format, 
you can write your own reports on top of it.

Bosco





On 11/3/15, 9:55 PM, "Bhavesh Mistry"  wrote:

>+ Kafka Dev team to see if Kafka Dev team know or recommend any Auth
>engine for Producers/Consumers.
>
>Thanks,
>
>Bhavesh
>
>Please pardon me,  I accidentally send previous blank email.
>
>On Tue, Nov 3, 2015 at 9:52 PM, Bhavesh Mistry
> wrote:
>> On Sun, Nov 1, 2015 at 11:15 PM, Bhavesh Mistry
>>  wrote:
>>> HI All,
>>>
>>> Have any one used Apache Ranger as Authorization Engine for Kafka Topic
>>> creation, consumption (read) and  write operation on a topic.  I am looking
>>> at having audit log and regulating consumption/ write to particular topic
>>> (for example, having production environment access does not mean that anyone
>>> can run console consumer etc on particular topic. Basically, regulate who
>>> can read/write to a topic as first use case).
>>>
>>> https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5+-+User+Guide#ApacheRanger0.5-UserGuide-KAFKA
>>>
>>> If you have used Apache Ranger in production, I have following question:
>>> 1) Is there any performance impact with Brokers/Producer/Consumer while
>>> using Apache Ranger ?
>>> 2) Is Audit log really useful out-of-box ? or let me know what sort of
>>> reports you run on audit logs (e.g pumping Apache Ranger audit log into any
>>> other system for reporting purpose).
>>>
>>> Please share your experience using Kafka with any other Authorization engine
>>> if you are not using Apache Ranger (This is based on
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface).
>>>
>>> Thanks and looking forward to hear back from Apache Kafka Community members.
>>>
>>> Thanks,
>>>
>>> Bhavesh

Re: Gauging Interest in adding Encryption to Kafka

[Don Bosco Durai]

The client-broker protocol would have to be augmented to carry the
encrypted encryption key, plus logic to handle redistribution to existing
clients due to key rotation.
This is a good point. HDFS has the encryption zone concept, which could be
akin to a topic. The keys in HDFS are per file level, not sure what would
be good compromise here for the granularity. For simplicity, it could be
at the topic level itself, but the master key is never given to the
client. 

Regardless, internal Kafka management of messages, batching, replication,
compression, compaction, performance, etc might be some of the key
deciding factors.


On Mon, Aug 3, 2015 at 10:22 AM, Gwen Shapira g...@confluent.io wrote:
If I understand you correctly, you are saying that the kerberos keytab
that
the broker uses to authenticate with the KMS will be somewhere on the
broker node and can be used by a malicious admin.

Yes. If broker is doing encryption/decryption, then you need to bootstrap
the broker(s) with the encryption key. The key could be in local disk as
java keystore or remote in KMS. If it is in remote KMS, then you will need
to authenticate with KMS via kerberos or other authentication scheme.
Regardless, the Kafka admin having shell login access as the linux user
used by Kafka broker process (assuming kafka) will have access to the key
and would be able to encrypt/decrypt the stored data.

This might seem not like a big deal, but in some enterprises ³separation
of duties² or ³privilege user access management (PIM)² are critical
compliance requirements.

More importantly, if the goal is just to store data in encrypted form in
the disk, then honestly, you just need to encrypt the Kafka data volume
using LUKS and with restricted file permissions. This will take care of
issues like disk being stolen, etc. You don¹t need to do any changes to
Kafka :-)

Thanks

Bosco







On 8/3/15, 12:28 PM, Alejandro Abdelnur tuc...@gmail.com wrote:

Doing encryption on the client has the following benefits (most of them
already mentioned in the thread):

* brokers don't have additional CPU load
* brokers never see the data in unencrypted form (Kafka admins cannot
snoop)
* secure multi-tenancy (keys are 100% on the client space)
* no need to secure Kafka wire transport, client-broker and broker-broker
(data is already encrypted)

It would be highly desirable, even if encryption is done on the client
side, that encryption is 'transparent'. Similar to how HDFS encryption
works, it is not the client writing/reading a topic the one that decides
to
encrypt/decrypt but the broker is the one telling that to the client and
providing encrypted encryption keys for the tasks.The client-broker
protocol would have to be augmented to carry the encrypted encryption key,
plus logic to handle redistribution to existing clients due to key
rotation.

A nice thing about doing broker side encryption though is that you can
shut
off clients at any time and they won't see unencrypted data anymore. But
this means the brokers will have to deal with the client ACLs for
encryption (i'd rather leave that outside of Kafka and being a concern of
the KMS system). You could achieve similar functionality on client side
encryption, by removing the client from the ACLs in the KMS and doing a
key
rotation, then the client will not be able to decrypt any messages using
the new key (though all previous ones using the key that the client
already
has will be visible to the client).


Thanks.


On Mon, Aug 3, 2015 at 10:22 AM, Gwen Shapira g...@confluent.io wrote:

 If I understand you correctly, you are saying that the kerberos keytab
that
 the broker uses to authenticate with the KMS will be somewhere on the
 broker node and can be used by a malicious admin.

 I agree this is a valid concern.
 I am not opposed to client-side encryption, I am more concerned that the
 modifications this will require in Kafka broker implementation make the
 idea impractical. And obviously, as in any security discussion - there
are
 lots of details regarding key exchange, management and protection that
are
 critical.

 Perhaps given a design doc, we can better evaluate the proposed
tradeoffs.

 Gwen



 On Sat, Aug 1, 2015 at 10:10 AM, Don Bosco Durai bo...@apache.org
wrote:

  Any reason you think its better to let the clients handle it?
  Gwen, I agree with Todd, depending on the goal, the requirements might
  vary. If the goal is that someone stills the disk, then they should be
  able to access the data, then encrypting at Broker is enough.
However, if
  the requirement is that the admin/operator should not be able to
access
  the data, then client side is the only option.
 
  Hadoop/HDFS transparent data encryption has a similar philosophy,
where
  the actual encryption/decryption happens at the client side.
 
  1. Key management
  Hadoop common has a KMS. And there are industry standards like KMIP.
If
  Broker does the encrypt/decrypt, then the solution is much easier. If
the
  client does

Re: [Vote] KIP-11 Authorization design for kafka security

[Don Bosco Durai]

+1 non-binding


On 5/15/15, 11:43 AM, Gwen Shapira gshap...@cloudera.com wrote:

+1 non-binding

On Fri, May 15, 2015 at 9:12 PM, Harsha harsh...@fastmail.fm wrote:

 +1 non-binding






 On Fri, May 15, 2015 at 9:18 AM -0700, Parth Brahmbhatt 
 pbrahmbh...@hortonworks.com wrote:










 Hi,

 Opening the voting thread for KIP-11.

 Link to the KIP:
 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+
Interface
 Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688

 Thanks
 Parth

Re: [VOTE] KIP-11- Authorization design for kafka security

[Don Bosco Durai]

Joe, these are good use cases, however in the firt phase the granularity
is at the Topic (your e.g. bucket) level and not what you are accessing
within the Topic. So in your use case, if you don’t have access to “Bucket
A”, then you won’t know who is in it, so you won’t know “Alice” or anyone
who as “X”.

The use case here, there is a HL7 topic with specific for “New Patients”,
then only users “A,B or C” can publish to it and only users “X, Y o Z”
can consume from it. In addition, only admin users “P, Q and R” can manage
the topic permissions.

I feel, keeping it simple should be good enough for the first phase.

Thanks

Bosco



On 4/30/15, 3:59 PM, Joe Stein joe.st...@stealth.ly wrote:

If you have bucket A and Bucket B and in Bucket A there are patients with
Disease X and Bucket B patients without Disease X.

Now you try to access Alice from bucket A and you get a 403  and then
from Bucket B you get a 404.

What does that tell you now about Alice? Yup, she has Disease X.

Uniform none existence is a good policy for protecting data. If you don't
have permission then 404 not found works too.

The context that I thought that applied with this discussion is because I
thought the authorization module was going to be a bit more integration
where the api responses were happening

~ Joe Stein
- - - - - - - - - - - - - - - - -

  http://www.stealth.ly
- - - - - - - - - - - - - - - - -

On Thu, Apr 30, 2015 at 6:51 PM, Suresh Srinivas sur...@hortonworks.com
wrote:

 Comment on AuthorizationException. I think the intent of exception
should
 be to capture why a request is rejected. It is important from API
 perspective to be specific to aid debugging. Having a generic or
obfuscated
 exception is not very useful. Does someone on getting an exceptionreach
 out to an admin to understand if a topic exists or it's an authorization
 issue?

 I am not getting the security concern. System must be ensure disallowing
 the access by implementing the security correctly. Not based on
security by
 obscurity.

 Regards,
 Suresh

 Sent from phone

 _
 From: Gwen Shapira gshap...@cloudera.commailto:gshap...@cloudera.com
 Sent: Thursday, April 30, 2015 10:14 AM
 Subject: Re: [VOTE] KIP-11- Authorization design for kafka security
 To: dev@kafka.apache.orgmailto:dev@kafka.apache.org


 * Regarding additional authorizers:
 Prasad, who is a PMC on Apache Sentry reviewed the design and confirmed
 Sentry can integrate with the current APIs. Dapeng Sun, a committer on
 Sentry had sme concerns about the IP privileges and how we prioritize
 privileges - but nothing that prevents Sentry from integrating with the
 existing solution, from what I coul see. It seems to me that the
design is
 very generic and adapters can be written for other authorization systems
 (after all, you just need to implement setACL, getACL and Authorize -
all
 pretty basic), although I can't speak for Oracle's Identity Manager
 specifically.

 * Regarding AuthorizationException to indicate that anoperation was
not
 authorized: Sorry I missed this in previous reviewed, but now that I
look
 at it - Many systems intentionally don't return AuthorizationException
when
 READ privilege is missing, since this already gives too much information
 (that the topic exists and that you don't have privileges on it).
Instead
 they return a variant of doesn't exist. I'm wondering if this
approach is
 applicable / desirable for Kafka as well.
 Note that this doesn't remove the need for AuthorizationException - I'm
 just suggesting apossible refinement on its use.

 Gwen



 On Thu, Apr 30, 2015 at 9:52 AM, Parth Brahmbhatt 
 pbrahmbh...@hortonworks.commailto:pbrahmbh...@hortonworks.com wrote:

  Hi Joe, Thanks for taking the time to review.
 
  * All the open issues already have a resolution , I can open a jira
for
  each one and add the resolution to it and resolve them immediately if
you
  want this for tracking purposes.
  * We will update system tests to verify that the code woks. We have
  thorough unit tests for all the new code except for modifications
made to
  KafkaAPI as that has way too many dependencies to be mocked which I
guess
  is the reason for no existing unit tests.
  * I don’t know if I completely understand the concern. We have talked
 with
  Ranger team (Don Bosco Durai) so we at least have one custom
authorizer
  implementation that has approved this design and they will be able to
  inject their authorization framework with current interfaces. Do you
see
  any issue with the design which will prevent anyone from providing a
  custom implementation?
  * Did not understand the concern around wire protocol, we are adding
  AuthorizationException to indicate that an operation was not
authorized.
 
  Thanks
  Parth
 
  On 4/30/15, 5:59 AM, Jun Rao
j...@confluent.iomailto:j...@confluent.io
 wrote:
 
  Joe,
  
  Could you elaborate on why we should not store JSON in ZK? So far,
all
  existing ZK data are in JSON.
  
  Thanks,
  
  Jun

Re: [VOTE] KIP-11- Authorization design for kafka security

[Don Bosco Durai]

Gwen  regarding additional authorizers

I think having these in the system tests duals as both good confidence in
language independency of the changes. It also makes sure that when we
release that we don't go breaking Sentry or Ranger or anyone else that
wants to integate.

As much I would like to have Ranger included in the Kafka system tests, I
don’t think it is the right thing to do. The interface and default Kafka
implementation should be final contract. This also ensures that there are
no 3rd party dependencies within Kafka.

Jun  Could you elaborate on why we should not store JSON in ZK? So
far, all existing ZK data are in JSON.

If I have 1,000,000 users in LDAP and 150 get access to Kafka topics
through this mechanism then I have to go and parse and push all of my
change into zookeeper for it to take affect?

I assume, the entries in the zookeeper will be limited based on the number
of users who will be accessing and the number topics they have permission.
So in your case, for one topic there might be only 150 entries. Or 1
entry, with an array of 150 elements (users).  I don’t know whether this
will be a limitation on the Zookeeper side or Kafka implementation. Based
on the discssion, each broker is going to store this in memory and on
regular interval refresh it. So the interaction with ZooKeeper will be
also limited. I hope this won’t be a huge burden.

If someone wanted to implement SAML I don't think this would work. Not
sure how it would work with NiFi either (something around here I think
maybe 
https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;a=blb;f=nar-
bundles/framework-bundle/framework/web/websecurity/src/main/java/org/apach
e/nifi/web/security/authorization/NiFiAuthorizationService.java;hb=e67eb4f
5).

I would put SAML and any other token based solution (OAuth/OpenId) in the
bucket of Authentication. These would be beyond the scope of the this
(authorization) KIP and should be parallel to the Authentication KIPs. If
any of these solution also provide Authorization capability, then it would
require another (SAML) Authorization implementation of the current
interface. The flow I guess will be, during authentication with SAML, get
the roles from the payload and store it in the session context and during
authorization, get the roles (or privileges) from the session context and
enforce it. 

I hope this addresses some of your concerns.

Thanks

Bosco





On 4/30/15, 11:24 AM, Joe Stein joe.st...@stealth.ly wrote:

Gwen  regarding additional authorizers

I think having these in the system tests duals as both good confidence in
language independency of the changes. It also makes sure that when we
release that we don't go breaking Sentry or Ranger or anyone else that
wants to integrate.

Gwen  Regarding AuthorizationException

Yeah so I have two issues. The one you raised yes, 100%. Also I don't
unerstand how that is not a broker wire protocol response and only a JVM
exception.

Jun  Could you elaborate on why we should not store JSON in ZK? So far,
all existing ZK data are in JSON.

If I have 1,000,000 users in LDAP and 150 get access to Kafka topics
through this mechanism then I have to go and parse and push all of my
changes into zookeeper for it to take affect?

If someone wanted to implement SAML I don't think this would work. Not
sure
how it would work with NiFi either (something around here I think maybe
https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;a=blob;f=nar-
bundles/framework-bundle/framework/web/web-security/src/main/java/org/apac
he/nifi/web/security/authorization/NiFiAuthorizationService.java;hb=e67eb4
f5).

Parth  All the open issues already have a resolution , I can open a jira
for each one and add the resolution to it and resolve them immediately if
you want thisfor tracking purposes.

Are those inline to the question with the li li I didn't quite get
that
section at all. If the open questions are answered then they aren't open
can you tidy that up then.

Parth   We will update system tests to verify that the code works. We
have thorough unit tests for all the new code except for modificaions
made
to KafkaAPI as that has way too many dependencies to be mocked which I
guess is the reason for no existing unit tests.

Can you update the KIP with some more detail about that please.

Parth  I don’t know if I completely understand the concern. We have
talked with Ranger team (Don Bosco Durai) so we at least have one custom
authorizer implementation that as approved this design and they will be
able to inject their authorization framework with current interfaces. Do
you see any issue with the design which will prevent anyone frm providing
a custom implementation?

Maybe a diagram for all of the different parts interacting. I still don't
get why there are no wire protocol changes and just change in the JVM.
What
do non-jvm clients do and how do they work with Kafka. Very confusing,
almost obfuscating.

~ Joestein


On Thu, Apr 30, 2015 at 1:14 PM, Gwen

Re: [DISCUSSION] KIP-11: ACL Management

[Don Bosco Durai]

Jun

Here is the recent interface from Hive. It is grossly simplified from what
you pasted before.
https://hive.apache.org/javadocs/r1.0.0/api/ql/org/apache/hadoop/hive/ql/se
curity/authorization/plugin/HiveAuthorizer.html


At the high level, there is only one authorization method (similar to
Parth’s proposal) for permission check. The method name is
checkPrivileges().

They have also exposed grantPrivileges() and revokePrivileges() APIs (as
suggested by Gwen). It is up to the plugin implementor to do as they wish.
Kafka interface can ignore all Role/Group related privileges for now.

On the Apache Ranger implementation, we store the ACLs in Ranger's central
database. The ACLs can be updated via Ranger UI or REST APIs. Ranger
doesn’t provide CLIs. In the Hive case, when someone uses Hive CLI for
grant/revoke, within HiveServer2 Ranger implementation of grant/revoke
Privileges() methods updates Ranger's database. So regardless how you
configure, the ACLs always lands into the Ranger database and Ranger
plugin uses the policies from Ranger database to enforce them.

Giving the flexibility for the plugin to store the ACLs in their store
helps the plugin implementor to extend the authorization features without
putting too much burden or dependency on Kafka development.

From my (may be biased) opinion, I feel we should keep the default Kafka
plugin implementation and configuration simple for the initial release. As
discussed so far, the default OOTB Kafka implementation should provide
topic level permission for Users and IPs. This will address the
requirements for most users.

I like Gwen simplicity of command line, but agree with Parth that we will
have to make it flexible to add groups, etc in the future.
kafka-topic --topic t1 --grant user --action action
kafka-topic --topic t1 --revoke user --action action


So something like this might be good (but doesn’t have to be exactly like
this):
kafka-topic --perm --topic t1,t2 --grantuser u1,u2 --granthost h1,h2
--revokeuser u3,u4 --revokehost h3,h4 --action a1,a2

Thanks

Bosco


On 4/17/15, 10:48 PM, Parth Brahmbhatt pbrahmbh...@hortonworks.com
wrote:

I have copied Thejas from hive team in cc list. Here is what I learnt from
him

* Hive calls the authorizer plugin if you execute “grant/revoke Operation
to User on Table.  They use this as hive provides the SQL layer and SQL
has standards for grant/revoke which they follow.
* If the plugin provides more entities then what can be expressed by the
above statement (like unix/ldap groups or host level control) you have to
go to the plugin’s CLI/UI to create this acl.

So as mentioned below you will have 2 tools. One for the very basic
grant/revoke access and for anything complex you have a secondary
interface provided by Authorizer plugin.

Thanks
Parth

On 4/17/15, 12:01 PM, Jun Rao j...@confluent.io wrote:

Hi, Parth,

How does this work in Hive? I thought authorization in Hive always goes
through it's SQL cli for any authorization plugin. When integrating with
Ranger(Argus), does Hive do authorization through a separate CLI?

Thanks,

Jun


On Fri, Apr 17, 2015 at 11:01 AM, Parth Brahmbhatt 
pbrahmbh...@hortonworks.com wrote:

 We could do this but I think its too simplistic plus now we are adding
 authorization related options in CLI which I thought everyone wants to
 avoid.

 When I say its too simplistic I mean there are missing options like
 —hosts, what happens when we start supporting group now we will
probably
 end up adding —grant —groups”. I think we will just endup polluting
kafka
 create CLI with all the different acl options or we will have 2 CLIs
one
 for the basic stuff and for anything advance you will have to use a
 different tool. It might be better to just have a single separate ACL
 management CLI.

 Thanks
 Parth

 On 4/17/15, 10:42 AM, Gwen Shapira gshap...@cloudera.com wrote:

 I've probably been a DBA for too long, but I imagined something like:
 kafka-topic --topic t1 --grant user --action action
 kafka-topic --topic t1 --revoke user --action action
 (i.e. the commandline equivalent of grant select on table1 to
 gwenshap and revoke select on table2 from gwenshap)
 
 When you need gazillion of them, you generate a script with gazillion
 of those and execute.
 
 Maybe it just looks reasonable to me because I'm used to it, though :)
 
 Or maybe including the json parsing code in TopicCommand is not so
bad?
 
 
 
 On Fri, Apr 17, 2015 at 10:30 AM, Parth Brahmbhatt
 pbrahmbh...@hortonworks.com wrote:
  * Yes, Acl pretty much captures everything. Originally I had
resource as
  part of Acls, we can go back to that.
  * The describe can call getAcl and I plan to do so. addAcl is tricky
  because the user will have to specify the acls through command
lines,
  which will probably be a location to some file. Basically the CLI
won¹t
  know how to parse user input and convert it to a principal/acl that
the
  plugin understands. We could add an API in authorizer that can take
a
 file
  as input if we 

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Don Bosco Durai]

I also feel, having just IP would be more appropriate. Host lookup will
unnecessary slow things down and would be insecure as you pointed out.

With IP, it will be also able to setup policies (in future if needed) with
ranges or netmasks and it would be more scalable.

Bosco


On 4/14/15, 1:40 PM, Michael Herstine mherst...@linkedin.com.INVALID
wrote:

Hi Parth,

Sorry to chime in so late, but I’ve got a minor question on the KIP.

Several methods take a parameter named “host” of type String. Is that
intended to be a hostname, or an IP address? If the former, I’m curious as
to how that’s found (in my experience, when accepting an incoming socket
connection, you only know the IP address, and there isn’t a way to map
that to a hostname without a round trip to a DNS server, which is insecure
anyway).


On 3/25/15, 1:07 PM, Parth Brahmbhatt pbrahmbh...@hortonworks.com
wrote:

Hi all,

I have modified the KIP to reflect the recent change request from the
reviewers. I have been working on the code and I have the server side
code
for authorization ready. I am now modifying the command line utilities. I
would really appreciate if some of the committers can spend sometime to
review the KIP so we can make progress on this.

Thanks
Parth

On 3/18/15, 2:20 PM, Michael Herstine mherst...@linkedin.com.INVALID
wrote:

Hi Parth,

Thanks! A few questions:

1. Do you want to permit rules in your ACLs that DENY access as well as
ALLOW? This can be handy setting up rules that have exceptions. E.g.
“Allow principal P to READ resource R from all hosts” with “Deny
principal
P READ access to resource R from host H1” in combination would allow P
to
READ R from all hosts *except* H1.

2. When a topic is newly created, will there be an ACL created for it?
If
not, would that not deny subsequent access to it?

(nit) Maybe use Principal instead of String to represent principals?


On 3/9/15, 11:48 AM, Don Bosco Durai bo...@apache.org wrote:

Parth

Overall it is looking good. Couple of questionsŠ

- Can you give an example how the policies will look like in the
default
implementation?
- In the operations, can we support ³CONNECT² also? This can be used
during Session connection
- Regarding access control for ³Topic Creation², since we can¹t do it
on
the server side, can we de-scope it for? And plan it as a future
feature
request?

Thanks

Bosco

 

On 3/6/15, 8:10 AM, Harsha ka...@harsha.io wrote:

Hi Parth,
Thanks for putting this together. Overall it looks good to
me. Although AdminUtils is a concern KIP-4  can probably
fix
that part.
Thanks,
Harsha

On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
 Forgot to add links to wiki and jira.
 
 Link to wiki:
 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizat
i
o
n
+
Interface
 Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
 
 Thanks
 Parth
 
 From: Parth Brahmbhatt
 pbrahmbh...@hortonworks.commailto:pbrahmbh...@hortonworks.com
 Date: Thursday, March 5, 2015 at 10:33 AM
 To: dev@kafka.apache.orgmailto:dev@kafka.apache.org
 dev@kafka.apache.orgmailto:dev@kafka.apache.org
 Subject: [DISCUSS] KIP-11- Authorization design for kafka security
 
 Hi,
 
 KIP-11 is open for discussion , I have updated the wiki with the
design
 and open questions.
 
 Thanks
 Parth

Re: [KIP-DISCUSSION] KIP-13 Quotas

[Don Bosco Durai]

Gwen

I feel, we should assume the metrics server is an external system and the
access to the server should be managed by the security features provided
by the system. This way, it would be the Kafka System Administrator
responsibility to ensure the metrics system is properly firewall¹ed or
access controlled. 

Also, we won¹t be dependent on the metrics system chosen by the user.

I hope, this makes sense.

Thanks

Bosco




On 4/7/15, 9:54 AM, Gwen Shapira gshap...@cloudera.com wrote:

Re (1):
We have no authorization story on the metrics collected by brokers, so I
assume that access to broker metrics means knowing exactly which topics
exist and their throughputs. (Prath and Don, correct me if I got it
wrong...)
Secure environments will strictly control access to this information, so I
am pretty sure the client developers will not have access to server
metrics
at all.

Gwen

On Tue, Apr 7, 2015 at 7:41 AM, Jay Kreps jay.kr...@gmail.com wrote:

 Totally. But is that the only use? What I wanted to flesh out was
whether
 the goal was:
 1. Expose throttling in the client metrics
 2. Enable programmatic response (i.e. stop sending stuff or something
like
 that)

 I think I kind of understand (1) but let's get specific on the metric we
 would be adding and what exactly you would expose  in a dashboard. For
 example if the goal is just monitoring do I really want a boolean flag
for
 is_throttled or do I want to know how much I am being throttled (i.e.
 throttle_pct might indicate the percent of your request time that was
due
 to throttling or something like that)? If I am 1% throttled that may be
 irrelevant but 99% throttled would be quite relevant? Not sure I agree,
 just throwing that out there...

 For (2) the prior discussion seemed to kind of allude to this but I
can't
 really come up with a use case. Is there one?

 If it is just (1) I think the question is whether it really helps much
to
 have the metric on the client vs the server. I suppose this is a bit
 environment specific. If you have a central metrics system it shouldn't
 make any difference, but if you don't I suppose it does.

 -Jay

 On Mon, Apr 6, 2015 at 7:57 PM, Gwen Shapira gshap...@cloudera.com
 wrote:

  Here's a wild guess:
 
  An app developer included a Kafka Producer in his app, and is not
happy
  with the throughput. He doesn't have visibility into the brokers since
 they
  are owned by a different team. Obviously the first instinct of a
 developer
  who knows that throttling exists is to blame throttling for any
slowdown
 in
  the app.
  If he doesn't have a way to know from the responses whether or not his
 app
  is throttled, he may end up calling Aditya at 4am asked Hey, is my
app
  throttled?.
 
  I assume Aditya is trying to avoid this scenario.
 
  On Mon, Apr 6, 2015 at 7:47 PM, Jay Kreps jay.kr...@gmail.com wrote:
 
   Hey Aditya,
  
   2. I kind of buy it, but I really like to understand the details of
the
  use
   case before we make protocol changes. What changes are you
proposing in
  the
   clients for monitoring and how would that be used?
  
   -Jay
  
   On Mon, Apr 6, 2015 at 10:36 AM, Aditya Auradkar 
   aaurad...@linkedin.com.invalid wrote:
  
Hi Jay,
   
2. At this time, the proposed response format changes are only for
monitoring/informing clients. As Jun mentioned, we get instance
level
monitoring in this case since each instance that got throttled
will
  have
   a
metric confirming the same. Without client level monitoring for
this,
   it's
hard for application developers to find if they are being
throttled
  since
they will also have to be aware of all the brokers in the cluster.
 This
   is
quite problematic for large clusters.
   
It seems nice for app developers to not have to think about kafka
   internal
metrics and only focus on the metrics exposed on their instances.
   Analogous
to having client-sde request latency metrics. Basically, we want
an
  easy
way for clients to be aware if they are being throttled.
   
4. For purgatory v delay queue, I think we are on the same page. I
 feel
   it
is nicer to use the purgatory but I'm happy to use a DelayQueue if
  there
are performance implications. I don't know enough about the
current
 and
Yasuhiro's new implementation to be sure one way or the other.
   
Stepping back, I think these two things are the only remaining
point
 of
discussion within the current proposal. Any concerns if I started
a
   voting
thread on the proposal after the KIP discussion tomorrow?
(assuming
 we
reach consensus on these items)
   
Thanks,
Aditya

From: Jay Kreps [jay.kr...@gmail.com]
Sent: Saturday, April 04, 2015 1:36 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
   
Hey Aditya,
   
2. For the return flag I'm not terribly particular. If we want to
add
  it
let's fully think through how it will 

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Don Bosco Durai]

 of forwarding requests to a broker host. Given all
 the code is executed on client side there is no easy way to perform
 authorization. “ - since we are adding the admin protocol requests in
 KIP-4, perhaps addressing those makes sense.

   *   Yes, We will have to wait for KIP-4 to be delivered.

 9. I didn’t see a specification of what is “resource”, I assume its an
 enum with things like Topic and… ?

   *   Resource is not an enum, Right now its just name of the topic.
 Currently we will not be able to support cluster actions like CREATE in
 which case the resource can be some constant string like
“Kafka-Cluster”.

 10. I’m also unclear on where and how “PermissionType” will be used
 and what does “DENY takes precedence” mean here.

   *   I originally did not have the “PermissionType” but someone
suggested
 in DISCUSS thread that we should add support for ALLOW and DENY acls.
The
 use case is to support acls like “Allow user U to Perform READ from all
 Hosts except from Host1 and Host2”. Deny take precedence only means if
you
 have define 2 ACLs for the same user/resource/operation. DENY acl will
take
 effect I.e. “Allow user X to read from *” and “Deny User X to Read from
 host1 and host2” will satisfy the use case described in the previous
 statement.

 11. What does acls on zookeeper node look like given all our admin
 APIs are currently performed directly from client?” - “secure mode”
 Kafka will need to set ACLs on ZK (using ZK’s model of ACLs) and both
 Kafka and everyone else will need to use them (this is limited to
 Kerberos authentication AFAIK.)

   *   Yes, again I think this can be a separate issue for now and can
only
 be worked on after KIP-4 is delivered.

 12. Do we want to support group acls as part of this authorizer? Do
 we want to support principal to local user mapping? If yes we need to
 add plugins for UserToGroupMapper and PrincipalToUserMapper.” -
 Sentry uses Groups for authorizing, so we need to support that. I
 figured that as long as the API specifies Principal, it typically
 contains both user and group, so nothing else is needed. Did I miss
 anything?

   *   Once we support group acls we will need someway to indicate if a
 principal is of type group or user(as part of acl) or we can have group
 acls and user acls stored separately with topic config. We will also
need a
 way to map an authenticated user to list of groups the user belongs to.

 13. It looks like the Authorizer stores the ACLs and not Kafka. So we
 need an API for Kafka to notify Authorizer when a topic is added and
 when ACLs are modified, right? I didn’t see that.

   *   ACLs will be stored under /topic/config at time of topic creation
in
 json format. Authorizer will get these acls using newly introduced
 TopicConfigCache which gets updated anytime topic config changes.

 14. Are we going to have any API for Kafka to give out the ACLs on a
 topic? Or we leave this to the Authorizer?

   *   I think it is better to leave this out side of Kafka. Mainly
because
 most 3rd party authorizers will have their own ACL stores and
configuration
 Uis like (Ranger-Argus not sure what are they calling it now).


 On Wed, Mar 25, 2015 at 9:26 PM, Neha Narkhede
n...@confluent.iomailto:
 n...@confluent.io wrote:
 Parth,

 We can make some 15 mins or so to discuss this at the next KIP hangout.

 Thanks,
 Neha

 On Wed, Mar 25, 2015 at 1:07 PM, Parth Brahmbhatt 
 pbrahmbh...@hortonworks.commailto:pbrahmbh...@hortonworks.com wrote:

 Hi all,

 I have modified the KIP to reflect the recent change request from the
 reviewers. I have been working on the code and I have the server side
code
 for authorization ready. I am now modifying the command line utilities.
I
 would really appreciate if some of the committers can spend sometime to
 review the KIP so we can make progress on this.

 Thanks
 Parth

 On 3/18/15, 2:20 PM, Michael Herstine mherst...@linkedin.com.INVALID
 mailto:mherst...@linkedin.com.INVALID
 wrote:

 Hi Parth,
 
 Thanks! A few questions:
 
 1. Do you want to permit rules in your ACLs that DENY access as well as
 ALLOW? This can be handy setting up rules that have exceptions. E.g.
 “Allow principal P to READ resource R from all hosts” with “Deny
principal
 P READ access to resource R from host H1” in combination would allow P
to
 READ R from all hosts *except* H1.
 
 2. When a topic is newly created, will there be an ACL created for it?
If
 not, would that not deny subsequent access to it?
 
 (nit) Maybe use Principal instead of String to represent principals?
 
 
 On 3/9/15, 11:48 AM, Don Bosco Durai bo...@apache.orgmailto:
 bo...@apache.org wrote:
 
 Parth
 
 Overall it is looking good. Couple of questionsŠ
 
 - Can you give an example how the policies will look like in the
default
 implementation?
 - In the operations, can we support ³CONNECT² also? This can be used
 during Session connection
 - Regarding access control for ³Topic Creation², since we can¹t do it
on
 the server side, can we de-scope

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Don Bosco Durai]

Related interesting question:
Since a broker is a consumer (of lead replicas), how do we handle the
broker level of permissions? Do we hardcode a broker-principal name
and automatically authorize brokers to do anything? Or is there a
cleaner way?

I feel, in Kerberos environment, “kafka” keytab would be the ideal
solution. And “kafka” principal will need to be white listed. SSL
certificate is another option, but it would be painful to set it up. IP
whitelisting is another low impact, but less secure option.

Bosco



On 3/31/15, 10:20 AM, Gwen Shapira gshap...@cloudera.com wrote:

Related interesting question:
Since a broker is a consumer (of lead replicas), how do we handle the
broker level of permissions? Do we hardcode a broker-principal name
and automatically authorize brokers to do anything? Or is there a
cleaner way?


On Tue, Mar 31, 2015 at 10:17 AM, Don Bosco Durai bo...@apache.org
wrote:
21. Operation: What about other types of requests not covered in the
list,
 such as committing and fetching offsets, list topics, fetching consumer
 metadata, heartbeat, join group, etc?

 Would “CONFIGURE”, “DESCRIBE”, etc take care of this? Or should we add
 high level grouping like “ADMIN”, “OPERATIONS/MANAGEMENT” to cover
related
 permissions?

 Bosco



 On 3/31/15, 9:21 AM, Jun Rao j...@confluent.io wrote:

Thanks for the writeup. A few more comments.

20. I agree that it would be better to do this after KIP-4 (admin
commands)
is done. With KIP-4, all admin operations will be sent as requests to
the
brokers instead of accessing ZK directly. This will make authorization
easier.

21. Operation: What about other types of requests not covered in the
list,
such as committing and fetching offsets, list topics, fetching consumer
metadata, heartbeat, join group, etc?

22. TopicConfigCache: We will need such a cache in KIP-4 as well. It
would
be useful to make sure that the implementation can be reused.

23. Authorizer:
23.1 Do cluster level operations go through authorize() too? If so, what
will be the resource?
23.2 I assume that the authorize() check will be called on every
request.
So, we will have to make sure that the check is cheap.

24. The acl json string in the config: Should we version this so that we
can evolve it in the future (e.g., adding group support)?

Jun

On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt 
pbrahmbh...@hortonworks.com wrote:

 Hi Gwen,

 Thanks a lot for taking the time to review this. I have tried to
address
 all your questions below.

 Thanks
 Parth
 On 3/28/15, 8:08 PM, Gwen Shapira gshap...@cloudera.commailto:
 gshap...@cloudera.com wrote:

 Preparing for Tuesday meeting, I went over the KIP :)

 First, Parth did an amazing job, the KIP is fantastic - detailed and
 readable. Thank you!

 Second, I have a lng list of questions :) No objections, just some
 things I'm unclear on and random minor comments. In general, I like
 the design, I just feel I'm missing parts of the picture.

 1. Yes, Create topic will have an optional acls, the output of
 describe will display owner and acls and alter topic will allow to
 modify the acls.”  - will be nice to see what the CLI will look like.

   *   I will modify the KIP but I was going to add “—acl
acl-file.json”
 to create-topic and alter-topic.

 2. I like the addition of Topic owner. We made the mistake of
 forgetting about it when adding authorization to Sqoop2. We probably
 want to add “chown” command to the topic commands.

   *   Again we can add “—owner user-name” to alter topic.

 3. Kafka server will read authorizer.class” config value at startup
 time, create an instance of the specified class and call initialize
 method.
 We’ll need to validate that users specify only one of those.

   *   The config type will be string so type validation should take
care
 of it.

 4. One added assumption is that on non-secure connections the session
 will have principal set to an object whose name() method will return
 Anonymous”.
 Can we keep DrWho? :)

   *   Sure, its up to you actually as you are the owner of the jira
that
 introduces session concept.

 5. For cluster actions that do not apply to a specific topic like
 CREATE we have 2 options. We can either add a broker config called
 broker.acls which will point to a json file. This file will be
 available on all broker hosts and authorizer will read the acls on
 initialization and keep refreshing it every X minutes. Any changes
 will require re-distribution of the acl json file. Alternatively we
 can add a zookeeper path /brokers/acls and store the acl json as data.
 Authorizer can refresh the acl from json every X minutes. In absence
 of broker acls the authorizer will fail open, in other words it will
 allow all users from all hosts to perform all cluster actions”
 I prefer a file to ZK - since thats where we store all use-defined
 configurations for now. Everyone knows how to secure a file system :)

   *   I will let everyone vote, file system is fine by me.

 6. When an Acl

[jira] [Commented] (KAFKA-1688) Add authorization interface and naive implementation

[Don Bosco Durai (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378100#comment-14378100
 ] 

Don Bosco Durai commented on KAFKA-1688:


[~prasadm], 

I agree with you, we should support ALL or equivalent keyword like (*) in the 
default implementation. 

I remember the Hierarchical subject brought up during one the google hangout 
discussion around authorization. I don't think there were any resolutions 
around it.

Does it make sense to make this as a custom implementation feature? So for 
OOTB, it would be be just topic name, but anyone who want to implement 
hierarchical privileges can parse the topic name and use . or any other 
supported character has delimiter and provide namespace/database like 
permissions.

FYI, it seems, hierarchical Topics was discussed back in 2012 
https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics


 Add authorization interface and naive implementation
 

 Key: KAFKA-1688
 URL: https://issues.apache.org/jira/browse/KAFKA-1688
 Project: Kafka
  Issue Type: Sub-task
  Components: security
Reporter: Jay Kreps
Assignee: Parth Brahmbhatt
 Fix For: 0.8.3


 Add a PermissionManager interface as described here:
 https://cwiki.apache.org/confluence/display/KAFKA/Security
 (possibly there is a better name?)
 Implement calls to the PermissionsManager in KafkaApis for the main requests 
 (FetchRequest, ProduceRequest, etc). We will need to add a new error code and 
 exception to the protocol to indicate permission denied.
 Add a server configuration to give the class you want to instantiate that 
 implements that interface. That class can define its own configuration 
 properties from the main config file.
 Provide a simple implementation of this interface which just takes a user and 
 ip whitelist and permits those in either of the whitelists to do anything, 
 and denies all others.
 Rather than writing an integration test for this class we can probably just 
 use this class for the TLS and SASL authentication testing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Don Bosco Durai]

Parth

Overall it is looking good. Couple of questionsŠ

- Can you give an example how the policies will look like in the default
implementation?
- In the operations, can we support ³CONNECT² also? This can be used
during Session connection
- Regarding access control for ³Topic Creation², since we can¹t do it on
the server side, can we de-scope it for? And plan it as a future feature
request?

Thanks

Bosco

 

On 3/6/15, 8:10 AM, Harsha ka...@harsha.io wrote:

Hi Parth,
Thanks for putting this together. Overall it looks good to
me. Although AdminUtils is a concern KIP-4  can probably fix
that part.
Thanks,
Harsha

On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
 Forgot to add links to wiki and jira.
 
 Link to wiki:
 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+
Interface
 Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
 
 Thanks
 Parth
 
 From: Parth Brahmbhatt
 pbrahmbh...@hortonworks.commailto:pbrahmbh...@hortonworks.com
 Date: Thursday, March 5, 2015 at 10:33 AM
 To: dev@kafka.apache.orgmailto:dev@kafka.apache.org
 dev@kafka.apache.orgmailto:dev@kafka.apache.org
 Subject: [DISCUSS] KIP-11- Authorization design for kafka security
 
 Hi,
 
 KIP-11 is open for discussion , I have updated the wiki with the design
 and open questions.
 
 Thanks
 Parth

Re: Review Request 28769: Patch for KAFKA-1809

[Don Bosco Durai]

+1

I also feel, having security.* would be easy going forward.

Thanks

Bosco


On 1/29/15, 6:08 AM, Jeff Holoman jeff.holo...@gmail.com wrote:



 On Jan. 23, 2015, 1:57 a.m., Jun Rao wrote:
  core/src/main/scala/kafka/server/KafkaConfig.scala, lines 182-183
  
https://reviews.apache.org/r/28769/diff/12/?file=820431#file820431line18
2
 
  Since this is also used for communication btw the controller and
the brokers, perhaps it's better named as sth like
intra.broker.security.protocol?

Maybe it makese sense to prepend all security related configs with
security, eg: security.intra.broker.protocol,
security.new.param.for.future.jira With all of the upcoming changes it
would make security realted configs easy to spot.


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28769/#review69281
---


On Jan. 28, 2015, 6:26 p.m., Gwen Shapira wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/28769/
 ---
 
 (Updated Jan. 28, 2015, 6:26 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1809
 https://issues.apache.org/jira/browse/KAFKA-1809
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1890 Fix bug preventing Mirror Maker from successful rebalance;
reviewed by Gwen Shapira and Neha Narkhede
 
 
 first commit of refactoring.
 
 
 changed topicmetadata to include brokerendpoints and fixed few unit
tests
 
 
 fixing systest and support for binding to default address
 
 
 fixed system tests
 
 
 fix default address binding and ipv6 support
 
 
 fix some issues regarding endpoint parsing. Also, larger segments for
systest make the validation much faster
 
 
 added link to security wiki in doc
 
 
 fixing unit test after rename of ProtocolType to SecurityProtocol
 
 
 Following Joe's advice, added security protocol enum on client side,
and modified protocol to use ID instead of string.
 
 
 validate producer config against enum
 
 
 add a second protocol for testing and modify SocketServerTests to check
on multi-ports
 
 
 Reverted the metadata request changes and removed the explicit security
protocol argument. Instead the socketserver will determine the protocol
based on the port and add this to the request
 
 
 bump version for UpdateMetadataRequest and added support for rolling
upgrades with new config
 
 
 following tests - fixed LeaderAndISR protocol and ZK registration for
backward compatibility
 
 
 cleaned up some changes that were actually not necessary. hopefully
making this patch slightly easier to review
 
 
 undoing some changes that don't belong here
 
 
 bring back config lost in cleanup
 
 
 fixes neccessary for an all non-plaintext cluster to work
 
 
 minor modifications following comments by Jun
 
 
 added missing license
 
 
 formatting
 
 
 clean up imports
 
 
 cleaned up V2 to not include host+port field. Using use.new.protocol
flag to decide which version to serialize
 
 
 change endpoints collection in Broker to Map[protocol, endpoint],
mostly to be clear that we intend to have one endpoint per protocol
 
 
 validate that listeners and advertised listeners have unique ports and
protocols
 
 
 support legacy configs
 
 
 some fixes following rebase
 
 
 Reverted to backward compatible zk registration, changed
use.new.protocol to support multiple versions and few minor fixes
 
 
 Diffs
 -
 
   
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.ja
va 8b3e565edd1ae04d8d34bd9f1a41e9fa8c880a75
   
clients/src/main/java/org/apache/kafka/common/protocol/ApiVersion.java
PRE-CREATION 
   
clients/src/main/java/org/apache/kafka/common/protocol/SecurityProtocol.j
ava PRE-CREATION 
   clients/src/main/java/org/apache/kafka/common/utils/Utils.java
527dd0f9c47fce7310b7a37a9b95bf87f1b9c292
   clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java
a39fab532f73148316a56c0f8e9197f38ea66f79
   config/server.properties 1614260b71a658b405bb24157c8f12b1f1031aa5
   core/src/main/scala/kafka/admin/AdminUtils.scala
28b12c7b89a56c113b665fbde1b95f873f8624a3
   core/src/main/scala/kafka/admin/TopicCommand.scala
285c0333ff43543d3e46444c1cd9374bb883bb59
   core/src/main/scala/kafka/api/ConsumerMetadataResponse.scala
84f60178f6ebae735c8aa3e79ed93fe21ac4aea7
   core/src/main/scala/kafka/api/LeaderAndIsrRequest.scala
4ff7e8f8cc695551dd5d2fe65c74f6b6c571e340
   core/src/main/scala/kafka/api/TopicMetadata.scala
0190076df0adf906ecd332284f222ff974b315fc
   core/src/main/scala/kafka/api/TopicMetadataResponse.scala
92ac4e687be22e4800199c0666bfac5e0059e5bb
   core/src/main/scala/kafka/api/UpdateMetadataRequest.scala
530982e36b17934b8cc5fb668075a5342e142c59
   core/src/main/scala/kafka/client/ClientUtils.scala
ebba87f0566684c796c26cb76c64b4640a5ccfde
   

[jira] [Commented] (KAFKA-1688) Add authorization interface and naive implementation

[Don Bosco Durai (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283104#comment-14283104
 ] 

Don Bosco Durai commented on KAFKA-1688:


[~charmalloc], can you help me create a new Wiki page to put the high level 
design for this feature? It will be easier to discuss based on that.

Thanks


 Add authorization interface and naive implementation
 

 Key: KAFKA-1688
 URL: https://issues.apache.org/jira/browse/KAFKA-1688
 Project: Kafka
  Issue Type: Sub-task
  Components: security
Reporter: Jay Kreps
Assignee: Sriharsha Chintalapani
 Fix For: 0.8.3


 Add a PermissionManager interface as described here:
 https://cwiki.apache.org/confluence/display/KAFKA/Security
 (possibly there is a better name?)
 Implement calls to the PermissionsManager in KafkaApis for the main requests 
 (FetchRequest, ProduceRequest, etc). We will need to add a new error code and 
 exception to the protocol to indicate permission denied.
 Add a server configuration to give the class you want to instantiate that 
 implements that interface. That class can define its own configuration 
 properties from the main config file.
 Provide a simple implementation of this interface which just takes a user and 
 ip whitelist and permits those in either of the whitelists to do anything, 
 and denies all others.
 Rather than writing an integration test for this class we can probably just 
 use this class for the TLS and SASL authentication testing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1688) Add authorization interface and naive implementation

[Don Bosco Durai (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283155#comment-14283155
 ] 

Don Bosco Durai commented on KAFKA-1688:


My id is bosco. I think I might already have write access to the Wiki. It 
would be good if you can create a new page where I can start putting content, 
which can be linked from the requirement document 
https://cwiki.apache.org/confluence/display/KAFKA/Security. If there is already 
a design page for security, then I can use that also.

Thanks

 Add authorization interface and naive implementation
 

 Key: KAFKA-1688
 URL: https://issues.apache.org/jira/browse/KAFKA-1688
 Project: Kafka
  Issue Type: Sub-task
  Components: security
Reporter: Jay Kreps
Assignee: Sriharsha Chintalapani
 Fix For: 0.8.3


 Add a PermissionManager interface as described here:
 https://cwiki.apache.org/confluence/display/KAFKA/Security
 (possibly there is a better name?)
 Implement calls to the PermissionsManager in KafkaApis for the main requests 
 (FetchRequest, ProduceRequest, etc). We will need to add a new error code and 
 exception to the protocol to indicate permission denied.
 Add a server configuration to give the class you want to instantiate that 
 implements that interface. That class can define its own configuration 
 properties from the main config file.
 Provide a simple implementation of this interface which just takes a user and 
 ip whitelist and permits those in either of the whitelists to do anything, 
 and denies all others.
 Rather than writing an integration test for this class we can probably just 
 use this class for the TLS and SASL authentication testing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1722) static analysis code coverage for pci audit needs

[Don Bosco Durai (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14282002#comment-14282002
 ] 

Don Bosco Durai commented on KAFKA-1722:


There few things to note here:
- Instrumentation and scanning takes significant amount time (at least in java)
- There is a upfront cost to review and write rules to eliminate false positives
- There is routine cost to eliminate false positives

If we can setup this process, then it will be very ideal and beneficial. It 
would be good to have an build option to optionally run the scanning before 
committing the code. 

Also, by increase/decrease code coverage, do you mean by number of lines or 
issues? Because number of lines can decrease if a piece of code is optimized.


 static analysis code coverage for pci audit needs
 -

 Key: KAFKA-1722
 URL: https://issues.apache.org/jira/browse/KAFKA-1722
 Project: Kafka
  Issue Type: Bug
  Components: security
Reporter: Joe Stein
Assignee: Ashish Kumar Singh
 Fix For: 0.9.0


 Code coverage is a measure used to describe the degree to which the source 
 code of a product is tested. A product with high code coverage has been more 
 thoroughly tested and has a lower chance of containing software bugs than a 
 product with low code coverage. Apart from PCI audit needs, increasing user 
 base of Kafka makes it important to increase code coverage of Kafka. 
 Something just can not be improved without being measured.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1722) static analysis code coverage for pci audit needs

[Don Bosco Durai (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281981#comment-14281981
 ] 

Don Bosco Durai commented on KAFKA-1722:


Ashish, Coverity is another option. They are free for open source projects. I 
have been scanning for most of the Hadoop projects.

There is already a project created for Kafka 
(https://scan.coverity.com/projects/1340). I am not sure who is the owner, but 
if you want I can investigate that path. 

I had checked with Coverity before and they don't support Scala yet. So it will 
be only for the java components.


 static analysis code coverage for pci audit needs
 -

 Key: KAFKA-1722
 URL: https://issues.apache.org/jira/browse/KAFKA-1722
 Project: Kafka
  Issue Type: Bug
  Components: security
Reporter: Joe Stein
Assignee: Ashish Kumar Singh
 Fix For: 0.9.0


 Code coverage is a measure used to describe the degree to which the source 
 code of a product is tested. A product with high code coverage has been more 
 thoroughly tested and has a lower chance of containing software bugs than a 
 product with low code coverage. Apart from PCI audit needs, increasing user 
 base of Kafka makes it important to increase code coverage of Kafka. 
 Something just can not be improved without being measured.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1722) static analysis code coverage for pci audit needs

[Don Bosco Durai (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281992#comment-14281992
 ] 

Don Bosco Durai commented on KAFKA-1722:


coverall also seems to be good. It says on it's website that it supports scala. 
Not sure to what level.

Have you thought about automating the build and submission? Coverity can be be 
integrated with Travis CI, so it is easy to schedule the build and have results 
shared with everyone.


 static analysis code coverage for pci audit needs
 -

 Key: KAFKA-1722
 URL: https://issues.apache.org/jira/browse/KAFKA-1722
 Project: Kafka
  Issue Type: Bug
  Components: security
Reporter: Joe Stein
Assignee: Ashish Kumar Singh
 Fix For: 0.9.0


 Code coverage is a measure used to describe the degree to which the source 
 code of a product is tested. A product with high code coverage has been more 
 thoroughly tested and has a lower chance of containing software bugs than a 
 product with low code coverage. Apart from PCI audit needs, increasing user 
 base of Kafka makes it important to increase code coverage of Kafka. 
 Something just can not be improved without being measured.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1688) Add authorization interface and naive implementation

[Don Bosco Durai (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178799#comment-14178799
 ] 

Don Bosco Durai edited comment on KAFKA-1688 at 10/21/14 6:33 PM:
--

Joe/Jay, I can look into this. Can you assign this to me? Thanks


was (Author: bosco):
Joe/Jay, I can look into this. Thanks

 Add authorization interface and naive implementation
 

 Key: KAFKA-1688
 URL: https://issues.apache.org/jira/browse/KAFKA-1688
 Project: Kafka
  Issue Type: Sub-task
  Components: security
Reporter: Jay Kreps

 Add a PermissionManager interface as described here:
 https://cwiki.apache.org/confluence/display/KAFKA/Security
 (possibly there is a better name?)
 Implement calls to the PermissionsManager in KafkaApis for the main requests 
 (FetchRequest, ProduceRequest, etc). We will need to add a new error code and 
 exception to the protocol to indicate permission denied.
 Add a server configuration to give the class you want to instantiate that 
 implements that interface. That class can define its own configuration 
 properties from the main config file.
 Provide a simple implementation of this interface which just takes a user and 
 ip whitelist and permits those in either of the whitelists to do anything, 
 and denies all others.
 Rather than writing an integration test for this class we can probably just 
 use this class for the TLS and SASL authentication testing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)