Hi, Yes, the token details and scram credentials are stored in Zookeeper and clients connects using tokenId and hmac for scram authentication. We plan to implement token storage as pluggable interface.
Thanks, Manikumar On Thu, Nov 2, 2017 at 2:09 PM, Postmann, P. (Peter) < peter.postm...@ing.com.invalid> wrote: > Manikumar, Sönke! > > Thanks, the Patch looks very promising. > > As far as I understood the token is stored in Zookeeper and when the > clients reconnects or connects to another broker it uses the tokened and > hmac for authentication. > > I think that’s an optimization. It could be not used and instead we send > the token with every new connection made (and don’t store anything in > Zookeeper). > > Is this right or do I overlook something? > > Kind Regards, > Peter > > -----Original Message----- > From: Manikumar [mailto:manikumar.re...@gmail.com] > Sent: Montag, 30. Oktober 2017 08:53 > To: dev@kafka.apache.org > Subject: Re: Use self contained tokens instead of ACL > > Hi, > > In the first phase, we are trying implement the components/design > discussed in the KIP. > Yes, we can definitely improve some of the components to be more > extensible. > We are planning to implement in future KIPs/PRs. > > Thanks > > > On Fri, Oct 27, 2017 at 8:22 PM, Sönke Liebau < soenke.lie...@opencore.com > .invalid> wrote: > > > Hi Manikumar, > > > > I've looked over the KIP and had a quick look at the code in the PR as > > well. In principle I think this would help Peter along depending on > > how plugable some of the components are. Since Peter wants to generate > > Tokens not in Kafka but in an external System the entire part in Kafka > > of generating DelegationTokens would simply not be used, which I think > > would be fine. To validate externally generated tokens an option to > > substitute for example the TokenCache for a custom implementation > > or/and substitute the method of authenticating a delegation token for a > custom class. > > > > Apologies for asking questions I could look up in the code myself, but > > at a first glance I haven't seen any indications of this token system > > being extendable, do you plan to allow extending the system to > > different external token providers? OAuth would come to mind as a > > fairly wide spread candidate that could probably be implemented fairly > easily. > > > > Kind regards, > > Sönke > > > > On Fri, Oct 27, 2017 at 11:17 AM, Manikumar > > <manikumar.re...@gmail.com> > > wrote: > > > > > Hi, > > > > > > We have a accepted KIP for adding delegation token support for Kafka. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 48+Delegation+token+support+for+Kafka > > > > > > currently the PR is under review process. Maybe this can used be as > > > starting point for your requirement. > > > > > > https://github.com/apache/kafka/pull/3616 > > > > > > > > > > > > On Fri, Oct 27, 2017 at 2:34 PM, Sönke Liebau < > > > soenke.lie...@opencore.com.invalid> wrote: > > > > > > > Hi Peter, > > > > > > > > thanks for the explanation, it all makes sense now :) > > > > > > > > I can't say that I immediately see an easy way forward though to > > > > be > > > honest. > > > > The big issue, I think, is getting the token to Kafka (and > > > > hopefully > > > there > > > > is an easy way that I simply don't know of and someone will > > > > correct > > me) - > > > > implementing a custom principalbuilder and authorizer should be > > > > almost trivial. > > > > > > > > If transmitting the token as part of the ssl certificate or a > > > > Kerberos ticket is out though the air gets a bit thin if you don't > > > > want to > > > maintain > > > > your own fork of Kafka. The only potential solution that I can > > > > come up > > > with > > > > is to piggyback on SASL and provide your own LoginModule in > > > > Kafka's > > jaas > > > > file. If you use the SASL_SSL endpoint certificate checking should > > still > > > > have occured before the SASL handshake is initialized, so you > > > authenticated > > > > the user at that point. You could then use that handshake to > > > > transmit > > > your > > > > token, have your custom principalbuilder extract the topics from > > > > that > > and > > > > your custom authorizer authorize based on the extracted topicnames. > > > > A word of caution though: this is based on a few minutes looking > > > > at > > code > > > > and my dangerous half knowledge of SASL, so there are any number > > > > of > > > things > > > > that could make this impossible, either with SASL or in the Kafka > > > codebase > > > > itself. Might be a direction to explore though. > > > > > > > > Hopefully that makes sense and is targeted at least in the > > > > vicinity of > > > whet > > > > you are looking for? > > > > > > > > Kind regards, > > > > Sönke > > > > > > > > On Fri, Oct 27, 2017 at 9:33 AM, Postmann, P. (Peter) < > > > > peter.postm...@ing.com.invalid> wrote: > > > > > > > > > Hi Sönke, > > > > > > > > > > Thanks for your feedback, sorry that I didn’t gave you the whole > > > picture > > > > > in first place: > > > > > > > > > > We are using an Architecture, which tries to avoid to fetch or > > > > > pull anything from a 3rd party during runtime. Therefore we are > > > > > using self-contained tokens and client side load balancing with > > > > > a micro > > > service > > > > > alike architecture. > > > > > > > > > > In this architecture we have two tokens: > > > > > - the manifest which enabled services to provide APIs > > > > > - the peer token which enables services to call APIs > > > > > > > > > > API providers publish their APIs in a Portal. API consumers > > > > > subscribe > > > to > > > > > those APIs. The portal generates a manifest for the provider and > > > > > a > > peer > > > > > token for the consumer. Both tokens contain a list of endpoints > > > > > and > > are > > > > > signed by the portal. The tokens are valid for a certain amount > > > > > of > > > time. > > > > > > > > > > Furthermore we use a Service Registry to discover those > > > > > services. The > > > > flow > > > > > works as follows: > > > > > > > > > > - A service instance registers itself (endpoint --> IP:Port) at > > > > > the Service Discovery (SD) using the manifest. > > > > > - A client queries SD for instances of a specific Endpoint and > > > receives a > > > > > list of IP:Port combinations > > > > > - The client connects to the service and provides its peer token > > > > > > > > > > The client and the service use mutual TLS for authentication and > > > > > the > > > peer > > > > > token for authorization. The token is signed to check its > > > > > integrity > > and > > > > > linked to the CN of the client certificate to check its validity > > > > > (and prevent forwarding of the token). > > > > > > > > > > The benefit is, that we do not have any critical runtime > > dependencies: > > > SD > > > > > results can be cached and the tokens are valid for a certain > > > > > amount > > of > > > > > time. A client can perform client side load balancing and call a > > > service > > > > > even if the SD or the Portal are currently unavailable. > > > > > Furthermore > > we > > > > > avoid bottlenecks like load balancers. > > > > > > > > > > If you have 20mins time, our Principal Architect for API Design > > > > > gave > > a > > > > > talk about this: https://www.youtube.com/watch?v=Yke6Vut2Shc > > > > > > > > > > We want to use the same mechanism with Kafka: > > > > > - Connect via Mutual TLS > > > > > - Provide a peer token which contains metod:topics > > > > > > > > > > I understood that it should be possible to get the details from > > > > > the certificate, but we also need the token. The combination of > > > certificate + > > > > > token is need to prevent impersonation of APIs (one could easily > > > forward > > > > a > > > > > token, if it wasn’t bound to a certificate). > > > > > > > > > > I agree with the revocation part, but we are only looking at > > > > > internal clients. The process would be to revoke access in the > > > > > Portal which > > > > prevents > > > > > new instances to connect to the service. If we really encounter > > > malicious > > > > > behaviour, the prefer way is to shut down the misbehaving client. > > > > > > > > > > Kind Regards, > > > > > Peter > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Sönke Liebau [mailto:soenke.lie...@opencore.com.INVALID] > > > > > Sent: Donnerstag, 26. Oktober 2017 14:59 > > > > > To: dev@kafka.apache.org > > > > > Subject: Re: Use self contained tokens instead of ACL > > > > > > > > > > Hi Peter, > > > > > > > > > > I am not entirely sure that I understand what you mean when you > > > > > say > > "at > > > > > application level" to be honest, but I do understand that you > > > > > want to forego 3rd party tools. However, this would mean that > > > > > you have to > > > > implement > > > > > some form of trust between your portal which issues the tokens > > > > > and > > > Kafka > > > > > which checks the tokens. I am not sure that I'd recommend > > > > > rolling > > your > > > > own > > > > > solution here, authentication has a lot of pitfalls that can > > > > > turn > > > around > > > > to > > > > > bite you. > > > > > Same thing as what I proposed with Kerberos could be done via > > > > certificates > > > > > of course. If your user requests access to a topic in the portal > > > > > he > > > > uploads > > > > > a certificate signing request with the topic name encoded in > > > > > some > > field > > > > > (lots of extensions to pick from). The portal signs the request > > > > > and > > > > returns > > > > > it to the user. When the user now connects to Kafka he does so > > > > > using > > > this > > > > > certificate and it should be fairly easy for you to extend the > > > > > KafkaPrincipalBuilder class and extract the list of these topics > > > > > from > > > the > > > > > certificate. Then you'd need to also extend SimpleAclAuthorizer > > > > > to > > > check > > > > > for these topics in the principal name and allow access if the > > > > > topic > > is > > > > > present in the certificate. > > > > > > > > > > I am unsure of the benefits of this system over for example > > > > > having > > the > > > > > portal create ACLs in Kafka, when the user is granted access to > > > > > a > > > topic. > > > > > One scenario that comes to mind is if you dynamically spin up > > > > > new clusters, if these new clusters all trust the same ca they > > > > > would also accept these tokens right away, no need to create > > > > > ACLs in fresh > > > clusters. > > > > > But this could also be solved by having a central repository for > > > > > ACLs > > > > that > > > > > are applied to all clusters regularly.. > > > > > A drawback of this system is that you'd need to revoke > > > > > certificates > > if > > > > you > > > > > want to withdraw access to a topic, which is significantly > > > > > harder > > that > > > > just > > > > > deleting an ACL. > > > > > > > > > > Anyway, not sure if this helps you at all, just some random > > > > > musings - > > > if > > > > > you explain your specific needs a bit more we can discuss further. > > > > > > > > > > Kind regards, > > > > > Sönke > > > > > > > > > > On Wed, Oct 25, 2017 at 1:10 PM, Postmann, P. (Peter) < > > > > > peter.postm...@ing.com.invalid> wrote: > > > > > > > > > > > Hi Sönke, > > > > > > > > > > > > Thanks for the fast replay. We don’t want to use Kerberos > > > > > > since we want to do the authorization on Application level and > > > > > > without involvement of a 3rd party during runtime. > > > > > > > > > > > > -----Original Message----- > > > > > > From: Sönke Liebau [mailto:soenke.lie...@opencore.com.INVALID] > > > > > > Sent: Mittwoch, 25. Oktober 2017 12:37 > > > > > > To: dev@kafka.apache.org > > > > > > Subject: Re: Use self contained tokens instead of ACL > > > > > > > > > > > > The concept you describe sounds similar to what Microsoft > > > > > > calls "claims based authorization". > > > > > > > > > > > > At a high level I should think that using Kerberos as a > > > > > > vehicle to transport the information would be the way to go, > > > > > > as it is > > > established > > > > > > and already supported by Kafka. I believe tickets have a field > > > > > > that can be used for authorization information, so if > > > > > > information about > > > the > > > > > > topics that a user has access to were to be encoded in this > > > > > > field > > you > > > > > > could probably extend Kafka to extract that information and > > > > > > use it > > > > > instead of ACLs. > > > > > > > > > > > > I am not well versed in what exactly Microsoft does and how > > > > > > you can control the granting side of things, but I do believe > > > > > > that AD > > server > > > > > > has support for something along those lines already. > > > > > > > > > > > > The upside of this would be that you don't have to implement > > anything > > > > > > around security, trust, encryption, etc. because everything is > > > > > > provided by Kerberos. > > > > > > > > > > > > Not much information in here I am afraid, but maybe a useful > > > direction > > > > > > for future research. > > > > > > > > > > > > Kind regards, > > > > > > Sönke > > > > > > > > > > > > On Wed, Oct 25, 2017 at 11:55 AM, Postmann, P. (Peter) < > > > > > > peter.postm...@ing.com.invalid> wrote: > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > I´m working on a concept to use Kafka with self-contained > > > > > > > tokens (instead of ACL). > > > > > > > > > > > > > > The idea: > > > > > > > > > > > > > > - A client requests access to a certain topic (in some > > > kind > > > > of > > > > > > > portal) > > > > > > > > > > > > > > - The owner of the topic approves the request (in some > > > kind > > > > of > > > > > > > portal) > > > > > > > > > > > > > > - The client receives a signed tokens which contains > the > > > > topic > > > > > > > (in some kind of portal) > > > > > > > > > > > > > > - The client sends the token when he connects to Kafka > > > > > > > > > > > > > > - Kafka validates the token and grants access > > > > > > > > > > > > > > Token Format: > > > > > > > > > > > > > > - List of Topics and methods > > > > > > > > > > > > > > o E.g. read /topic1 > > > > > > > > > > > > > > - Expire Date > > > > > > > > > > > > > > - Signature > > > > > > > > > > > > > > Implementation Idea: > > > > > > > > > > > > > > - Create a custom Authorization Class which checks the > > > > > signature > > > > > > > > > > > > > > - Implement the possibility to send arbitrary data > > > > > (key->value) > > > > > > > along with the request when the client connects to the > > > > > > > cluster > > > > > > > > > > > > > > I´m looking forward for feedback on this approach and would > > > > > > > be > > > happy > > > > > > > if you could give me a starting where to start with the > > > > > > > implementation (or if there already is a way to send > > > > > > > arbitrary > > data > > > > > > > to a custom > > > > > > Authorizer). > > > > > > > > > > > > > > Kind Regards, > > > > > > > Peter > > > > > > > > > > > > > > ------------------------------------------------------------ > > ----- > > > > > > > ATTENTION: > > > > > > > The information in this e-mail is confidential and only > > > > > > > meant for the intended recipient. If you are not the > > > > > > > intended recipient, > > > don't > > > > > > > use or disclose it in any way. Please let the sender know > > > > > > > and > > > delete > > > > > > > the message immediately. > > > > > > > ------------------------------------------------------------ > > ----- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Sönke Liebau > > > > > > Partner > > > > > > Tel. +49 179 7940878 > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - > > Germany > > > > > > > > > > > > -------------------------------------------------------------- > > > > > > --- > > > > > > ATTENTION: > > > > > > The information in this e-mail is confidential and only meant > > > > > > for > > the > > > > > > intended recipient. If you are not the intended recipient, > > > > > > don't > > use > > > > > > or disclose it in any way. Please let the sender know and > > > > > > delete > > the > > > > > > message immediately. > > > > > > -------------------------------------------------------------- > > > > > > --- > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Sönke Liebau > > > > > Partner > > > > > Tel. +49 179 7940878 > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - > > > > > Germany > > > > > > > > > > ---------------------------------------------------------------- > > > > > - > > > > > ATTENTION: > > > > > The information in this e-mail is confidential and only meant > > > > > for the intended recipient. If you are not the intended > > > > > recipient, don't use > > or > > > > > disclose it in any way. Please let the sender know and delete > > > > > the > > > message > > > > > immediately. > > > > > ---------------------------------------------------------------- > > > > > - > > > > > > > > > > > > > > > > > > > > > -- > > > > Sönke Liebau > > > > Partner > > > > Tel. +49 179 7940878 > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - > > > > Germany > > > > > > > > > > > > > > > -- > > Sönke Liebau > > Partner > > Tel. +49 179 7940878 > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany > > > > ----------------------------------------------------------------- > ATTENTION: > The information in this e-mail is confidential and only meant for the > intended recipient. If you are not the intended recipient, don't use or > disclose it in any way. Please let the sender know and delete the message > immediately. > ----------------------------------------------------------------- >
