+1. Colm.
On Fri, Dec 15, 2017 at 3:52 AM, 郑锴(铁杰) <zhengkai...@alibaba-inc.com> wrote: > Thanks Jiajia for the update and major progress. It would be good to see > HAS feature consolidated into Kerby KDC offering. > > Regards, > Kai > > ------------------------------------------------------------------ > 发件人：Li, Jiajia <jiajia...@intel.com> > 发送时间：2017年12月15日(星期五) 10:38 > 收件人：Apache Directory Developers List <d...@directory.apache.org>; > cohei...@apache.org <cohei...@apache.org> > 抄 送：kerby@directory.apache.org <kerby@directory.apache.org> > 主 题：RE: [DISCUSS] Merge HAS to Apache Kerby > > Hi all, <#m_8069559770700476617_this> > > > > Status update: > > > > 1. I've checked both Intel and Alibaba have signed the CCLA. > > 2. We have fixed dependency issues: mysql JDBC driver is replaced with > Drizzle JDBC and some dependencies cannot find a license have been removed. > > 3. If there are no more questions, we will start to merge under the master > JIRA(https://issues.apache.org/jira/browse/DIRKRB-671), please help to > review the patches. > > > > Thanks, > > Jiajia > > > > <#m_8069559770700476617_this>*From:* Colm O hEigeartaigh [mailto: > cohei...@apache.org] > *Sent:* Wednesday, December 6, 2017 11:23 PM > *To:* Li, Jiajia <jiajia...@intel.com> > *Cc:* Apache Directory Developers List <d...@directory.apache.org>; > kerby@directory.apache.org > *Subject:* Re: [DISCUSS] Merge HAS to Apache Kerby > > > > Hi Jiajia, > > Perhaps you could get one of the Alibaba contributors to mail " > secret...@apache.org" and ask if there is a CCLA on record? > > Colm. > > > > On Tue, Dec 5, 2017 at 1:49 AM, Li, Jiajia <jiajia...@intel.com> wrote: > > I think Intel have provided the CCLA when contributing Kerby to Apache. > But I'm not sure whether Alibaba already provided, is there one place we > could check it? > > > > Thanks, > > Jiajia > > <#m_8069559770700476617_this> > > <#m_8069559770700476617_this>*From:* Colm O hEigeartaigh [mailto: > cohei...@apache.org] > *Sent:* Tuesday, December 5, 2017 1:50 AM > *To:* kerby@directory.apache.org > *Cc:* Apache Directory Developers List <d...@directory.apache.org> > *Subject:* Re: [DISCUSS] Merge HAS to Apache Kerby > > > > Do we have both CCLAs filed for Intel and Alibaba? > > Colm. > > > > On Mon, Dec 4, 2017 at 6:36 AM, Li, Jiajia <jiajia...@intel.com> wrote: > > Hi all, > > Here with some status update, now all the contributors have provided the > ICLA to secret...@apache.org and I've create the master JIRA( > https://issues.apache.org/jira/browse/DIRKRB-671) for this merging. > Any more suggestions on how to merge? > > Thanks, > Jiajia > > From: Li, Jiajia [mailto:jiajia...@intel.com] > Sent: Thursday, November 30, 2017 1:38 PM > To: cohei...@apache.org > Cc: kerby@directory.apache.org; Apache Directory Developers List < > d...@directory.apache.org> > Subject: RE: [DISCUSS] Merge HAS to Apache Kerby > > Hi Colm, > > > What I meant with the point about the backend, is that it should be > configurable whether to just trust the signature of the presented auth > token as sufficient validation, without requiring any MySQL backend. For > example, the token might be issued by an IdP that HAS "trusts", where the > IdP has an identity backend of which HAS knows nothing about. > > Now I understand what you mean. There are there reasons for using backend: > 1. If user using the new authentication mechanism(Kerberos-based token > authentication), the TGT(ticket granting ticket) could be got without > backend. But TGT is not enough to access the service, after getting the > TGT, next step is to get SGT(Ticket for Service), in this step, the service > principal is needed in backend. > 2. The new authentication mechanism is used by the end users instead of > service level, services are still strongly authenticated by Kerberos, they > through the keytabs to login. > 3. Users or admins sometimes want to using "kinit" to get credential cache > to manage the cluster, for the compatibility. > > > One final overall point, is that HAS looks a bit like a > SecurityTokenService (STS). Apache CXF ships with a STS that I am very > familiar with. It is a web application that supports a SOAP and REST > interface to issue, validate tokens etc, where you can "plug in" the tokens > that are supported. It might be worth exploring if the functionality of HAS > could be integrated with the CXF STS. > > I do not know much about SecurityTokenService, from your introduction, I > think STS could issue token and validate token, that is exactly the > existing authentication system HAS wants to plugin, we can write the client > and server plugins for STS, then using STS in HAS framework. Please correct > me if I'm wrong. > > We think it's more suitable to be integrated with kerby with following > reasons: > 1. The new authentication mechanism ("Kerberos-based token > authentication") is based on the "TokenPreauth" provided in Kerby, using > AuthToken to exchange a Kerberos ticket. > 2. The REST APIs not only for the new authentication, also provide some > useful interfaces, such as: config Kerby KDC, manage the Kerby backend, > export keytab files. These could help Kerby KDC to be stronger. > 3. HAS binds webserver and Kerby KDC very closely, they are all included > in HasServer(we can rename it after merging), we could also think the > webserver is one part of Kerby KDC, we using the webserver for KDC to > receive some requests from HTTPs client. > > Thanks > Jiajia > From: Colm O hEigeartaigh [mailto:cohei...@apache.org] > Sent: Wednesday, November 29, 2017 10:58 PM > > To: Li, Jiajia <jiajia...@intel.com<mailto:jiajia...@intel.com>> > Cc: kerby@directory.apache.org<mailto:kerby@directory.apache.org>; Apache > Directory Developers List <d...@directory.apache.org<mailto: > d...@directory.apache.org>> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby > > Hi Jiajia, > What I meant with the point about the backend, is that it should be > configurable whether to just trust the signature of the presented auth > token as sufficient validation, without requiring any MySQL backend. For > example, the token might be issued by an IdP that HAS "trusts", where the > IdP has an identity backend of which HAS knows nothing about. > > One final overall point, is that HAS looks a bit like a > SecurityTokenService (STS). Apache CXF ships with a STS that I am very > familiar with. It is a web application that supports a SOAP and REST > interface to issue, validate tokens etc, where you can "plug in" the tokens > that are supported. It might be worth exploring if the functionality of HAS > could be integrated with the CXF STS. > > Colm. > > > Thanks, > Jiajia > > -----Original Message----- > From: Colm O hEigeartaigh [mailto:cohei...@apache.org<mailto: > cohei...@apache.org>] > Sent: Tuesday, November 28, 2017 9:12 PM > To: Li, Jiajia <jiajia...@intel.com<mailto:jiajia...@intel.com>> > Cc: kerby@directory.apache.org<mailto:kerby@directory.apache.org>; Apache > Directory Developers List <d...@directory.apache.org<mailto: > d...@directory.apache.org>> > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby > Thanks Jiajia, that was very helpful. I have some questions: > > There are no HasClientPlugin implementations in the commit (unless I > missed them). Is the plan to provide some later on, or is the user supposed > to implement their own? > > If we want to get Kerby to issue a TGT using an AuthToken currently, we > have to use a token armor cache. In HAS, when it queries Kerby to get a TGT > using the verified AuthToken, is this just an "internal" call so we can > avoid this step? > > I'm not sure why we need to verify the user information in the SQL backend. > If the received AuthToken is signed by a trusted IdP, can we not just > accept the identity of the user "as is" and skip this step? > > KinitTool and KinitOption in has-client-tool duplicate the Kerby versions > with just a few changes. Can the changes be rolled into Kerby to prevent > code duplication? > > Colm. > > On Tue, Nov 28, 2017 at 2:16 AM, Li, Jiajia <jiajia...@intel.com<mailto:ji > ajia...@intel.com>> wrote: > > > Thanks Colm. > > > > > It sounds like a really interesting project. > > I'm glad to here that. > > > > > Have you got any feedback from the Hadoop project about it? > > We haven't proposed this solution in the hadoop community. > > > > > I'm finding it hard to understand exactly how it works though based > > > on > > the README. Could you describe how it works from a really basic point > > of view for say a simple Hadoop client? Normally I just have to use > > "kinit" to get a kerberos ticket and then I am authenticated to invoke > > on HDFS. How does HAS work differently? Where does the token pre-auth > stuff fit in? > > > > Following are the steps of user accessing HDFS service, taking the cmd > > "hadoop fs -ls /" as an example: > > 1. user runs the command "hadoop fs -ls /" > > 2. Hadoop client will call the "HasLoginModule", > > https://github.com/apache/directory-kerby/blob/has- > > project/has/has-client/src/main/java/org/apache/hadoop/ > > has/client/HasLoginModule.java > > 3. "HasLoginModule" will call the "HasClient", > > https://github.com/apache/ > > directory-kerby/blob/438904f7e557a085c8c336efd2d2be > > 304291d246/has/has-client/src/main/java/org/apache/hadoop/ > > has/client/HasLoginModule.java#L237 > > 4. "HasClient" will get the plugin type from config, then choose the > > right client plugin, the client plugin will collect and add some user > > info to "AuthToken", the following is the client plugin interface: > > > > // Get the login module type ID, used to distinguish this module from > > others. > > // Should correspond to the server side module. > > String getLoginType() > > > > // Perform all the client side login logics, the results wrapped in an > > AuthToken, // will be validated by HAS server. > > AuthToken login(Conf loginConf) throws HasLoginException > > > > 5. Then "HasClient" sends the "AuthToken" to HAS Server through HTTPS; > > 6. After HAS server receives the message, it will call the server > > plugin, server plugin will verify the user info in AuthToken, the > > following is the server plugin interface: > > > > // Get the login module type ID, used to distinguish this module from > > others. > > // Should correspond to the client side module. > > String getLoginType() > > > > // Perform all the server side authentication logics, the results > > wrapped in an "AuthToken", // will be used to exchange a Kerberos > > ticket. > > AuthToken authenticate(AuthToken userToken) throws HasAuthenException > > > > 7. If the user info is verified in existing user authentication > > system, server plugin will return the verified "AuthToken" to Kerby > > KDC 8. Kerby KDC will issue the TGT ticket using the TokenPreauth, > > then send the TGT to HasClient through HTTPS 9. Now user login > > successful, could continue the others steps, such as: > > getting SGT ticket. > > > > We replace the step through "kinit" to get Kerberos Ticket. There are > > two important benefits: > > 1. The user's principal may not be in the backend, security admins > > won't have to migrate and sync up their user accounts to Kerberos back > and forth. > > 2. Multiple users could run the job at the same time and in the same > > machine, through collecting user info from environment variables in > step4. > > > > > > Thanks, > > Jiajia > > > > -----Original Message----- > > From: Colm O hEigeartaigh [mailto:cohei...@apache.org<mailto: > cohei...@apache.org>] > > Sent: Monday, November 27, 2017 6:54 PM > > To: kerby@directory.apache.org<mailto:kerby@directory.apache.org> > > Cc: Apache Directory Developers List <d...@directory.apache.org<mailto: > d...@directory.apache.org>> > > Subject: Re: [DISCUSS] Merge HAS to Apache Kerby > > > > Hi Jiajia, > > > > It sounds like a really interesting project. Have you got any feedback > > from the Hadoop project about it? > > > > I'm finding it hard to understand exactly how it works though based on > > the README. Could you describe how it works from a really basic point > > of view for say a simple Hadoop client? Normally I just have to use > > "kinit" to get a kerberos ticket and then I am authenticated to invoke > > on HDFS. How does HAS work differently? Where does the token pre-auth > stuff fit in? > > > > Colm. > > > > > > On Fri, Nov 24, 2017 at 3:30 AM, Li, Jiajia <jiajia...@intel.com<mailto: > jiajia...@intel.com>> wrote: > > > > > Hi all, > > > > > > I would like to post a proposal about merging a new project HAS > > > (Hadoop Authentication Service) to Apache Kerby. HAS is led by Intel > > > and Alibaba, it is a solution to support the authentication of open > > > source big data ecosystem in cloud computing platforms. I've created > > > a new branch "has-project" in Kerby, HAS is under "has" folder. > > > Please look at > > > https://github.com/apache/directory-kerby/tree/has-project/has > > > for details. > > > > > > Background and motivation: > > > At present, the open source big data ecosystems (Hadoop/Spark) only > > > has the built-in Kerberos support on the security authentication. > > > HAS aims to build a standalone authentication service for the big > > > data ecosystem that simplifies the support of Kerberos and allows to > > > use more authentication methods. > > > > > > Targets users: > > > HAS supports various authentication mechanisms other than just > > > Kerberos, and it provides a new authentication mechanism can be easy > > > customized and plugin with existing user authentication and > > > authorization system, and security admins won't have to migrate and > > > sync up their user accounts to Kerberos back and forth. > > > > > > Architecture & Design: > > > HAS provides a new authentication mechanism ("Kerberos-based token > > > authentication"), depending on the "TokenPreauth" provided by Apache > > Kerby. > > > Please look at > > > https://github.com/apache/directory-kerby/blob/has-project/ > > > has/README.md for details. > > > > > > Features: > > > 1. Provides new authentication mechanism plugin APIs to customize > > and > > > plugin with existing user authentication and authorization system. > > > Please look at > > > https://github.com/apache/directory-kerby/blob/has-project/ > > > has/README.md for details. > > > 2. Provides lots of REST APIs and facility tools to simplify the > > > support of Kerberos. Kerberos is essentially a protocol, or secure > > > channel, doesn't have to be that complex to users. Please look at > > > https://github.com/apache/directory-kerby/blob/has-project/ > > > has/doc/rest-api.md<http://rest-api.md> for details. > > > 3. Provides MySQL backend for High Availability. Please look at > > > https://github.com/apache/directory-kerby/blob/has-project/ > > > has/doc/mysql-backend.md<http://mysql-backend.md> for details. > > > > 4. New authentication mechanism now supports most of the > components > > > of open source big data ecosystem with little or no changes to > > > components, including HDFS, HBase, Zookeeper, Hive, Spark.... Please > > > look at > > > https://github.com/apache/directory-kerby/tree/has-project/has/suppo > > > rt > > > s > > > for details. > > > > > > Practice > > > This solution has been deployed in Alibaba Cloud E-MapReduce > production. > > > > > > Why to merge? > > > HAS provides a complete Hadoop/Spark authentication framework and > > > solution based on Kerberos, HAS can help to upgrade Kerby KDC, make > > > it more solid and stronger. And if HAS can be merged to Apache > > > Kerby, community will help HAS grow faster and users can more easily > > > using this solution in their own production. We have two suggestions > > > about how > > to merge: > > > - Option1: > > > Create a standalone module "kerby-has", putting HAS project under > > > this module. > > > - Option2: > > > Suggest replacing kerby-kdc module with HAS, upgrade the Kerby KDC. > > > > > > Contributors: > > > Jiajia, Li (Intel) > > > Lin, Zeng (Intel) > > > Zhiqiang, Zhang (Intel) > > > Kai, Zheng (Intel) > > > Wei, Wu (Alibaba) > > > Jun, Song (Alibaba) > > > Long, Cao (Alibaba) > > > Zhenyuan, Wei (Alibaba) > > > > > > Your review efforts are truly appreciated, please feel free to > > > provide us your feedback. > > > > > > Regards, > > > Jiajia > > > > > > > > > > > > > > > > > > -- > > Colm O hEigeartaigh > > > > Talend Community Coder > > http://coders.talend.com > > > > > > -- > Colm O hEigeartaigh > > Talend Community Coder > http://coders.talend.com > > > > -- > Colm O hEigeartaigh > > Talend Community Coder > http://coders.talend.com > > > > > -- > > Colm O hEigeartaigh > > Talend Community Coder > http://coders.talend.com > > > > > -- > > Colm O hEigeartaigh > > Talend Community Coder > http://coders.talend.com > > -- Colm O hEigeartaigh Talend Community Coder http://coders.talend.com
