On Mon, Oct 16, 2017 at 2:29 PM, Matteo Durighetto <[email protected]> wrote:
> Hello Todd, > thank you very much for the answer. I think I have > found something interesting. > > Kudu is doing the ACL list with the sAMAccountName or CN as it writes in > the logs: > > "Logged in from keytab as kudu/<host>@REALM (short username > <sAMAccountName> )" > > I begin to think that the problem is between sssd with plugin > sssd_krb5_localauth_plugin.so > so for every principal kudu/<host>@REALM kudu maps to <sAMAccountName>, > so seems impossible to have the all kudu/<host>@REALM mapped to the same > "kudu" > as suggested "So, basically, it's critical that the username that the > master determines for itself (from this function) > matches the username that it has determined for the tablet servers when > they authenticate > (what you pasted as 'abcdefgh1234' above)." > Interesting. What is the sAMAccountName in this case? Wouldn't all of the 'kudu' have the same account name? > > The strange thing is that with hadoop I have the correct mapping ( > probably because I have no rule, so > it switch to default rule ) > > hadoop org.apache.hadoop.security.HadoopKerberosName kudu/<host>@REALM > ==> kudu > > This makes sense since Hadoop uses its own auth_to_local configuration defined in its core-site.xml rather than relying on the system krb5 library. Kudu, being implemented in C++, uses the system krb5 API to do mapping, and hence picks upthe sssd localauth plugin as you found. As a temporary workaround, one thing you could try is to duplicate your krb5.conf and make a new one like krb5.kudu.conf. In this file remove the localauth plugin and any auth_to_local mappings that are causing trouble. You can then start the kudu servers with KRB5_CONFIG=/path/to/krb5.kudu.conf . I believe this will ensure that the local-auth mapping performs as you'd like. Given your setup, do you think we should provide an advanced option to override the system-configured localauth mapping and instead just use the "simple" mapping of using the short principal name? Generally we'd prefer to have as simple a configuration as possible but if your configuration is relatively commonplace it seems we might want an easier workaround than duplicating krb5.conf. -Todd > > > 2017-10-13 1:32 GMT+02:00 Todd Lipcon <[email protected]>: > >> Hey Matteo, >> >> Looks like you did quite a bit of digging in the code! Responses inline >> below. >> >> On Wed, Oct 11, 2017 at 1:24 PM, Matteo Durighetto < >> [email protected]> wrote: >> >>> Hello, >>> I have a strange behaviour with Kudu 1.4 and kerberos. >>> I enabled kerberos on kudu, I have the principal correctly in the OU of >>> an AD, but >>> at startup I got a lot of errors on method TSHeartbeat between tablet >>> server and >>> master server as unauthorized. There's no firewall between nodes. >>> >> >> right, "unauthorized" indicates that the connection was made fine, but >> the individual RPC call was determined to not be allowed for the identity >> presented on the other side of the connection. >> >> >>> >>> W1011 <time> server_base.cc:316] Unauthorized access attempt >>> to method kudu.master.MasterService.TSHeartbeat >>> from {username='abcdefgh1234', principal='kudu/[email protected]'} >>> at <IP>:37360 >>> >>> the "abcdefgh1234" it's an example of the the string created by the >>> cloudera manager during the enable kerberos. >>> >> >> This output indicates that it successfully authenticated via Kerberos as >> the principal listed above. That's good news and means you don't need to >> worry about rdns, etc (if you had issues with that it would have had >> trouble finding a service ticket or authenticating the connection). This >> means you got past the "authentication" step and having problems at the >> "authorization" step. >> >> >>> >>> The other services (hdfs and so on ) are under kerberos without problem >>> and there is the rdns at true in the /etc/krb5.conf ( KUDU-2032 ). >>> As I understand the problem is something about the 3° level of >>> authorization between master servers and tablet servers. >>> >> >> Right. >> >> >>> ... <snipped> >>> So I think the problem, as I say before, could be >>> in ContainsKey(users_, username); : >>> >>> bool SimpleAcl::UserAllowed(const string& username) { >>> return ContainsKey(users_, "*") || ContainsKey(users_, username); >>> } >>> >>> >>> At this point It's not clear for me how Kudu build the array/key list >>> users for daemon service ( it's not as super users or user ACL an external >>> parameter). >>> >> >> Exactly. The users here for the 'service' ACL are set in >> ServerBase::InitAcls(): >> >> boost::optional<string> keytab_user = security::GetLoggedInUsernameF >> romKeytab(); >> if (keytab_user) { >> // If we're logged in from a keytab, then everyone should be, and we >> expect them >> // to use the same mapped username. >> service_user = *keytab_user; >> } else { >> // If we aren't logged in from a keytab, then just assume that the >> services >> // will be running as the same Unix user as we are. >> RETURN_NOT_OK_PREPEND(GetLoggedInUser(&service_user), >> "could not deterine local username"); >> } >> >> Since you're using Kerberos, the top branch here would apply -- it's >> calling GetLoggedInUsernameFromKeytab() from init.cc. >> >> You can see what username the server is getting by looking for a log >> message at startup like "Logged in from keytab as kudu/<host>@REALM (short >> username <XYZ>)". Here 'XYZ' is the username that ends up in the service >> ACL. >> >> So, basically, it's critical that the username that the master determines >> for itself (from this function) matches the username that it has determined >> for the tablet servers when they authenticate (what you pasted as >> 'abcdefgh1234' above). >> >> That brings us to the next question: how do we convert from a principal >> like kudu/<HOST>@REALM to a short "username"? The answer there is the >> function 'MapPrincipalToLocalName' again from security/init.cc. This >> function delegates the mapping to the krb5 library itself using the >> krb5_aname_to_localname() API. The results of this API can vary depending >> on the kerberos configuration, but in typical configurations it's >> determined by the 'auth_to_local' configuration in your krb5.conf. See the >> corresponding section in the docs here: >> >> https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/ >> krb5_conf.html >> >> My guess is that your host has been configured such that when the master >> maps its own principal, it's getting a different result than when it maps >> the principal being used by the tservers. >> >> Hope that gets you on the right track. >> >> Thanks >> -Todd >> -- >> Todd Lipcon >> Software Engineer, Cloudera >> > > -- Todd Lipcon Software Engineer, Cloudera
