kudu 1.4 kerberos

[Matteo Durighetto]

Hello,
           I have a strange behaviour with Kudu 1.4 and kerberos.
I enabled kerberos on kudu, I have the principal correctly in the OU of an
AD, but
at startup I got a lot of errors on method TSHeartbeat between tablet
server and
master server as unauthorized. There's no firewall between nodes.

W1011 <time>   server_base.cc:316] Unauthorized access attempt
to method kudu.master.MasterService.TSHeartbeat
from {username='abcdefgh1234', principal='kudu/hostn...@domain.xyz'}
at <IP>:37360

the "abcdefgh1234" it's an example of the the string created by the
cloudera manager during the enable kerberos.

The other services (hdfs and so on ) are under kerberos without problem and
there is the rdns at true in the /etc/krb5.conf (  KUDU-2032 ).
As I understand the problem is something about the 3° level of
authorization between master servers and tablet servers.
So I try to understand better how it works.

The server_base.cc:316 is this call on branch 1.4 :

void ServerBase::LogUnauthorizedAccess(rpc::RpcContext* rpc) const {
  LOG(WARNING) << "Unauthorized access attempt to method "
               << rpc->service_name() << "." << rpc->method_name()
               << " from " << rpc->requestor_string();
}

was called by :

bool ServerBase::Authorize(rpc::RpcContext* rpc, uint32_t allowed_roles) {
  if ((allowed_roles & SUPER_USER) &&
      superuser_acl_.UserAllowed(rpc->remote_user().username())) {
    return true;
  }

  if ((allowed_roles & USER) &&
      user_acl_.UserAllowed(rpc->remote_user().username())) {
    return true;
  }

  if ((allowed_roles & SERVICE_USER) &&
      service_acl_.UserAllowed(rpc->remote_user().username())) {
    return true;
  }

  LogUnauthorizedAccess(rpc);
  rpc->RespondFailure(Status::NotAuthorized("unauthorized access to method",
                                            rpc->method_name()));
  return false;
}

The part of the code called for a service is this :

  if ((allowed_roles & SERVICE_USER) &&
      service_acl_.UserAllowed(rpc->remote_user().username())) {
    return true;
  }

the  service_acl_.UserAllowed(rpc->remote_user().username())  I think in
the could be
the problem.
because :

It is a kerberized service so, with the keytab ( for exemple   with kudu/
hostn...@domain.xyz )

rpc->remote_user().username()

it will run this code ( rpc/remote_user.cc )

string RemoteUser::ToString() const {
  string ret;
  strings::SubstituteAndAppend(&ret, "{username='$0'", username_);
  if (principal_) {
    strings::SubstituteAndAppend(&ret, ", principal='$0'", *principal_);
  }
  ret.append("}");
  return ret;
}

and it will returns the "{username='abcdefgh1234',principal='kudu/HOSTNAME@
DOMAIN.XYZ'} "
as in the error message

and the code  service_acl_.UserAllowed call (src/kudu/security/simple_acl.cc
)

bool SimpleAcl::UserAllowed(const string& username) {
  return ContainsKey(users_, "*") || ContainsKey(users_, username);
}

==> users_ , I not understand for service where is builded or read

the other part of code :

the allowed_roles & SERVICE_USER is a bitwise operation,
As I understand from server/server_base.h

SERVICE_USER = 1 << 2

kudu is using a bitmap: 100 for service user , 010 for user, and 001 for
superuser

  enum {
    SUPER_USER = 1,
    USER = 1 << 1,
    SERVICE_USER = 1 << 2
  };

So I expectes it will be true.

So I think the problem, as I say before, could be in  ContainsKey(users_,
username);  :

bool SimpleAcl::UserAllowed(const string& username) {
  return ContainsKey(users_, "*") || ContainsKey(users_, username);
}


At this point It's not clear for me how Kudu build the array/key list users
for daemon service ( it's not as super users or user ACL an external
parameter).
I would like to debug more, but I not have found more instructions than the
log level ( INFO ).

Kind Regards

Matteo