Jungtaek Lim created STORM-2906:
-----------------------------------

             Summary: HDFS and HBase bolt on the same worker fails with GSS no 
valid credentials exception
                 Key: STORM-2906
                 URL: https://issues.apache.org/jira/browse/STORM-2906
             Project: Apache Storm
          Issue Type: Bug
          Components: storm-hbase
            Reporter: Jungtaek Lim
            Assignee: Jungtaek Lim


We have observed that the topology which has HDFS and HBase bolts fails with 
GSS exception no valid credentials. It doesn't occur when HDFS and HBase bolts 
are running in different workers.

After some investigation I found the cause, 

[https://github.com/apache/storm/blob/6402d436a8700eb743ccd84f7f562c51d8cf9be7/external/storm-hbase/src/main/java/org/apache/storm/hbase/common/Utils.java#L39-L65]

 
{code:java}
    public static HTable getTable(UserProvider provider, final Configuration 
config, final String tableName)
            throws IOException, InterruptedException {
        UserGroupInformation ugi;
        if (provider != null) {
            ugi = provider.getCurrent().getUGI();
            LOG.debug("Current USER for provider: {}", ugi.getUserName());
        } else {
            // autocreds puts delegation token into current user UGI
            ugi = UserGroupInformation.getCurrentUser();


            LOG.debug("UGI for current USER : {}", ugi.getUserName());
            for (Token<? extends TokenIdentifier> token : ugi.getTokens()) {
                LOG.debug("Token in UGI (delegation token): {} / {}", 
token.toString(),
                        token.decodeIdentifier().getUser());


                // use UGI from token
                ugi = token.decodeIdentifier().getUser();
                ugi.addToken(token);
            }
        }


        return ugi.doAs(new PrivilegedExceptionAction<HTable>() {
            @Override public HTable run() throws IOException {
                return new HTable(config, tableName);
            }
        });
    }{code}

UGI is always selected from last element of tokens, and unfortunately tokens 
have HDFS delegation tokens as well. So we should check kind of token 
(HBASE_AUTH_TOKEN) before leveraging it.

If nimbus can distribute tokens based on worker, then it will not occur when 
HDFS and HBase bolts are running different workers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to