Jungtaek Lim created STORM-2906:
-----------------------------------
Summary: HDFS and HBase bolt on the same worker fails with GSS no
valid credentials exception
Key: STORM-2906
URL: https://issues.apache.org/jira/browse/STORM-2906
Project: Apache Storm
Issue Type: Bug
Components: storm-hbase
Reporter: Jungtaek Lim
Assignee: Jungtaek Lim
We have observed that the topology which has HDFS and HBase bolts fails with
GSS exception no valid credentials. It doesn't occur when HDFS and HBase bolts
are running in different workers.
After some investigation I found the cause,
[https://github.com/apache/storm/blob/6402d436a8700eb743ccd84f7f562c51d8cf9be7/external/storm-hbase/src/main/java/org/apache/storm/hbase/common/Utils.java#L39-L65]
{code:java}
public static HTable getTable(UserProvider provider, final Configuration
config, final String tableName)
throws IOException, InterruptedException {
UserGroupInformation ugi;
if (provider != null) {
ugi = provider.getCurrent().getUGI();
LOG.debug("Current USER for provider: {}", ugi.getUserName());
} else {
// autocreds puts delegation token into current user UGI
ugi = UserGroupInformation.getCurrentUser();
LOG.debug("UGI for current USER : {}", ugi.getUserName());
for (Token<? extends TokenIdentifier> token : ugi.getTokens()) {
LOG.debug("Token in UGI (delegation token): {} / {}",
token.toString(),
token.decodeIdentifier().getUser());
// use UGI from token
ugi = token.decodeIdentifier().getUser();
ugi.addToken(token);
}
}
return ugi.doAs(new PrivilegedExceptionAction<HTable>() {
@Override public HTable run() throws IOException {
return new HTable(config, tableName);
}
});
}{code}
UGI is always selected from last element of tokens, and unfortunately tokens
have HDFS delegation tokens as well. So we should check kind of token
(HBASE_AUTH_TOKEN) before leveraging it.
If nimbus can distribute tokens based on worker, then it will not occur when
HDFS and HBase bolts are running different workers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)