[
https://issues.apache.org/jira/browse/HDFS-4881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
takeshi.miao updated HDFS-4881:
-------------------------------
Resolution: Won't Fix
Fix Version/s: (was: 1.0.0)
Status: Resolved (was: Patch Available)
Close it without any fix
> fine tune "Access token verification failed" error msg in datanode log
> ----------------------------------------------------------------------
>
> Key: HDFS-4881
> URL: https://issues.apache.org/jira/browse/HDFS-4881
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 1.0.0
> Environment: CentOS-5.3, java-version-1.6.0_26
> Reporter: takeshi.miao
> Priority: Trivial
> Attachments: HDFS-4881-branch-1.0-v1.patch,
> HDFS-4881-branch-1.0.patch, HDFS-4881-branch-1.patch
>
>
> I'd like to issue this ticket is due to we suffered a datanode access token
> verification failure issue recently. The client is HBase who is accessing the
> local datanode via DFSClient. The details log snippets as follows...
> *regionserver log*
> {code}
> ...
> [2013-05-24 08:33:37,553][regionserver8120-compactions-1369288874174][INFO
> ][org.apache.hadoop.hbase.regionserver.Store]: Started compaction of 1
> file(s) in cf=ho, hasReferences=true, into
> hdfs://sjdc-s-hdd-001.sjdc.ispn.trendmicro.com:8020/user/SPN-hbase/spn.guidcensus.ho/f99c6fb26f488034bf0e6ddd7a647ba4/.tmp,
> seqid=3, totalSize=4.2g
> [2013-05-24 08:33:37,554][regionserver8120-compactions-1369288874174][INFO
> ][org.apache.hadoop.hdfs.DFSClient]: Access token was invalid when connecting
> to /10.31.6.49:1004 :
> org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException: Got
> access token error for OP_READ_BLOCK, self=/10.31.6.49:36530,
> remote=/10.31.6.49:1004, for file
> /user/SPN-hbase/spn.guidcensus.ho/a565dd142933e3abf9bec33d59210d1b/ho/c5b37b9dd8801275c8fb160c0fb32ce5c48b56f4,
> for block 4549293737579979499_205814042
> ...
> {code}
> *datanode log*
> {code}
> ...
> [2013-05-24 08:33:37,554][DataXceiver for client /10.31.6.49:36530 [Waiting
> for operation #1]][ERROR][org.apache.hadoop.hdfs.server.datanode.DataNode]:
> DatanodeRegistration(10.31.6.49:1004,
> storageID=DS-1953102179-10.31.6.49-1004- 1342490559943, infoPort=1006,
> ipcPort=50020):DataXceiver
> java.io.IOException: Access token verification failed, for client
> /10.31.6.49:36530 for OP_READ_BLOCK for block
> blk_4549293737579979499_205814042
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:252)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:175)
> ...
> {code}
> After trace o.a.h.hdfs.security.token.block.BlockTokenSecretManager.java, I
> found that there are more further details error description written in code.
> *o.a.h.hdfs.security.token.block.BlockTokenSecretManager.java*
> {code}
> public void checkAccess(BlockTokenIdentifier id, String userId, Block block,
> AccessMode mode) throws InvalidToken {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Checking access for user=" + userId + ", block=" + block
> + ", access mode=" + mode + " using " + id.toString());
> }
> if (userId != null && !userId.equals(id.getUserId())) {
> throw new InvalidToken("Block token with " + id.toString()
> + " doesn't belong to user " + userId);
> }
> if (id.getBlockId() != block.getBlockId()) {
> throw new InvalidToken("Block token with " + id.toString()
> + " doesn't apply to block " + block);
> }
> if (isExpired(id.getExpiryDate())) {
> throw new InvalidToken("Block token with " + id.toString()
> + " is expired.");
> }
> if (!id.getAccessModes().contains(mode)) {
> throw new InvalidToken("Block token with " + id.toString()
> + " doesn't have " + mode + " permission");
> }
> }
> {code}
> But actually, this InvalidTokenException will not be handled further (but
> caught), so I can not trace what kind of this access block token verification
> is...
> *o.a.h.hdfs.server.datanode.DataXceiver.java*
> {code}
> ...
> if (datanode.isBlockTokenEnabled) {
> try {
> datanode.blockTokenSecretManager.checkAccess(accessToken, null, block,
> BlockTokenSecretManager.AccessMode.READ);
> } catch (InvalidToken e) {
> // the e object not handled further...
> try {
> out.writeShort(DataTransferProtocol.OP_STATUS_ERROR_ACCESS_TOKEN);
> out.flush();
> throw new IOException("Access token verification failed, for client
> "
> + remoteAddress + " for OP_READ_BLOCK for block " + block);
> } finally {
> IOUtils.closeStream(out);
> }
> }
> }
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
