[
https://issues.apache.org/jira/browse/HDFS-4881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13676997#comment-13676997
]
Daryn Sharp commented on HDFS-4881:
-----------------------------------
The pre-commit is trying to apply the patch to trunk which has a different path
layout. It's best to run test-patch manually and paste the result in a comment.
As for the patch, I'd suggest considering if it would be better to wrap the
InvalidToken (which is an IOException) in another InvalidToken exception,
and/or throw a InvalidToken exception with the new message prepended to the
original InvalidToken's exception message, so callers may easily distinguish
and better handle the IOException with a catch clause for InvalidToken. I
don't know if it's true in this case, but most code doesn't bother to check an
exception's cause before blindly issuing retries - even if an exception like
this one is not transient and will never work. I'm not suggesting you go so
far as modifying the caller's catch clauses, just discussing why this approach
is worth considering.
If this is also an issue on trunk/branch-2, please post patches for those
branches.
> fine tune "Access token verification failed" error msg in datanode log
> ----------------------------------------------------------------------
>
> Key: HDFS-4881
> URL: https://issues.apache.org/jira/browse/HDFS-4881
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 1.0.0
> Environment: CentOS-5.3, java-version-1.6.0_26
> Reporter: takeshi.miao
> Priority: Trivial
> Fix For: 1.0.0
>
> Attachments: HDFS-4881-branch-1.0.patch,
> HDFS-4881-branch-1.0-v1.patch, HDFS-4881-branch-1.patch
>
>
> I'd like to issue this ticket is due to we suffered a datanode access token
> verification failure issue recently. The client is HBase who is accessing the
> local datanode via DFSClient. The details log snippets as follows...
> *regionserver log*
> {code}
> ...
> [2013-05-24 08:33:37,553][regionserver8120-compactions-1369288874174][INFO
> ][org.apache.hadoop.hbase.regionserver.Store]: Started compaction of 1
> file(s) in cf=ho, hasReferences=true, into
> hdfs://sjdc-s-hdd-001.sjdc.ispn.trendmicro.com:8020/user/SPN-hbase/spn.guidcensus.ho/f99c6fb26f488034bf0e6ddd7a647ba4/.tmp,
> seqid=3, totalSize=4.2g
> [2013-05-24 08:33:37,554][regionserver8120-compactions-1369288874174][INFO
> ][org.apache.hadoop.hdfs.DFSClient]: Access token was invalid when connecting
> to /10.31.6.49:1004 :
> org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException: Got
> access token error for OP_READ_BLOCK, self=/10.31.6.49:36530,
> remote=/10.31.6.49:1004, for file
> /user/SPN-hbase/spn.guidcensus.ho/a565dd142933e3abf9bec33d59210d1b/ho/c5b37b9dd8801275c8fb160c0fb32ce5c48b56f4,
> for block 4549293737579979499_205814042
> ...
> {code}
> *datanode log*
> {code}
> ...
> [2013-05-24 08:33:37,554][DataXceiver for client /10.31.6.49:36530 [Waiting
> for operation #1]][ERROR][org.apache.hadoop.hdfs.server.datanode.DataNode]:
> DatanodeRegistration(10.31.6.49:1004,
> storageID=DS-1953102179-10.31.6.49-1004- 1342490559943, infoPort=1006,
> ipcPort=50020):DataXceiver
> java.io.IOException: Access token verification failed, for client
> /10.31.6.49:36530 for OP_READ_BLOCK for block
> blk_4549293737579979499_205814042
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:252)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:175)
> ...
> {code}
> After trace o.a.h.hdfs.security.token.block.BlockTokenSecretManager.java, I
> found that there are more further details error description written in code.
> *o.a.h.hdfs.security.token.block.BlockTokenSecretManager.java*
> {code}
> public void checkAccess(BlockTokenIdentifier id, String userId, Block block,
> AccessMode mode) throws InvalidToken {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Checking access for user=" + userId + ", block=" + block
> + ", access mode=" + mode + " using " + id.toString());
> }
> if (userId != null && !userId.equals(id.getUserId())) {
> throw new InvalidToken("Block token with " + id.toString()
> + " doesn't belong to user " + userId);
> }
> if (id.getBlockId() != block.getBlockId()) {
> throw new InvalidToken("Block token with " + id.toString()
> + " doesn't apply to block " + block);
> }
> if (isExpired(id.getExpiryDate())) {
> throw new InvalidToken("Block token with " + id.toString()
> + " is expired.");
> }
> if (!id.getAccessModes().contains(mode)) {
> throw new InvalidToken("Block token with " + id.toString()
> + " doesn't have " + mode + " permission");
> }
> }
> {code}
> But actually, this InvalidTokenException will not be handled further (but
> caught), so I can not trace what kind of this access block token verification
> is...
> *o.a.h.hdfs.server.datanode.DataXceiver.java*
> {code}
> ...
> if (datanode.isBlockTokenEnabled) {
> try {
> datanode.blockTokenSecretManager.checkAccess(accessToken, null, block,
> BlockTokenSecretManager.AccessMode.READ);
> } catch (InvalidToken e) {
> // the e object not handled further...
> try {
> out.writeShort(DataTransferProtocol.OP_STATUS_ERROR_ACCESS_TOKEN);
> out.flush();
> throw new IOException("Access token verification failed, for client
> "
> + remoteAddress + " for OP_READ_BLOCK for block " + block);
> } finally {
> IOUtils.closeStream(out);
> }
> }
> }
> ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
