[
https://issues.apache.org/jira/browse/HDFS-9476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265396#comment-15265396
]
Masatake Iwasaki commented on HDFS-9476:
----------------------------------------
The issue was reproduced even after the 001 was applied.
{noformat}
2016-04-30 22:09:02,681 [IPC Server handler 1 on 38857] INFO
FSNamesystem.audit (FSNamesystem.java:logAuditMessage(6997)) - allowed=true
ugi=iwasakims (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/1kb-multiple-checksum-blocks-64-16 dst=null perm=null
proto=rpc
2016-04-30 22:09:02,753 [main] INFO hdfs.TestDFSUpgradeFromImage
(TestDFSUpgradeFromImage.java:dfsOpenFileWithRetries(187)) - Open failed. 0
times. Retrying.
...(snip)
2016-04-30 22:09:32,048 [IPC Server handler 4 on 38857] INFO
FSNamesystem.audit (FSNamesystem.java:logAuditMessage(6997)) - allowed=true
ugi=iwasakims (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/1kb-multiple-checksum-blocks-64-16 dst=null perm=null
proto=rpc
2016-04-30 22:09:32,056 [main] INFO hdfs.TestDFSUpgradeFromImage
(TestDFSUpgradeFromImage.java:dfsOpenFileWithRetries(187)) - Open failed. 29
times. Retrying.
{noformat}
The test seems to fail if lease recovery happens (due to client's request)
before the datanode's storage report is processed. This looks like not a test
problem but real race.
{noformat}
2016-04-30 22:09:02,537 [IPC Server handler 2 on 38857] INFO
FSNamesystem.audit (FSNamesystem.java:logAuditMessage(6997)) - allowed=true
ugi=iwasakims (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo
src=/1kb-multiple-checksum-blocks-64-16 dst=null perm=null
proto=rpc
2016-04-30 22:09:02,544 [IPC Server handler 4 on 38857] INFO
namenode.FSNamesystem (FSNamesystem.java:recoverLeaseInternal(2365)) -
recoverLease: [Lease. Holder: DFSClient_8256078, pending creates: 13],
src=/1kb-multiple-checksum-blocks-64-16 from client DFSClient_8256078
2016-04-30 22:09:02,544 [IPC Server handler 4 on 38857] INFO
namenode.FSNamesystem (FSNamesystem.java:internalReleaseLease(3096)) -
Recovering [Lease. Holder: DFSClient_8256078, pending creates: 13],
src=/1kb-multiple-checksum-blocks-64-16
2016-04-30 22:09:02,545 [IPC Server handler 4 on 38857] WARN BlockStateChange
(BlockUnderConstructionFeature.java:initializeBlockRecovery(205)) - BLOCK*
BlockUnderConstructionFeature.initializeBlockRecovery: No blocks found, lease
removed.
2016-04-30 22:09:02,545 [IPC Server handler 4 on 38857] WARN hdfs.StateChange
(FSNamesystem.java:internalReleaseLease(3220)) - DIR*
NameSystem.internalReleaseLease: File /1kb-multiple-checksum-blocks-64-16 has
not been closed. Lease recovery is in progress. RecoveryId = 1031 for block
blk_7162739548153522810_1020
2016-04-30 22:09:02,545 [Block report processor] INFO
blockmanagement.BlockManager (BlockManager.java:processReport(2174)) -
Processing first storage report for DS-a78b43f5-967a-47e3-afe1-68bb2f73a7eb
from datanode e6f67da5-812b-402b-b5ac-dad5deab2666
{noformat}
> TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail
> ---------------------------------------------------------------------
>
> Key: HDFS-9476
> URL: https://issues.apache.org/jira/browse/HDFS-9476
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Wei-Chiu Chuang
> Assignee: Akira AJISAKA
> Attachments: HDFS-9476.01.patch
>
>
> This test occasionally fail. For example, the most recent one is:
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/2587/
> Error Message
> {noformat}
> Cannot obtain block length for
> LocatedBlock{BP-1371507683-67.195.81.153-1448798439809:blk_7162739548153522810_1020;
> getBlockSize()=1024; corrupt=false; offset=0;
> locs=[DatanodeInfoWithStorage[127.0.0.1:33080,DS-c5eaf2b4-2ee6-419d-a8a0-44a5df5ef9a1,DISK]]}
> {noformat}
> Stacktrace
> {noformat}
> java.io.IOException: Cannot obtain block length for
> LocatedBlock{BP-1371507683-67.195.81.153-1448798439809:blk_7162739548153522810_1020;
> getBlockSize()=1024; corrupt=false; offset=0;
> locs=[DatanodeInfoWithStorage[127.0.0.1:33080,DS-c5eaf2b4-2ee6-419d-a8a0-44a5df5ef9a1,DISK]]}
> at
> org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:399)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:343)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:275)
> at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:265)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1046)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1011)
> at
> org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.dfsOpenFileWithRetries(TestDFSUpgradeFromImage.java:177)
> at
> org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.verifyDir(TestDFSUpgradeFromImage.java:213)
> at
> org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.verifyFileSystem(TestDFSUpgradeFromImage.java:228)
> at
> org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.upgradeAndVerify(TestDFSUpgradeFromImage.java:600)
> at
> org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.testUpgradeFromRel1BBWImage(TestDFSUpgradeFromImage.java:622)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
