[
https://issues.apache.org/jira/browse/HDFS-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154478#comment-17154478
]
fengwu edited comment on HDFS-14396 at 7/9/20, 12:10 PM:
---------------------------------------------------------
Hi, Found in my test roll downgrade from 3.1.3 to 2.7.2, namenode successful
， but datanode failed （2.8+ successfully ）. because different datanode
layout versions is -56 in hdfs 2.7
So, Is there a way to solve datanode roll downgrade from layout version -57 to
-56 ?
{code:java}
// code placeholder
2020-07-06 14:45:01,313 WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data/hadoop/dfs. Reported: -57. Expecting = -56.
2020-07-06 14:45:01,315 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock
on /data/hadoop/dfs/in_use.lock acquired by nodename 21258@test-v03 2020-07-06
14:45:01,315 WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data/hadoop/dfs. Reported: -57. Expecting = -56.
2020-07-06 14:45:01,315 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode:
Initialization failed for Block pool <registering> (Datanode Uuid unassigned)
service to test-v01/10.110.228.21:8020. Exiting. java.io.IOException: All
specified directories are failed to load. at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1358)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1323)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:802)
at java.lang.Thread.run(Thread.java:748)
{code}
was (Author: fengwu99):
Hi, Found in my test roll downgrade from 3.1.3 to 2.7.2, namenode successful
， but datanode failed （2.8+ successfully ）. because different datanode
layout versions is -56 in hdfs 2.7.
So, Is there a way to solve datanode roll downgrade from layout version -57 to
-56 ?
// code placeholder2020-07-06 14:45:01,313 WARN
org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data/hadoop/dfs. Reported: -57. Expecting = -56.
2020-07-06 14:45:01,315 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock
on /data/hadoop/dfs/in_use.lock acquired by nodename 21258@test-v03
2020-07-06 14:45:01,315 WARN org.apache.hadoop.hdfs.server.common.Storage:
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /data/hadoop/dfs. Reported: -57. Expecting = -56.
2020-07-06 14:45:01,315 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode:
Initialization failed for Block pool <registering> (Datanode Uuid unassigned)
service to test-v01/10.110.228.21:8020. Exiting.
java.io.IOException: All specified directories are failed to load.
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1358)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1323)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:802)
at java.lang.Thread.run(Thread.java:748)
> Failed to load image from FSImageFile when downgrade from 3.x to 2.x
> --------------------------------------------------------------------
>
> Key: HDFS-14396
> URL: https://issues.apache.org/jira/browse/HDFS-14396
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rolling upgrades
> Reporter: Fei Hui
> Assignee: Fei Hui
> Priority: Blocker
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14396.001.patch, HDFS-14396.002.patch
>
>
> After fixing HDFS-13596, try to downgrade from 3.x to 2.x. But namenode can't
> start because exception occurs. The message follows
> {code:java}
> 2019-01-23 17:22:18,730 ERROR org.apache.hadoop.hdfs.server.namenode.FSImage:
> Failed to load image from
> FSImageFile(file=/data1/hadoopdata/hadoop-namenode/current/fsimage_0000000000000025310,
> cpktTxId=0000000000
> 000025310)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:179)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:885)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:869)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:742)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:673)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:290)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:998)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:612)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:672)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:839)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:823)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1517)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1583)
> 2019-01-23 17:22:19,023 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: Failed to load FSImage file, see error(s) above for more
> info.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:688)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:290)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:998)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:612)
> {code}
> This issue occurs because 3.x namenode saves image with EC fields during
> upgrade
> Try to fix it
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
