[ https://issues.apache.org/jira/browse/HDFS-15624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17222863#comment-17222863 ]
Ayush Saxena commented on HDFS-15624: ------------------------------------- No, I didn't try. I just saw the code and thought it would be a problem and folks there at HDFS-15025 confirmed too. I was just bothered it should get fixed and frankly didn't follow much. (Limited Bandwidth, Saw the code again today, because [~huangtianhua] asked for) I just tried, First the upgrade issue, due to the change of ordinal, {noformat} ayushsaxena@ayushsaxena-MBP16 hadoop % bin/hdfs dfsadmin -setSpaceQuota 5 -storageType ARCHIVE /dir 2020-10-29 16:14:03,072 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ayushsaxena@ayushsaxena-MBP16 hadoop % bin/hdfs dfs -count -q -h -t ARCHIVE /dir 2020-10-29 16:15:43,660 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 5 5 /dir ayushsaxena@ayushsaxena-MBP16 hadoop % bin/hdfs dfs -count -q -h -t DISK /dir 2020-10-29 16:15:52,942 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable none inf /dir Upgrade Namenode : Magic, Had set quota for ARCHIVE not for DISK ayushsaxena@ayushsaxena-MBP16 hadoop % bin/hdfs dfs -count -q -h -t DISK /dir 2020-10-29 16:31:39,677 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 5 5 /dir ayushsaxena@ayushsaxena-MBP16 hadoop % bin/hdfs dfs -count -q -h -t ARCHIVE /dir 2020-10-29 16:31:48,698 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable none inf /dir {noformat} Now the Rolling Upgrade stuff, I had set quota on Provided before finalising the upgrade, then Rollback, My NN crashed with : {noformat} 2020-10-29 16:51:06,261 ERROR org.apache.hadoop.hdfs.server.namenode.FSImage: Error replaying edit log at offset 25. Expected transaction ID was 9 Recent opcode offsets: 25 java.lang.ArrayIndexOutOfBoundsException: 5 at org.apache.hadoop.fs.StorageType.parseStorageType(StorageType.java:80) at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$SetQuotaByStorageTypeOp.readFields(FSEditLogOp.java:2363) at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$LengthPrefixedReader.decodeOp(FSEditLogOp.java:5161) at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$Reader.readOp(FSEditLogOp.java:5021) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:229) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:276) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:200) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:243) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:182) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:914) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:761) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:338) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1197) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:779) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:761) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1015) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:988) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1757) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1822) 2020-10-29 16:51:06,263 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception loading fsimage org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 25. Expected transaction ID was 9 Recent opcode offsets: 25 at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:256) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:182) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:914) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:761) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:338) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1197) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:779) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:761) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1015) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:988) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1757) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1822) Caused by: java.lang.ArrayIndexOutOfBoundsException: 5 at org.apache.hadoop.fs.StorageType.parseStorageType(StorageType.java:80) at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$SetQuotaByStorageTypeOp.readFields(FSEditLogOp.java:2363) at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$LengthPrefixedReader.decodeOp(FSEditLogOp.java:5161) at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$Reader.readOp(FSEditLogOp.java:5021) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:229) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:276) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:200) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:243) ... 12 more 2020-10-29 16:51:06,265 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.w.WebAppContext@1c39680d{hdfs,/,null,UNAVAILABLE}{file:/Users/ayushsaxena/code/hadoop-code/osCode/hadoop-3.3.1-SNAPSHOT/share/hadoop/hdfs/webapps/hdfs} 2020-10-29 16:51:06,269 INFO org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@6b09fb41{HTTP/1.1,[http/1.1]}{0.0.0.0:9870} 2020-10-29 16:51:06,269 INFO org.eclipse.jetty.server.session: node0 Stopped scavenging 2020-10-29 16:51:06,269 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@6f10d5b6{static,/static,file:///Users/ayushsaxena/code/hadoop-code/osCode/hadoop-3.3.1-SNAPSHOT/share/hadoop/hdfs/webapps/static/,UNAVAILABLE} 2020-10-29 16:51:06,269 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@1972e513{logs,/logs,file:///Users/ayushsaxena/code/hadoop-code/osCode/hadoop-3.3.1-SNAPSHOT/logs/,UNAVAILABLE} 2020-10-29 16:51:06,270 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system... 2020-10-29 16:51:06,270 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped. 2020-10-29 16:51:06,270 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete. 2020-10-29 16:51:06,270 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode. org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 25. Expected transaction ID was 9 Recent opcode offsets: 25 at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:256) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:182) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:914) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:761) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:338) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1197) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:779) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:674) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:761) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1015) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:988) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1757) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1822) Caused by: java.lang.ArrayIndexOutOfBoundsException: 5 at org.apache.hadoop.fs.StorageType.parseStorageType(StorageType.java:80) at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$SetQuotaByStorageTypeOp.readFields(FSEditLogOp.java:2363) at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$LengthPrefixedReader.decodeOp(FSEditLogOp.java:5161) at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$Reader.readOp(FSEditLogOp.java:5021) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:229) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:276) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:200) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:243) ... 12 more 2020-10-29 16:51:06,271 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 25. Expected transaction ID was 9 Recent opcode offsets: 25 2020-10-29 16:51:06,272 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at ayushsaxena-MBP16.local ************************************************************/{noformat} I am not able to capture screenshot, Couldn't get the log fit on one screen(I tried but.) That much discount I can expect from you. :) Wanted to check HDFS-15660 too, but now my namenode isn't starting now because of this, and It will take time to sort it. But I am still convinced it is different, so no need to hold this, But still I will respect opinions on how to go ahead. But in case people want to hold, I think, If I have not messed up. It's good to go back. > Fix the SetQuotaByStorageTypeOp problem after updating hadoop > --------------------------------------------------------------- > > Key: HDFS-15624 > URL: https://issues.apache.org/jira/browse/HDFS-15624 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs > Reporter: YaYun Wang > Priority: Major > Labels: pull-request-available > Time Spent: 5h > Remaining Estimate: 0h > > HDFS-15025 adds a new storage Type NVDIMM, changes the ordinal() of the enum > of StorageType. And, setting the quota by storageType depends on the > ordinal(), therefore, it may cause the setting of quota to be invalid after > upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org