ConfX created HDFS-17105:
----------------------------
Summary: mistakenly purge editLogs even after it is empty in
NNStorageRetentionManager
Key: HDFS-17105
URL: https://issues.apache.org/jira/browse/HDFS-17105
Project: Hadoop HDFS
Issue Type: Bug
Reporter: ConfX
Attachments: reproduce.sh
h2. What happened:
Got {{IndexOutOfBoundsException}} after setting
{{dfs.namenode.max.extra.edits.segments.retained}} to a negative value and
purging old record with {{{}NNStorageRetentionManager{}}}.
h2. Where's the bug:
In line 156 of {{{}NNStorageRetentionManager{}}}, the manager trims
{{editLogs}} until it is under the {{{}maxExtraEditsSegmentsToRetain{}}}:
{noformat}
while (editLogs.size() > maxExtraEditsSegmentsToRetain) {
purgeLogsFrom = editLogs.get(0).getLastTxId() + 1;
editLogs.remove(0);
}{noformat}
However, if {{dfs.namenode.max.extra.edits.segments.retained}} is set to below
0 the size of {{editLogs}} would never be below, resulting in ultimately
{{editLog.size()=0}} and thus {{editLogs.get(0)}} is out of range.
h2. How to reproduce:
(1) Set {{dfs.namenode.max.extra.edits.segments.retained}} to -1974676133
(2) Run test:
{{org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionManager#testNoLogs}}
h2. Stacktrace:
{noformat}
java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at
java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
at
java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
at
java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
at java.base/java.util.Objects.checkIndex(Objects.java:372)
at java.base/java.util.ArrayList.get(ArrayList.java:459)
at
org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:157)
at
org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionManager.runTest(TestNNStorageRetentionManager.java:299)
at
org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionManager.testNoLogs(TestNNStorageRetentionManager.java:143){noformat}
For an easy reproduction, run the reproduce.sh in the attachment.
We are happy to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
