[jira] [Comment Edited] (HDFS-15421) IBR leak causes standby NN to be stuck in safe mode
[
https://issues.apache.org/jira/browse/HDFS-15421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144579#comment-17144579
]
Takanobu Asanuma edited comment on HDFS-15421 at 6/25/20, 1:41 AM:
---
Thanks for working on this, [~aajisaka]. Thanks for reporting it, [~kihwal].
The patch looks good to me for the cases of append and truncate. But it may
still leak when lease recovery(block recovery) runs. The following code creates
a new GS.
[https://github.com/apache/hadoop/blob/4c53fb9ce102c46c6956b4aecdfd9dd513280b35/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L3724-L3735]
was (Author: tasanuma0829):
Thanks for working on this, [~aajisaka].
The patch looks good to me for the cases of append and truncate. But it may
still leak when lease recovery(block recovery) runs. The following code creates
a new GS.
https://github.com/apache/hadoop/blob/4c53fb9ce102c46c6956b4aecdfd9dd513280b35/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L3724-L3735
> IBR leak causes standby NN to be stuck in safe mode
> ---
>
> Key: HDFS-15421
> URL: https://issues.apache.org/jira/browse/HDFS-15421
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Kihwal Lee
>Assignee: Akira Ajisaka
>Priority: Blocker
> Labels: release-blocker
> Attachments: HDFS-15421-000.patch, HDFS-15421-001.patch,
> HDFS-15421.002.patch, HDFS-15421.003.patch, HDFS-15421.004.patch
>
>
> After HDFS-14941, update of the global gen stamp is delayed in certain
> situations. This makes the last set of incremental block reports from append
> "from future", which causes it to be simply re-queued to the pending DN
> message queue, rather than processed to complete the block. The last set of
> IBRs will leak and never cleaned until it transitions to active. The size of
> {{pendingDNMessages}} constantly grows until then.
> If a leak happens while in a startup safe mode, the namenode will never be
> able to come out of safe mode on its own.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15421) IBR leak causes standby NN to be stuck in safe mode
[
https://issues.apache.org/jira/browse/HDFS-15421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142884#comment-17142884
]
Akira Ajisaka edited comment on HDFS-15421 at 6/23/20, 1:33 PM:
{quote}I think we need to update genstamp when rolling {{OP_APPEND}}. In
{{OP_TRUNCATE}}, it is the same.
{quote}
This change does not fix the problem for append. When appending a block without
{{CreateFlag.NEW_BLOCK}}, the edit log becomes as follows:
* {{OP_APPEND}}: prepare for append
* {{OP_SET_GENSTAMP_V2}}: update pipeline
* (edited) {{OP_UPDATE_BLOCKS}}: update blocks
That way SNN will tail {{OP_SET_GENSTAMP_V2}} after {{OP_APPEND}}, so apply
impending genstamp in {{OP_APPEND}} does not fix this problem.
I'll attach a patch with some regression tests.
was (Author: ajisakaa):
{quote}I think we need to update genstamp when rolling {{OP_APPEND}}. In
{{OP_TRUNCATE}}, it is the same.
{quote}
This change does not fix the problem for append. When appending a block without
{{CreateFlag.NEW_BLOCK}}, the edit log becomes as follows:
* {{OP_APPEND}}: prepare for append
* {{OP_SET_GENSTAMP_V2}}: update pipeline
That way SNN will tail {{OP_SET_GENSTAMP_V2}} after {{OP_APPEND}}, so apply
impending genstamp in {{OP_APPEND}} does not fix this problem.
I'll attach a patch with some regression tests.
> IBR leak causes standby NN to be stuck in safe mode
> ---
>
> Key: HDFS-15421
> URL: https://issues.apache.org/jira/browse/HDFS-15421
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Kihwal Lee
>Assignee: Akira Ajisaka
>Priority: Blocker
> Labels: release-blocker
> Attachments: HDFS-15421-000.patch, HDFS-15421-001.patch
>
>
> After HDFS-14941, update of the global gen stamp is delayed in certain
> situations. This makes the last set of incremental block reports from append
> "from future", which causes it to be simply re-queued to the pending DN
> message queue, rather than processed to complete the block. The last set of
> IBRs will leak and never cleaned until it transitions to active. The size of
> {{pendingDNMessages}} constantly grows until then.
> If a leak happens while in a startup safe mode, the namenode will never be
> able to come out of safe mode on its own.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15421) IBR leak causes standby NN to be stuck in safe mode
[
https://issues.apache.org/jira/browse/HDFS-15421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140607#comment-17140607
]
Kihwal Lee edited comment on HDFS-15421 at 6/19/20, 2:56 PM:
-
Example of a leak itself: (single replica shown for simplicity)
1) IBRs queued. The file was created, data written to it and closed. Then it
was opened for append, additional data written and closed.
{noformat}
2020-06-19 02:38:27,423 [Block report processor] INFO
blockmanagement.BlockManager: Queueing reported block
blk_1521788462_1099975774416 in state FINALIZED from datanode 1.2.3.4:1004 for
later processing because generation stamp is in the future.
2020-06-19 02:38:28,190 [Block report processor] INFO
blockmanagement.BlockManager: Queueing reported block
blk_1521788462_1099975774420 in state FINALIZED from datanode 1.2.3.4:1004 for
later processing because generation stamp is in the future.
{noformat}
2) Processing of queued IBRs as edits replayed. The IBR with the first gen
stamp for the initial file is processed. The one from append is not, as the gen
stamp is still in the future. It is re-queued.
{noformat}
2020-06-19 02:42:22,774 [Edit log tailer] INFO blockmanagement.BlockManager:
Processing previouly queued message ReportedBlockInfo
[block=blk_1521788462_1099975774416, dn=1.2.3.4:1004, reportedState=FINALIZED]
2020-06-19 02:42:22,774 [Edit log tailer] INFO blockmanagement.BlockManager:
Processing previouly queued message ReportedBlockInfo
[block=blk_1521788462_1099975774420, dn=1.2.3.4:1004, reportedState=FINALIZED]
2020-06-19 02:42:22,774 [Edit log tailer] INFO blockmanagement.BlockManager:
Queueing reported block blk_1521788462_1099975774420 in state FINALIZED from
datanode 1.2.3.4:1004 for later processing because generation stamp is in the
future.
{noformat}
3) When the edits for append is replayed. The IBR is still identified as from
future and re-queued. Since there is no more edits regarding this file, the
IBR is leaked.
{noformat}
2020-06-19 02:42:22,776 [Edit log tailer] INFO blockmanagement.BlockManager:
Processing previouly queued message ReportedBlockInfo
[block=blk_1521788462_1099975774420, dn=1.2.3.4:1004, reportedState=FINALIZED]
2020-06-19 02:42:22,776 [Edit log tailer] INFO blockmanagement.BlockManager:
Queueing reported block blk_1521788462_1099975774420 in state FINALIZED from
datanode 1.2.3.4:1004 for later processing because generation stamp is in the
future.
{noformat}
With HDFS-14941 reverted, the last IBR is processed as expected and the leak
does not happen anymore.
Note: The original logging level of above lines are DEBUG, but was changed to
INFO temporarily.
was (Author: kihwal):
Example of a leak itself: (single replica shown for simplicity)
1) IBRs queued. The file was created, data written to it and closed. Then it
was opened for append, additional data written and closed.
{noformat}
2020-06-19 02:38:27,423 [Block report processor] INFO
blockmanagement.BlockManager: Queueing reported block
blk_1521788462_1099975774416 in state FINALIZED from datanode 1.2.3.4:1004 for
later processing because generation stamp is in the future.
2020-06-19 02:38:28,190 [Block report processor] INFO
blockmanagement.BlockManager: Queueing reported block
blk_1521788462_1099975774420 in state FINALIZED from datanode 1.2.3.4:1004 for
later processing because generation stamp is in the future.
{noformat}
2) Processing of queued IBRs as edits replayed. The IBR with the first gen
stamp for the initial file is processed. The one from append is not, as the gen
stamp is still in the future. It is re-queued.
{noformat}
2020-06-19 02:42:22,774 [Edit log tailer] INFO blockmanagement.BlockManager:
Processing previouly queued message ReportedBlockInfo
[block=blk_1521788462_1099975774416, dn=1.2.3.4:1004, reportedState=FINALIZED]
2020-06-19 02:42:22,774 [Edit log tailer] INFO blockmanagement.BlockManager:
Processing previouly queued message ReportedBlockInfo
[block=blk_1521788462_1099975774420, dn=1.2.3.4:1004, reportedState=FINALIZED]
2020-06-19 02:42:22,774 [Edit log tailer] INFO blockmanagement.BlockManager:
Queueing reported block blk_1521788462_1099975774420 in state FINALIZED from
datanode 1.2.3.4:1004 for later processing because generation stamp is in the
future.
{noformat}
3) When the edits for append is replayed. The IBR is still identified as from
future and re-queued. Since there is no more edits regarding this file, the
IBR is leaked.
{noformat}
2020-06-19 02:42:22,776 [Edit log tailer] INFO blockmanagement.BlockManager:
Processing previouly queued message ReportedBlockInfo
[block=blk_1521788462_1099975774420, dn=1.2.3.4:1004, reportedState=FINALIZED]
2020-06-19 02:42:22,776 [Edit log tailer] INFO blockmanagement.BlockManager:
Queueing reported block blk_1521788462_1099975774420 in state FINALIZED from
datanode 1.2.3.4:1004 for later processing
