[
https://issues.apache.org/jira/browse/HDFS-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133013#comment-15133013
]
Kihwal Lee commented on HDFS-9752:
----------------------------------
The fix looks good, but we don't need to restart the datanode 10 times in the
test.
With the sleeps, it take about 102 seconds to run this test case. If we just do
6 times, it becomes like 64 seconds and still fails without the fix. The use of
{{Thread.sleep()}} in the test may not always be reliable , I guess there is no
other choice. The shutdown should be quick since the datanode is only serving
one writer. So the total sleep time per iteration has to be a bit longer than
the client wait time of 4 seconds. I think we can get away with making the
second sleep 2 seconds instead of 5 seconds. After making it restart 6 times
and shrinking the sleep time, the test case finished in about 48 seconds.
> Permanent write failures may happen to slow writers during datanode rolling
> upgrades
> ------------------------------------------------------------------------------------
>
> Key: HDFS-9752
> URL: https://issues.apache.org/jira/browse/HDFS-9752
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Kihwal Lee
> Assignee: Walter Su
> Priority: Critical
> Attachments: HDFS-9752.01.patch
>
>
> When datanodes are being upgraded, an out-of-band ack is sent upstream and
> the client does a pipeline recovery. The client may hit this multiple times
> as more nodes get upgraded. This normally does not cause any issue, but if
> the client is holding the stream open without writing any data during this
> time, a permanent write failure can occur.
> This is because there is a limit of 5 recovery trials for the same packet,
> which is tracked by "last acked sequence number". Since the empty heartbeat
> packets for an idle output stream does not increment the sequence number, the
> write will fail after it seeing 5 pipeline breakages by datanode upgrades.
> This check/limit was added to avoid spinning until running out of nodes in
> the cluster due to a corruption or any other irrecoverable conditions. The
> datanode upgrade-restart should be excluded from the count.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
