[
https://issues.apache.org/jira/browse/HDFS-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14282985#comment-14282985
]
Chris Nauroth commented on HDFS-3519:
-------------------------------------
[~mingma], thank you for working on this.
I think the test failures shown in the last Jenkins run are unrelated. I did
multiple test runs locally, and they always passed.
A few comments on the patch:
* {{FSImage#saveNamespace}}: It's possible to leave this method after adding
the transaction ID to the checkpointing set, but without removing it. This
would leave the transaction ID in the set permanently, and I believe it would
then be impossible to checkpoint at that transaction ID again. Even though
{{removeFromCheckpointing}} is called in a {{finally}} block, it is preceded by
a call to {{FSEditLog#startLogSegmentAndWriteHeaderTxn}}, which can throw
{{IOException}}. I think we'll need to wrap the whole logic in a second layer
of try-finally to guarantee the transaction ID gets removed from the set.
* {{FSImage#saveFSImageInAllDirs}}: Now that this is wrapped in try-finally,
there is an existing line of code that needs to be indented.
> Checkpoint upload may interfere with a concurrent saveNamespace
> ---------------------------------------------------------------
>
> Key: HDFS-3519
> URL: https://issues.apache.org/jira/browse/HDFS-3519
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Reporter: Todd Lipcon
> Assignee: Ming Ma
> Priority: Critical
> Attachments: HDFS-3519-2.patch, HDFS-3519.patch, test-output.txt
>
>
> TestStandbyCheckpoints failed in [precommit build
> 2620|https://builds.apache.org/job/PreCommit-HDFS-Build/2620//testReport/]
> due to the following issue:
> - both nodes were in Standby state, and configured to checkpoint "as fast as
> possible"
> - NN1 starts to save its own namespace
> - NN2 starts to upload a checkpoint for the same txid. So, both threads are
> writing to the same file fsimage.ckpt_12, but the actual file contents
> correspond to the uploading thread's data.
> - NN1 finished its saveNamespace operation while NN2 was still uploading. So,
> it renamed the ckpt file. However, the contents of the file are still empty
> since NN2 hasn't sent any bytes
> - NN2 finishes the upload, and the rename() call fails, which causes the
> directory to be marked failed, etc.
> The result is that there is a file fsimage_12 which appears to be a finalized
> image but in fact is incompletely transferred. When the transfer completes,
> the problem "heals itself" so there wouldn't be persistent corruption unless
> the machine crashes at the same time. And even then, we'd still have the
> earlier checkpoint to restore from.
> This same race could occur in a non-HA setup if a user puts the NN in safe
> mode and issues saveNamespace operations concurrent with a 2NN checkpointing,
> I believe.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
