[
https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324607#comment-16324607
]
Ajay Kumar edited comment on HDFS-12942 at 1/12/18 9:58 PM:
------------------------------------------------------------
{quote}While cleaning up the new replica on failure, we call
volume.onBlockFileDeletion which decrements the space used for the block on the
new target volume. However I am not sure that the space used was incremented in
the first place by copyReplicaToVolume. This may be a pre-existing bug
actually.{quote}
[~arpitagarwal],[~virajith] good catch.
{code} try (AutoCloseableLock lock = datasetLock.acquire()) {
// Increment numBlocks here as this block moved without knowing to BPS
FsVolumeImpl volume = (FsVolumeImpl) newReplicaInfo.getVolume();
volume.incrNumBlocks(block.getBlockPoolId());
}{code}
Seems in case of success we should increment the "dfs used" and "no of blocks"
for new volume and decrement the same for old volume. Currently we only
increment no of blocks. Is this bug or intentional? [~arpitagarwal], thanks for
offline discussion on this.
In case of failure we should free up memory on disk. (will update patch for
same)
{quote}Adding to this, in patch v4, the generation stamp check in
finalizeReplica is done after a call to v.addFinalizedBlock which seems wasted
work if the check doesn't pass. Can the generation stamp check be done before
the call to v.addFinalizedBlock?{quote}
Agree,Will move check in start of finalize call.
was (Author: ajayydv):
{quote}While cleaning up the new replica on failure, we call
volume.onBlockFileDeletion which decrements the space used for the block on the
new target volume. However I am not sure that the space used was incremented in
the first place by copyReplicaToVolume. This may be a pre-existing bug
actually.{quote}
[~arpitagarwal],[~virajith] good catch.
{code} try (AutoCloseableLock lock = datasetLock.acquire()) {
// Increment numBlocks here as this block moved without knowing to BPS
FsVolumeImpl volume = (FsVolumeImpl) newReplicaInfo.getVolume();
volume.incrNumBlocks(block.getBlockPoolId());
}{code}
Seems in case of success we should increment the dfs used and no of blocks for
new volume and decrement the same for old block. Currently we only increment no
of blocks. Is this bug or intentional? [~arpitagarwal], thanks for offline
discussion on this.
In case of failure we should free up memory on disk. (will update patch for
same)
{quote}Adding to this, in patch v4, the generation stamp check in
finalizeReplica is done after a call to v.addFinalizedBlock which seems wasted
work if the check doesn't pass. Can the generation stamp check be done before
the call to v.addFinalizedBlock?{quote}
Agree,Will move check in start of finalize call.
> Synchronization issue in FSDataSetImpl#moveBlock
> ------------------------------------------------
>
> Key: HDFS-12942
> URL: https://issues.apache.org/jira/browse/HDFS-12942
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Ajay Kumar
> Assignee: Ajay Kumar
> Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch,
> HDFS-12942.003.patch, HDFS-12942.004.patch
>
>
> FSDataSetImpl#moveBlock works in following following 3 steps:
> # first creates a new replicaInfo object
> # calls finalizeReplica to finalize it.
> # Calls removeOldReplica to remove oldReplica.
> A client can potentially append to the old replica between step 1 and 2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
