[
https://issues.apache.org/jira/browse/HDFS-17658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17907153#comment-17907153
]
ASF GitHub Bot commented on HDFS-17658:
---------------------------------------
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1892968157
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:
##########
@@ -1744,6 +1744,27 @@ public class DFSConfigKeys extends
CommonConfigurationKeys {
public static final boolean
DFS_DATANODE_LOCKMANAGER_TRACE_DEFAULT = false;
+ /**
+ * A datanode should only enter decommissioned state if all blocks
+ * on the datanode are sufficiently replicated to other live datanodes.
+ *
+ * When this setting is disabled, the Namenode does not consider
+ * Under Construction blocks in determining if a datanode can be
+ * decommissioned; this results in scenarios where datanodes enter
+ * decommissioned state before their blocks are sufficiently replicated
+ * to other live datanodes. This can lead to HDFS write failures and
+ * data loss, if all the datanodes in the block write pipeline are
+ * decommissioned and terminated at around the same time.
+ *
+ * Enable the following setting to have the Namenode track and consider
+ * Under Construction blocks when identifying if a datanode can be
+ * decommissioned.
+ */
+ public static final String DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS =
Review Comment:
I will use the version of the comment which you have suggested above:
```
/**
* Determines whether the NameNode tracks under-construction blocks when
decommissioning DataNodes.
*
* A DataNode should only enter decommissioned state if all blocks on the
DataNode are
* sufficiently replicated to other live DataNodes.
*
* When this setting is disabled, the NameNode does not consider
under-construction blocks
* in determining if a DataNode can be decommissioned. This can result in
scenarios where
* DataNodes enter decommissioned state before their blocks are sufficiently
replicated
* to other live DataNodes.
*
* This situation can lead to HDFS write failures and data loss if all the
DataNodes in
* the block write pipeline are decommissioned and terminated at around the
same time.
*
* Enable this setting to have the NameNode track and consider
under-construction blocks
* when identifying if a DataNode can be decommissioned.
*/
```
Please let me know if there are still any concerns about the wording
> HDFS decommissioning does not consider if Under Construction blocks are
> sufficiently replicated which causes HDFS Data Loss
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17658
> URL: https://issues.apache.org/jira/browse/HDFS-17658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 3.4.0
> Reporter: Kevin Wikant
> Priority: Major
> Labels: pull-request-available
>
> h2. Background
> The HDFS Namenode manages datanode decommissioning using the
> DatanodeAdminManager
> The DatanodeAdminManager has logic to prevent transitioning datanodes to
> decommissioned state if they contain open blocks (i.e. Under Construction
> blocks) which are not sufficiently replicated to other datanodes. In the
> context of decomissioning, the reason that open blocks are important is
> because they cannot be replicated to other datanodes given they are actively
> being appended by an HDFS client. Because open blocks cannot be moved during
> decommissioning, they should prevent the associated datanodes from becoming
> decommissioned until the block is completed/finalized and it can be safely
> moved to other live datanode(s).
>
> h2. Problem
> The logic for DatanodeAdminManager to avoid decommissioning datanodes which
> contain open blocks does not properly consider blocks which are in Under
> Construction state. The DatanodeAdminManager is only considering blocks which
> are in committed/finalized state. For reference:
> * a block which is actively being appended by a DFSOutputStream is in Under
> Construction state
> * only when a DFSOutputStream is closed does the block transition from Under
> Construction state to Committed/Finalized state
> * then later when the block is reported to the namenode in a block report,
> it will transition from Committed to Completed state
> Furthermore:
> * this is true for a new file/block that was just created via
> "DFSClient.create"
> * this is true for an existing file/block that was just opened via
> "DFSClient.append"
> * this is true for all different dfs.replication factor values
>
> h2. Impact
> In some environments, a datanode being decommissioned is taken as a signal
> that all the blocks on that datanodes are sufficiently replicated to other
> live datanodes & therefore it is safe to terminate the underlying virtual
> host running the datanode.
> These types of environments can be impacted by HDFS data loss for open blocks
> which are not considered in the datanode decommissioning process. Since open
> blocks are not considered in determining if a datanode can be decommissioned,
> a datanode may become decommissioned before its open blocks are replicated to
> other datanodes. If the decommissioned datanode is then terminated, the open
> blocks will be lost.
> For open blocks with replication of 1, it takes a single
> decommissioned/terminated datanode to cause data loss. For open blocks with
> greater replication, all the datanodes holding the open block must be
> decommissioned/terminated for there to be data loss.
> I would also break the impact down into 2 cases:
> * for a new HDFS block that has never been closed, arguably if the
> DFSOutputStream encounters failure then the client should be able to replay
> the data from source
> * for an existing HDFS block that has previously been closed, when this
> block is opened via a new DFSOutputStream the block is susceptible to being
> lost & the client cannot replay data which was appended in the past by a
> different client. This is arguably the worse case of impact.
>
> h2. Testing
> This behaviour has been verified via testing on Hadoop 3.4.0; however, I
> suspect it also applies to many other older/newer Hadoop versions.
> See JIRA comments for detailed test methodology & results.
> The following is a summary of the test cases & test results:
> {quote}*Test#1: Create Block & Repeatedly Append in Loop → Decommission
> Datanode*
> ^ Expectation: block is considered during decommissioning; datanode is not
> decommissioned until the write operation is finished & block is replicated to
> another datanode.
> ^ Observation: block is not considered during decommissioning & is lost when
> decommissioned data is terminated.
> *Test#2: Create Block & Repeatedly Append in Loop → Close DFSOutputStream →
> Decommission Datanode*
> ^ Expectation: block is considered during decommissioning & is replicated to
> another datanode as part of decommissioning.
> ^ Observation: block is considered during decommissioning & is replicated to
> another datanode as part of decommissioning.
> *Test#3: Create Block & Repeatedly Append in Loop → Close DFSOutputStream →
> Re-open Block & Repeatedly Append in Loop → Decommission Datanode*
> ^ Expectation: block is considered during decommissioning; datanode is not
> decommissioned until the write operation is finished & block is replicated to
> another datanode.
> ^ Observation: block is not considered during decommissioning & is lost when
> decommissioned data is terminated.
> *Test#4: Create Block & Repeatedly Append in Loop → Close DFSOutputStream →
> Re-open Block & Repeatedly Append in Loop → Close DFSOutputStream →
> Decommission Datanode*
> ^ Expectation: block is considered during decommissioning & is replicated to
> another datanode as part of decommissioning.
> ^ Observation: block is not considered during decommissioning & is lost when
> decommissioned data is terminated.
> {quote}
> These were all tested with replication factor of 1 & 2, observation is the
> same in both cases.
>
> h2. Root Cause Theory
> The DatanodeAdminManager relies on the DatanodeDescriptor StorageInfos to
> identify which blocks to consider as part of wether or not a datanode can be
> decommissioned.
> Based on an examination of the HDFS Namenode & Datanode DEBUG logs during
> testing, I believe the root cause has to do with the following 2 behaviours:
> {{*1.* When a DFSOutputStream is created for a block, that block enters Under
> Construction state but the Namenode does not add Under Construction blocks to
> the StorageInfos unless they have been committed/finalized which only occurs
> when the DFSOutputStream is closed. Therefore, by checking the StorageInfos
> only, the DatanodeAdminManager is not actually checking Under Construction
> blocks.}}
> During the testing, we see that the Under Construction blocks are not in the
> StorageInfos for the corresponding DatanodeDescriptor:
> {quote}2024-11-05 14:32:19,206 DEBUG BlockStateChange: BLOCK* block
> RECEIVING_BLOCK: blk_1073741825_1001 is received from 172.31.93.123:9866
> ...
> 2024-11-05 14:36:02,805 INFO blockmanagement.DatanodeAdminManager: Starting
> decommission of 172.31.93.123:9866
> [DISK]DS-bb1d316c-a47a-4a3a-bf6e-0781945a50d1:NORMAL:172.31.93.123:9866 with
> 0 blocks
> 2024-11-05 14:36:02,805 INFO blockmanagement.DatanodeAdminManager: Starting
> decommission of 172.31.93.123:9866
> [DISK]DS-95cfd2fa-25b0-4b20-aacf-e259201cf2eb:NORMAL:172.31.93.123:9866 with
> 0 blocks
> {quote}
> Whereas, if the DFSOutputStream is closed, we see the block is included in
> the StorageInfos:
> {quote}2024-11-05 14:49:49,563 DEBUG BlockStateChange: BLOCK* block
> RECEIVED_BLOCK: blk_1073741825_1001 is received from 172.31.95.159:9866
> ...
> 2024-11-05 14:52:36,770 INFO blockmanagement.DatanodeAdminManager: Starting
> decommission of 172.31.95.159:9866
> [DISK]DS-07f2256b-0cbc-4b5e-9c6c-9108650ee896:NORMAL:172.31.95.159:9866 with
> 0 blocks
> 2024-11-05 14:52:36,770 INFO blockmanagement.DatanodeAdminManager: Starting
> decommission of 172.31.95.159:9866
> [DISK]DS-91919280-df53-4dac-b983-0eb12c81e4bf:NORMAL:172.31.95.159:9866 with
> 1 blocks
> 2024-11-05 14:52:48,231 INFO BlockStateChange: Block: blk_1073741825_1001,
> Expected Replicas: 1, live replicas: 0, corrupt replicas: 0, decommissioned
> replicas: 0, decommissioning replicas: 1, maintenance replicas: 0, live
> entering maintenance replicas: 0, replicas on stale nodes:0, readonly
> replicas: 0, excess replicas: 0, Is Open File: false, Datanodes having this
> block: 172.31.95.159:9866 , Current Datanode: 172.31.95.159:9866, Is current
> datanode decommissioning: true, Is current datanode entering maintenance:
> false
> {quote}
> I think this behaviour might be related to the following code which has been
> in Hadoop for the past 10+ years:
> [https://github.com/apache/hadoop/blob/51ebc3c20e8ae7d4dced41cdd2f52715aea604cc/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L3708]
>
>
> {{*2.* When a DFSOutputStream is created for an existing block, the Namenode
> marks the existing block with previous generation stamp as stale & removes it
> from the StorageInfos.}}
> We can see the following DEBUG logs during repro testing:
> {quote}2024-11-05 15:47:24,131 INFO namenode.FSNamesystem:
> updatePipeline(blk_1073741825_1001, newGS=1002, newLength=307200,
> newNodes=[172.31.95.208:9866], client=DFSClient_NONMAPREDUCE_-1408310900_1)
> 2024-11-05 15:47:24,132 DEBUG BlockStateChange: BLOCK* Removing stale replica
> ReplicaUC[[DISK]DS-1bd20290-4662-4e5e-af0b-9b644d78d4f8:NORMAL:172.31.95.208:9866|RBW]
> of blk_1073741825_1001
> 2024-11-05 15:47:24,132 INFO namenode.FSNamesystem:
> updatePipeline(blk_1073741825_1001 => blk_1073741825_1002) success
> {quote}
> Tracing the code from the logs we see where this occurs:
> *
> [https://github.com/apache/hadoop/blob/f7651e2f63ddba9ed1ae4052e38464f85dd445f0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L4476]
> *
> [https://github.com/apache/hadoop/blob/f7651e2f63ddba9ed1ae4052e38464f85dd445f0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L4433]
> *
> [https://github.com/apache/hadoop/blob/f7651e2f63ddba9ed1ae4052e38464f85dd445f0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java#L202]
>
>
> h3. Potential Solution
> If possible, can consider adding Under Construction blocks to StorageInfos.
> If this would have other adverse impacts, then another solution is to expose
> the Under Construction blocks to the DatanodeAdminManager via another
> separate data structure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
