[
https://issues.apache.org/jira/browse/HDFS-17976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-17976:
----------------------------------
Labels: pull-request-available (was: )
> HDFS DataNode add configurable inactivity-based timeout for DataNode block
> transfers
> ------------------------------------------------------------------------------------
>
> Key: HDFS-17976
> URL: https://issues.apache.org/jira/browse/HDFS-17976
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
> Reporter: Rajan Dhabalia
> Priority: Major
> Labels: pull-request-available
>
> h2. Summary
> Add a configurable inactivity-based timeout to safely terminate stalled
> DataNode block transfers, reclaim resources, and preserve partially written
> replicas for recovery.
> h2. Issue Type
> Improvement
> h2. Component/s
> datanode
> h2. Description
> h3. Motivation
> A block write remains active until the client sends the final packet and the
> block is finalized. If a client crashes, loses connectivity, or becomes
> unresponsive, the DataNode transfer thread can remain blocked waiting for
> additional data.
> Stalled transfers can accumulate over time, consuming transfer threads and
> other resources and potentially affecting healthy client operations.
> h3. Approach
> Introduce a configurable inactivity timeout for DataNode block transfers.
> When enabled:
> * Track packet-arrival activity for each ongoing block write.
> * Schedule an inactivity check for each active transfer.
> * Terminate the transfer if no packet is received within the configured
> timeout.
> * Flush buffered data before termination.
> * Close the input stream to interrupt blocked reads.
> * Keep the replica in *RBW (Replica Being Written)* state.
> * Allow existing NameNode lease recovery and block synchronization to
> recover and finalize the replica.
> The timeout uses a shared `ScheduledExecutorService` and is created lazily
> only when enabled. When disabled, there are no additional scheduler threads
> or timeout processing.
> h3. Configuration
> ||Property||Default||Description||
> |`dfs.datanode.last.packet.receive.timeout.ms`|0|Inactivity timeout for an
> ongoing block transfer. If no packet is received within the configured
> window, the transfer is considered stalled and terminated. `0` or negative
> disables the feature.|
> h3. Transfer Lifecycle
> {code:java}
> Client starts block write
> |
> v
> DataNode receives packets
> |
> v
> Track packet activity
> |
> v
> Packet received?
> / \
> Yes No
> | |
> v v
> Reset timer Timeout reached
> |
> v
> Flush buffered data
> |
> v
> Close input stream
> |
> v
> Replica remains RBW
> |
> v
> Existing lease recovery /
> block synchronization
> |
> v
> Block finalized
> {code}
> h3. Data Safety
> When a stalled transfer is terminated:
> # Buffered data is flushed.
> # The input stream is closed.
> # The replica remains in RBW state.
> # Existing HDFS recovery mechanisms handle subsequent recovery/finalization.
> This preserves data already received by the DataNode while reclaiming stalled
> transfer resources.
> h3. Results
> The inactivity timeout provides:
> * Bounded resource usage for stalled transfers.
> * Prevention of indefinite transfer-thread retention after client failures.
> * Reduced risk of transfer-thread exhaustion during client failure bursts.
> * Preservation of partially written replica data.
> * Compatibility with existing HDFS recovery mechanisms.
> * No additional overhead when disabled.
> h3. Estimated Impact
> This is primarily a reliability and resource-reclamation improvement rather
> than a throughput optimization.
> ||Dimension||Without Timeout||With Inactivity Timeout||
> |Stalled transfer lifetime|Potentially unbounded|Bounded by configured
> timeout|
> |Transfer threads|Can accumulate|Reclaimed after inactivity|
> |Client failure bursts|Risk of resource exhaustion|Resource usage remains
> bounded|
> |Partially written data|Preserved through existing recovery|Preserved;
> replica remains RBW|
> |Normal transfers|Existing behavior|Unchanged|
> |Disabled overhead|Existing behavior|No additional scheduler/timeout
> processing|
> h3. Operational Considerations
> The timeout should be configured based on workload characteristics. An overly
> aggressive timeout could terminate legitimately slow transfers, while a
> sufficiently large timeout allows temporary network or client stalls while
> still reclaiming resources from genuinely stalled transfers.
> The feature is therefore *disabled by default* and can be enabled explicitly
> by operators.
> h2. Expected Impact
> * Improve DataNode resilience to crashed, disconnected, or hung clients.
> * Prevent stalled transfers from holding resources indefinitely.
> * Bound transfer-resource usage during client failure bursts.
> * Preserve partially written data for existing recovery mechanisms.
> * Keep normal block-transfer behavior unchanged when disabled.
> h2. Backward Compatibility
> The feature is {*}disabled by default{*}. Existing DataNode block-transfer
> behavior remains unchanged unless
> `dfs.datanode.last.packet.receive.timeout.ms` is explicitly configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
