rdhabalia opened a new pull request, #8714:
URL: https://github.com/apache/hadoop/pull/8714
### Description of PR
HDFS blocks are processed as smaller packets. Concurrent writes across
multiple blocks/files can interleave these packets, creating non-sequential
disk I/O.
This is particularly expensive on HDDs, resulting in:
* Small and fragmented writes
* Higher IOPS and seek overhead
* Page-cache pressure from dirty pages
* Read/write contention
* Higher read and write tail latency
* Lower disk and DataNode throughput
**Goal:** Improve DataNode vertical efficiency and provide more predictable
performance for mixed read/write workloads.
---
## Proposed Solution
Introduce a DataNode-managed in-memory write buffer:
```text
Small Packets
|
v
In-Memory Buffer
|
v
Large Batched Writes
|
v
O_DIRECT
|
v
Disk
```
### Key Design
1. **Large write buffers**
* Accumulate small packets before flushing.
* Reduce I/O operations and physical I/O fragmentation.
* Improve sequential writes and disk bandwidth utilization.
2. **Direct I/O**
* Use `O_DIRECT` for buffered flushes.
* Avoid dirty-page pressure and preserve OS page cache for reads and
read-ahead.
3. **Controlled flush concurrency**
* Limit concurrent flush bytes per volume.
* Balance disk parallelism, sequential I/O, and read latency.
4. **Last-replica-only mode**
* Optionally enable buffering only on the last DataNode in the replication
pipeline.
5. **Bounded memory**
* Configurable limits for total buffer capacity, per-block buffer size,
and flush concurrency.
6. **Idle flush**
* Flush partially filled buffers after a configurable timeout.
---
## Read Path Benefit
By bypassing the OS page cache for DataNode writes, system memory can be
used more effectively for:
* Read cache
* Read-ahead
* Other filesystem activity
Expected benefits:
* Lower read latency
* Better read-ahead effectiveness
* Reduced eviction of useful read pages
* Less read/write contention
* More stable mixed-workload performance
---
## Key Configuration
| Property | Description
|
| ---------------------------------------------------- |
-------------------------------------------- |
| `dfs.datanode.write.memory.buffer.enabled` | Enable write
buffering |
| `dfs.datanode.write.memory.buffer.last-replica-only` | Buffer only the
last replica |
| `dfs.datanode.write.memory.buffer.max.capacity.mb` | Maximum total
buffer capacity |
| `dfs.datanode.write.buffer.size.bytes` | Per-block buffer
size |
| `dfs.datanode.write.buffer.idle.flush.timeout.ms` | Idle flush timeout
|
| `dfs.datanode.concurrent.flush.mb.per.volume` | Maximum concurrent
flush capacity per volume |
The feature is **disabled by default**.
---
## Implementation
The change introduces:
* `BufferedBlockWriter`
* `BufferedBlockWriterImpl`
Integrated with:
* `BlockReceiver`
* `DataXceiver`
* `FsVolumeImpl`
When disabled, the existing DataNode write path remains unchanged.
---
## Benchmark Results
| Metric | Improvement |
| ------------------------------------- | --------------------: |
| Mixed 80/20 read/write throughput | ~30% higher |
| Read-only throughput | Up to ~45% higher |
| Large-block P99 write latency | ~80% lower |
| Read latency during concurrent writes | Significant reduction |
Disk-level results also show:
* `r_await`: Reduced from frequently >100 ms to consistently <50 ms
* `w_await`: Reduced from frequently >200 ms to <90 ms
* Per-disk throughput: Improved from ~120 MB/s to >170 MB/s under mixed
workloads
---
## Expected Impact
For I/O-bound workloads:
* **25-40% higher throughput** for write-heavy workloads
* **Up to ~45% higher throughput** for read-heavy workloads
* **~80% lower large-block P99 write latency**
* Lower read latency during concurrent writes
* Reduced random I/O and disk contention
* Better disk bandwidth utilization
* More predictable mixed read/write performance
Actual gains depend on workload characteristics and storage hardware. CPU-,
network-, or metadata-bound workloads are expected to see limited benefit.
### Performance impact and IO Stats
IO stats for a single disk: Before the change, the disk achieved around 120
MBps for certain read-heavy workloads, but the throughput was bursty and
inconsistent. After the change, the disk consistently achieved around 170 MBps,
as shown in Image 1.
Image 2 compares the read_await time before and after the change. After the
change, r_await consistently remains low, enabling higher and more stable read
throughput compared to the previous results.
#### Read, Write, Total Throughput
<img width="2314" height="1372" alt="Image"
src="https://github.com/user-attachments/assets/f1179190-5c39-4375-99b1-f34b5009871c";
/>
#### Read await_time and RPS
<img width="2328" height="1312" alt="Image"
src="https://github.com/user-attachments/assets/905044a2-32e6-48ce-84ab-52389b75283e";
/>
---
## Rollout
* Disabled by default
* Enable selectively for benchmarking and validation
* Monitor throughput, disk utilization, IOPS, `await`, read latency, write
P99, CPU, and memory
* Validate sustained mixed read/write workloads before broad rollout
* Tune buffer size and flush concurrency based on workload and storage
characteristics
### Recommended Kernel tuning with this change
The following kernel settings can complement the DataNode optimization:
```
Read-ahead and larger block-layer requests
/sys/block/sd${disk}/queue/read_ahead_kb => >4096
/sys/block/sd${disk}/queue/max_sectors_kb => 4096
```
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### How was this patch tested?
Tested by newly added unit test
### For code changes:
- [ ] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint
declared according to the connector-specific documentation? *Note:
Automated CI
testing doesn't cover all cases so manual testing with cloud storage
is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
