smengcl opened a new pull request, #11617:
URL: https://github.com/apache/ozone/pull/11617
<!-- claude-projects-attribution: {"github_login":"smengcl"} -->
Generated-by: Claude Code (Opus 5.5)
## What changes were proposed in this pull request?
When a client runs on a datanode host (for example an HBase RegionServer
colocated with a datanode), SCM picks a Ratis pipeline for a new block without
looking at where the client is, so often no replica is local to the writer.
HDFS writes the first replica on the writer's datanode, which later lets reads
of that data (such as HBase compactions) use short circuit reads.
This change adds `hdds.scm.block.allocation.prefer.client.local` (default
`false`). When it is set:
- OM passes the client address to SCM on every block allocation, even when
OM sorts the datanodes itself.
- SCM first tries open Ratis pipelines led by the client's datanode, then
other open pipelines that contain it, and falls back to the pipeline choose
policy when none of those has a usable container. A pipeline led by the
client's datanode also keeps the client to leader hop of each write on the same
host (see HDDS-11000).
EC allocation and the default behavior are unchanged.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17113
## How was this patch tested?
- New unit tests in `TestWritableRatisContainerProvider` (prefers a pipeline
led by the client, then one containing it, and falls back when the local
pipeline has no container or the client is in no pipeline) and in
`TestOMAllocateBlockRequest` (OM sends the client address to SCM when the flag
is set).
- Existing `TestSCMBlockProtocolServer`, `TestBlockManager`,
`TestOMKeyCreateRequest` and `TestOMFileCreateRequest` pass.
- On a 6 datanode docker compose cluster with the flag set, each writer
running on a datanode that leads a pipeline got blocks on that pipeline. hsync
latency differences between local leader, local follower and remote writers
were within noise there, since all containers share one host. A real network
A/B is still needed to measure the latency effect.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
