[
https://issues.apache.org/jira/browse/HDFS-11313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966458#comment-15966458
]
Daryn Sharp commented on HDFS-11313:
------------------------------------
Very clever design, but as I hinted before, I have strong concerns/objections
to a sorted order requirement upon which this design appears predicated.
Restricting the block data structures, by design, to some form of a tree does
not scale well compared to other data structures. Ex. hashed or indexed. In
fact it completely eliminates them.
The answer may be much simpler.
Back when the ipc handlers processed the IBR and FBRs, yielding the fsn lock
was not possible, but awhile back I offloaded the BRs into a queue for
processing by a dedicated thread. This reduced fsn lock contention (1 vs
n-many waiters), and increased throughput via batching multiple BRs under the
same write lock subject to a time limit. I think this may be extended to yield
the lock during FBR processing. The serialized nature of BR processing removes
the IBR races.
There's probably just a few races to consider with systems like the decom
manager, repl monitor, etc.
> Segmented Block Reports
> -----------------------
>
> Key: HDFS-11313
> URL: https://issues.apache.org/jira/browse/HDFS-11313
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
> Affects Versions: 2.6.2
> Reporter: Konstantin Shvachko
> Assignee: Vinitha Reddy Gankidi
> Attachments: SegmentedBlockReports.pdf
>
>
> Block reports from a single DataNode can be currently split into multiple
> RPCs each reporting a single DataNode storage (disk). The reports are still
> large since disks are getting bigger. Splitting blockReport RPCs into
> multiple smaller calls would improve NameNode performance and overall HDFS
> stability.
> This was discussed in multiple jiras. Here the approach is to let NameNode
> divide blockID space into segments and then ask DataNodes to report replicas
> in a particular range of IDs.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
