[
https://issues.apache.org/jira/browse/HDFS-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17363696#comment-17363696
]
Stephen O'Donnell commented on HDFS-15150:
------------------------------------------
[~ahussein] Thanks for backporting this to 2.10. I have a couple of comments:
{code}
---
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
+++
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
@@ -3614,7 +3614,7 @@ void incrUserConnections(String user) {
if (count == null) {
count = 1;
} else {
- count++;
+ count = count + 1;
}
userToConnectionsMap.put(user, count);
}
@@ -3625,9 +3625,8 @@ void decrUserConnections(String user) {
Integer count = userToConnectionsMap.get(user);
if (count == null) {
return;
- } else {
- count--;
}
+ count = count - 1;
if (count == 0) {
userToConnectionsMap.remove(user);
} else {
{code}
Are the above changes intentional? They seem to be unrelated to this Jira?
{code}
@@ -1045,7 +1056,7 @@ public ReplicaInfo moveBlockAcrossStorage(ExtendedBlock
block,
newReplicaInfo.setNumBytes(blockFiles[1].length());
// Finalize the copied files
newReplicaInfo = finalizeReplica(block.getBlockPoolId(), newReplicaInfo);
- try(AutoCloseableLock lock = datasetLock.acquire()) {
+ try(AutoCloseableLock lock = datasetReadLock.acquire()) {
{code}
I think this should be datasetWriteLock.acquire()? This first patch just makes
everything use the Write lock, so we have added the RW lock, but never use the
Read lock. Then HDFS-15160 switches some write locks to read locks.
> Introduce read write lock to Datanode
> -------------------------------------
>
> Key: HDFS-15150
> URL: https://issues.apache.org/jira/browse/HDFS-15150
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 3.3.0
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-15150-branch-2.10.001.patch,
> HDFS-15150-branch-2.10.002.patch, HDFS-15150.001.patch, HDFS-15150.002.patch,
> HDFS-15150.003.patch
>
>
> HDFS-9668 pointed out the issues around the DN lock being a point of
> contention some time ago, but that Jira went in a direction of creating a new
> FSDataset implementation which is very risky, and activity on the Jira has
> stalled for a few years now. Edit: Looks like HDFS-9668 eventually went in a
> similar direction to what I was thinking, so I will review that Jira in more
> detail to see if this one is necessary.
> I feel there could be significant gains by moving to a ReentrantReadWrite
> lock within the DN. The current implementation is simply a ReentrantLock so
> any locker blocks all others.
> Once place I think a read lock would benefit us significantly, is when the DN
> is serving a lot of small blocks and there are jobs which perform a lot of
> reads. The start of reading any blocks right now takes the lock, but if we
> moved this to a read lock, many reads could do this at the same time.
> The first conservative step, would be to change the current lock and then
> make all accesses to it obtain the write lock. That way, we should keep the
> current behaviour and then we can selectively move some lock accesses to the
> readlock in separate Jiras.
> I would appreciate any thoughts on this, and also if anyone has attempted it
> before and found any blockers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
