[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HADOOP-11772:
Labels: (was: BB2015-05-RFC)
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
Attachments: HADOOP-11772-001.patch, HADOOP-11772-002.patch,
HADOOP-11772-003.patch, HADOOP-11772-wip-001.patch,
HADOOP-11772-wip-002.patch, HADOOP-11772.004.patch, after-ipc-fix.png,
cached-connections.png, cached-locking.png, dfs-sync-ipc.png,
sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
!dfs-sync-ipc.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HADOOP-11772:
Resolution: Fixed
Fix Version/s: 2.8.0
Release Note: The Client#call() methods that are deprecated since 0.23
have been removed.
Hadoop Flags: Incompatible change,Reviewed
Status: Resolved (was: Patch Available)
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Haohui Mai
Fix For: 2.8.0
Attachments: HADOOP-11772-001.patch, HADOOP-11772-002.patch,
HADOOP-11772-003.patch, HADOOP-11772-wip-001.patch,
HADOOP-11772-wip-002.patch, HADOOP-11772.004.patch, after-ipc-fix.png,
cached-connections.png, cached-locking.png, dfs-sync-ipc.png,
sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
!dfs-sync-ipc.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HADOOP-11772:
-
Attachment: cached-locking.png
cached-connections.png
The patch looks good, there's no performance degradation due to locking
!cached-locking.png!
Tried to check total held connections, which seems to also be contained with
the guava cache patch.
!cached-connections.png!
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
Labels: BB2015-05-RFC
Attachments: HADOOP-11772-001.patch, HADOOP-11772-002.patch,
HADOOP-11772-003.patch, HADOOP-11772-wip-001.patch,
HADOOP-11772-wip-002.patch, HADOOP-11772.004.patch, after-ipc-fix.png,
cached-connections.png, cached-locking.png, dfs-sync-ipc.png,
sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
!dfs-sync-ipc.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HADOOP-11772:
Attachment: HADOOP-11772.004.patch
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
Labels: BB2015-05-RFC
Attachments: HADOOP-11772-001.patch, HADOOP-11772-002.patch,
HADOOP-11772-003.patch, HADOOP-11772-wip-001.patch,
HADOOP-11772-wip-002.patch, HADOOP-11772.004.patch, after-ipc-fix.png,
dfs-sync-ipc.png, sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
!dfs-sync-ipc.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray Chiang updated HADOOP-11772:
Labels: BB2015-05-RFC (was: BB2015-05-TBR)
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
Labels: BB2015-05-RFC
Attachments: HADOOP-11772-001.patch, HADOOP-11772-002.patch,
HADOOP-11772-003.patch, HADOOP-11772-wip-001.patch,
HADOOP-11772-wip-002.patch, after-ipc-fix.png, dfs-sync-ipc.png,
sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
!dfs-sync-ipc.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-11772:
--
Labels: BB2015-05-TBR (was: )
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
Labels: BB2015-05-TBR
Attachments: HADOOP-11772-001.patch, HADOOP-11772-002.patch,
HADOOP-11772-003.patch, HADOOP-11772-wip-001.patch,
HADOOP-11772-wip-002.patch, after-ipc-fix.png, dfs-sync-ipc.png,
sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
!dfs-sync-ipc.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-11772:
---
Attachment: HADOOP-11772-003.patch
v3 patch removes an unused {{@VisibleForTesting}} method.
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
Attachments: HADOOP-11772-001.patch, HADOOP-11772-002.patch,
HADOOP-11772-003.patch, HADOOP-11772-wip-001.patch,
HADOOP-11772-wip-002.patch, after-ipc-fix.png, dfs-sync-ipc.png,
sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
!dfs-sync-ipc.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-11772:
---
Attachment: HADOOP-11772-002.patch
Thanks [~gopalv] for answering the questions! Good to hear that.
Attaching v2 patch.
* Added test cases
* Changed the default value of the new parameter to 1.
* Refactored {{ClientCache#getClientWithLeastReferences}}
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
Attachments: HADOOP-11772-001.patch, HADOOP-11772-002.patch,
HADOOP-11772-wip-001.patch, HADOOP-11772-wip-002.patch, after-ipc-fix.png,
dfs-sync-ipc.png, sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
!dfs-sync-ipc.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HADOOP-11772:
---
Target Version/s: 2.7.1 (was: 2.8.0)
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
Attachments: HADOOP-11772-001.patch, HADOOP-11772-wip-001.patch,
HADOOP-11772-wip-002.patch, after-ipc-fix.png, dfs-sync-ipc.png,
sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
!dfs-sync-ipc.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HADOOP-11772:
-
Attachment: after-ipc-fix.png
[~ajisakaa]: looks much better with the patch.
!after-ipc-fix.png!
I still see the occasional blocked getConnection(), but that's because I'm
running 24 threads in parallel with 10 IPC Client instances.
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
Attachments: HADOOP-11772-001.patch, HADOOP-11772-wip-001.patch,
HADOOP-11772-wip-002.patch, after-ipc-fix.png, dfs-sync-ipc.png,
sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
!dfs-sync-ipc.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-11772:
---
Attachment: HADOOP-11772-wip-002.patch
Fixed the test failure.
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
Attachments: HADOOP-11772-001.patch, HADOOP-11772-wip-001.patch,
HADOOP-11772-wip-002.patch, dfs-sync-ipc.png, sync-client-bt.png,
sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
!dfs-sync-ipc.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-11772:
---
Attachment: HADOOP-11772-wip-001.patch
Thanks [~gopalv] for your comment. Attaching a sample patch to create pool for
Clients.
TODO:
* Add a document for the new parameter
* Create a test
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
Attachments: HADOOP-11772-001.patch, HADOOP-11772-wip-001.patch,
dfs-sync-ipc.png, sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
!dfs-sync-ipc.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HADOOP-11772:
-
Attachment: dfs-sync-ipc.png
[~ajisakaa]: Does this patch fix the requirement of needing 1 IPC client per
socket-factory?
From a quick read, the single factory - single IPC client mapping still
exists, so the same hash-bucket will be locked by all processes using regular
RPC invoker.
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
Attachments: HADOOP-11772-001.patch, dfs-sync-ipc.png,
sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HADOOP-11772:
-
Description:
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking up.
!sync-client-threads.png!
!sync-client-bt.png!
!dfs-sync-ipc.png!
was:
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking up.
!sync-client-threads.png!
!sync-client-bt.png!
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
Attachments: HADOOP-11772-001.patch, dfs-sync-ipc.png,
sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
!dfs-sync-ipc.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-11772:
---
Component/s: performance
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
Attachments: HADOOP-11772-001.patch, sync-client-bt.png,
sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-11772:
---
Target Version/s: 2.8.0
Status: Patch Available (was: Open)
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Gopal V
Assignee: Akira AJISAKA
Attachments: HADOOP-11772-001.patch, sync-client-bt.png,
sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-11772:
---
Attachment: HADOOP-11772-001.patch
Thanks [~gopalv] for the report. Attaching a patch to
* Use ConcurrentHashMap for caching clients
* Remove unnecessarily synchronization
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Gopal V
Assignee: Akira AJISAKA
Attachments: HADOOP-11772-001.patch, sync-client-bt.png,
sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
[
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HADOOP-11772:
-
Attachment: sync-client-threads.png
sync-client-bt.png
RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Gopal V
Attachments: sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking
up.
!sync-client-threads.png!
!sync-client-bt.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
