[ http://issues.apache.org/jira/browse/HADOOP-312?page=comments#action_12418508 ]
Devaraj Das commented on HADOOP-312: ------------------------------------ The way I have designed this is to have the connection caching configurable. Given any IPC server (like the namenode or jobtracker) the caching can range from fully cached (which is the current behavior) to number of transactions per created connection. The configuration related to caching can be set in the client's hadoop config file like: <property> <name>ipc.connection.notcached.address</name> <value>server1:port=num_transactions, server2:port=num_transactions</value> <description>Defines the connections that should not be cached. </description> </property> The above means that clients close the connections to server1:port after it has done num_transactions (a transaction is defined as one request-response). Any number of such servers can be specified by a comma-separated list. Connections to servers not explicitly mentioned in the config is cached (though the server will disconnect clients that have not communicated with it for a specified period of time). Doing this will limit the number of connections cached significantly. For example, one can always specify in the config that connections to the namenode should not be cached at all (num_transactions = 1) and so on. The patch is ready but I can probably incorporate any inputs you may have before I submit that. > Connections should not be cached > -------------------------------- > > Key: HADOOP-312 > URL: http://issues.apache.org/jira/browse/HADOOP-312 > Project: Hadoop > Type: Improvement > Components: ipc > Versions: 0.4.0 > Reporter: Devaraj Das > > Servers and clients (client include datanodes, tasktrackers, DFSClients & > tasks) should not cache connections or maybe cache them for very short > periods of time. Clients should set up & tear down connections to the servers > everytime they need to contact the servers (including the heartbeats). If > connection is cached, then reuse the existing connection for a few subsequent > transactions until the connection expires. The heartbeat interval should be > more so that many more clients (order of tens of thousands) can be > accomodated within 1 heartbeat interval. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
