[ https://issues.apache.org/jira/browse/HADOOP-10203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HADOOP-10203: ------------------------------------ Resolution: Fixed Fix Version/s: 2.4.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Andrei. > Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata > ---------------------------------------------------------------- > > Key: HADOOP-10203 > URL: https://issues.apache.org/jira/browse/HADOOP-10203 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 > Environment: CDH 2.0.0-cdh4.5.0 > (30821ec616ee7a21ee8447949b7c6208a8f1e7d8) > Reporter: Andrei Savu > Assignee: Andrei Savu > Fix For: 2.4.0 > > Attachments: HADOOP-10203-trunk.patch, HADOOP-10203.patch > > > Jets3tNativeFileSystemStore#retrieveMetadata is leaking connections. > This affects any client that tries to read many small files very quickly > (e.g. distcp from s3 to hdfs with small files blocks due to connection pool > starvation). > This is not a problem for larger files because when the GC runs any > connection that's out of scope will be released in #finalize(). > We are seeing the following log messages as a symptom of this problem: > {noformat} > 13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Attempting to > release HttpMethod in finalize() as its response data stream has gone out of > scope. This attempt will not always succeed and cannot be relied upon! Please > ensure response data streams are always fully consumed or closed to avoid > HTTP connection starvation. > 13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Successfully > released HttpMethod in finalize(). You were lucky this time... Please ensure > response data streams are always fully consumed or closed. > {noformat} -- This message was sent by Atlassian JIRA (v6.1.5#6160)