[
https://issues.apache.org/jira/browse/HADOOP-10203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13863251#comment-13863251
]
Andrei Savu commented on HADOOP-10203:
--------------------------------------
[[email protected]] I've moved the StorageObject because I've switched to
s3Service.getObjectDetails. This method will perform a HEAD request rather than
attempt a full file download.
> Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
> ----------------------------------------------------------------
>
> Key: HADOOP-10203
> URL: https://issues.apache.org/jira/browse/HADOOP-10203
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Environment: CDH 2.0.0-cdh4.5.0
> (30821ec616ee7a21ee8447949b7c6208a8f1e7d8)
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Attachments: HADOOP-10203-trunk.patch, HADOOP-10203.patch
>
>
> Jets3tNativeFileSystemStore#retrieveMetadata is leaking connections.
> This affects any client that tries to read many small files very quickly
> (e.g. distcp from s3 to hdfs with small files blocks due to connection pool
> starvation).
> This is not a problem for larger files because when the GC runs any
> connection that's out of scope will be released in #finalize().
> We are seeing the following log messages as a symptom of this problem:
> {noformat}
> 13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Attempting to
> release HttpMethod in finalize() as its response data stream has gone out of
> scope. This attempt will not always succeed and cannot be relied upon! Please
> ensure response data streams are always fully consumed or closed to avoid
> HTTP connection starvation.
> 13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Successfully
> released HttpMethod in finalize(). You were lucky this time... Please ensure
> response data streams are always fully consumed or closed.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
