[jira] [Updated] (HADOOP-10203) Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
[
https://issues.apache.org/jira/browse/HADOOP-10203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-10203:
Resolution: Fixed
Fix Version/s: 2.4.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've just committed this to trunk and branch-2.
Thanks a lot for the contribution, Andrei.
Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
Key: HADOOP-10203
URL: https://issues.apache.org/jira/browse/HADOOP-10203
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Environment: CDH 2.0.0-cdh4.5.0
(30821ec616ee7a21ee8447949b7c6208a8f1e7d8)
Reporter: Andrei Savu
Assignee: Andrei Savu
Fix For: 2.4.0
Attachments: HADOOP-10203-trunk.patch, HADOOP-10203.patch
Jets3tNativeFileSystemStore#retrieveMetadata is leaking connections.
This affects any client that tries to read many small files very quickly
(e.g. distcp from s3 to hdfs with small files blocks due to connection pool
starvation).
This is not a problem for larger files because when the GC runs any
connection that's out of scope will be released in #finalize().
We are seeing the following log messages as a symptom of this problem:
{noformat}
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Attempting to
release HttpMethod in finalize() as its response data stream has gone out of
scope. This attempt will not always succeed and cannot be relied upon! Please
ensure response data streams are always fully consumed or closed to avoid
HTTP connection starvation.
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Successfully
released HttpMethod in finalize(). You were lucky this time... Please ensure
response data streams are always fully consumed or closed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HADOOP-10203) Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
[
https://issues.apache.org/jira/browse/HADOOP-10203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-10203:
Status: Patch Available (was: Open)
Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
Key: HADOOP-10203
URL: https://issues.apache.org/jira/browse/HADOOP-10203
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Environment: CDH 2.0.0-cdh4.5.0
(30821ec616ee7a21ee8447949b7c6208a8f1e7d8)
Reporter: Andrei Savu
Assignee: Andrei Savu
Attachments: HADOOP-10203-trunk.patch, HADOOP-10203.patch
Jets3tNativeFileSystemStore#retrieveMetadata is leaking connections.
This affects any client that tries to read many small files very quickly
(e.g. distcp from s3 to hdfs with small files blocks due to connection pool
starvation).
This is not a problem for larger files because when the GC runs any
connection that's out of scope will be released in #finalize().
We are seeing the following log messages as a symptom of this problem:
{noformat}
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Attempting to
release HttpMethod in finalize() as its response data stream has gone out of
scope. This attempt will not always succeed and cannot be relied upon! Please
ensure response data streams are always fully consumed or closed to avoid
HTTP connection starvation.
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Successfully
released HttpMethod in finalize(). You were lucky this time... Please ensure
response data streams are always fully consumed or closed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HADOOP-10203) Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
[
https://issues.apache.org/jira/browse/HADOOP-10203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrei Savu updated HADOOP-10203:
-
Attachment: HADOOP-10203.patch
The attached patch fixes the problem by adding a finalize block. I've also
changed the implementation to use a HEAD request rather than attempt an
expensive full file download.
Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
Key: HADOOP-10203
URL: https://issues.apache.org/jira/browse/HADOOP-10203
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Environment: CDH 2.0.0-cdh4.5.0
(30821ec616ee7a21ee8447949b7c6208a8f1e7d8)
Reporter: Andrei Savu
Attachments: HADOOP-10203.patch
Jets3tNativeFileSystemStore#retrieveMetadata is leaking connections.
This affects any client that tries to read many small files very quickly
(e.g. distcp from s3 to hdfs with small files blocks due to connection pool
starvation).
This is not a problem for larger files because when the GC runs any
connection that's out of scope will be released in #finalize().
We are seeing the following log messages as a symptom of this problem:
{noformat}
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Attempting to
release HttpMethod in finalize() as its response data stream has gone out of
scope. This attempt will not always succeed and cannot be relied upon! Please
ensure response data streams are always fully consumed or closed to avoid
HTTP connection starvation.
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Successfully
released HttpMethod in finalize(). You were lucky this time... Please ensure
response data streams are always fully consumed or closed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HADOOP-10203) Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
[
https://issues.apache.org/jira/browse/HADOOP-10203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrei Savu updated HADOOP-10203:
-
Attachment: HADOOP-10203-trunk.patch
Attached an updated version of the patch for the current Hadoop trunk.
Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
Key: HADOOP-10203
URL: https://issues.apache.org/jira/browse/HADOOP-10203
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Environment: CDH 2.0.0-cdh4.5.0
(30821ec616ee7a21ee8447949b7c6208a8f1e7d8)
Reporter: Andrei Savu
Assignee: Andrei Savu
Attachments: HADOOP-10203-trunk.patch, HADOOP-10203.patch
Jets3tNativeFileSystemStore#retrieveMetadata is leaking connections.
This affects any client that tries to read many small files very quickly
(e.g. distcp from s3 to hdfs with small files blocks due to connection pool
starvation).
This is not a problem for larger files because when the GC runs any
connection that's out of scope will be released in #finalize().
We are seeing the following log messages as a symptom of this problem:
{noformat}
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Attempting to
release HttpMethod in finalize() as its response data stream has gone out of
scope. This attempt will not always succeed and cannot be relied upon! Please
ensure response data streams are always fully consumed or closed to avoid
HTTP connection starvation.
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Successfully
released HttpMethod in finalize(). You were lucky this time... Please ensure
response data streams are always fully consumed or closed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
