[jira] [Commented] (HADOOP-10203) Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
[
https://issues.apache.org/jira/browse/HADOOP-10203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13862284#comment-13862284
]
Steve Loughran commented on HADOOP-10203:
-
why the move from the S3 classes to the Storage ones? I don't see it compiling
without imports and some other changes -but I'll wait to see what Jenkins says
there
Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
Key: HADOOP-10203
URL: https://issues.apache.org/jira/browse/HADOOP-10203
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Environment: CDH 2.0.0-cdh4.5.0
(30821ec616ee7a21ee8447949b7c6208a8f1e7d8)
Reporter: Andrei Savu
Assignee: Andrei Savu
Attachments: HADOOP-10203-trunk.patch, HADOOP-10203.patch
Jets3tNativeFileSystemStore#retrieveMetadata is leaking connections.
This affects any client that tries to read many small files very quickly
(e.g. distcp from s3 to hdfs with small files blocks due to connection pool
starvation).
This is not a problem for larger files because when the GC runs any
connection that's out of scope will be released in #finalize().
We are seeing the following log messages as a symptom of this problem:
{noformat}
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Attempting to
release HttpMethod in finalize() as its response data stream has gone out of
scope. This attempt will not always succeed and cannot be relied upon! Please
ensure response data streams are always fully consumed or closed to avoid
HTTP connection starvation.
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Successfully
released HttpMethod in finalize(). You were lucky this time... Please ensure
response data streams are always fully consumed or closed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HADOOP-10203) Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
[
https://issues.apache.org/jira/browse/HADOOP-10203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-10203:
Status: Patch Available (was: Open)
Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
Key: HADOOP-10203
URL: https://issues.apache.org/jira/browse/HADOOP-10203
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Environment: CDH 2.0.0-cdh4.5.0
(30821ec616ee7a21ee8447949b7c6208a8f1e7d8)
Reporter: Andrei Savu
Assignee: Andrei Savu
Attachments: HADOOP-10203-trunk.patch, HADOOP-10203.patch
Jets3tNativeFileSystemStore#retrieveMetadata is leaking connections.
This affects any client that tries to read many small files very quickly
(e.g. distcp from s3 to hdfs with small files blocks due to connection pool
starvation).
This is not a problem for larger files because when the GC runs any
connection that's out of scope will be released in #finalize().
We are seeing the following log messages as a symptom of this problem:
{noformat}
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Attempting to
release HttpMethod in finalize() as its response data stream has gone out of
scope. This attempt will not always succeed and cannot be relied upon! Please
ensure response data streams are always fully consumed or closed to avoid
HTTP connection starvation.
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Successfully
released HttpMethod in finalize(). You were lucky this time... Please ensure
response data streams are always fully consumed or closed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HADOOP-10204) ThrottledInputStream should #close() the underlying stream
[ https://issues.apache.org/jira/browse/HADOOP-10204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-10204: Status: Patch Available (was: Open) ThrottledInputStream should #close() the underlying stream --- Key: HADOOP-10204 URL: https://issues.apache.org/jira/browse/HADOOP-10204 Project: Hadoop Common Issue Type: Bug Components: fs Environment: CDH 2.0.0-cdh4.5.0 (30821ec616ee7a21ee8447949b7c6208a8f1e7d8) Reporter: Andrei Savu Assignee: Andrei Savu Attachments: HADOOP-10204-trunk.patch, HADOOP-10204.patch While working on HADOOP-10203 I've noticed that ThrottledInputStream (DistCP V2) does not override #close(). This can also leak connection. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HADOOP-10204) ThrottledInputStream should #close() the underlying stream
[ https://issues.apache.org/jira/browse/HADOOP-10204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-10204: Priority: Minor (was: Major) ThrottledInputStream should #close() the underlying stream --- Key: HADOOP-10204 URL: https://issues.apache.org/jira/browse/HADOOP-10204 Project: Hadoop Common Issue Type: Bug Components: fs Environment: CDH 2.0.0-cdh4.5.0 (30821ec616ee7a21ee8447949b7c6208a8f1e7d8) Reporter: Andrei Savu Assignee: Andrei Savu Priority: Minor Attachments: HADOOP-10204-trunk.patch, HADOOP-10204.patch While working on HADOOP-10203 I've noticed that ThrottledInputStream (DistCP V2) does not override #close(). This can also leak connection. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10204) ThrottledInputStream should #close() the underlying stream
[
https://issues.apache.org/jira/browse/HADOOP-10204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13862290#comment-13862290
]
Steve Loughran commented on HADOOP-10204:
-
looking at the code, probably good idea to tag the constructor arg as
{{@Nonnull}} and have the ctor raise the NPE if it is null, avoids having a
check on the close(). That'll let you add a test too, to keep jenkins happy
ThrottledInputStream should #close() the underlying stream
---
Key: HADOOP-10204
URL: https://issues.apache.org/jira/browse/HADOOP-10204
Project: Hadoop Common
Issue Type: Bug
Components: fs
Environment: CDH 2.0.0-cdh4.5.0
(30821ec616ee7a21ee8447949b7c6208a8f1e7d8)
Reporter: Andrei Savu
Assignee: Andrei Savu
Attachments: HADOOP-10204-trunk.patch, HADOOP-10204.patch
While working on HADOOP-10203 I've noticed that ThrottledInputStream (DistCP
V2) does not override #close(). This can also leak connection.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10204) ThrottledInputStream should #close() the underlying stream
[
https://issues.apache.org/jira/browse/HADOOP-10204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13862293#comment-13862293
]
Hadoop QA commented on HADOOP-10204:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12621438/HADOOP-10204-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3398//console
This message is automatically generated.
ThrottledInputStream should #close() the underlying stream
---
Key: HADOOP-10204
URL: https://issues.apache.org/jira/browse/HADOOP-10204
Project: Hadoop Common
Issue Type: Bug
Components: fs
Environment: CDH 2.0.0-cdh4.5.0
(30821ec616ee7a21ee8447949b7c6208a8f1e7d8)
Reporter: Andrei Savu
Assignee: Andrei Savu
Priority: Minor
Attachments: HADOOP-10204-trunk.patch, HADOOP-10204.patch
While working on HADOOP-10203 I've noticed that ThrottledInputStream (DistCP
V2) does not override #close(). This can also leak connection.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10203) Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
[
https://issues.apache.org/jira/browse/HADOOP-10203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13862295#comment-13862295
]
Hadoop QA commented on HADOOP-10203:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12621436/HADOOP-10203-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3399//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3399//console
This message is automatically generated.
Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
Key: HADOOP-10203
URL: https://issues.apache.org/jira/browse/HADOOP-10203
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Environment: CDH 2.0.0-cdh4.5.0
(30821ec616ee7a21ee8447949b7c6208a8f1e7d8)
Reporter: Andrei Savu
Assignee: Andrei Savu
Attachments: HADOOP-10203-trunk.patch, HADOOP-10203.patch
Jets3tNativeFileSystemStore#retrieveMetadata is leaking connections.
This affects any client that tries to read many small files very quickly
(e.g. distcp from s3 to hdfs with small files blocks due to connection pool
starvation).
This is not a problem for larger files because when the GC runs any
connection that's out of scope will be released in #finalize().
We are seeing the following log messages as a symptom of this problem:
{noformat}
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Attempting to
release HttpMethod in finalize() as its response data stream has gone out of
scope. This attempt will not always succeed and cannot be relied upon! Please
ensure response data streams are always fully consumed or closed to avoid
HTTP connection starvation.
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Successfully
released HttpMethod in finalize(). You were lucky this time... Please ensure
response data streams are always fully consumed or closed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-9992) Modify the NN loadGenerator to optionally run as a MapReduce job
[ https://issues.apache.org/jira/browse/HADOOP-9992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13862502#comment-13862502 ] Sanjay Radia commented on HADOOP-9992: -- Looks good. Update the javadoc to reflect the MR option. Also you have left some debugging printfs in the code. Modify the NN loadGenerator to optionally run as a MapReduce job Key: HADOOP-9992 URL: https://issues.apache.org/jira/browse/HADOOP-9992 Project: Hadoop Common Issue Type: Bug Reporter: Akshay Radia Assignee: Akshay Radia Attachments: hadoop-9992.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-9992) Modify the NN loadGenerator to optionally run as a MapReduce job
[ https://issues.apache.org/jira/browse/HADOOP-9992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13862506#comment-13862506 ] Sanjay Radia commented on HADOOP-9992: -- Update MapredTestDriver to include the loadGenerator for convenience. Modify the NN loadGenerator to optionally run as a MapReduce job Key: HADOOP-9992 URL: https://issues.apache.org/jira/browse/HADOOP-9992 Project: Hadoop Common Issue Type: Bug Reporter: Akshay Radia Assignee: Akshay Radia Attachments: hadoop-9992.patch -- This message was sent by Atlassian JIRA (v6.1.5#6160)
