[jira] [Commented] (HADOOP-9623) Update jets3t dependency to 0.9.0
[ https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14012940#comment-14012940 ] Steve Loughran commented on HADOOP-9623: I'm just going to highlight that this patch actually appears to have caused some significant regressions in some cases that weren't tested by the default contract. Specifically seek() out of range and some other problems don't raise exceptions any more, just return a null ref that causes NPEs later. I've done fixes for this in the next HADOOP-9361, but the changes in s3n error handling are pretty significant [https://github.com/steveloughran/hadoop-trunk/blob/7e674307256b3c853d7ef91e58b74df4607e3200/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/Jets3tNativeFileSystemStore.java#L423] Update jets3t dependency to 0.9.0 --- Key: HADOOP-9623 URL: https://issues.apache.org/jira/browse/HADOOP-9623 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 3.0.0, 2.1.0-beta, 2.2.0 Reporter: Timothy St. Clair Assignee: Amandeep Khurana Labels: maven Fix For: 2.3.0 Attachments: HADOOP-9623.patch, HADOOP-9623.patch, HADOOP-9623_2.patch, HADOOP-9623_3.patch Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0 enables mvn-rpmbuild to build against system dependencies. http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9623) Update jets3t dependency to 0.9.0
[ https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814685#comment-13814685 ] Colin Patrick McCabe commented on HADOOP-9623: -- +1, thanks Amandeep Update jets3t dependency to 0.9.0 --- Key: HADOOP-9623 URL: https://issues.apache.org/jira/browse/HADOOP-9623 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 3.0.0, 2.1.0-beta, 2.2.0 Reporter: Timothy St. Clair Assignee: Amandeep Khurana Labels: maven Attachments: HADOOP-9623.patch, HADOOP-9623.patch, HADOOP-9623_2.patch, HADOOP-9623_3.patch Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0 enables mvn-rpmbuild to build against system dependencies. http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9623) Update jets3t dependency to 0.9.0
[
https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13810860#comment-13810860
]
Colin Patrick McCabe commented on HADOOP-9623:
--
Looks straightforward in general.
{code}
+} catch (ServiceException e) {
+ return null;
}
}
@@ -194,6 +197,8 @@ private InputStream get(String key, long byteRangeStart)
throws IOException {
throw (IOException) e.getCause();
}
throw new S3Exception(e);
+} catch (ServiceException e) {
+ return null;
}
}
{code}
Why are these exceptions ignored? Should they be logged? Or could we at least
check the exception text before doing this?
Update jets3t dependency to 0.9.0
---
Key: HADOOP-9623
URL: https://issues.apache.org/jira/browse/HADOOP-9623
Project: Hadoop Common
Issue Type: Improvement
Components: fs/s3
Affects Versions: 3.0.0, 2.1.0-beta, 2.2.0
Reporter: Timothy St. Clair
Assignee: Amandeep Khurana
Labels: maven
Attachments: HADOOP-9623.patch, HADOOP-9623.patch, HADOOP-9623_2.patch
Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0
enables mvn-rpmbuild to build against system dependencies.
http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9623) Update jets3t dependency to 0.9.0
[ https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13810863#comment-13810863 ] Amandeep Khurana commented on HADOOP-9623: -- The hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/Jets3tFileSystemStore.java file, where you saw those is a piece of the code that's not recommended or used. It's for the S3 interface, not S3N, and there have been conversations about deprecating it (no jira filed yet afaik though). I didn't pay much attention to that code base. Just made sure that no API conflict happened and that it compiled. Update jets3t dependency to 0.9.0 --- Key: HADOOP-9623 URL: https://issues.apache.org/jira/browse/HADOOP-9623 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 3.0.0, 2.1.0-beta, 2.2.0 Reporter: Timothy St. Clair Assignee: Amandeep Khurana Labels: maven Attachments: HADOOP-9623.patch, HADOOP-9623.patch, HADOOP-9623_2.patch Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0 enables mvn-rpmbuild to build against system dependencies. http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9623) Update jets3t dependency to 0.9.0
[ https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13810877#comment-13810877 ] Amandeep Khurana commented on HADOOP-9623: -- Thinking about this more. I'll log the exceptions and do some smoke tests on the s3 connector too. Already done that for the S3N connector. Creating jira for deprecating the S3 connector in a future release. Update jets3t dependency to 0.9.0 --- Key: HADOOP-9623 URL: https://issues.apache.org/jira/browse/HADOOP-9623 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 3.0.0, 2.1.0-beta, 2.2.0 Reporter: Timothy St. Clair Assignee: Amandeep Khurana Labels: maven Attachments: HADOOP-9623.patch, HADOOP-9623.patch, HADOOP-9623_2.patch Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0 enables mvn-rpmbuild to build against system dependencies. http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9623) Update jets3t dependency to 0.9.0
[
https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13810882#comment-13810882
]
Colin Patrick McCabe commented on HADOOP-9623:
--
If there are no alternate suggestions, I think it would be preferable to log
these exceptions in {{Jets3tFileSystemStore.java}}. We can always change it
later if it becomes too verbose.
Let's also make sure that we pay some attention to the (non-S3N) S3 code base
as well. It may be removed in the future, but it hasn't been yet.
Update jets3t dependency to 0.9.0
---
Key: HADOOP-9623
URL: https://issues.apache.org/jira/browse/HADOOP-9623
Project: Hadoop Common
Issue Type: Improvement
Components: fs/s3
Affects Versions: 3.0.0, 2.1.0-beta, 2.2.0
Reporter: Timothy St. Clair
Assignee: Amandeep Khurana
Labels: maven
Attachments: HADOOP-9623.patch, HADOOP-9623.patch, HADOOP-9623_2.patch
Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0
enables mvn-rpmbuild to build against system dependencies.
http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9623) Update jets3t dependency
[
https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13793956#comment-13793956
]
Hadoop QA commented on HADOOP-9623:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608242/HADOOP-9623_2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3211//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3211//console
This message is automatically generated.
Update jets3t dependency
Key: HADOOP-9623
URL: https://issues.apache.org/jira/browse/HADOOP-9623
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Timothy St. Clair
Labels: maven
Attachments: HADOOP-9623_2.patch, HADOOP-9623.patch, HADOOP-9623.patch
Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0
enables mvn-rpmbuild to build against system dependencies.
http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9623) Update jets3t dependency
[ https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13793276#comment-13793276 ] Amandeep Khurana commented on HADOOP-9623: -- I did the following testing: 1. distcp files of 1G, 5G and 10G a) hdfs - s3 b) s3 - hdfs 2. Teragen to create a 5G file in s3 directly These were all done from a single node hadoop deployment. Update jets3t dependency Key: HADOOP-9623 URL: https://issues.apache.org/jira/browse/HADOOP-9623 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.0.0, 2.1.0-beta Reporter: Timothy St. Clair Labels: maven Attachments: HADOOP-9623.patch, HADOOP-9623.patch Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0 enables mvn-rpmbuild to build against system dependencies. http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9623) Update jets3t dependency
[
https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13793277#comment-13793277
]
Hadoop QA commented on HADOOP-9623:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608135/HADOOP-9623.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 1532 javac
compiler warnings (more than the trunk's current 1525 warnings).
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3210//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3210//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3210//console
This message is automatically generated.
Update jets3t dependency
Key: HADOOP-9623
URL: https://issues.apache.org/jira/browse/HADOOP-9623
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Timothy St. Clair
Labels: maven
Attachments: HADOOP-9623.patch, HADOOP-9623.patch
Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0
enables mvn-rpmbuild to build against system dependencies.
http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9623) Update jets3t dependency
[ https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701069#comment-13701069 ] Timothy St. Clair commented on HADOOP-9623: --- Canceling in favor of: https://issues.apache.org/jira/browse/HADOOP-9680 Update jets3t dependency Key: HADOOP-9623 URL: https://issues.apache.org/jira/browse/HADOOP-9623 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.0.0, 2.1.0-beta Reporter: Timothy St. Clair Labels: maven Attachments: HADOOP-9623.patch Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0 enables mvn-rpmbuild to build against system dependencies. http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9623) Update jets3t dependency
[ https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682321#comment-13682321 ] Tom White commented on HADOOP-9623: --- Overall it looks fine. Do Jets3tNativeS3FileSystemContractTest and Jets3tS3FileSystemContractTest pass with this change? They are live tests run against the real S3 service and are not run by default. You'll need to put S3 credentials into hadoop-common-project/hadoop-common/src/test/resources/core-site.xml to run them. Nit: catch blocks should be on the same line as the preceding brace. Update jets3t dependency Key: HADOOP-9623 URL: https://issues.apache.org/jira/browse/HADOOP-9623 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.0.0, 2.1.0-beta Reporter: Timothy St. Clair Labels: maven Attachments: HADOOP-9623.patch Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0 enables mvn-rpmbuild to build against system dependencies. http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9623) Update jets3t dependency
[ https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682722#comment-13682722 ] Timothy St. Clair commented on HADOOP-9623: --- Hi Tom - Thanks for the feedback, I will try to scrounge up some time to do the tests and report back the results. Cheers, Tim Update jets3t dependency Key: HADOOP-9623 URL: https://issues.apache.org/jira/browse/HADOOP-9623 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.0.0, 2.1.0-beta Reporter: Timothy St. Clair Labels: maven Attachments: HADOOP-9623.patch Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0 enables mvn-rpmbuild to build against system dependencies. http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9623) Update jets3t dependency
[ https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679939#comment-13679939 ] Timothy St. Clair commented on HADOOP-9623: --- Hi [~ste...@apache.org], do you know of anyone else who could review? Update jets3t dependency Key: HADOOP-9623 URL: https://issues.apache.org/jira/browse/HADOOP-9623 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.0.0, 2.1.0-beta Reporter: Timothy St. Clair Labels: maven Attachments: HADOOP-9623.patch Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0 enables mvn-rpmbuild to build against system dependencies. http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9623) Update jets3t dependency
[ https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677035#comment-13677035 ] Timothy St. Clair commented on HADOOP-9623: --- This patch is ready for review. Recommend taking latest(0.9.0) at this point, as 2.X stabilizes. Update jets3t dependency Key: HADOOP-9623 URL: https://issues.apache.org/jira/browse/HADOOP-9623 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.0.0, 2.1.0-beta Reporter: Timothy St. Clair Labels: maven Attachments: HADOOP-9623.patch Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0 enables mvn-rpmbuild to build against system dependencies. http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9623) Update jets3t dependency
[
https://issues.apache.org/jira/browse/HADOOP-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13676394#comment-13676394
]
Hadoop QA commented on HADOOP-9623:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586387/HADOOP-9623.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 1172 javac
compiler warnings (more than the trunk's current 1156 warnings).
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2603//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2603//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2603//console
This message is automatically generated.
Update jets3t dependency
Key: HADOOP-9623
URL: https://issues.apache.org/jira/browse/HADOOP-9623
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Timothy St. Clair
Labels: maven
Attachments: HADOOP-9623.patch
Current version referenced in pom is 0.6.1 (Aug 2008), updating to 0.9.0
enables mvn-rpmbuild to build against system dependencies.
http://jets3t.s3.amazonaws.com/RELEASE_NOTES.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
