[jira] [Commented] (HADOOP-15552) Move logging APIs over to slf4j in hadoop-tools - Part2
[
https://issues.apache.org/jira/browse/HADOOP-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567807#comment-16567807
]
genericqa commented on HADOOP-15552:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 35 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 30m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 12m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 30m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 30m
0s{color} | {color:green} root generated 0 new + 1462 unchanged - 6 fixed =
1462 total (was 1468) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 13m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
55s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 0s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
55s{color} | {color:green} hadoop-streaming in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
18s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
16s{color} | {color:green} hadoop-archives in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
0s{color} | {color:green} hadoop-archive-logs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
42s{color} | {color:green} hadoop-rumen in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
59s{color} | {color:green} hadoop-gridmix in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
47s{color} | {color:green} hadoop-datajoin in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
[jira] [Commented] (HADOOP-15626) FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append.
[ https://issues.apache.org/jira/browse/HADOOP-15626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567776#comment-16567776 ] Aaron Fabbri commented on HADOOP-15626: --- +1 LTGM thanks for fixing this. > FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails > on filesystems without append. > -- > > Key: HADOOP-15626 > URL: https://issues.apache.org/jira/browse/HADOOP-15626 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15626-001.patch > > > After HADOOP-14396. one of the new tests fails on S3A because append() isn't > supported there -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[
https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567775#comment-16567775
]
Aaron Fabbri commented on HADOOP-14154:
---
Thanks for the v2 patch. The DDBPathMetadata stuff turned out pretty good.
Was thinking we could probably reduce garbage a bit with some additional
constructors for that class, but actually I think JVM escape analysis will
handle most of those extra allocations on the stack. The lists of metadata are
harder to deal with but I think your code looks fine here.
Some inline comments...
{noformat}
@@ -840,21 +861,36 @@ public void prune(long modTime, String keyPrefix) throws
IOException {
new ArrayList<>(S3GUARD_DDB_BATCH_WRITE_REQUEST_LIMIT);
int delay = conf.getInt(S3GUARD_DDB_BACKGROUND_SLEEP_MSEC_KEY,
S3GUARD_DDB_BACKGROUND_SLEEP_MSEC_DEFAULT);
+ Set parentPathSet = new HashSet<>();
for (Item item : expiredFiles(modTime, keyPrefix)) {
-PathMetadata md = PathMetadataDynamoDBTranslation
+DDBPathMetadata md = PathMetadataDynamoDBTranslation
.itemToPathMetadata(item, username);
Path path = md.getFileStatus().getPath();
deletionBatch.add(path);
+
+// add parent path of what we remove
+Path parentPath = path.getParent();
+parentPathSet.add(parentPath);
{noformat}
What if parentPath is root dir? I think we want a null check here.
{noformat}
+ private void removeAuthoritativeDirFlag(Set pathSet) {
+Set metas = pathSet.stream().map(path -> {
+ try {
+DDBPathMetadata ddbPathMetadata = get(path);
+if(ddbPathMetadata == null) {
+ return null;
+}
+LOG.debug("Setting false isAuthoritativeDir on {}", ddbPathMetadata);
+ddbPathMetadata.setAuthoritativeDir(false);
+return ddbPathMetadata;
+ } catch (IOException e) {
+String msg = String.format("IOException while getting PathMetadata "
++ "on path: %s.", path);
+LOG.error(msg, e);
+return null;
+ }
+}).filter(Objects::nonNull).collect(Collectors.toSet());
{noformat}
I like that the stream keeps running if one of the paths fail, but should we
also save a reference to the IOException and then throw it at the end of the
function? That way we keep working even if we get a failure, but the failure
still gets propagated to the caller.
{noformat}
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStoreTestBase.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStoreTestBase.java
@@ -727,6 +727,13 @@ public void testPruneUnsetsAuthoritative() throws
Exception {
new FileStatus(0, false, 0, 0, time + 1, strToPath(freshFile)),
Tristate.FALSE, false));
+// set parent dir as authoritative
+if (!allowMissing()) {
+ DirListingMetadata parentDirMd = ms.listChildren(strToPath(parentDir));
+ parentDirMd.setAuthoritative(true);
+ ms.put(parentDirMd);
+}
{noformat}
Looks like you found a bug in the existing test case? Nice work.
I was wondering if we want an integration test to confirm forward and backward
compatibility with the schema change (old S3a works with new schema rows and
vice versa). Not sure how we'd implement that though. Would probably need a
separate copy of a couple of the PathMetadataDynamoDBTranslation functions with
the old logic in them (or better yet, a boolean flag to select old behavior w/o
the read/write of the auth flag), and then use those in the DDB integration
test to confirm it all works. I'm not sure what this buys us in terms of
regression testing though--so I could see the argument for manual testing. What
do you think?
> Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
> -
>
> Key: HADOOP-14154
> URL: https://issues.apache.org/jira/browse/HADOOP-14154
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Rajesh Balamohan
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-14154-HADOOP-13345.001.patch,
> HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf,
> HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch
>
>
> Add support for "authoritative mode" for DynamoDBMetadataStore.
> The missing feature is to persist the bit set in
> {{DirListingMetadata.isAuthoritative}}.
> This topic has been super confusing for folks so I will also file a
> documentation Jira to explain the design better.
> We may want to also rename the DirListingMetadata.isAuthoritative field to
> .isFullListing to eliminate the multiple uses and meanings of the word
>
[jira] [Commented] (HADOOP-15107) Stabilize/tune S3A committers; review correctness & docs
[
https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567754#comment-16567754
]
genericqa commented on HADOOP-15107:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 7 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
49s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
37s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15107 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934192/HADOOP-15107-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 55a50e6cd99b 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 40ab8ee |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| whitespace |
[jira] [Commented] (HADOOP-15626) FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append.
[
https://issues.apache.org/jira/browse/HADOOP-15626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567732#comment-16567732
]
genericqa commented on HADOOP-15626:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
2s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
23s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15626 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934194/HADOOP-15626-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux f9590bc33118 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 40ab8ee |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14987/testReport/ |
| Max. process+thread count | 319 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14987/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails
> on filesystems without append.
>
[jira] [Commented] (HADOOP-15583) Stabilize S3A Assumed Role support
[
https://issues.apache.org/jira/browse/HADOOP-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567698#comment-16567698
]
genericqa commented on HADOOP-15583:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
35s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 23m
26s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 30m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 30m 2s{color}
| {color:red} root generated 184 new + 1280 unchanged - 0 fixed = 1464 total
(was 1280) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 12 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
5s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
34s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}138m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15583 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934184/HADOOP-15583-005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml findbugs checkstyle |
| uname | Linux da649f590b98 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git
[jira] [Updated] (HADOOP-15107) Stabilize/tune S3A committers; review correctness & docs
[ https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15107: Summary: Stabilize/tune S3A committers; review correctness & docs (was: Prove the correctness of the new committers, or fix where they are not correct) > Stabilize/tune S3A committers; review correctness & docs > > > Key: HADOOP-15107 > URL: https://issues.apache.org/jira/browse/HADOOP-15107 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15107-001.patch, HADOOP-15107-002.patch, > HADOOP-15107-003.patch > > > I'm writing about the paper on the committers, one which, being a proper > paper, requires me to show the committers work. > # define the requirements of a "Correct" committed job (this applies to the > FileOutputCommitter too) > # show that the Staging committer meets these requirements (most of this is > implicit in that it uses the V1 FileOutputCommitter to marshall .pendingset > lists from committed tasks to the final destination, where they are read and > committed. > # Show the magic committer also works. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15645) ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB
[ https://issues.apache.org/jira/browse/HADOOP-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15645: Priority: Blocker (was: Major) > ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding > to DDB > --- > > Key: HADOOP-15645 > URL: https://issues.apache.org/jira/browse/HADOOP-15645 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15645-001.patch > > > If your bucket has a per-bucket setting to use S3Guard, then the > ITestS3GuardToolLocal.testDiffCommand test can fail as fs-only data can creep > into the metastore. > The rawfs should use clearBucketOption to clear the metastore binding, so > guarantee a real raw fs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15626) FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append.
[
https://issues.apache.org/jira/browse/HADOOP-15626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567694#comment-16567694
]
Steve Loughran edited comment on HADOOP-15626 at 8/3/18 2:38 AM:
-
HADOOP-15626 patch 001; mark testBuilderCreateAppendExistingFile() as @ignored
for S3A
Testing: us-west1 with s3guard
{code}
Tests run: 68, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 295.271 s - in
org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
{code}
This is really urgent as its breaking all my local build/test runs
was (Author: ste...@apache.org):
HADOOP-15626 patch 001; mark testBuilderCreateAppendExistingFile() as @ignored
for S3A
Testing: us-west1 with s3guard
> FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails
> on filesystems without append.
> --
>
> Key: HADOOP-15626
> URL: https://issues.apache.org/jira/browse/HADOOP-15626
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15626-001.patch
>
>
> After HADOOP-14396. one of the new tests fails on S3A because append() isn't
> supported there
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15626) FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append.
[ https://issues.apache.org/jira/browse/HADOOP-15626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15626: Assignee: Steve Loughran Status: Patch Available (was: Open) HADOOP-15626 patch 001; mark testBuilderCreateAppendExistingFile() as @ignored for S3A Testing: us-west1 with s3guard > FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails > on filesystems without append. > -- > > Key: HADOOP-15626 > URL: https://issues.apache.org/jira/browse/HADOOP-15626 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15626-001.patch > > > After HADOOP-14396. one of the new tests fails on S3A because append() isn't > supported there -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15626) FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append.
[ https://issues.apache.org/jira/browse/HADOOP-15626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15626: Attachment: HADOOP-15626-001.patch > FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails > on filesystems without append. > -- > > Key: HADOOP-15626 > URL: https://issues.apache.org/jira/browse/HADOOP-15626 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15626-001.patch > > > After HADOOP-14396. one of the new tests fails on S3A because append() isn't > supported there -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15107) Prove the correctness of the new committers, or fix where they are not correct
[ https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15107: Status: Patch Available (was: Open) rebase/merge to trunk, final quick review. Testing US-west 1, all good apart from failures which have outstanding fixes for on separate patches > Prove the correctness of the new committers, or fix where they are not correct > -- > > Key: HADOOP-15107 > URL: https://issues.apache.org/jira/browse/HADOOP-15107 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15107-001.patch, HADOOP-15107-002.patch, > HADOOP-15107-003.patch > > > I'm writing about the paper on the committers, one which, being a proper > paper, requires me to show the committers work. > # define the requirements of a "Correct" committed job (this applies to the > FileOutputCommitter too) > # show that the Staging committer meets these requirements (most of this is > implicit in that it uses the V1 FileOutputCommitter to marshall .pendingset > lists from committed tasks to the final destination, where they are read and > committed. > # Show the magic committer also works. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15107) Prove the correctness of the new committers, or fix where they are not correct
[ https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15107: Attachment: HADOOP-15107-003.patch > Prove the correctness of the new committers, or fix where they are not correct > -- > > Key: HADOOP-15107 > URL: https://issues.apache.org/jira/browse/HADOOP-15107 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15107-001.patch, HADOOP-15107-002.patch, > HADOOP-15107-003.patch > > > I'm writing about the paper on the committers, one which, being a proper > paper, requires me to show the committers work. > # define the requirements of a "Correct" committed job (this applies to the > FileOutputCommitter too) > # show that the Staging committer meets these requirements (most of this is > implicit in that it uses the V1 FileOutputCommitter to marshall .pendingset > lists from committed tasks to the final destination, where they are read and > committed. > # Show the magic committer also works. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15107) Prove the correctness of the new committers, or fix where they are not correct
[ https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15107: Status: Open (was: Patch Available) > Prove the correctness of the new committers, or fix where they are not correct > -- > > Key: HADOOP-15107 > URL: https://issues.apache.org/jira/browse/HADOOP-15107 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15107-001.patch, HADOOP-15107-002.patch > > > I'm writing about the paper on the committers, one which, being a proper > paper, requires me to show the committers work. > # define the requirements of a "Correct" committed job (this applies to the > FileOutputCommitter too) > # show that the Staging committer meets these requirements (most of this is > implicit in that it uses the V1 FileOutputCommitter to marshall .pendingset > lists from committed tasks to the final destination, where they are read and > committed. > # Show the magic committer also works. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15107) Prove the correctness of the new committers, or fix where they are not correct
[
https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567639#comment-16567639
]
genericqa commented on HADOOP-15107:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-15107 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15107 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12914510/HADOOP-15107-002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14985/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Prove the correctness of the new committers, or fix where they are not correct
> --
>
> Key: HADOOP-15107
> URL: https://issues.apache.org/jira/browse/HADOOP-15107
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15107-001.patch, HADOOP-15107-002.patch
>
>
> I'm writing about the paper on the committers, one which, being a proper
> paper, requires me to show the committers work.
> # define the requirements of a "Correct" committed job (this applies to the
> FileOutputCommitter too)
> # show that the Staging committer meets these requirements (most of this is
> implicit in that it uses the V1 FileOutputCommitter to marshall .pendingset
> lists from committed tasks to the final destination, where they are read and
> committed.
> # Show the magic committer also works.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15470) S3A staging committers to not log FNFEs on job abort listings
[ https://issues.apache.org/jira/browse/HADOOP-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-15470. - Resolution: Duplicate > S3A staging committers to not log FNFEs on job abort listings > - > > Key: HADOOP-15470 > URL: https://issues.apache.org/jira/browse/HADOOP-15470 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Priority: Minor > > When aborting a job, the staging committers list staged files in the cluster > FS to abort...all exceptions are caught & downgraded to log events. > We shouldn't even log FNFEs except at debug level, as all it means is "the > job is aborting before things got that far. Printing the full stack simply > creates confusion about what the problem is -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15642) Update to latest/recent version of aws-sdk for Hadoop 3.2
[ https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567633#comment-16567633 ] Steve Loughran commented on HADOOP-15642: - Patch 005 of HADOOP-15583 works with this SDK as well as the current one > Update to latest/recent version of aws-sdk for Hadoop 3.2 > - > > Key: HADOOP-15642 > URL: https://issues.apache.org/jira/browse/HADOOP-15642 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15642-001.patch, HADOOP-15642-003.patch, Screen > Shot 2018-07-30 at 14.11.22.png > > > Move to a later version of the AWS SDK library for a different set of > features and issues. > proposed version: 1.11.375 > One thing which doesn't work on the one we ship with is the ability to create > assumed role sessions >1h, as there's a check in the client lib for > role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration > checks to the far end. > see: > [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/] > * assuming later versions will extend assumed role life, docs will need > changing, > * Adding a test in HADOOP-15583 which expects an error message if you ask for > a duration of 3h; this should act as the test to see what happens. > * think this time would be good to explicitly write down the SDK update > process -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15583) Stabilize S3A Assumed Role support
[
https://issues.apache.org/jira/browse/HADOOP-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15583:
Status: Open (was: Patch Available)
> Stabilize S3A Assumed Role support
> --
>
> Key: HADOOP-15583
> URL: https://issues.apache.org/jira/browse/HADOOP-15583
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15583-001.patch, HADOOP-15583-002.patch,
> HADOOP-15583-003.patch, HADOOP-15583-004.patch, HADOOP-15583-005.patch
>
>
> started off just on sharing credentials across S3A and S3Guard, but in the
> process it has grown to becoming one of stabilising the assumed role support
> so it can be used for more than just testing.
> Was: "S3Guard to get AWS Credential chain from S3AFS; credentials closed() on
> shutdown"
> h3. Issue: lack of auth chain sharing causes ddb and s3 to get out of sync
> S3Guard builds its DDB auth chain itself, which stops it having to worry
> about being created standalone vs part of an S3AFS, but it means its
> authenticators are in a separate chain.
> When you are using short-lived assumed roles or other session credentials
> updated in the S3A FS authentication chain, you need that same set of
> credentials picked up by DDB. Otherwise, at best you are doubling load, at
> worse: the DDB connector may not get refreshed credentials.
> Proposed: {{DynamoDBClientFactory.createDynamoDBClient()}} to take an
> optional ref to aws credentials. If set: don't create a new set.
> There's one little complication here: our {{AWSCredentialProviderList}} list
> is autocloseable; it's close() will go through all children and close them.
> Apparently the AWS S3 client (And hopefully the DDB client) will close this
> when they are closed themselves. If DDB has the same set of credentials as
> the FS, then there could be trouble if they are closed in one place when the
> other still wants to use them.
> Solution; have a use count the uses of the credentials list, starting at one:
> every close() call decrements, and when this hits zero the cleanup is kicked
> off
> h3. Issue: {{AssumedRoleCredentialProvider}} connector to STS not picking up
> the s3a connection settings, including proxy.
> h3. issue: we're not using getPassword() to get user/password for proxy
> binding for STS. Fix: use that and pass down the bucket ref for per-bucket
> secrets in a JCEKS file.
> h3. Issue; hard to debug what's going wrong :)
> h3. Issue: docs about KMS permissions for SSE-KMS are wrong, and the
> ITestAssumedRole* tests don't request KMS permissions, so fail in a bucket
> when the base s3 FS is using SSE-KMS. KMS permissions need to be included in
> generated profiles
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15583) Stabilize S3A Assumed Role support
[
https://issues.apache.org/jira/browse/HADOOP-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15583:
Status: Patch Available (was: Open)
* do best to shut up findbugs
* assumed role range tests compatible with AWS SDKs which hand off duration
range checks to STS itself: different errors/strings are accepted, and the 3h
timed exception is allowed to work (i.e ready for HADOOP-15642).
* added a minimal test to guarantee coverage
{{AssumedRoleCredentialProvider.operationRetried}}; There's only a log
statement when retries == 0; this test is purely for regression/completeness. I
don't like error handlers which aren't tested
Tested: US-west 1 with both old and new SDKs, bucket to require SSE-KMS and
s3guard. All well excluding known failures (/HADOOP-15645, HADOOP-14528)
> Stabilize S3A Assumed Role support
> --
>
> Key: HADOOP-15583
> URL: https://issues.apache.org/jira/browse/HADOOP-15583
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15583-001.patch, HADOOP-15583-002.patch,
> HADOOP-15583-003.patch, HADOOP-15583-004.patch, HADOOP-15583-005.patch
>
>
> started off just on sharing credentials across S3A and S3Guard, but in the
> process it has grown to becoming one of stabilising the assumed role support
> so it can be used for more than just testing.
> Was: "S3Guard to get AWS Credential chain from S3AFS; credentials closed() on
> shutdown"
> h3. Issue: lack of auth chain sharing causes ddb and s3 to get out of sync
> S3Guard builds its DDB auth chain itself, which stops it having to worry
> about being created standalone vs part of an S3AFS, but it means its
> authenticators are in a separate chain.
> When you are using short-lived assumed roles or other session credentials
> updated in the S3A FS authentication chain, you need that same set of
> credentials picked up by DDB. Otherwise, at best you are doubling load, at
> worse: the DDB connector may not get refreshed credentials.
> Proposed: {{DynamoDBClientFactory.createDynamoDBClient()}} to take an
> optional ref to aws credentials. If set: don't create a new set.
> There's one little complication here: our {{AWSCredentialProviderList}} list
> is autocloseable; it's close() will go through all children and close them.
> Apparently the AWS S3 client (And hopefully the DDB client) will close this
> when they are closed themselves. If DDB has the same set of credentials as
> the FS, then there could be trouble if they are closed in one place when the
> other still wants to use them.
> Solution; have a use count the uses of the credentials list, starting at one:
> every close() call decrements, and when this hits zero the cleanup is kicked
> off
> h3. Issue: {{AssumedRoleCredentialProvider}} connector to STS not picking up
> the s3a connection settings, including proxy.
> h3. issue: we're not using getPassword() to get user/password for proxy
> binding for STS. Fix: use that and pass down the bucket ref for per-bucket
> secrets in a JCEKS file.
> h3. Issue; hard to debug what's going wrong :)
> h3. Issue: docs about KMS permissions for SSE-KMS are wrong, and the
> ITestAssumedRole* tests don't request KMS permissions, so fail in a bucket
> when the base s3 FS is using SSE-KMS. KMS permissions need to be included in
> generated profiles
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15583) Stabilize S3A Assumed Role support
[
https://issues.apache.org/jira/browse/HADOOP-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15583:
Status: Patch Available (was: Open)
> Stabilize S3A Assumed Role support
> --
>
> Key: HADOOP-15583
> URL: https://issues.apache.org/jira/browse/HADOOP-15583
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15583-001.patch, HADOOP-15583-002.patch,
> HADOOP-15583-003.patch, HADOOP-15583-004.patch, HADOOP-15583-005.patch
>
>
> started off just on sharing credentials across S3A and S3Guard, but in the
> process it has grown to becoming one of stabilising the assumed role support
> so it can be used for more than just testing.
> Was: "S3Guard to get AWS Credential chain from S3AFS; credentials closed() on
> shutdown"
> h3. Issue: lack of auth chain sharing causes ddb and s3 to get out of sync
> S3Guard builds its DDB auth chain itself, which stops it having to worry
> about being created standalone vs part of an S3AFS, but it means its
> authenticators are in a separate chain.
> When you are using short-lived assumed roles or other session credentials
> updated in the S3A FS authentication chain, you need that same set of
> credentials picked up by DDB. Otherwise, at best you are doubling load, at
> worse: the DDB connector may not get refreshed credentials.
> Proposed: {{DynamoDBClientFactory.createDynamoDBClient()}} to take an
> optional ref to aws credentials. If set: don't create a new set.
> There's one little complication here: our {{AWSCredentialProviderList}} list
> is autocloseable; it's close() will go through all children and close them.
> Apparently the AWS S3 client (And hopefully the DDB client) will close this
> when they are closed themselves. If DDB has the same set of credentials as
> the FS, then there could be trouble if they are closed in one place when the
> other still wants to use them.
> Solution; have a use count the uses of the credentials list, starting at one:
> every close() call decrements, and when this hits zero the cleanup is kicked
> off
> h3. Issue: {{AssumedRoleCredentialProvider}} connector to STS not picking up
> the s3a connection settings, including proxy.
> h3. issue: we're not using getPassword() to get user/password for proxy
> binding for STS. Fix: use that and pass down the bucket ref for per-bucket
> secrets in a JCEKS file.
> h3. Issue; hard to debug what's going wrong :)
> h3. Issue: docs about KMS permissions for SSE-KMS are wrong, and the
> ITestAssumedRole* tests don't request KMS permissions, so fail in a bucket
> when the base s3 FS is using SSE-KMS. KMS permissions need to be included in
> generated profiles
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15625) S3A input stream to use etags to detect changed source files
[ https://issues.apache.org/jira/browse/HADOOP-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-15625: --- Assignee: Brahma Reddy Battula > S3A input stream to use etags to detect changed source files > > > Key: HADOOP-15625 > URL: https://issues.apache.org/jira/browse/HADOOP-15625 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Major > Attachments: HADOOP-15625-001.patch > > > S3A input stream doesn't handle changing source files any better than the > other cloud store connectors. Specifically: it doesn't noticed it has > changed, caches the length from startup, and whenever a seek triggers a new > GET, you may get one of: old data, new data, and even perhaps go from new > data to old data due to eventual consistency. > We can't do anything to stop this, but we could detect changes by > # caching the etag of the first HEAD/GET (we don't get that HEAD on open with > S3Guard, BTW) > # on future GET requests, verify the etag of the response > # raise an IOE if the remote file changed during the read. > It's a more dramatic failure, but it stops changes silently corrupting things. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15625) S3A input stream to use etags to detect changed source files
[ https://issues.apache.org/jira/browse/HADOOP-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15625: Parent Issue: HADOOP-15220 (was: HADOOP-15620) > S3A input stream to use etags to detect changed source files > > > Key: HADOOP-15625 > URL: https://issues.apache.org/jira/browse/HADOOP-15625 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Brahma Reddy Battula >Priority: Major > Attachments: HADOOP-15625-001.patch > > > S3A input stream doesn't handle changing source files any better than the > other cloud store connectors. Specifically: it doesn't noticed it has > changed, caches the length from startup, and whenever a seek triggers a new > GET, you may get one of: old data, new data, and even perhaps go from new > data to old data due to eventual consistency. > We can't do anything to stop this, but we could detect changes by > # caching the etag of the first HEAD/GET (we don't get that HEAD on open with > S3Guard, BTW) > # on future GET requests, verify the etag of the response > # raise an IOE if the remote file changed during the read. > It's a more dramatic failure, but it stops changes silently corrupting things. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14237: Parent Issue: HADOOP-15620 (was: HADOOP-15220) > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura >Assignee: Kazuyuki Tanimura >Priority: Major > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567625#comment-16567625 ] Steve Loughran commented on HADOOP-14237: - I'm looking at this again, going to move to 3.3 along with most of the other outstainding s3 for 3.2 features. * I don't like saving the full secrets (unencrypted) to HDFS * session secrets could work, though of course they'll expire within 24h. once HADOOP-15883 is in I'm going to revisit HADOOP-14556, which lets the s3a client to serialize its secrets as a filesystem delegation token, something apps (hive, spark, MR) know to ask for -and which YARN knows how to securely marshall to launched apps. With this feature you could launch things into a pool of VMs with reduced privilege IAM roles, sending in higher privilege credentials with the request. Would that work? I've also created HADOOP-15650 to cover the issue of better retry logic on credential retrieval. I see there's an async option, which might be more responsive, but could put even more load on the service unless managed carefully. What it could do though, is handle retries much better (though it'd also be a more more complicated) > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura >Assignee: Kazuyuki Tanimura >Priority: Major > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15650) Add custom InstanceProfileCredentialsProvider with more resilience to throttling
Steve Loughran created HADOOP-15650:
---
Summary: Add custom InstanceProfileCredentialsProvider with more
resilience to throttling
Key: HADOOP-15650
URL: https://issues.apache.org/jira/browse/HADOOP-15650
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.1.0
Reporter: Steve Loughran
Add our own InstanceProfileCredentialsProvider class which uses the AWS
implementation to retrieve credentials from EC2's instance info, but more
resilient to overloading.
# pass in client config with retry logic (HADOOP-15603)
# use Invoke.retry() to retry
# log/measure failures
# maybe use the Async feature of the AWS SDK class, so that credential renewer
doesn't block IO.
# be shared amongst all AWS auth chains which need these credentials.
The singleton we current use for IAM auth doesn't do async, which is good as it
ensures that we don't prematurely close it when
{{AWSCredentialProviderList.close()}} closes its children.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15603) S3A to support configuring various AWS S3 client extended options
[ https://issues.apache.org/jira/browse/HADOOP-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567612#comment-16567612 ] Steve Loughran commented on HADOOP-15603: - This config also need to be passed to all clients of AWS services, not just DDB but anything going near STS, even IAM credential providers, which inevitably means: new ctor for those providers which take it > S3A to support configuring various AWS S3 client extended options > - > > Key: HADOOP-15603 > URL: https://issues.apache.org/jira/browse/HADOOP-15603 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Priority: Minor > > There's a fair few options in the S3 client connection (IPv4 vs v6, use of > accelerated S3...) which we don't configure from fs.s3a options. > Add them, document them and where possible test them -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14943) Add common getFileBlockLocations() emulation for object stores, including S3A
[
https://issues.apache.org/jira/browse/HADOOP-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567606#comment-16567606
]
genericqa commented on HADOOP-14943:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-14943 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14943 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910618/HADOOP-14943-004.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14983/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add common getFileBlockLocations() emulation for object stores, including S3A
> -
>
> Key: HADOOP-14943
> URL: https://issues.apache.org/jira/browse/HADOOP-14943
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14943-001.patch, HADOOP-14943-002.patch,
> HADOOP-14943-002.patch, HADOOP-14943-003.patch, HADOOP-14943-004.patch
>
>
> It looks suspiciously like S3A isn't providing the partitioning data needed
> in {{listLocatedStatus}} and {{getFileBlockLocations()}} needed to break up a
> file by the blocksize. This will stop tools using the MRv1 APIS doing the
> partitioning properly if the input format isn't doing it own split logic.
> FileInputFormat in MRv2 is a bit more configurable about input split
> calculation & will split up large files. but otherwise, the partitioning is
> being done more by the default values of the executing engine, rather than
> any config data from the filesystem about what its "block size" is,
> NativeAzureFS does a better job; maybe that could be factored out to
> hadoop-common and reused?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14943) Add common getFileBlockLocations() emulation for object stores, including S3A
[
https://issues.apache.org/jira/browse/HADOOP-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14943:
Parent Issue: HADOOP-15620 (was: HADOOP-15220)
> Add common getFileBlockLocations() emulation for object stores, including S3A
> -
>
> Key: HADOOP-14943
> URL: https://issues.apache.org/jira/browse/HADOOP-14943
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14943-001.patch, HADOOP-14943-002.patch,
> HADOOP-14943-002.patch, HADOOP-14943-003.patch, HADOOP-14943-004.patch
>
>
> It looks suspiciously like S3A isn't providing the partitioning data needed
> in {{listLocatedStatus}} and {{getFileBlockLocations()}} needed to break up a
> file by the blocksize. This will stop tools using the MRv1 APIS doing the
> partitioning properly if the input format isn't doing it own split logic.
> FileInputFormat in MRv2 is a bit more configurable about input split
> calculation & will split up large files. but otherwise, the partitioning is
> being done more by the default values of the executing engine, rather than
> any config data from the filesystem about what its "block size" is,
> NativeAzureFS does a better job; maybe that could be factored out to
> hadoop-common and reused?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14936) S3Guard: remove "experimental" from documentation
[ https://issues.apache.org/jira/browse/HADOOP-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567602#comment-16567602 ] Steve Loughran commented on HADOOP-14936: - +review those S3guard constants tagged as @Unstable and remove the markings where we are confident that they are now stable > S3Guard: remove "experimental" from documentation > - > > Key: HADOOP-14936 > URL: https://issues.apache.org/jira/browse/HADOOP-14936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Priority: Major > > I think it is time to remove the "experimental feature" designation in the > site docs for S3Guard. Discuss. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15552) Move logging APIs over to slf4j in hadoop-tools - Part2
[ https://issues.apache.org/jira/browse/HADOOP-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Pickering updated HADOOP-15552: --- Attachment: HADOOP-15552.v6.patch > Move logging APIs over to slf4j in hadoop-tools - Part2 > --- > > Key: HADOOP-15552 > URL: https://issues.apache.org/jira/browse/HADOOP-15552 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Ian Pickering >Priority: Major > Attachments: HADOOP-15552.v1.patch, HADOOP-15552.v2.patch, > HADOOP-15552.v3.patch, HADOOP-15552.v4.patch, HADOOP-15552.v5.patch, > HADOOP-15552.v6.patch > > > Some classes in Hadoop-tools were not moved to slf4j > e.g. AliyunOSSInputStream.java, HadoopArchiveLogs.java, > HadoopArchiveLogsRunner.java -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13637) improve setting of max connections in AWS client
[ https://issues.apache.org/jira/browse/HADOOP-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13637: Parent Issue: HADOOP-15620 (was: HADOOP-15220) > improve setting of max connections in AWS client > > > Key: HADOOP-13637 > URL: https://issues.apache.org/jira/browse/HADOOP-13637 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > things can go badly wrong if the S3A FS creates a thread pool for IO > than > the number of pooled AWS http connections (set by property > MAXIMUM_CONNECTIONS); you also need some for any other IO requests coming in. > The max connections property is currently independent of thread pool size, > and has a default value of 1. > this is why there is a troubleshooting section in the docs showing the stack > trace and instructions to fix". > Better: have a dynamic minimum like thread pool size + n, for a value of n to > be chosen. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15552) Move logging APIs over to slf4j in hadoop-tools - Part2
[
https://issues.apache.org/jira/browse/HADOOP-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567598#comment-16567598
]
genericqa commented on HADOOP-15552:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 35 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
56s{color} | {color:green} root generated 0 new + 1462 unchanged - 6 fixed =
1462 total (was 1468) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 13m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
9s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 5s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
9s{color} | {color:green} hadoop-streaming in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
12s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
13s{color} | {color:green} hadoop-archives in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-archive-logs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-rumen in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
23s{color} | {color:green} hadoop-gridmix in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
41s{color} | {color:green} hadoop-datajoin in the patch passed. {color}
[jira] [Updated] (HADOOP-15627) S3A ITestS3GuardWriteBack failing if bucket explicitly set to s3guard+DDB
[
https://issues.apache.org/jira/browse/HADOOP-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15627:
Summary: S3A ITestS3GuardWriteBack failing if bucket explicitly set to
s3guard+DDB (was: S3A ITests failing if bucket explicitly set to s3guard+DDB)
> S3A ITestS3GuardWriteBack failing if bucket explicitly set to s3guard+DDB
> -
>
> Key: HADOOP-15627
> URL: https://issues.apache.org/jira/browse/HADOOP-15627
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> Repeatable failure in {{ITestS3GuardWriteBack.testListStatusWriteBack}}
> Possible causes could include
> * test not setting up the three fs instances
> * (disabled) caching not isolating properly
> * something more serious
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15583) Stabilize S3A Assumed Role support
[
https://issues.apache.org/jira/browse/HADOOP-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15583:
Status: Open (was: Patch Available)
> Stabilize S3A Assumed Role support
> --
>
> Key: HADOOP-15583
> URL: https://issues.apache.org/jira/browse/HADOOP-15583
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15583-001.patch, HADOOP-15583-002.patch,
> HADOOP-15583-003.patch, HADOOP-15583-004.patch, HADOOP-15583-005.patch
>
>
> started off just on sharing credentials across S3A and S3Guard, but in the
> process it has grown to becoming one of stabilising the assumed role support
> so it can be used for more than just testing.
> Was: "S3Guard to get AWS Credential chain from S3AFS; credentials closed() on
> shutdown"
> h3. Issue: lack of auth chain sharing causes ddb and s3 to get out of sync
> S3Guard builds its DDB auth chain itself, which stops it having to worry
> about being created standalone vs part of an S3AFS, but it means its
> authenticators are in a separate chain.
> When you are using short-lived assumed roles or other session credentials
> updated in the S3A FS authentication chain, you need that same set of
> credentials picked up by DDB. Otherwise, at best you are doubling load, at
> worse: the DDB connector may not get refreshed credentials.
> Proposed: {{DynamoDBClientFactory.createDynamoDBClient()}} to take an
> optional ref to aws credentials. If set: don't create a new set.
> There's one little complication here: our {{AWSCredentialProviderList}} list
> is autocloseable; it's close() will go through all children and close them.
> Apparently the AWS S3 client (And hopefully the DDB client) will close this
> when they are closed themselves. If DDB has the same set of credentials as
> the FS, then there could be trouble if they are closed in one place when the
> other still wants to use them.
> Solution; have a use count the uses of the credentials list, starting at one:
> every close() call decrements, and when this hits zero the cleanup is kicked
> off
> h3. Issue: {{AssumedRoleCredentialProvider}} connector to STS not picking up
> the s3a connection settings, including proxy.
> h3. issue: we're not using getPassword() to get user/password for proxy
> binding for STS. Fix: use that and pass down the bucket ref for per-bucket
> secrets in a JCEKS file.
> h3. Issue; hard to debug what's going wrong :)
> h3. Issue: docs about KMS permissions for SSE-KMS are wrong, and the
> ITestAssumedRole* tests don't request KMS permissions, so fail in a bucket
> when the base s3 FS is using SSE-KMS. KMS permissions need to be included in
> generated profiles
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15583) Stabilize S3A Assumed Role support
[
https://issues.apache.org/jira/browse/HADOOP-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15583:
Attachment: HADOOP-15583-005.patch
> Stabilize S3A Assumed Role support
> --
>
> Key: HADOOP-15583
> URL: https://issues.apache.org/jira/browse/HADOOP-15583
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15583-001.patch, HADOOP-15583-002.patch,
> HADOOP-15583-003.patch, HADOOP-15583-004.patch, HADOOP-15583-005.patch
>
>
> started off just on sharing credentials across S3A and S3Guard, but in the
> process it has grown to becoming one of stabilising the assumed role support
> so it can be used for more than just testing.
> Was: "S3Guard to get AWS Credential chain from S3AFS; credentials closed() on
> shutdown"
> h3. Issue: lack of auth chain sharing causes ddb and s3 to get out of sync
> S3Guard builds its DDB auth chain itself, which stops it having to worry
> about being created standalone vs part of an S3AFS, but it means its
> authenticators are in a separate chain.
> When you are using short-lived assumed roles or other session credentials
> updated in the S3A FS authentication chain, you need that same set of
> credentials picked up by DDB. Otherwise, at best you are doubling load, at
> worse: the DDB connector may not get refreshed credentials.
> Proposed: {{DynamoDBClientFactory.createDynamoDBClient()}} to take an
> optional ref to aws credentials. If set: don't create a new set.
> There's one little complication here: our {{AWSCredentialProviderList}} list
> is autocloseable; it's close() will go through all children and close them.
> Apparently the AWS S3 client (And hopefully the DDB client) will close this
> when they are closed themselves. If DDB has the same set of credentials as
> the FS, then there could be trouble if they are closed in one place when the
> other still wants to use them.
> Solution; have a use count the uses of the credentials list, starting at one:
> every close() call decrements, and when this hits zero the cleanup is kicked
> off
> h3. Issue: {{AssumedRoleCredentialProvider}} connector to STS not picking up
> the s3a connection settings, including proxy.
> h3. issue: we're not using getPassword() to get user/password for proxy
> binding for STS. Fix: use that and pass down the bucket ref for per-bucket
> secrets in a JCEKS file.
> h3. Issue; hard to debug what's going wrong :)
> h3. Issue: docs about KMS permissions for SSE-KMS are wrong, and the
> ITestAssumedRole* tests don't request KMS permissions, so fail in a bucket
> when the base s3 FS is using SSE-KMS. KMS permissions need to be included in
> generated profiles
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[
https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567527#comment-16567527
]
genericqa commented on HADOOP-14154:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
46s{color} | {color:red} hadoop-tools/hadoop-aws generated 1 new + 0 unchanged
- 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
22s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-aws |
| | org.apache.hadoop.fs.s3a.s3guard.DDBPathMetadata doesn't override
PathMetadata.equals(Object) At DDBPathMetadata.java:At
DDBPathMetadata.java:[line 1] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-14154 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934174/HADOOP-14154.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 16309d718418 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 889df6f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14981/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14981/testReport/ |
| Max. process+thread count | 336 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[
https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567473#comment-16567473
]
Gabor Bota commented on HADOOP-14154:
-
The patch still does not include the docs. I'll upload another patch with the
docs once the implementation got +1.
> Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
> -
>
> Key: HADOOP-14154
> URL: https://issues.apache.org/jira/browse/HADOOP-14154
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Rajesh Balamohan
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-14154-HADOOP-13345.001.patch,
> HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf,
> HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch
>
>
> Add support for "authoritative mode" for DynamoDBMetadataStore.
> The missing feature is to persist the bit set in
> {{DirListingMetadata.isAuthoritative}}.
> This topic has been super confusing for folks so I will also file a
> documentation Jira to explain the design better.
> We may want to also rename the DirListingMetadata.isAuthoritative field to
> .isFullListing to eliminate the multiple uses and meanings of the word
> "authoritative".
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[
https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567469#comment-16567469
]
Gabor Bota commented on HADOOP-14154:
-
Thanks for the review [~fabbri], and for the idea of DDBPathMetadata.
Uploaded my v2 patch (HADOOP-14154.002.patch) where I've fixed all the issues
you mentioned.
For the usual testing part:
Tested against eu-west-1, and got some issues, but seems unrelated/known:
{noformat}
[ERROR] Failures:
[ERROR]
ITestS3AContractGetFileStatusV1List>AbstractContractGetFileStatusTest.testListLocatedStatusFiltering:499->AbstractContractGetFileStatusTest.verifyListStatus:534->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88
length of listStatus(s3a://cloudera-dev-gabor-ireland/fork-0003/test,
org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest$AllPathsFilter@5beca598
) expected:<2> but was:<3>
[ERROR] Errors:
[ERROR]
ITestS3AFileContextMainOperations>FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile:840
? UnsupportedOperation
[ERROR]
ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testDestroyNoBucket:309->AbstractS3GuardToolTestBase.run:110
? IllegalArgument
[ERROR]
ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testDestroyNoBucket:309->AbstractS3GuardToolTestBase.run:110
? IllegalArgument
{noformat}
> Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
> -
>
> Key: HADOOP-14154
> URL: https://issues.apache.org/jira/browse/HADOOP-14154
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Rajesh Balamohan
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-14154-HADOOP-13345.001.patch,
> HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf,
> HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch
>
>
> Add support for "authoritative mode" for DynamoDBMetadataStore.
> The missing feature is to persist the bit set in
> {{DirListingMetadata.isAuthoritative}}.
> This topic has been super confusing for folks so I will also file a
> documentation Jira to explain the design better.
> We may want to also rename the DirListingMetadata.isAuthoritative field to
> .isFullListing to eliminate the multiple uses and meanings of the word
> "authoritative".
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[
https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-14154:
Attachment: HADOOP-14154.002.patch
> Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
> -
>
> Key: HADOOP-14154
> URL: https://issues.apache.org/jira/browse/HADOOP-14154
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Rajesh Balamohan
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-14154-HADOOP-13345.001.patch,
> HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf,
> HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch
>
>
> Add support for "authoritative mode" for DynamoDBMetadataStore.
> The missing feature is to persist the bit set in
> {{DirListingMetadata.isAuthoritative}}.
> This topic has been super confusing for folks so I will also file a
> documentation Jira to explain the design better.
> We may want to also rename the DirListingMetadata.isAuthoritative field to
> .isFullListing to eliminate the multiple uses and meanings of the word
> "authoritative".
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567454#comment-16567454
]
Xiao Chen commented on HADOOP-14445:
Thanks again Wei-Chiu and Ajay.
I agree more testing is a good thing in general. For this case though, the KMS
HA behavior is orthogonal to the authentication method used. Specifically,
{{LBKMSCP#doOp}} (the HA behavior) does not depend on the config. So I didn't
add the test for now, because it wouldn't add more coverage. But feel free to
elaborate and convince me otherwise.
It seems [~daryn] isn't around for the rest of the week, I'll wait for a few
more days, and commit on next Wednesday (8/8) if no more comments. :)
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566084#comment-16566084
]
Ajay Kumar edited comment on HADOOP-14445 at 8/2/18 6:29 PM:
-
[~xiaochen] thanks for updating the patch. LGTM.
{quote}This patch does not change that code path and is unrelated to client
retry behavior, but on the delegation token side. So I don't think adding the
config would add reasonable value here.
{quote}
Although this patch is mostly about KMS DT i was thinking of failover testing
for KMS HA when auth method will be DT. But i don't feel too strongly about it,
feel free to commit it as this is important improvement over current state.
was (Author: ajayydv):
[~xiaochen] thanks for updating the patch. LGTM.
{quote}This patch does not change that code path and is unrelated to client
retry behavior, but on the delegation token side. So I don't think adding the
config would add reasonable value here.{quote}
Although this patch is mostly about KMS DT i was thinking of failover testing
for KMS HA when auth method will be DT.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15642) Update to latest/recent version of aws-sdk for Hadoop 3.2
[
https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567314#comment-16567314
]
Steve Loughran commented on HADOOP-15642:
-
with this SDK, the new test in HADOOP-15583 for assumed role duration >1h
(correctly) fails, indicating that the client SDK is no longer failing if the
requested duration > 1h.
{code}
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:492)
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:377)
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:446)
at
org.apache.hadoop.fs.s3a.S3ATestUtils.interceptClosing(S3ATestUtils.java:480)
at
org.apache.hadoop.fs.s3a.auth.ITestAssumeRole.expectFileSystemCreateFailure(ITestAssumeRole.java:122)
at
org.apache.hadoop.fs.s3a.auth.ITestAssumeRole.testAssumeRoleThreeHourSessionDuration(ITestAssumeRole.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
A test for duration of 36h now fails with a 400 error coming back from the STS
service itself, which implies that SDK isn't doing any checks; it's all in the
service.
{code}
Caused by:
com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: 1
validation error detected: Value '129600' at 'durationSeconds' failed to
satisfy constraint: Member must have value less than or equal to 43200
(Service: AWSSecurityTokenService; Status Code: 400; Error Code:
ValidationError; Request ID: 4c6ef3ea-9680-11e8-aa3d-815d32cf1bdf)
{code}
> Update to latest/recent version of aws-sdk for Hadoop 3.2
> -
>
> Key: HADOOP-15642
> URL: https://issues.apache.org/jira/browse/HADOOP-15642
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15642-001.patch, HADOOP-15642-003.patch, Screen
> Shot 2018-07-30 at 14.11.22.png
>
>
> Move to a later version of the AWS SDK library for a different set of
> features and issues.
> proposed version: 1.11.375
> One thing which doesn't work on the one we ship with is the ability to create
> assumed role sessions >1h, as there's a check in the client lib for
> role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration
> checks to the far end.
> see:
> [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/]
> * assuming later versions will extend assumed role life, docs will need
> changing,
> * Adding a test in HADOOP-15583 which expects an error message if you ask for
> a duration of 3h; this should act as the test to see what happens.
> * think this time would be good to explicitly write down the SDK update
> process
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15552) Move logging APIs over to slf4j in hadoop-tools - Part2
[ https://issues.apache.org/jira/browse/HADOOP-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Pickering updated HADOOP-15552: --- Attachment: HADOOP-15552.v5.patch > Move logging APIs over to slf4j in hadoop-tools - Part2 > --- > > Key: HADOOP-15552 > URL: https://issues.apache.org/jira/browse/HADOOP-15552 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Ian Pickering >Priority: Major > Attachments: HADOOP-15552.v1.patch, HADOOP-15552.v2.patch, > HADOOP-15552.v3.patch, HADOOP-15552.v4.patch, HADOOP-15552.v5.patch > > > Some classes in Hadoop-tools were not moved to slf4j > e.g. AliyunOSSInputStream.java, HadoopArchiveLogs.java, > HadoopArchiveLogsRunner.java -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567086#comment-16567086
]
Steve Loughran commented on HADOOP-15576:
-
Failing tests all look unrelated; if they involved MPU stuff things would be
different.
{code}
org.apache.hadoop.hdfs.TestMaintenanceState.testFileCloseAfterEnteringMaintenance
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancerWithStripedFile
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancerCliWithExcludeListInAFile
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancerCliWithIncludeListWithPorts
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancerCliWithExcludeList
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testMaxIterationTime
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancerWithExcludeListWithPorts
org.apache.hadoop.hdfs.server.datanode.TestDataNodeUUID.testUUIDRegeneration
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure.testUnderReplicationAfterVolFailure
{code}
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576-005.patch, HADOOP-15576.001.patch,
> HADOOP-15576.002.patch, HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14212) Expose SecurityEnabled boolean field in JMX for other services besides NameNode
[
https://issues.apache.org/jira/browse/HADOOP-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566921#comment-16566921
]
Adam Antal commented on HADOOP-14212:
-
As I see the InterfaceAudience annotation is changed in HDFS-11217 - so the
patch is good as for the annotation.
> Expose SecurityEnabled boolean field in JMX for other services besides
> NameNode
> ---
>
> Key: HADOOP-14212
> URL: https://issues.apache.org/jira/browse/HADOOP-14212
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ray Burgemeestre
>Assignee: Adam Antal
>Priority: Minor
> Labels: security
> Attachments: HADOOP-14212.001.patch, HADOOP-14212.002.patch,
> HADOOP-14212.003.patch, HADOOP-14212.004.patch, HADOOP-14212.005.patch,
> HADOOP-14212.005.patch, HADOOP-14212.005.patch, HADOOP-14212.006.patch,
> HADOOP-14212.007.patch, HADOOP-14212.008.patch, HADOOP-14212.009.patch
>
>
> The following commit
> https://github.com/apache/hadoop/commit/dc17bda4b677e30c02c2a9a053895a43e41f7a12
> introduced a "SecurityEnabled" field in the JMX output for the NameNode. I
> believe it would be nice to add this same change to the JMX output of other
> services: Secondary Namenode, ResourceManager, NodeManagers, DataNodes, etc.
> So that it can be queried whether Security is enabled in all JMX resources.
> The reason I am suggesting this feature / improvement is that I think it
> would provide a clean way to check whether your cluster is completely
> Kerberized or not. I don't think there is an easy/clean way to do this now,
> other than checking the logs, checking ports etc.?
> The file where the change was made is
> hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> has the following function now:
> {code:java}
> @Override // NameNodeStatusMXBean
> public boolean isSecurityEnabled() {
> return UserGroupInformation.isSecurityEnabled();
> }
> {code}
> I would be happy to develop a patch if it seems useful by others as well?
> This is a snippet from the JMX output from the NameNode in case security is
> not enabled:
> {code}
> {
> "name" : "Hadoop:service=NameNode,name=NameNodeStatus",
> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.NameNode",
> "NNRole" : "NameNode",
> "HostAndPort" : "node001.cm.cluster:8020",
> "SecurityEnabled" : false,
> "LastHATransitionTime" : 0,
> "State" : "standby"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-9069) FileSystem.get leads to stack overflow if default FS is not configured with a scheme
[
https://issues.apache.org/jira/browse/HADOOP-9069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HADOOP-9069.
-
Resolution: Duplicate
> FileSystem.get leads to stack overflow if default FS is not configured with a
> scheme
>
>
> Key: HADOOP-9069
> URL: https://issues.apache.org/jira/browse/HADOOP-9069
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 0.23.3, 2.0.1-alpha
>Reporter: Jason Lowe
>Assignee: Andras Bokor
>Priority: Minor
>
> If fs.defaultFS is configured without a scheme, e.g.: "/", then
> FileSystem.get will infinitely recurse and lead to a stack overflow. An
> example stacktrace from 0.23:
> {noformat}
> java.lang.StackOverflowError
> at java.util.AbstractCollection.(AbstractCollection.java:66)
> at java.util.AbstractList.(AbstractList.java:76)
> at java.util.ArrayList.(ArrayList.java:128)
> at java.util.ArrayList.(ArrayList.java:139)
> at
> org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:430)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:852)
> at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:171)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:163)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:163)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:163)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15649) The Resource Estimator Service cannot parse the default resourcemanager log
[
https://issues.apache.org/jira/browse/HADOOP-15649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566706#comment-16566706
]
tangshangwen commented on HADOOP-15649:
---
I think we should support parsing the default log format
> The Resource Estimator Service cannot parse the default resourcemanager log
> ---
>
> Key: HADOOP-15649
> URL: https://issues.apache.org/jira/browse/HADOOP-15649
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 3.1.0
>Reporter: tangshangwen
>Priority: Major
>
> When I used the Resource Estimator Service for Resource emulation, I found
> that it could not parse the default resourcemanager log，It parses the log
> format as
> {code:java}
> 253114:e,06/21/2017 16:18:35,yarn resourcemanager,DefaultTag,Pid="5156"
> Tid="19824" TS="0x01D2EAAA089569FD" String1="17/06/21 16:18:35 INFO
> resourcemanager.RMAppManager$ApplicationSummary:
> appId=application_1497832133857_0330,name=FraudDetection-1,user=hadoop,
> queue=PROD,state=FINISHED,trackingUrl=http://BY1PR00OC0019.namprd00.prod.outlook.com:8088/proxy/application_1497832133857_0330/,appMasterHost=by2pr08mb1799.namprd08.prod.outlook.com,startTime=1498061413073,finishTime=1498061905698,finalStatus=SUCCEEDED,memorySeconds=1330655,vcoreSeconds=704,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources= vCores:0>,applicationType=MAPREDUCE"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15649) The Resource Estimator Service cannot parse the default resourcemanager log
[
https://issues.apache.org/jira/browse/HADOOP-15649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566705#comment-16566705
]
tangshangwen commented on HADOOP-15649:
---
我认为我们应该支持解析默认的日志格式
> The Resource Estimator Service cannot parse the default resourcemanager log
> ---
>
> Key: HADOOP-15649
> URL: https://issues.apache.org/jira/browse/HADOOP-15649
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 3.1.0
>Reporter: tangshangwen
>Priority: Major
>
> When I used the Resource Estimator Service for Resource emulation, I found
> that it could not parse the default resourcemanager log，It parses the log
> format as
> {code:java}
> 253114:e,06/21/2017 16:18:35,yarn resourcemanager,DefaultTag,Pid="5156"
> Tid="19824" TS="0x01D2EAAA089569FD" String1="17/06/21 16:18:35 INFO
> resourcemanager.RMAppManager$ApplicationSummary:
> appId=application_1497832133857_0330,name=FraudDetection-1,user=hadoop,
> queue=PROD,state=FINISHED,trackingUrl=http://BY1PR00OC0019.namprd00.prod.outlook.com:8088/proxy/application_1497832133857_0330/,appMasterHost=by2pr08mb1799.namprd08.prod.outlook.com,startTime=1498061413073,finishTime=1498061905698,finalStatus=SUCCEEDED,memorySeconds=1330655,vcoreSeconds=704,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources= vCores:0>,applicationType=MAPREDUCE"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-15649) The Resource Estimator Service cannot parse the default resourcemanager log
[
https://issues.apache.org/jira/browse/HADOOP-15649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tangshangwen updated HADOOP-15649:
--
Comment: was deleted
(was: 我认为我们应该支持解析默认的日志格式)
> The Resource Estimator Service cannot parse the default resourcemanager log
> ---
>
> Key: HADOOP-15649
> URL: https://issues.apache.org/jira/browse/HADOOP-15649
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 3.1.0
>Reporter: tangshangwen
>Priority: Major
>
> When I used the Resource Estimator Service for Resource emulation, I found
> that it could not parse the default resourcemanager log，It parses the log
> format as
> {code:java}
> 253114:e,06/21/2017 16:18:35,yarn resourcemanager,DefaultTag,Pid="5156"
> Tid="19824" TS="0x01D2EAAA089569FD" String1="17/06/21 16:18:35 INFO
> resourcemanager.RMAppManager$ApplicationSummary:
> appId=application_1497832133857_0330,name=FraudDetection-1,user=hadoop,
> queue=PROD,state=FINISHED,trackingUrl=http://BY1PR00OC0019.namprd00.prod.outlook.com:8088/proxy/application_1497832133857_0330/,appMasterHost=by2pr08mb1799.namprd08.prod.outlook.com,startTime=1498061413073,finishTime=1498061905698,finalStatus=SUCCEEDED,memorySeconds=1330655,vcoreSeconds=704,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources= vCores:0>,applicationType=MAPREDUCE"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15649) The Resource Estimator Service cannot parse the default resourcemanager log
tangshangwen created HADOOP-15649:
-
Summary: The Resource Estimator Service cannot parse the default
resourcemanager log
Key: HADOOP-15649
URL: https://issues.apache.org/jira/browse/HADOOP-15649
Project: Hadoop Common
Issue Type: Bug
Components: tools
Affects Versions: 3.1.0
Reporter: tangshangwen
When I used the Resource Estimator Service for Resource emulation, I found that
it could not parse the default resourcemanager log，It parses the log format as
{code:java}
253114:e,06/21/2017 16:18:35,yarn resourcemanager,DefaultTag,Pid="5156"
Tid="19824" TS="0x01D2EAAA089569FD" String1="17/06/21 16:18:35 INFO
resourcemanager.RMAppManager$ApplicationSummary:
appId=application_1497832133857_0330,name=FraudDetection-1,user=hadoop,
queue=PROD,state=FINISHED,trackingUrl=http://BY1PR00OC0019.namprd00.prod.outlook.com:8088/proxy/application_1497832133857_0330/,appMasterHost=by2pr08mb1799.namprd08.prod.outlook.com,startTime=1498061413073,finishTime=1498061905698,finalStatus=SUCCEEDED,memorySeconds=1330655,vcoreSeconds=704,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources=,applicationType=MAPREDUCE"
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[
https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566638#comment-16566638
]
wujinhu commented on HADOOP-15607:
--
Thanks [~Sammi]
> AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
> -
>
> Key: HADOOP-15607
> URL: https://issues.apache.org/jira/browse/HADOOP-15607
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Critical
> Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch,
> HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch
>
>
> When I generated data with hive-tpcds tool, I got exception below:
> 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id :
> attempt_1531723399698_0001_m_52_0, Status : FAILED
> Error: com.aliyun.oss.OSSException: The list of parts was not in ascending
> order. Parts list must specified in order by part number.
> [ErrorCode]: InvalidPartOrder
> [RequestId]: 5B4C40425FCC208D79D1EAF5
> [HostId]: 100.103.0.137
> [ResponseError]:
>
>
> InvalidPartOrder
> The list of parts was not in ascending order. Parts list must
> specified in order by part number.
> 5B4C40425FCC208D79D1EAF5
> xx.xx.xx.xx
> current PartNumber 3, you given part number 3is not in
> ascending order
>
> at
> com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99)
> at
> com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69)
> at
> com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248)
> at
> com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130)
> at
> com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68)
> at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94)
> at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149)
> at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113)
> at
> com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185)
> at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574)
> at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
>
> I reviewed code below,
> {code:java}
> blockId {code}
> has thread synchronization problem
> {code:java}
> // code placeholder
> private void uploadCurrentPart() throws IOException {
> blockFiles.add(blockFile);
> blockStream.flush();
> blockStream.close();
> if (blockId == 0) {
> uploadId = store.getUploadId(key);
> }
> ListenableFuture partETagFuture =
> executorService.submit(() -> {
> PartETag partETag = store.uploadPart(blockFile, key, uploadId,
> blockId + 1);
> return partETag;
> });
> partETagsFutures.add(partETagFuture);
> blockFile = newBlockFile();
> blockId++;
> blockStream = new BufferedOutputStream(new FileOutputStream(blockFile));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566593#comment-16566593
]
genericqa commented on HADOOP-15576:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 10 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 29m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
28s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}121m 8s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
9s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}277m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestMaintenanceState |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15576 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934013/HADOOP-15576-005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs
[jira] [Comment Edited] (HADOOP-11794) Enable distcp to copy blocks in parallel
[
https://issues.apache.org/jira/browse/HADOOP-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566294#comment-16566294
]
smarthan edited comment on HADOOP-11794 at 8/2/18 7:42 AM:
---
Hi [~yzhangal], I have a doubt that whether there is a bug in
RetriableFileCopyCommand.copyBytes( ) when enable blocksperchunk .
{code:java}
int bytesRead = readBytes(inStream, buf, sourceOffset);
while (bytesRead >= 0) {
if (chunkLength > 0 &&
(totalBytesRead + bytesRead) >= chunkLength) {
bytesRead = (int)(chunkLength - totalBytesRead);
finished = true;
}
totalBytesRead += bytesRead;
//action == FileAction.APPEND always false when blocksperchunk >= 1,
//the sourceOffset will never been change
//readBytes(inStream, buf, sourceOffset) will always start from the same
position 'sourceOffset'
//so the buf[] always read same data, and result in wrong split copy finally
if (action == FileAction.APPEND) {
sourceOffset += bytesRead;
}
outStream.write(buf, 0, bytesRead);
updateContextStatus(totalBytesRead, context, source2);
if (finished) {
break;
}
bytesRead = readBytes(inStream, buf, sourceOffset);
}
{code}
I merge this patch to branch *cdh5.10*, and when I set blocksperchunk >= 1, the
copy of data will be different form source.
I debug and find this problem, and when I modify the condition as follow, it
works.
{code:java}
if (action == FileAction.APPEND || (source2.isSplit() && sourceOffset > 0)) {
sourceOffset += bytesRead;
}{code}
I am not sure whether it is a bug or just caused by version, please look at it
if possible, thanks.
was (Author: smarthan):
Hi [~yzhangal], I have a doubt that whether there is a bug in
RetriableFileCopyCommand.copyBytes( ) when enable blocksperchunk .
{code:java}
int bytesRead = readBytes(inStream, buf, sourceOffset);
while (bytesRead >= 0) {
if (chunkLength > 0 &&
(totalBytesRead + bytesRead) >= chunkLength) {
bytesRead = (int)(chunkLength - totalBytesRead);
finished = true;
}
totalBytesRead += bytesRead;
//action == FileAction.APPEND always false when blocksperchunk >= 1,
//the sourceOffset will never been change
//readBytes(inStream, buf, sourceOffset) will always start from the same
position 'sourceOffset'
//so the buf[] always read same data, and result in wrong split copy finally
if (action == FileAction.APPEND) {
sourceOffset += bytesRead;
}
outStream.write(buf, 0, bytesRead);
updateContextStatus(totalBytesRead, context, source2);
if (finished) {
break;
}
bytesRead = readBytes(inStream, buf, sourceOffset);
}
{code}
I merge this patch to branch cdh5.10, and when I set blocksperchunk >= 1, the
copy of data will be different form source.
I debug and find this problem, and when I modify the condition as follow, it
works.
{code:java}
if (action == FileAction.APPEND || (source2.isSplit() && sourceOffset > 0)) {
sourceOffset += bytesRead;
}{code}
I am not sure whether it is a bug or just caused by version, please look at it
if possible, thanks.
> Enable distcp to copy blocks in parallel
>
>
> Key: HADOOP-11794
> URL: https://issues.apache.org/jira/browse/HADOOP-11794
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 0.21.0
>Reporter: dhruba borthakur
>Assignee: Yongjun Zhang
>Priority: Major
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-11794.001.patch, HADOOP-11794.002.patch,
> HADOOP-11794.003.patch, HADOOP-11794.004.patch, HADOOP-11794.005.patch,
> HADOOP-11794.006.patch, HADOOP-11794.007.patch, HADOOP-11794.008.patch,
> HADOOP-11794.009.patch, HADOOP-11794.010.branch2.002.patch,
> HADOOP-11794.010.branch2.patch, HADOOP-11794.010.patch, MAPREDUCE-2257.patch
>
>
> The minimum unit of work for a distcp task is a file. We have files that are
> greater than 1 TB with a block size of 1 GB. If we use distcp to copy these
> files, the tasks either take a long long long time or finally fails. A better
> way for distcp would be to copy all the source blocks in parallel, and then
> stich the blocks back to files at the destination via the HDFS Concat API
> (HDFS-222)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
