[jira] [Commented] (HADOOP-14107) ITestS3GuardListConsistency fails intermittently
[
https://issues.apache.org/jira/browse/HADOOP-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967385#comment-15967385
]
Steve Loughran commented on HADOOP-14107:
-
+1 for the patch, let's see if the hypothesised-consistency-problem arises. If
so, LambaTestUtils.eventually() is designed for precisely this: you can wrap
the operation and keep trying
> ITestS3GuardListConsistency fails intermittently
>
>
> Key: HADOOP-14107
> URL: https://issues.apache.org/jira/browse/HADOOP-14107
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HADOOP-14107-HADOOP-13345.000.patch
>
>
> {code}
> mvn -Dit.test='ITestS3GuardListConsistency' -Dtest=none -Dscale -Ds3guard
> -Ddynamo -q clean verify
> ---
> T E S T S
> ---
> Running org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.544 sec <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
> testListStatusWriteBack(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)
> Time elapsed: 3.147 sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of results from metastore.
> Metastore should only know about /XYZ:
> DirListingMetadata{path=s3a://mliu-s3guard/test/ListStatusWriteBack,
> listMap={s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ;
> isDirectory=true; modification_time=0; access_time=0; owner=mliu;
> group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true},
> s3a://mliu-s3guard/test/ListStatusWriteBack/123=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/123;
> isDirectory=true; modification_time=0; access_time=0; owner=mliu;
> group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true}},
> isAuthoritative=false}
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testListStatusWriteBack(ITestS3GuardListConsistency.java:127)
> {code}
> See discussion on the parent JIRA [HADOOP-13345].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14107) ITestS3GuardListConsistency fails intermittently
[
https://issues.apache.org/jira/browse/HADOOP-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966881#comment-15966881
]
Hadoop QA commented on HADOOP-14107:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
36s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
22s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HADOOP-14107 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861966/HADOOP-14107-HADOOP-13345.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d9749415fe7e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / af8250a |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12093/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12093/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ITestS3GuardListConsistency fails intermittently
>
>
> Key: HADOOP-14107
> URL: https://issues.apache.org/jira/browse/HADOOP-14107
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HADOOP-14107-HADOOP-13345.000.patch
>
>
> {code}
> mvn -Dit.test='ITestS3GuardListConsistenc
[jira] [Commented] (HADOOP-14107) ITestS3GuardListConsistency fails intermittently
[
https://issues.apache.org/jira/browse/HADOOP-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956028#comment-15956028
]
Mingliang Liu commented on HADOOP-14107:
[~ste...@apache.org], thanks for your input.
I reran the test but did not reproduce the failure. There are two possibilities
that may fail this test:
# In {{noWriteBack.listStatus(directory);}}, the directory on S3 only may not
be listed because of the eventually consistency in S3 list. S3Guard is not
guaranteed to discover new S3 objects in a timely manner. This is possible but
I did not reproduce this in test.
# If the test data was not cleaned up during last test process, then this test
will fail, as the stack trace shows in description section.
The #1 is a corner case and we can simply sleep for 3~5 seconds. The #2 can be
addressed by deleting the existing test directory explicitly. We may still need
to sleep for a while before delete tracking is supported. Meanwhile, per your
comment, I changed the coding style, comments, variable names etc in the v0
patch to make the test purpose clearer.
The major point is that, is the test doing the right thing? I think it is. It
creates a directory on S3 only (via noS3Guard), and another directory on S3 and
DDB aka metadata store (via noWriteBack). Then if we list the file system with
S3Guard enabled, regardless of S3Guard writing back or not, it should return
both of the data. The only exception is the point #1 above: if the data only in
S3 is not visible yet. That's being said, both noWriteBack and yesWriteBack
will discover S3-only object, and union that with DDB result. However,
noWriteBack should not populate the newly found S3-only object to DDB while
yesWriteBack should always populate the S3-only object to DDB. We assert that
via querying DDB directly.
> ITestS3GuardListConsistency fails intermittently
>
>
> Key: HADOOP-14107
> URL: https://issues.apache.org/jira/browse/HADOOP-14107
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14107-HADOOP-13345.000.patch
>
>
> {code}
> mvn -Dit.test='ITestS3GuardListConsistency' -Dtest=none -Dscale -Ds3guard
> -Ddynamo -q clean verify
> ---
> T E S T S
> ---
> Running org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.544 sec <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
> testListStatusWriteBack(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)
> Time elapsed: 3.147 sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of results from metastore.
> Metastore should only know about /XYZ:
> DirListingMetadata{path=s3a://mliu-s3guard/test/ListStatusWriteBack,
> listMap={s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ;
> isDirectory=true; modification_time=0; access_time=0; owner=mliu;
> group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true},
> s3a://mliu-s3guard/test/ListStatusWriteBack/123=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/123;
> isDirectory=true; modification_time=0; access_time=0; owner=mliu;
> group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true}},
> isAuthoritative=false}
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testListStatusWriteBack(ITestS3GuardListConsistency.java:127)
> {code}
> See discussion on the parent JIRA [HADOOP-13345].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14107) ITestS3GuardListConsistency fails intermittently
[
https://issues.apache.org/jira/browse/HADOOP-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880310#comment-15880310
]
Steve Loughran commented on HADOOP-14107:
-
* Is the test doing the write thing? It's asserting that the noWriteBack
metastore should only see the entry it created, *and not that created by
yesWriteback*
Except shouldn't it be that because yesWriteback is writing back the entry,
listing operations which do still go to s3 may pick up the created entry? What
Mingliang may be seeing on his runs is that the S3 LIST command is up to date
with the written back entry, so return both. Other people (including myself)
have our tests working because we are doing the LIST before the inconsistent s3
listing has grabbed the entry
Looking at the test, I've some recommendations to aid diagnostics
# we add a small delay between write and probe, so that s3 will list stuff if
it has it.
# the asserts should really be assertEquals, though that isn't the cause of
this problem
# the names of the paths should be writtenBack and notWrittenBack not XYZ and
123. That will make what's going on clearer
# assert text to indicate whether its writeback/not writeback
> ITestS3GuardListConsistency fails intermittently
>
>
> Key: HADOOP-14107
> URL: https://issues.apache.org/jira/browse/HADOOP-14107
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>
> {code}
> mvn -Dit.test='ITestS3GuardListConsistency' -Dtest=none -Dscale -Ds3guard
> -Ddynamo -q clean verify
> ---
> T E S T S
> ---
> Running org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.544 sec <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
> testListStatusWriteBack(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)
> Time elapsed: 3.147 sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of results from metastore.
> Metastore should only know about /XYZ:
> DirListingMetadata{path=s3a://mliu-s3guard/test/ListStatusWriteBack,
> listMap={s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ;
> isDirectory=true; modification_time=0; access_time=0; owner=mliu;
> group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true},
> s3a://mliu-s3guard/test/ListStatusWriteBack/123=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/123;
> isDirectory=true; modification_time=0; access_time=0; owner=mliu;
> group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true}},
> isAuthoritative=false}
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testListStatusWriteBack(ITestS3GuardListConsistency.java:127)
> {code}
> See discussion on the parent JIRA [HADOOP-13345].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
