[jira] [Comment Edited] (HADOOP-15420) s3guard ITestS3GuardToolLocal failures in diff tests
[ https://issues.apache.org/jira/browse/HADOOP-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467244#comment-16467244 ] Gabor Bota edited comment on HADOOP-15420 at 5/8/18 11:20 AM: -- *using {{standardize(Path)}}* Do you mean using it like: {code:java} private boolean expired(FileStatus status, long expiry, String keyPrefix) { Path path = standardize(status.getPath()); String bucket = path.toUri().getHost(); String translatedPath = ""; if(bucket != null && !bucket.isEmpty()){ translatedPath = PathMetadataDynamoDBTranslation.pathToParentKey(path); } else { translatedPath = path.toString(); } return status.getModificationTime() < expiry && !status.isDirectory() && translatedPath.startsWith(keyPrefix); } {code} I need to check for the bucket. Removing the check for the {{getHost}} (existing bucket) the following tests will fail: org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase#testPruneUnsetsAuthoritative org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase#testPruneFiles org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase#testPruneDirs in these cases the {{status.getPath()}}=file:/unpruned-root-dir; and it can not be digested by {{PathMetadataDynamoDBTranslation.pathToParentKey()}}, the test will fail with the following: {noformat} java.lang.IllegalArgumentException: Path missing bucket at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88) at org.apache.hadoop.fs.s3a.s3guard.PathMetadataDynamoDBTranslation.pathToParentKey(PathMetadataDynamoDBTranslation.java:255) at org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.expired(LocalMetadataStore.java:358) at org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(LocalMetadataStore.java:318) at org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(LocalMetadataStore.java:308) at org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase.testPruneUnsetsAuthoritative(MetadataStoreTestBase.java:730) (...) {noformat} *Testing with local dynamo* Here is my output for {{mvn clean test -Ds3guard -Ddynamo}}: [https://pastebin.com/raw/qGuwuS4F] Short summary: Tests run: 398, Failures: 0, Errors: 0, Skipped: 2 was (Author: gabor.bota): *using {{standardize(Path)}}* Do you mean using it like: {code:java} private boolean expired(FileStatus status, long expiry, String keyPrefix) { Path path = standardize(status.getPath()); String bucket = path.toUri().getHost(); String translatedPath = ""; if(bucket != null && !bucket.isEmpty()){ translatedPath = PathMetadataDynamoDBTranslation.pathToParentKey(path); } else { translatedPath = path.toString(); } return status.getModificationTime() < expiry && !status.isDirectory() && translatedPath.startsWith(keyPrefix); } {code} I need to check for the bucket. Removing the check for the {{getHost}} (existing bucket) the following tests will fail: org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase#testPruneUnsetsAuthoritative org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase#testPruneFiles org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase#testPruneDirs in these cases the {{status.getPath()}}=file:/unpruned-root-dir; and it can not be digested by {{PathMetadataDynamoDBTranslation.pathToParentKey()}}, the test will fail with the following: {noformat} java.lang.IllegalArgumentException: Path missing bucket at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88) at org.apache.hadoop.fs.s3a.s3guard.PathMetadataDynamoDBTranslation.pathToParentKey(PathMetadataDynamoDBTranslation.java:255) at org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.expired(LocalMetadataStore.java:358) at org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(LocalMetadataStore.java:318) at org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(LocalMetadataStore.java:308) at org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase.testPruneUnsetsAuthoritative(MetadataStoreTestBase.java:730) (...) {noformat} *Testing with local dynamo* Here is my output for {{mvn clean test -Ds3guard -Ddynamo}}: [https://pastebin.com/qGuwuS4F] Short summary: Tests run: 398, Failures: 0, Errors: 0, Skipped: 2 > s3guard ITestS3GuardToolLocal failures in diff tests > > > Key: HADOOP-15420 > URL: https://issues.apache.org/jira/browse/HADOOP-15420 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Aaron Fabbri >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-15420.001.patch, HADOOP-15420.002.patch > > > Noticed this when testing the patch for HADOOP-13756. > > {code:java} > [ERROR]
[jira] [Comment Edited] (HADOOP-15420) s3guard ITestS3GuardToolLocal failures in diff tests
[ https://issues.apache.org/jira/browse/HADOOP-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467244#comment-16467244 ] Gabor Bota edited comment on HADOOP-15420 at 5/8/18 11:16 AM: -- *using {{standardize(Path)}}* Do you mean using it like: {code:java} private boolean expired(FileStatus status, long expiry, String keyPrefix) { Path path = standardize(status.getPath()); String bucket = path.toUri().getHost(); String translatedPath = ""; if(bucket != null && !bucket.isEmpty()){ translatedPath = PathMetadataDynamoDBTranslation.pathToParentKey(path); } else { translatedPath = path.toString(); } return status.getModificationTime() < expiry && !status.isDirectory() && translatedPath.startsWith(keyPrefix); } {code} I need to check for the bucket. Removing the check for the {{getHost}} (existing bucket) the following tests will fail: org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase#testPruneUnsetsAuthoritative org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase#testPruneFiles org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase#testPruneDirs in these cases the {{status.getPath()}}=file:/unpruned-root-dir; and it can not be digested by {{PathMetadataDynamoDBTranslation.pathToParentKey()}}, the test will fail with the following: {noformat} java.lang.IllegalArgumentException: Path missing bucket at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88) at org.apache.hadoop.fs.s3a.s3guard.PathMetadataDynamoDBTranslation.pathToParentKey(PathMetadataDynamoDBTranslation.java:255) at org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.expired(LocalMetadataStore.java:358) at org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(LocalMetadataStore.java:318) at org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(LocalMetadataStore.java:308) at org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase.testPruneUnsetsAuthoritative(MetadataStoreTestBase.java:730) (...) {noformat} *Testing with local dynamo* Here is my output for {{mvn clean test -Ds3guard -Ddynamo}}: [https://pastebin.com/qGuwuS4F] Short summary: Tests run: 398, Failures: 0, Errors: 0, Skipped: 2 was (Author: gabor.bota): *using {{standardize(Path)}} Do you mean using it like: {code:java} private boolean expired(FileStatus status, long expiry, String keyPrefix) { Path path = standardize(status.getPath()); String bucket = path.toUri().getHost(); String translatedPath = ""; if(bucket != null && !bucket.isEmpty()){ translatedPath = PathMetadataDynamoDBTranslation.pathToParentKey(path); } else { translatedPath = path.toString(); } return status.getModificationTime() < expiry && !status.isDirectory() && translatedPath.startsWith(keyPrefix); } {code} I need to check for the bucket. Removing the check for the {{getHost}} (existing bucket) the following tests will fail: org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase#testPruneUnsetsAuthoritative org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase#testPruneFiles org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase#testPruneDirs in these cases the {{status.getPath()}}=file:/unpruned-root-dir; and it can not be digested by {{PathMetadataDynamoDBTranslation.pathToParentKey()}}, the test will fail with the following: {noformat} java.lang.IllegalArgumentException: Path missing bucket at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88) at org.apache.hadoop.fs.s3a.s3guard.PathMetadataDynamoDBTranslation.pathToParentKey(PathMetadataDynamoDBTranslation.java:255) at org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.expired(LocalMetadataStore.java:358) at org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(LocalMetadataStore.java:318) at org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(LocalMetadataStore.java:308) at org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase.testPruneUnsetsAuthoritative(MetadataStoreTestBase.java:730) (...) {noformat} *Testing with local dynamo* Here is my output for {{mvn clean test -Ds3guard -Ddynamo}}: [https://pastebin.com/qGuwuS4F] Short summary: Tests run: 398, Failures: 0, Errors: 0, Skipped: 2 > s3guard ITestS3GuardToolLocal failures in diff tests > > > Key: HADOOP-15420 > URL: https://issues.apache.org/jira/browse/HADOOP-15420 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Aaron Fabbri >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-15420.001.patch, HADOOP-15420.002.patch > > > Noticed this when testing the patch for HADOOP-13756. > > {code:java} > [ERROR]
[jira] [Comment Edited] (HADOOP-15420) s3guard ITestS3GuardToolLocal failures in diff tests
[ https://issues.apache.org/jira/browse/HADOOP-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466764#comment-16466764 ] Aaron Fabbri edited comment on HADOOP-15420 at 5/8/18 2:44 AM: --- Thank you for working on this issue [~gabor.bota]. Good work identifying the bug. Couple comments on v1 patch: {noformat} private boolean expired(FileStatus status, long expiry, String keyPrefix) { +// remove the protocol from path string to be able to compare +String bucket = status.getPath().toUri().getHost(); + statusTranslatedPath = status.getPath().toUri().getPath(); +} + {noformat} Can you use helper func {{standardize(Path)}} here instead? Thanks for moving {{ testDiffCommand() }} to the base class. Did you test this with Dynamo? (`mvn clean test -Ds3guard -Ddynamo`) Unfortunately dynamodb metadatastore test the Local (in-memory) Test Dynamo thing still (until we finish HADOOP-14918). was (Author: fabbri): Thank you for working on this issue [~gabor.bota]. Good work identifying the bug. Couple comments on v1 patch: {noformat} private boolean expired(FileStatus status, long expiry, String keyPrefix) { +// remove the protocol from path string to be able to compare +String bucket = status.getPath().toUri().getHost(); + statusTranslatedPath = status.getPath().toUri().getPath(); +} + {noformat} Can you use helper func {{standardize(Path)}} here instead? Thanks for moving {{ testDiffCommand() }} to the base class. Did you test this with Dynamo? (`mvn clean test -Ds3guard -Ddynamo`) Unfortunately dynamodb metadatastore test the Local (in-memory) Test Dynamo thing still (until we finish HADOOP-14918). Also a reminder please declare which AWS region you ran integration tests in. > s3guard ITestS3GuardToolLocal failures in diff tests > > > Key: HADOOP-15420 > URL: https://issues.apache.org/jira/browse/HADOOP-15420 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Aaron Fabbri >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-15420.001.patch, HADOOP-15420.002.patch > > > Noticed this when testing the patch for HADOOP-13756. > > {code:java} > [ERROR] Failures: > [ERROR] > ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testPruneCommandCLI:221->AbstractS3GuardToolTestBase.testPruneCommand:201->AbstractS3GuardToolTestBase.assertMetastoreListingCount:214->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 > Pruned children count > [PathMetadata{fileStatus=S3AFileStatus{path=s3a://bucket-new/test/testPruneCommandCLI/stale; > isDirectory=false; length=100; replication=1; blocksize=512; > modification_time=1524798258286; access_time=0; owner=hdfs; group=hdfs; > permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=false; > isErasureCoded=false} isEmptyDirectory=FALSE; isEmptyDirectory=UNKNOWN; > isDeleted=false}, > PathMetadata{fileStatus=S3AFileStatus{path=s3a://bucket-new/test/testPruneCommandCLI/fresh; > isDirectory=false; length=100; replication=1; blocksize=512; > modification_time=1524798262583; access_time=0; owner=hdfs; group=hdfs; > permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=false; > isErasureCoded=false} isEmptyDirectory=FALSE; isEmptyDirectory=UNKNOWN; > isDeleted=false}] expected:<1> but was:<2>{code} > > Looking through the code, I'm noticing a couple of issues. > > 1. {{testDiffCommand()}} is in {{ITestS3GuardToolLocal}}, but it should > really be running for all MetadataStore implementations. Seems like it > should live in {{AbstractS3GuardToolTestBase}}. > 2. {{AbstractS3GuardToolTestBase#createFile()}} seems wrong. When > {{onMetadataStore}} is false, it does a {{ContractTestUtils.touch(file)}}, > but the fs is initialized with a MetadataStore present, so seem like the fs > will still put the file in the MetadataStore? > There are other tests which explicitly go around the MetadataStore by using > {{fs.setMetadataStore(nullMS)}}, e.g. ITestS3AInconsistency. We should do > something similar in {{AbstractS3GuardToolTestBase#createFile()}}, minding > any issues with parallel test runs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org