http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md index 3b83f1f..3b9b5c4 100644 --- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md +++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md @@ -107,6 +107,10 @@ each filesystem for its testing. 1. `test.fs.s3n.name` : the URL of the bucket for S3n tests 1. `test.fs.s3a.name` : the URL of the bucket for S3a tests +*Note* that running s3a and s3n tests in parallel mode, against the same bucket +is unreliable. We recommend using separate buckets or testing one connector +at a time. + The contents of each bucket will be destroyed during the test process: do not use the bucket for any purpose other than testing. Furthermore, for s3a, all in-progress multi-part uploads to the bucket will be aborted at the @@ -691,7 +695,7 @@ use requires the presence of secret credentials, where tests may be slow, and where finding out why something failed from nothing but the test output is critical. -#### Subclasses Existing Shared Base Blasses +#### Subclasses Existing Shared Base Classes Extend `AbstractS3ATestBase` or `AbstractSTestS3AHugeFiles` unless justifiable. These set things up for testing against the object stores, provide good threadnames, @@ -798,7 +802,7 @@ We really appreciate this — you will too. ### How to keep your credentials really safe -Although the `auth-keys.xml` file is marged as ignored in git and subversion, +Although the `auth-keys.xml` file is marked as ignored in git and subversion, it is still in your source tree, and there's always that risk that it may creep out. @@ -813,3 +817,283 @@ using an absolute XInclude reference to it. </configuration> ``` + +# Failure Injection + +**Warning do not enable any type of failure injection in production. The +following settings are for testing only.** + +One of the challenges with S3A integration tests is the fact that S3 is an +eventually-consistent storage system. In practice, we rarely see delays in +visibility of recently created objects both in listings (`listStatus()`) and +when getting a single file's metadata (`getFileStatus()`). Since this behavior +is rare and non-deterministic, thorough integration testing is challenging. + +To address this, S3A supports a shim layer on top of the `AmazonS3Client` +class which artificially delays certain paths from appearing in listings. +This is implemented in the class `InconsistentAmazonS3Client`. + +## Simulating List Inconsistencies + +### Enabling the InconsistentAmazonS3CClient + +There are two ways of enabling the `InconsistentAmazonS3Client`: at +config-time, or programmatically. For an example of programmatic test usage, +see `ITestS3GuardListConsistency`. + +To enable the fault-injecting client via configuration, switch the +S3A client to use the "Inconsistent S3 Client Factory" when connecting to +S3: + +```xml +<property> + <name>fs.s3a.s3.client.factory.impl</name> + <value>org.apache.hadoop.fs.s3a.InconsistentS3ClientFactory</value> +</property> +``` + +The inconsistent client works by: + +1. Choosing which objects will be "inconsistent" at the time the object is +created or deleted. +2. When `listObjects()` is called, any keys that we have marked as +inconsistent above will not be returned in the results (until the +configured delay has elapsed). Similarly, deleted items may be *added* to +missing results to delay the visibility of the delete. + +There are two ways of choosing which keys (filenames) will be affected: By +substring, and by random probability. + +```xml +<property> + <name>fs.s3a.failinject.inconsistency.key.substring</name> + <value>DELAY_LISTING_ME</value> +</property> + +<property> + <name>fs.s3a.failinject.inconsistency.probability</name> + <value>1.0</value> +</property> +``` + +By default, any object which has the substring "DELAY_LISTING_ME" in its key +will subject to delayed visibility. For example, the path +`s3a://my-bucket/test/DELAY_LISTING_ME/file.txt` would match this condition. +To match all keys use the value "\*" (a single asterisk). This is a special +value: *We don't support arbitrary wildcards.* + +The default probability of delaying an object is 1.0. This means that *all* +keys that match the substring will get delayed visibility. Note that we take +the logical *and* of the two conditions (substring matches *and* probability +random chance occurs). Here are some example configurations: + +``` +| substring | probability | behavior | +|-----------|-------------|--------------------------------------------| +| | 0.001 | An empty <value> tag in .xml config will | +| | | be interpreted as unset and revert to the | +| | | default value, "DELAY_LISTING_ME" | +| | | | +| * | 0.001 | 1/1000 chance of *any* key being delayed. | +| | | | +| delay | 0.01 | 1/100 chance of any key containing "delay" | +| | | | +| delay | 1.0 | All keys containing substring "delay" .. | +``` + +You can also configure how long you want the delay in visibility to last. +The default is 5000 milliseconds (five seconds). + +```xml +<property> + <name>fs.s3a.failinject.inconsistency.msec</name> + <value>5000</value> +</property> +``` + +Future versions of this client will introduce new failure modes, +with simulation of S3 throttling exceptions the next feature under +development. + +### Limitations of Inconsistency Injection + +Although `InconsistentAmazonS3Client` can delay the visibility of an object +or parent directory, it does not prevent the key of that object from +appearing in all prefix searches. For example, if we create the following +object with the default configuration above, in an otherwise empty bucket: + +``` +s3a://bucket/a/b/c/DELAY_LISTING_ME +``` + +Then the following paths will still be visible as directories (ignoring +possible real-world inconsistencies): + +``` +s3a://bucket/a +s3a://bucket/a/b +``` + +Whereas `getFileStatus()` on the following *will* be subject to delayed +visibility (`FileNotFoundException` until delay has elapsed): + +``` +s3a://bucket/a/b/c +s3a://bucket/a/b/c/DELAY_LISTING_ME +``` + +In real-life S3 inconsistency, however, we expect that all the above paths +(including `a` and `b`) will be subject to delayed visiblity. + +### Using the `InconsistentAmazonS3CClient` in downstream integration tests + +The inconsistent client is shipped in the `hadoop-aws` JAR, so it can +be used in applications which work with S3 to see how they handle +inconsistent directory listings. + +## Testing S3Guard + +The basic strategy for testing S3Guard correctness consists of: + +1. MetadataStore Contract tests. + + The MetadataStore contract tests are inspired by the Hadoop FileSystem and + `FileContext` contract tests. Each implementation of the `MetadataStore` interface + subclasses the `MetadataStoreTestBase` class and customizes it to initialize + their MetadataStore. This test ensures that the different implementations + all satisfy the semantics of the MetadataStore API. + +2. Running existing S3A unit and integration tests with S3Guard enabled. + + You can run the S3A integration tests on top of S3Guard by configuring your + `MetadataStore` in your + `hadoop-tools/hadoop-aws/src/test/resources/core-site.xml` or + `hadoop-tools/hadoop-aws/src/test/resources/auth-keys.xml` files. + Next run the S3A integration tests as outlined in the *Running the Tests* section + of the [S3A documentation](./index.html) + +3. Running fault-injection tests that test S3Guard's consistency features. + + The `ITestS3GuardListConsistency` uses failure injection to ensure + that list consistency logic is correct even when the underlying storage is + eventually consistent. + + The integration test adds a shim above the Amazon S3 Client layer that injects + delays in object visibility. + + All of these tests will be run if you follow the steps listed in step 2 above. + + No charges are incurred for using this store, and its consistency + guarantees are that of the underlying object store instance. <!-- :) --> + +## Testing S3A with S3Guard Enabled + +All the S3A tests which work with a private repository can be configured to +run with S3Guard by using the `s3guard` profile. When set, this will run +all the tests with local memory for the metadata set to "non-authoritative" mode. + +```bash +mvn -T 1C verify -Dparallel-tests -DtestsThreadCount=6 -Ds3guard +``` + +When the `s3guard` profile is enabled, following profiles can be specified: + +* `dynamo`: use an AWS-hosted DynamoDB table; creating the table if it does + not exist. You will have to pay the bills for DynamoDB web service. +* `dynamodblocal`: use an in-memory DynamoDBLocal server instead of real AWS + DynamoDB web service; launch the server and creating the table. + You won't be charged bills for using DynamoDB in test. As it runs in-JVM, + the table isn't shared across other tests running in parallel. +* `non-auth`: treat the S3Guard metadata as authorative. + +```bash +mvn -T 1C verify -Dparallel-tests -DtestsThreadCount=6 -Ds3guard -Ddynamo -Dauth +``` + +When experimenting with options, it is usually best to run a single test suite +at a time until the operations appear to be working. + +```bash +mvn -T 1C verify -Dtest=skip -Dit.test=ITestS3AMiscOperations -Ds3guard -Ddynamo +``` + +### Notes + +1. If the `s3guard` profile is not set, then the S3Guard properties are those +of the test configuration set in `contract-test-options.xml` or `auth-keys.xml` + +If the `s3guard` profile *is* set, +1. The S3Guard options from maven (the dynamo and authoritative flags) + overwrite any previously set in the configuration files. +1. DynamoDB will be configured to create any missing tables. + +### Warning About Concurrent Tests + +You must not run S3A and S3N tests in parallel on the same bucket. This is +especially true when S3Guard is enabled. S3Guard requires that all clients +that are modifying the bucket have S3Guard enabled, so having S3N +integration tests running in parallel with S3A tests will cause strange +failures. + +### Scale Testing MetadataStore Directly + +There are some scale tests that exercise Metadata Store implementations +directly. These ensure that S3Guard is are robust to things like DynamoDB +throttling, and compare performance for different implementations. These +are included in the scale tests executed when `-Dscale` is passed to +the maven command line. + +The two S3Guard scale testse are `ITestDynamoDBMetadataStoreScale` and +`ITestLocalMetadataStoreScale`. To run the DynamoDB test, you will need to +define your table name and region in your test configuration. For example, +the following settings allow us to run `ITestDynamoDBMetadataStoreScale` with +artificially low read and write capacity provisioned, so we can judge the +effects of being throttled by the DynamoDB service: + +```xml +<property> + <name>scale.test.operation.count</name> + <value>10</value> +</property> +<property> + <name>scale.test.directory.count</name> + <value>3</value> +</property> +<property> + <name>fs.s3a.scale.test.enabled</name> + <value>true</value> +</property> +<property> + <name>fs.s3a.s3guard.ddb.table</name> + <value>my-scale-test</value> +</property> +<property> + <name>fs.s3a.s3guard.ddb.region</name> + <value>us-west-2</value> +</property> +<property> + <name>fs.s3a.s3guard.ddb.table.create</name> + <value>true</value> +</property> +<property> + <name>fs.s3a.s3guard.ddb.table.capacity.read</name> + <value>10</value> +</property> +<property> + <name>fs.s3a.s3guard.ddb.table.capacity.write</name> + <value>10</value> +</property> +``` + +### Testing only: Local Metadata Store + +There is an in-memory Metadata Store for testing. + +```xml +<property> + <name>fs.s3a.metadatastore.impl</name> + <value>org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore</value> +</property> +``` + +This is not for use in production.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractCreate.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractCreate.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractCreate.java index d2a858f..fd9497b 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractCreate.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractCreate.java @@ -22,11 +22,25 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.contract.AbstractContractCreateTest; import org.apache.hadoop.fs.contract.AbstractFSContract; +import static org.apache.hadoop.fs.s3a.S3ATestUtils.maybeEnableS3Guard; + /** * S3A contract tests creating files. */ public class ITestS3AContractCreate extends AbstractContractCreateTest { + /** + * Create a configuration, possibly patching in S3Guard options. + * @return a configuration + */ + @Override + protected Configuration createConfiguration() { + Configuration conf = super.createConfiguration(); + // patch in S3Guard options + maybeEnableS3Guard(conf); + return conf; + } + @Override protected AbstractFSContract createContract(Configuration conf) { return new S3AContract(conf); http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractDelete.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractDelete.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractDelete.java index a47dcaef..95ea410 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractDelete.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractDelete.java @@ -22,11 +22,25 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.contract.AbstractContractDeleteTest; import org.apache.hadoop.fs.contract.AbstractFSContract; +import static org.apache.hadoop.fs.s3a.S3ATestUtils.maybeEnableS3Guard; + /** * S3A contract tests covering deletes. */ public class ITestS3AContractDelete extends AbstractContractDeleteTest { + /** + * Create a configuration, possibly patching in S3Guard options. + * @return a configuration + */ + @Override + protected Configuration createConfiguration() { + Configuration conf = super.createConfiguration(); + // patch in S3Guard options + maybeEnableS3Guard(conf); + return conf; + } + @Override protected AbstractFSContract createContract(Configuration conf) { return new S3AContract(conf); http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractDistCp.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractDistCp.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractDistCp.java index 50ce0c2..587dbbc 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractDistCp.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractDistCp.java @@ -20,6 +20,7 @@ package org.apache.hadoop.fs.contract.s3a; import static org.apache.hadoop.fs.s3a.Constants.*; import static org.apache.hadoop.fs.s3a.S3ATestConstants.SCALE_TEST_TIMEOUT_MILLIS; +import static org.apache.hadoop.fs.s3a.S3ATestUtils.maybeEnableS3Guard; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.tools.contract.AbstractContractDistCpTest; @@ -38,12 +39,18 @@ public class ITestS3AContractDistCp extends AbstractContractDistCpTest { return SCALE_TEST_TIMEOUT_MILLIS; } + /** + * Create a configuration, possibly patching in S3Guard options. + * @return a configuration + */ @Override protected Configuration createConfiguration() { Configuration newConf = super.createConfiguration(); newConf.setLong(MULTIPART_SIZE, MULTIPART_SETTING); newConf.setBoolean(FAST_UPLOAD, true); newConf.set(FAST_UPLOAD_BUFFER, FAST_UPLOAD_BUFFER_DISK); + // patch in S3Guard options + maybeEnableS3Guard(newConf); return newConf; } http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractGetFileStatus.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractGetFileStatus.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractGetFileStatus.java index c7ed5a3..cb9819c 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractGetFileStatus.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractGetFileStatus.java @@ -23,6 +23,8 @@ import org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest; import org.apache.hadoop.fs.s3a.Constants; import org.apache.hadoop.fs.s3a.S3ATestUtils; +import static org.apache.hadoop.fs.s3a.S3ATestUtils.maybeEnableS3Guard; + /** * S3A contract tests covering getFileStatus. */ @@ -46,6 +48,8 @@ public class ITestS3AContractGetFileStatus S3ATestUtils.disableFilesystemCaching(conf); // aggressively low page size forces tests to go multipage conf.setInt(Constants.MAX_PAGING_KEYS, 2); + // patch in S3Guard options + maybeEnableS3Guard(conf); return conf; } } http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java index d953e7e..dba52e1 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java @@ -22,11 +22,25 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.contract.AbstractContractMkdirTest; import org.apache.hadoop.fs.contract.AbstractFSContract; +import static org.apache.hadoop.fs.s3a.S3ATestUtils.maybeEnableS3Guard; + /** * Test dir operations on S3A. */ public class ITestS3AContractMkdir extends AbstractContractMkdirTest { + /** + * Create a configuration, possibly patching in S3Guard options. + * @return a configuration + */ + @Override + protected Configuration createConfiguration() { + Configuration conf = super.createConfiguration(); + // patch in S3Guard options + maybeEnableS3Guard(conf); + return conf; + } + @Override protected AbstractFSContract createContract(Configuration conf) { return new S3AContract(conf); http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractOpen.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractOpen.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractOpen.java index a7bdc0d..8e338b7 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractOpen.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractOpen.java @@ -22,11 +22,25 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.contract.AbstractContractOpenTest; import org.apache.hadoop.fs.contract.AbstractFSContract; +import static org.apache.hadoop.fs.s3a.S3ATestUtils.maybeEnableS3Guard; + /** * S3A contract tests opening files. */ public class ITestS3AContractOpen extends AbstractContractOpenTest { + /** + * Create a configuration, possibly patching in S3Guard options. + * @return a configuration + */ + @Override + protected Configuration createConfiguration() { + Configuration conf = super.createConfiguration(); + // patch in S3Guard options + maybeEnableS3Guard(conf); + return conf; + } + @Override protected AbstractFSContract createContract(Configuration conf) { return new S3AContract(conf); http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java index 5dba03d..4339649 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java @@ -26,12 +26,25 @@ import org.apache.hadoop.fs.Path; import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset; import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset; +import static org.apache.hadoop.fs.s3a.S3ATestUtils.maybeEnableS3Guard; /** * S3A contract tests covering rename. */ public class ITestS3AContractRename extends AbstractContractRenameTest { + /** + * Create a configuration, possibly patching in S3Guard options. + * @return a configuration + */ + @Override + protected Configuration createConfiguration() { + Configuration conf = super.createConfiguration(); + // patch in S3Guard options + maybeEnableS3Guard(conf); + return conf; + } + @Override protected AbstractFSContract createContract(Configuration conf) { return new S3AContract(conf); http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRootDir.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRootDir.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRootDir.java index 8383a77..5c2e2cd 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRootDir.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRootDir.java @@ -28,6 +28,8 @@ import org.apache.hadoop.fs.contract.AbstractFSContract; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import static org.apache.hadoop.fs.s3a.S3ATestUtils.maybeEnableS3Guard; + /** * root dir operations against an S3 bucket. */ @@ -37,6 +39,18 @@ public class ITestS3AContractRootDir extends private static final Logger LOG = LoggerFactory.getLogger(ITestS3AContractRootDir.class); + /** + * Create a configuration, possibly patching in S3Guard options. + * @return a configuration + */ + @Override + protected Configuration createConfiguration() { + Configuration conf = super.createConfiguration(); + // patch in S3Guard options + maybeEnableS3Guard(conf); + return conf; + } + @Override protected AbstractFSContract createContract(Configuration conf) { return new S3AContract(conf); http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractSeek.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractSeek.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractSeek.java index 1572fbc..379ace8 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractSeek.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractSeek.java @@ -22,11 +22,25 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.contract.AbstractContractSeekTest; import org.apache.hadoop.fs.contract.AbstractFSContract; +import static org.apache.hadoop.fs.s3a.S3ATestUtils.maybeEnableS3Guard; + /** * S3A contract tests covering file seek. */ public class ITestS3AContractSeek extends AbstractContractSeekTest { + /** + * Create a configuration, possibly patching in S3Guard options. + * @return a configuration + */ + @Override + protected Configuration createConfiguration() { + Configuration conf = super.createConfiguration(); + // patch in S3Guard options + maybeEnableS3Guard(conf); + return conf; + } + @Override protected AbstractFSContract createContract(Configuration conf) { return new S3AContract(conf); http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java index 6734947..0c7f7df 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java @@ -26,6 +26,8 @@ import com.amazonaws.services.s3.AmazonS3; import java.net.URI; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.s3a.s3guard.MetadataStore; +import org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore; import org.junit.After; import org.junit.Before; @@ -33,7 +35,8 @@ import org.junit.Rule; import org.junit.rules.ExpectedException; /** - * Abstract base class for S3A unit tests using a mock S3 client. + * Abstract base class for S3A unit tests using a mock S3 client and a null + * metadata store. */ public abstract class AbstractS3AMockTest { @@ -55,6 +58,10 @@ public abstract class AbstractS3AMockTest { Configuration conf = new Configuration(); conf.setClass(S3_CLIENT_FACTORY_IMPL, MockS3ClientFactory.class, S3ClientFactory.class); + // We explicitly disable MetadataStore even if it's configured. For unit + // test we don't issue request to AWS DynamoDB service. + conf.setClass(S3_METADATA_STORE_IMPL, NullMetadataStore.class, + MetadataStore.class); fs = new S3AFileSystem(); URI uri = URI.create(FS_S3A + "://" + BUCKET); fs.initialize(uri, conf); http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java index c19b72c..f0c389d 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java @@ -33,6 +33,7 @@ import java.io.IOException; import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset; import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset; +import static org.apache.hadoop.fs.s3a.S3ATestUtils.maybeEnableS3Guard; /** * An extension of the contract test base set up for S3A tests. @@ -65,6 +66,18 @@ public abstract class AbstractS3ATestBase extends AbstractFSContractTestBase return S3A_TEST_TIMEOUT; } + /** + * Create a configuration, possibly patching in S3Guard options. + * @return a configuration + */ + @Override + protected Configuration createConfiguration() { + Configuration conf = super.createConfiguration(); + // patch in S3Guard options + maybeEnableS3Guard(conf); + return conf; + } + protected Configuration getConfiguration() { return getContract().getConf(); } @@ -99,10 +112,21 @@ public abstract class AbstractS3ATestBase extends AbstractFSContractTestBase */ protected Path writeThenReadFile(String name, int len) throws IOException { Path path = path(name); + writeThenReadFile(path, len); + return path; + } + + /** + * Write a file, read it back, validate the dataset. Overwrites the file + * if it is present + * @param path path to file + * @param len length of file + * @throws IOException any IO problem + */ + protected void writeThenReadFile(Path path, int len) throws IOException { byte[] data = dataset(len, 'a', 'z'); writeDataset(getFileSystem(), path, data, data.length, 1024 * 1024, true); ContractTestUtils.verifyFileContents(getFileSystem(), path, data); - return path; } /** http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AAWSCredentialsProvider.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AAWSCredentialsProvider.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AAWSCredentialsProvider.java index 22c4f7e..6601233 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AAWSCredentialsProvider.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AAWSCredentialsProvider.java @@ -140,6 +140,10 @@ public class ITestS3AAWSCredentialsProvider { createFailingFS(conf); } catch (AccessDeniedException e) { // expected + } catch (AWSServiceIOException e) { + GenericTestUtils.assertExceptionContains( + "UnrecognizedClientException", e); + // expected } } http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java index dd75cb6..b9fe0fd 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java @@ -25,6 +25,7 @@ import com.amazonaws.services.s3.S3ClientOptions; import org.apache.commons.lang.StringUtils; import org.apache.commons.lang.reflect.FieldUtils; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.contract.ContractTestUtils; import org.apache.hadoop.fs.s3native.S3xLoginHelper; @@ -483,7 +484,7 @@ public class ITestS3AConfiguration { } }); assertEquals("username", alice, fs.getUsername()); - S3AFileStatus status = fs.getFileStatus(new Path("/")); + FileStatus status = fs.getFileStatus(new Path("/")); assertEquals("owner in " + status, alice, status.getOwner()); assertEquals("group in " + status, alice, status.getGroup()); } http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACopyFromLocalFile.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACopyFromLocalFile.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACopyFromLocalFile.java index 71776ac..7dc286d 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACopyFromLocalFile.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACopyFromLocalFile.java @@ -29,6 +29,7 @@ import org.apache.commons.io.Charsets; import org.apache.commons.io.FileUtils; import org.apache.commons.io.IOUtils; import org.apache.hadoop.fs.FileAlreadyExistsException; +import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.Path; import static org.apache.hadoop.test.LambdaTestUtils.intercept; @@ -63,7 +64,7 @@ public class ITestS3ACopyFromLocalFile extends AbstractS3ATestBase { Path dest = upload(file, true); assertPathExists("uploaded file not found", dest); S3AFileSystem fs = getFileSystem(); - S3AFileStatus status = fs.getFileStatus(dest); + FileStatus status = fs.getFileStatus(dest); assertEquals("File length of " + status, message.getBytes(ASCII).length, status.getLen()); assertFileTextEquals(dest, message); http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACredentialsInURL.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACredentialsInURL.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACredentialsInURL.java index b3d7abf..95d44cc 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACredentialsInURL.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACredentialsInURL.java @@ -19,6 +19,7 @@ package org.apache.hadoop.fs.s3a; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IOUtils; import org.junit.After; @@ -37,6 +38,7 @@ import java.net.URLEncoder; import java.nio.file.AccessDeniedException; import static org.apache.hadoop.fs.s3a.S3ATestConstants.TEST_FS_S3A_NAME; +import static org.apache.hadoop.fs.s3a.S3ATestUtils.assumeS3GuardState; /** * Tests that credentials can go into the URL. This includes a valid @@ -63,6 +65,11 @@ public class ITestS3ACredentialsInURL extends Assert { public void testInstantiateFromURL() throws Throwable { Configuration conf = new Configuration(); + + // Skip in the case of S3Guard with DynamoDB because it cannot get + // credentials for its own use if they're only in S3 URLs + assumeS3GuardState(false, conf); + String accessKey = conf.get(Constants.ACCESS_KEY); String secretKey = conf.get(Constants.SECRET_KEY); String fsname = conf.getTrimmed(TEST_FS_S3A_NAME, ""); @@ -84,6 +91,7 @@ public class ITestS3ACredentialsInURL extends Assert { conf.unset(Constants.ACCESS_KEY); conf.unset(Constants.SECRET_KEY); fs = S3ATestUtils.createTestFileSystem(conf); + String fsURI = fs.getUri().toString(); assertFalse("FS URI contains a @ symbol", fsURI.contains("@")); assertFalse("FS URI contains a % symbol", fsURI.contains("%")); @@ -119,13 +127,14 @@ public class ITestS3ACredentialsInURL extends Assert { Configuration conf = new Configuration(); String fsname = conf.getTrimmed(TEST_FS_S3A_NAME, ""); Assume.assumeNotNull(fsname); + assumeS3GuardState(false, conf); URI original = new URI(fsname); URI testURI = createUriWithEmbeddedSecrets(original, "user", "//"); conf.set(TEST_FS_S3A_NAME, testURI.toString()); - fs = S3ATestUtils.createTestFileSystem(conf); try { - S3AFileStatus status = fs.getFileStatus(new Path("/")); + fs = S3ATestUtils.createTestFileSystem(conf); + FileStatus status = fs.getFileStatus(new Path("/")); fail("Expected an AccessDeniedException, got " + status); } catch (AccessDeniedException e) { // expected http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADelayedFNF.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADelayedFNF.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADelayedFNF.java new file mode 100644 index 0000000..7abd474 --- /dev/null +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADelayedFNF.java @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a; + +import org.apache.hadoop.fs.FSDataInputStream; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.contract.ContractTestUtils; +import org.apache.hadoop.test.LambdaTestUtils; +import org.junit.Test; + +import java.io.FileNotFoundException; +import java.util.concurrent.Callable; + +/** + * Tests behavior of a FileNotFound error that happens after open(), i.e. on + * the first read. + */ +public class ITestS3ADelayedFNF extends AbstractS3ATestBase { + + + /** + * See debugging documentation + * <a href="https://cwiki.apache.org/confluence/display/HADOOP/S3A%3A+FileNotFound+Exception+on+Read";>here</a>. + * @throws Exception + */ + @Test + public void testNotFoundFirstRead() throws Exception { + FileSystem fs = getFileSystem(); + Path p = path("some-file"); + ContractTestUtils.createFile(fs, p, false, new byte[] {20, 21, 22}); + + final FSDataInputStream in = fs.open(p); + assertDeleted(p, false); + + // This should fail since we deleted after the open. + LambdaTestUtils.intercept(FileNotFoundException.class, + new Callable<Integer>() { + @Override + public Integer call() throws Exception { + return in.read(); + } + }); + } + +} http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEmptyDirectory.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEmptyDirectory.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEmptyDirectory.java new file mode 100644 index 0000000..c55be5b --- /dev/null +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEmptyDirectory.java @@ -0,0 +1,83 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a; + +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.contract.ContractTestUtils; +import org.junit.Test; + +import java.io.IOException; + +/** + * Tests which exercise treatment of empty/non-empty directories. + */ +public class ITestS3AEmptyDirectory extends AbstractS3ATestBase { + + @Test + public void testDirectoryBecomesEmpty() throws Exception { + S3AFileSystem fs = getFileSystem(); + + // 1. set up non-empty dir + Path dir = path("testEmptyDir"); + Path child = path("testEmptyDir/dir2"); + mkdirs(child); + + S3AFileStatus status = getS3AFileStatus(fs, dir); + assertEmptyDirectory(false, status); + + // 2. Make testEmptyDir empty + assertDeleted(child, false); + status = getS3AFileStatus(fs, dir); + + assertEmptyDirectory(true, status); + } + + private static void assertEmptyDirectory(boolean isEmpty, S3AFileStatus s) { + String msg = "dir is empty"; + // Should *not* be Tristate.UNKNOWN since we request a definitive value + // in getS3AFileStatus() below + Tristate expected = Tristate.fromBool(isEmpty); + assertEquals(msg, expected, s.isEmptyDirectory()); + } + + @Test + public void testDirectoryBecomesNonEmpty() throws Exception { + S3AFileSystem fs = getFileSystem(); + + // 1. create empty dir + Path dir = path("testEmptyDir"); + mkdirs(dir); + + S3AFileStatus status = getS3AFileStatus(fs, dir); + assertEmptyDirectory(true, status); + + // 2. Make testEmptyDir non-empty + + ContractTestUtils.touch(fs, path("testEmptyDir/file1")); + status = getS3AFileStatus(fs, dir); + + assertEmptyDirectory(false, status); + } + + private S3AFileStatus getS3AFileStatus(S3AFileSystem fs, Path p) throws + IOException { + return fs.innerGetFileStatus(p, true /* want isEmptyDirectory value */); + } + +} http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEC.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEC.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEC.java index 91be8b9..8b7e031 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEC.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEC.java @@ -18,19 +18,21 @@ package org.apache.hadoop.fs.s3a; -import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset; -import static org.apache.hadoop.fs.contract.ContractTestUtils.rm; -import static org.apache.hadoop.fs.s3a.S3ATestUtils.skipIfEncryptionTestsDisabled; -import static org.apache.hadoop.test.LambdaTestUtils.intercept; - import java.io.IOException; +import java.nio.file.AccessDeniedException; + +import org.junit.Test; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.contract.ContractTestUtils; import org.apache.hadoop.fs.contract.s3a.S3AContract; -import org.junit.Test; +import org.apache.hadoop.io.IOUtils; + +import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset; +import static org.apache.hadoop.fs.s3a.S3ATestUtils.*; +import static org.apache.hadoop.test.LambdaTestUtils.intercept; /** * Concrete class that extends {@link AbstractTestS3AEncryption} @@ -38,17 +40,39 @@ import org.junit.Test; */ public class ITestS3AEncryptionSSEC extends AbstractTestS3AEncryption { + private static final String SERVICE_AMAZON_S3_STATUS_CODE_403 + = "Service: Amazon S3; Status Code: 403;"; + private static final String KEY_1 + = "4niV/jPK5VFRHY+KNb6wtqYd4xXyMgdJ9XQJpcQUVbs="; + private static final String KEY_2 + = "G61nz31Q7+zpjJWbakxfTOZW4VS0UmQWAq2YXhcTXoo="; + private static final String KEY_3 + = "NTx0dUPrxoo9+LbNiT/gqf3z9jILqL6ilismFmJO50U="; + private static final String KEY_4 + = "msdo3VvvZznp66Gth58a91Hxe/UpExMkwU9BHkIjfW8="; + private static final int TEST_FILE_LEN = 2048; + + /** + * Filesystem created with a different key. + */ + private FileSystem fsKeyB; + @Override protected Configuration createConfiguration() { Configuration conf = super.createConfiguration(); - S3ATestUtils.disableFilesystemCaching(conf); + disableFilesystemCaching(conf); conf.set(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM, getSSEAlgorithm().getMethod()); - conf.set(Constants.SERVER_SIDE_ENCRYPTION_KEY, - "4niV/jPK5VFRHY+KNb6wtqYd4xXyMgdJ9XQJpcQUVbs="); + conf.set(Constants.SERVER_SIDE_ENCRYPTION_KEY, KEY_1); return conf; } + @Override + public void teardown() throws Exception { + super.teardown(); + IOUtils.closeStream(fsKeyB); + } + /** * This will create and write to a file using encryption key A, then attempt * to read from it again with encryption key B. This will not work as it @@ -64,26 +88,25 @@ public class ITestS3AEncryptionSSEC extends AbstractTestS3AEncryption { assumeEnabled(); skipIfEncryptionTestsDisabled(getConfiguration()); - final Path[] path = new Path[1]; - intercept(java.nio.file.AccessDeniedException.class, - "Service: Amazon S3; Status Code: 403;", () -> { - - int len = 2048; - describe("Create an encrypted file of size " + len); - String src = createFilename(len); - path[0] = writeThenReadFile(src, len); - - //extract the test FS - FileSystem fileSystem = createNewFileSystemWithSSECKey( - "kX7SdwVc/1VXJr76kfKnkQ3ONYhxianyL2+C3rPVT9s="); - byte[] data = dataset(len, 'a', 'z'); - ContractTestUtils.verifyFileContents(fileSystem, path[0], data); - throw new Exception("Fail"); - }); + intercept(AccessDeniedException.class, + SERVICE_AMAZON_S3_STATUS_CODE_403, + () -> { + int len = TEST_FILE_LEN; + describe("Create an encrypted file of size " + len); + Path src = path("testCreateFileAndReadWithDifferentEncryptionKey"); + writeThenReadFile(src, len); + + //extract the test FS + fsKeyB = createNewFileSystemWithSSECKey( + "kX7SdwVc/1VXJr76kfKnkQ3ONYhxianyL2+C3rPVT9s="); + byte[] data = dataset(len, 'a', 'z'); + ContractTestUtils.verifyFileContents(fsKeyB, src, data); + return fsKeyB.getFileStatus(src); + }); } /** - * While each object has it's own key and should be distinct, this verifies + * While each object has its own key and should be distinct, this verifies * that hadoop treats object keys as a filesystem path. So if a top level * dir is encrypted with keyA, a sublevel dir cannot be accessed with a * different keyB. @@ -96,25 +119,20 @@ public class ITestS3AEncryptionSSEC extends AbstractTestS3AEncryption { public void testCreateSubdirWithDifferentKey() throws Exception { assumeEnabled(); skipIfEncryptionTestsDisabled(getConfiguration()); - - final Path[] path = new Path[1]; - intercept(java.nio.file.AccessDeniedException.class, - "Service: Amazon S3; Status Code: 403;", () -> { - - path[0] = S3ATestUtils.createTestPath( - new Path(createFilename("dir/")) - ); - Path nestedDirectory = S3ATestUtils.createTestPath( - new Path(createFilename("dir/nestedDir/")) - ); - FileSystem fsKeyB = createNewFileSystemWithSSECKey( - "G61nz31Q7+zpjJWbakxfTOZW4VS0UmQWAq2YXhcTXoo="); - getFileSystem().mkdirs(path[0]); - fsKeyB.mkdirs(nestedDirectory); - - throw new Exception("Exception should be thrown."); - }); - rm(getFileSystem(), path[0], true, false); + assumeS3GuardState(false, getConfiguration()); + + intercept(AccessDeniedException.class, + SERVICE_AMAZON_S3_STATUS_CODE_403, + () -> { + Path base = path("testCreateSubdirWithDifferentKey"); + Path nestedDirectory = new Path(base, "nestedDir"); + fsKeyB = createNewFileSystemWithSSECKey( + KEY_2); + getFileSystem().mkdirs(base); + fsKeyB.mkdirs(nestedDirectory); + // expected to fail + return fsKeyB.getFileStatus(nestedDirectory); + }); } /** @@ -130,20 +148,17 @@ public class ITestS3AEncryptionSSEC extends AbstractTestS3AEncryption { assumeEnabled(); skipIfEncryptionTestsDisabled(getConfiguration()); - final Path[] path = new Path[1]; - intercept(java.nio.file.AccessDeniedException.class, - "Service: Amazon S3; Status Code: 403;", () -> { - - int len = 2048; - String src = createFilename(len); - path[0] = writeThenReadFile(src, len); - - FileSystem fsKeyB = createNewFileSystemWithSSECKey( - "NTx0dUPrxoo9+LbNiT/gqf3z9jILqL6ilismFmJO50U="); - fsKeyB.rename(path[0], new Path(createFilename("different-path.txt"))); - - throw new Exception("Exception should be thrown."); - }); + intercept(AccessDeniedException.class, + SERVICE_AMAZON_S3_STATUS_CODE_403, + () -> { + int len = TEST_FILE_LEN; + Path src = path(createFilename(len)); + writeThenReadFile(src, len); + fsKeyB = createNewFileSystemWithSSECKey(KEY_3); + Path dest = path(createFilename("different-path.txt")); + getFileSystem().mkdirs(dest.getParent()); + return fsKeyB.rename(src, dest); + }); } /** @@ -157,11 +172,11 @@ public class ITestS3AEncryptionSSEC extends AbstractTestS3AEncryption { assumeEnabled(); skipIfEncryptionTestsDisabled(getConfiguration()); - String src = createFilename("original-path.txt"); - Path path = writeThenReadFile(src, 2048); - Path newPath = path(createFilename("different-path.txt")); - getFileSystem().rename(path, newPath); - byte[] data = dataset(2048, 'a', 'z'); + Path src = path("original-path.txt"); + writeThenReadFile(src, TEST_FILE_LEN); + Path newPath = path("different-path.txt"); + getFileSystem().rename(src, newPath); + byte[] data = dataset(TEST_FILE_LEN, 'a', 'z'); ContractTestUtils.verifyFileContents(getFileSystem(), newPath, data); } @@ -175,30 +190,26 @@ public class ITestS3AEncryptionSSEC extends AbstractTestS3AEncryption { public void testListEncryptedDir() throws Exception { assumeEnabled(); skipIfEncryptionTestsDisabled(getConfiguration()); + assumeS3GuardState(false, getConfiguration()); + + Path pathABC = path("testListEncryptedDir/a/b/c/"); + Path pathAB = pathABC.getParent(); + Path pathA = pathAB.getParent(); - Path nestedDirectory = S3ATestUtils.createTestPath( - path(createFilename("/a/b/c/")) - ); + Path nestedDirectory = createTestPath(pathABC); assertTrue(getFileSystem().mkdirs(nestedDirectory)); - FileSystem fsKeyB = createNewFileSystemWithSSECKey( - "msdo3VvvZznp66Gth58a91Hxe/UpExMkwU9BHkIjfW8="); + fsKeyB = createNewFileSystemWithSSECKey(KEY_4); - fsKeyB.listFiles(S3ATestUtils.createTestPath( - path(createFilename("/a/")) - ), true); - fsKeyB.listFiles(S3ATestUtils.createTestPath( - path(createFilename("/a/b/")) - ), true); + fsKeyB.listFiles(pathA, true); + fsKeyB.listFiles(pathAB, true); //Until this point, no exception is thrown about access - intercept(java.nio.file.AccessDeniedException.class, - "Service: Amazon S3; Status Code: 403;", () -> { - fsKeyB.listFiles(S3ATestUtils.createTestPath( - path(createFilename("/a/b/c/")) - ), false); - throw new Exception("Exception should be thrown."); - }); + intercept(AccessDeniedException.class, + SERVICE_AMAZON_S3_STATUS_CODE_403, + () -> { + fsKeyB.listFiles(pathABC, false); + }); Configuration conf = this.createConfiguration(); conf.unset(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM); @@ -209,22 +220,13 @@ public class ITestS3AEncryptionSSEC extends AbstractTestS3AEncryption { FileSystem unencryptedFileSystem = contract.getTestFileSystem(); //unencrypted can access until the final directory - unencryptedFileSystem.listFiles(S3ATestUtils.createTestPath( - path(createFilename("/a/")) - ), true); - unencryptedFileSystem.listFiles(S3ATestUtils.createTestPath( - path(createFilename("/a/b/")) - ), true); - intercept(org.apache.hadoop.fs.s3a.AWSS3IOException.class, - "Bad Request (Service: Amazon S3; Status Code: 400; Error" + - " Code: 400 Bad Request;", () -> { - - unencryptedFileSystem.listFiles(S3ATestUtils.createTestPath( - path(createFilename("/a/b/c/")) - ), false); - throw new Exception("Exception should be thrown."); - }); - rm(getFileSystem(), path(createFilename("/")), true, false); + unencryptedFileSystem.listFiles(pathA, true); + unencryptedFileSystem.listFiles(pathAB, true); + AWSS3IOException ex = intercept(AWSS3IOException.class, + () -> { + unencryptedFileSystem.listFiles(pathABC, false); + }); + assertStatusCode(ex, 400); } /** @@ -236,31 +238,27 @@ public class ITestS3AEncryptionSSEC extends AbstractTestS3AEncryption { public void testListStatusEncryptedDir() throws Exception { assumeEnabled(); skipIfEncryptionTestsDisabled(getConfiguration()); + assumeS3GuardState(false, getConfiguration()); - Path nestedDirectory = S3ATestUtils.createTestPath( - path(createFilename("/a/b/c/")) - ); - assertTrue(getFileSystem().mkdirs(nestedDirectory)); + Path pathABC = path("testListStatusEncryptedDir/a/b/c/"); + Path pathAB = pathABC.getParent(); + Path pathA = pathAB.getParent(); + assertTrue(getFileSystem().mkdirs(pathABC)); - FileSystem fsKeyB = createNewFileSystemWithSSECKey( - "msdo3VvvZznp66Gth58a91Hxe/UpExMkwU9BHkIjfW8="); + fsKeyB = createNewFileSystemWithSSECKey(KEY_4); - fsKeyB.listStatus(S3ATestUtils.createTestPath( - path(createFilename("/a/")))); - fsKeyB.listStatus(S3ATestUtils.createTestPath( - path(createFilename("/a/b/")))); + fsKeyB.listStatus(pathA); + fsKeyB.listStatus(pathAB); //Until this point, no exception is thrown about access - intercept(java.nio.file.AccessDeniedException.class, - "Service: Amazon S3; Status Code: 403;", () -> { - fsKeyB.listStatus(S3ATestUtils.createTestPath( - path(createFilename("/a/b/c/")))); - - throw new Exception("Exception should be thrown."); + intercept(AccessDeniedException.class, + SERVICE_AMAZON_S3_STATUS_CODE_403, + () -> { + fsKeyB.listStatus(pathABC); }); //Now try it with an unencrypted filesystem. - Configuration conf = this.createConfiguration(); + Configuration conf = createConfiguration(); conf.unset(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM); conf.unset(Constants.SERVER_SIDE_ENCRYPTION_KEY); @@ -269,20 +267,14 @@ public class ITestS3AEncryptionSSEC extends AbstractTestS3AEncryption { FileSystem unencryptedFileSystem = contract.getTestFileSystem(); //unencrypted can access until the final directory - unencryptedFileSystem.listStatus(S3ATestUtils.createTestPath( - path(createFilename("/a/")))); - unencryptedFileSystem.listStatus(S3ATestUtils.createTestPath( - path(createFilename("/a/b/")))); - - intercept(org.apache.hadoop.fs.s3a.AWSS3IOException.class, - "Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400" + - " Bad Request;", () -> { - - unencryptedFileSystem.listStatus(S3ATestUtils.createTestPath( - path(createFilename("/a/b/c/")))); - throw new Exception("Exception should be thrown."); + unencryptedFileSystem.listStatus(pathA); + unencryptedFileSystem.listStatus(pathAB); + + AWSS3IOException ex = intercept(AWSS3IOException.class, + () -> { + unencryptedFileSystem.listStatus(pathABC); }); - rm(getFileSystem(), path(createFilename("/")), true, false); + assertStatusCode(ex, 400); } /** @@ -294,31 +286,24 @@ public class ITestS3AEncryptionSSEC extends AbstractTestS3AEncryption { public void testListStatusEncryptedFile() throws Exception { assumeEnabled(); skipIfEncryptionTestsDisabled(getConfiguration()); + assumeS3GuardState(false, getConfiguration()); + Path pathABC = path("testListStatusEncryptedFile/a/b/c/"); + assertTrue(getFileSystem().mkdirs(pathABC)); - Path nestedDirectory = S3ATestUtils.createTestPath( - path(createFilename("/a/b/c/")) - ); - assertTrue(getFileSystem().mkdirs(nestedDirectory)); - - String src = createFilename("/a/b/c/fileToStat.txt"); - Path fileToStat = writeThenReadFile(src, 2048); + Path fileToStat = new Path(pathABC, "fileToStat.txt"); + writeThenReadFile(fileToStat, TEST_FILE_LEN); - FileSystem fsKeyB = createNewFileSystemWithSSECKey( - "msdo3VvvZznp66Gth58a91Hxe/UpExMkwU9BHkIjfW8="); + fsKeyB = createNewFileSystemWithSSECKey(KEY_4); //Until this point, no exception is thrown about access - intercept(java.nio.file.AccessDeniedException.class, - "Service: Amazon S3; Status Code: 403;", () -> { - fsKeyB.listStatus(S3ATestUtils.createTestPath(fileToStat)); - - throw new Exception("Exception should be thrown."); - }); - rm(getFileSystem(), path(createFilename("/")), true, false); + intercept(AccessDeniedException.class, + SERVICE_AMAZON_S3_STATUS_CODE_403, + () -> { + fsKeyB.listStatus(fileToStat); + }); } - - /** * It is possible to delete directories without the proper encryption key and * the hierarchy above it. @@ -329,31 +314,26 @@ public class ITestS3AEncryptionSSEC extends AbstractTestS3AEncryption { public void testDeleteEncryptedObjectWithDifferentKey() throws Exception { assumeEnabled(); skipIfEncryptionTestsDisabled(getConfiguration()); - - Path nestedDirectory = S3ATestUtils.createTestPath( - path(createFilename("/a/b/c/")) - ); - assertTrue(getFileSystem().mkdirs(nestedDirectory)); - String src = createFilename("/a/b/c/filetobedeleted.txt"); - Path fileToDelete = writeThenReadFile(src, 2048); - - FileSystem fsKeyB = createNewFileSystemWithSSECKey( - "msdo3VvvZznp66Gth58a91Hxe/UpExMkwU9BHkIjfW8="); - intercept(java.nio.file.AccessDeniedException.class, - "Forbidden (Service: Amazon S3; Status Code: 403; Error Code: " + - "403 Forbidden", () -> { - - fsKeyB.delete(fileToDelete, false); - throw new Exception("Exception should be thrown."); - }); + assumeS3GuardState(false, getConfiguration()); + Path pathABC = path("testDeleteEncryptedObjectWithDifferentKey/a/b/c/"); + + Path pathAB = pathABC.getParent(); + Path pathA = pathAB.getParent(); + assertTrue(getFileSystem().mkdirs(pathABC)); + Path fileToDelete = new Path(pathABC, "filetobedeleted.txt"); + writeThenReadFile(fileToDelete, TEST_FILE_LEN); + fsKeyB = createNewFileSystemWithSSECKey(KEY_4); + intercept(AccessDeniedException.class, + SERVICE_AMAZON_S3_STATUS_CODE_403, + () -> { + fsKeyB.delete(fileToDelete, false); + }); //This is possible - fsKeyB.delete(S3ATestUtils.createTestPath( - path(createFilename("/a/b/c/"))), true); - fsKeyB.delete(S3ATestUtils.createTestPath( - path(createFilename("/a/b/"))), true); - fsKeyB.delete(S3ATestUtils.createTestPath( - path(createFilename("/a/"))), true); + fsKeyB.delete(pathABC, true); + fsKeyB.delete(pathAB, true); + fsKeyB.delete(pathA, true); + assertPathDoesNotExist("expected recursive delete", fileToDelete); } private FileSystem createNewFileSystemWithSSECKey(String sseCKey) throws @@ -371,4 +351,5 @@ public class ITestS3AEncryptionSSEC extends AbstractTestS3AEncryption { protected S3AEncryptionMethods getSSEAlgorithm() { return S3AEncryptionMethods.SSE_C; } + } http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileOperationCost.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileOperationCost.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileOperationCost.java index 00171f0..3e293f7 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileOperationCost.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileOperationCost.java @@ -18,6 +18,7 @@ package org.apache.hadoop.fs.s3a; +import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.contract.ContractTestUtils; @@ -32,8 +33,8 @@ import java.net.URI; import static org.apache.hadoop.fs.contract.ContractTestUtils.*; import static org.apache.hadoop.fs.s3a.Statistic.*; import static org.apache.hadoop.fs.s3a.S3ATestUtils.*; -import static org.apache.hadoop.fs.s3a.S3ATestUtils.MetricDiff; import static org.apache.hadoop.test.GenericTestUtils.getTestDir; +import static org.junit.Assume.assumeFalse; /** * Use metrics to assert about the cost of file status queries. @@ -62,9 +63,11 @@ public class ITestS3AFileOperationCost extends AbstractS3ATestBase { S3AFileSystem fs = getFileSystem(); touch(fs, simpleFile); resetMetricDiffs(); - S3AFileStatus status = fs.getFileStatus(simpleFile); + FileStatus status = fs.getFileStatus(simpleFile); assertTrue("not a file: " + status, status.isFile()); - metadataRequests.assertDiffEquals(1); + if (!fs.hasMetadataStore()) { + metadataRequests.assertDiffEquals(1); + } listRequests.assertDiffEquals(0); } @@ -79,9 +82,13 @@ public class ITestS3AFileOperationCost extends AbstractS3ATestBase { Path dir = path("empty"); fs.mkdirs(dir); resetMetricDiffs(); - S3AFileStatus status = fs.getFileStatus(dir); - assertTrue("not empty: " + status, status.isEmptyDirectory()); - metadataRequests.assertDiffEquals(2); + S3AFileStatus status = fs.innerGetFileStatus(dir, true); + assertTrue("not empty: " + status, + status.isEmptyDirectory() == Tristate.TRUE); + + if (!fs.hasMetadataStore()) { + metadataRequests.assertDiffEquals(2); + } listRequests.assertDiffEquals(0); } @@ -92,7 +99,7 @@ public class ITestS3AFileOperationCost extends AbstractS3ATestBase { Path path = path("missing"); resetMetricDiffs(); try { - S3AFileStatus status = fs.getFileStatus(path); + FileStatus status = fs.getFileStatus(path); fail("Got a status back from a missing file path " + status); } catch (FileNotFoundException expected) { // expected @@ -108,7 +115,7 @@ public class ITestS3AFileOperationCost extends AbstractS3ATestBase { Path path = path("missingdir/missingpath"); resetMetricDiffs(); try { - S3AFileStatus status = fs.getFileStatus(path); + FileStatus status = fs.getFileStatus(path); fail("Got a status back from a missing file path " + status); } catch (FileNotFoundException expected) { // expected @@ -126,16 +133,18 @@ public class ITestS3AFileOperationCost extends AbstractS3ATestBase { Path simpleFile = new Path(dir, "simple.txt"); touch(fs, simpleFile); resetMetricDiffs(); - S3AFileStatus status = fs.getFileStatus(dir); - if (status.isEmptyDirectory()) { + S3AFileStatus status = fs.innerGetFileStatus(dir, true); + if (status.isEmptyDirectory() == Tristate.TRUE) { // erroneous state String fsState = fs.toString(); fail("FileStatus says directory isempty: " + status + "\n" + ContractTestUtils.ls(fs, dir) + "\n" + fsState); } - metadataRequests.assertDiffEquals(2); - listRequests.assertDiffEquals(1); + if (!fs.hasMetadataStore()) { + metadataRequests.assertDiffEquals(2); + listRequests.assertDiffEquals(1); + } } @Test @@ -187,6 +196,13 @@ public class ITestS3AFileOperationCost extends AbstractS3ATestBase { + "In S3, rename deletes any fake directories as a part of " + "clean up activity"); S3AFileSystem fs = getFileSystem(); + + // As this test uses the s3 metrics to count the number of fake directory + // operations, it depends on side effects happening internally. With + // metadata store enabled, it is brittle to change. We disable this test + // before the internal behavior w/ or w/o metadata store. + assumeFalse(fs.hasMetadataStore()); + Path srcBaseDir = path("src"); mkdirs(srcBaseDir); MetricDiff deleteRequests = http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileSystemContract.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileSystemContract.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileSystemContract.java index 1b49d07..27af23a 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileSystemContract.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileSystemContract.java @@ -27,6 +27,7 @@ import org.slf4j.LoggerFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystemContractBaseTest; import org.apache.hadoop.fs.Path; + import static org.junit.Assume.*; import static org.junit.Assert.*; http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AInconsistency.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AInconsistency.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AInconsistency.java new file mode 100644 index 0000000..eb4f70b --- /dev/null +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AInconsistency.java @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.contract.AbstractFSContract; +import org.apache.hadoop.fs.contract.s3a.S3AContract; +import org.apache.hadoop.test.LambdaTestUtils; +import org.junit.Test; + +import java.io.FileNotFoundException; +import java.util.concurrent.Callable; + +import static org.apache.hadoop.fs.contract.ContractTestUtils.touch; +import static org.apache.hadoop.fs.s3a.Constants.*; +import static org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client.*; + +/** + * Tests S3A behavior under forced inconsistency via {@link + * InconsistentAmazonS3Client}. + * + * These tests are for validating expected behavior *without* S3Guard, but + * may also run with S3Guard enabled. For tests that validate S3Guard's + * consistency features, see {@link ITestS3GuardListConsistency}. + */ +public class ITestS3AInconsistency extends AbstractS3ATestBase { + + @Override + protected AbstractFSContract createContract(Configuration conf) { + conf.setClass(S3_CLIENT_FACTORY_IMPL, InconsistentS3ClientFactory.class, + S3ClientFactory.class); + conf.set(FAIL_INJECT_INCONSISTENCY_KEY, DEFAULT_DELAY_KEY_SUBSTRING); + conf.setFloat(FAIL_INJECT_INCONSISTENCY_PROBABILITY, 1.0f); + conf.setLong(FAIL_INJECT_INCONSISTENCY_MSEC, DEFAULT_DELAY_KEY_MSEC); + return new S3AContract(conf); + } + + @Test + public void testGetFileStatus() throws Exception { + S3AFileSystem fs = getFileSystem(); + + // 1. Make sure no ancestor dirs exist + Path dir = path("ancestor"); + fs.delete(dir, true); + waitUntilDeleted(dir); + + // 2. Create a descendant file, which implicitly creates ancestors + // This file has delayed visibility. + touch(getFileSystem(), + path("ancestor/file-" + DEFAULT_DELAY_KEY_SUBSTRING)); + + // 3. Assert expected behavior. If S3Guard is enabled, we should be able + // to get status for ancestor. If S3Guard is *not* enabled, S3A will + // fail to infer the existence of the ancestor since visibility of the + // child file is delayed, and its key prefix search will return nothing. + try { + FileStatus status = fs.getFileStatus(dir); + if (fs.hasMetadataStore()) { + assertTrue("Ancestor is dir", status.isDirectory()); + } else { + fail("getFileStatus should fail due to delayed visibility."); + } + } catch (FileNotFoundException e) { + if (fs.hasMetadataStore()) { + fail("S3Guard failed to list parent of inconsistent child."); + } + LOG.info("File not found, as expected."); + } + } + + private void waitUntilDeleted(final Path p) throws Exception { + LambdaTestUtils.eventually(30 * 1000, 1000, + new Callable<Void>() { + @Override + public Void call() throws Exception { + assertPathDoesNotExist("Dir should be deleted", p); + return null; + } + } + ); + } +} http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AMiscOperations.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AMiscOperations.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AMiscOperations.java index 59fcb05..869d64c 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AMiscOperations.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AMiscOperations.java @@ -22,10 +22,17 @@ import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.FileAlreadyExistsException; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.contract.ContractTestUtils; +import org.apache.hadoop.test.LambdaTestUtils; + +import com.amazonaws.services.s3.model.ObjectMetadata; +import com.amazonaws.services.s3.model.PutObjectRequest; +import com.amazonaws.services.s3.model.PutObjectResult; import org.junit.Test; +import java.io.ByteArrayInputStream; import java.io.FileNotFoundException; import java.io.IOException; +import java.util.concurrent.Callable; /** * Tests of the S3A FileSystem which don't have a specific home and can share @@ -55,6 +62,26 @@ public class ITestS3AMiscOperations extends AbstractS3ATestBase { createNonRecursive(new Path(parent, "fail")); } + @Test + public void testPutObjectDirect() throws Throwable { + final S3AFileSystem fs = getFileSystem(); + ObjectMetadata metadata = fs.newObjectMetadata(-1); + metadata.setContentLength(-1); + Path path = path("putDirect"); + final PutObjectRequest put = new PutObjectRequest(fs.getBucket(), + path.toUri().getPath(), + new ByteArrayInputStream("PUT".getBytes()), + metadata); + LambdaTestUtils.intercept(IllegalStateException.class, + new Callable<PutObjectResult>() { + @Override + public PutObjectResult call() throws Exception { + return fs.putObjectDirect(put); + } + }); + assertPathDoesNotExist("put object was created", path); + } + private FSDataOutputStream createNonRecursive(Path path) throws IOException { return getFileSystem().createNonRecursive(path, false, 4096, (short) 3, (short) 4096, http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardCreate.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardCreate.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardCreate.java new file mode 100644 index 0000000..dcc2538 --- /dev/null +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardCreate.java @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a; + +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata; +import org.apache.hadoop.fs.s3a.s3guard.MetadataStore; +import org.junit.Assume; +import org.junit.Test; + +import static org.apache.hadoop.fs.contract.ContractTestUtils.touch; + +/** + * Home for testing the creation of new files and directories with S3Guard + * enabled. + */ +public class ITestS3GuardCreate extends AbstractS3ATestBase { + + /** + * Test that ancestor creation during S3AFileSystem#create() is properly + * accounted for in the MetadataStore. This should be handled by the + * FileSystem, and be a FS contract test, but S3A does not handle ancestors on + * create(), so we need to take care in the S3Guard code to do the right + * thing. This may change: See HADOOP-13221 for more detail. + */ + @Test + public void testCreatePopulatesFileAncestors() throws Exception { + final S3AFileSystem fs = getFileSystem(); + Assume.assumeTrue(fs.hasMetadataStore()); + final MetadataStore ms = fs.getMetadataStore(); + final Path parent = path("testCreatePopulatesFileAncestors"); + + try { + fs.mkdirs(parent); + final Path nestedFile = new Path(parent, "dir1/dir2/file4"); + touch(fs, nestedFile); + + DirListingMetadata list = ms.listChildren(parent); + assertFalse("MetadataStore falsely reports authoritative empty list", + list.isEmpty() == Tristate.TRUE); + } finally { + fs.delete(parent, true); + } + } +} http://git-wip-us.apache.org/repos/asf/hadoop/blob/621b43e2/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardEmptyDirs.java ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardEmptyDirs.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardEmptyDirs.java new file mode 100644 index 0000000..fb6e370 --- /dev/null +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardEmptyDirs.java @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a; + +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.s3a.s3guard.MetadataStore; +import org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore; +import org.junit.Assume; +import org.junit.Test; + +import static org.apache.hadoop.fs.contract.ContractTestUtils.touch; + +/** + * Test logic around whether or not a directory is empty, with S3Guard enabled. + * The fact that S3AFileStatus has an isEmptyDirectory flag in it makes caching + * S3AFileStatus's really tricky, as the flag can change as a side effect of + * changes to other paths. + * After S3Guard is merged to trunk, we should try to remove the + * isEmptyDirectory flag from S3AFileStatus, or maintain it outside + * of the MetadataStore. + */ +public class ITestS3GuardEmptyDirs extends AbstractS3ATestBase { + + @Test + public void testEmptyDirs() throws Exception { + S3AFileSystem fs = getFileSystem(); + Assume.assumeTrue(fs.hasMetadataStore()); + MetadataStore configuredMs = fs.getMetadataStore(); + Path existingDir = path("existing-dir"); + Path existingFile = path("existing-dir/existing-file"); + try { + // 1. Simulate files already existing in the bucket before we started our + // cluster. Temporarily disable the MetadataStore so it doesn't witness + // us creating these files. + + fs.setMetadataStore(new NullMetadataStore()); + assertTrue(fs.mkdirs(existingDir)); + touch(fs, existingFile); + + + // 2. Simulate (from MetadataStore's perspective) starting our cluster and + // creating a file in an existing directory. + fs.setMetadataStore(configuredMs); // "start cluster" + Path newFile = path("existing-dir/new-file"); + touch(fs, newFile); + + S3AFileStatus status = fs.innerGetFileStatus(existingDir, true); + assertEquals("Should not be empty dir", Tristate.FALSE, + status.isEmptyDirectory()); + + // 3. Assert that removing the only file the MetadataStore witnessed + // being created doesn't cause it to think the directory is now empty. + fs.delete(newFile, false); + status = fs.innerGetFileStatus(existingDir, true); + assertEquals("Should not be empty dir", Tristate.FALSE, + status.isEmptyDirectory()); + + // 4. Assert that removing the final file, that existed "before" + // MetadataStore started, *does* cause the directory to be marked empty. + fs.delete(existingFile, false); + status = fs.innerGetFileStatus(existingDir, true); + assertEquals("Should be empty dir now", Tristate.TRUE, + status.isEmptyDirectory()); + } finally { + configuredMs.forgetMetadata(existingFile); + configuredMs.forgetMetadata(existingDir); + } + } +} --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
