[
https://issues.apache.org/jira/browse/HADOOP-17409?focusedWorklogId=667779&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-667779
]
ASF GitHub Bot logged work on HADOOP-17409:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 20/Oct/21 14:02
Start Date: 20/Oct/21 14:02
Worklog Time Spent: 10m
Work Description: bogthe commented on a change in pull request #3534:
URL: https://github.com/apache/hadoop/pull/3534#discussion_r732739468
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -3587,116 +3399,13 @@ S3AFileStatus innerGetFileStatus(final Path f,
String key = pathToKey(path);
LOG.debug("Getting path status for {} ({}); needEmptyDirectory={}",
path, key, needEmptyDirectoryFlag);
+ // there was no entry in S3Guard
+ // retrieve the data and update the metadata store in the process.
Review comment:
Does this still update the metadata store? If not, the description of
this method can be updated too since it mentions S3Guard too
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##########
@@ -494,27 +497,25 @@ private Constants() {
public static final String CUSTOM_SIGNERS = "fs.s3a.custom.signers";
/**
- * There's 3 parameters that can be used to specify a non-default signing
+ * Multiple can be used to specify a non-default signing
Review comment:
nit: + `parameters`
##########
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##########
@@ -554,7 +439,7 @@ public void checkLocatedFileStatusScanFile() throws
Throwable {
true,
TEXT_FILE,
true);
- accessDeniedIf(!guardedInAuthMode,
+ accessDeniedIf(!false,
Review comment:
nit: `!false` => `true`
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
##########
@@ -84,705 +89,31 @@ Once you are confident that all applications have been
restarted, _Delete the Dy
This is to avoid paying for a database you no longer need.
This is best done from the AWS GUI.
-## Setting up S3Guard
-
-### S3A to warn or fail if S3Guard is disabled
-A seemingly recurrent problem with S3Guard is that people think S3Guard is
-turned on but it isn't.
-You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level`
-to avoid this. The property sets what to do when an S3A FS is instantiated
-without S3Guard. The following values are available:
-
-* `SILENT`: Do nothing.
-* `INFORM`: Log at info level that FS is instantiated without S3Guard.
-* `WARN`: Warn that data may be at risk in workflows.
-* `FAIL`: S3AFileSystem instantiation will fail.
-
-The default setting is `SILENT`. The setting is case insensitive.
-The required level can be set in the `core-site.xml`.
-
----
-The latest configuration parameters are defined in `core-default.xml`. You
-should consult that file for full information, but a summary is provided here.
-
+## Removing S3Guard Configurations
-### 1. Choose the Database
+The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty
string "",
+or to the "Null" Metadata store
`org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`.
-A core concept of S3Guard is that the directory listing data of the object
-store, *the metadata* is replicated in a higher-performance, consistent,
-database. In S3Guard, this database is called *The Metadata Store*
-
-By default, S3Guard is not enabled.
-
-The Metadata Store to use in production is bonded to Amazon's DynamoDB
-database service. The following setting will enable this Metadata Store:
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
+ <value></value>
</property>
```
-Note that the `NullMetadataStore` store can be explicitly requested if desired.
-This offers no metadata storage, and effectively disables S3Guard.
-
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
<value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
</property>
```
-### 2. Configure S3Guard Settings
-
-More settings will may be added in the future.
-Currently the only Metadata Store-independent setting, besides the
-implementation class above, are the *allow authoritative* and *fail-on-error*
-flags.
-
-#### <a name="authoritative"></a> Authoritative S3Guard
-
-Authoritative S3Guard is a complicated configuration which delivers performance
-at the expense of being unsafe for other applications to use the same directory
-tree/bucket unless configured consistently.
-
-It can also be used to support [directory marker
retention](directory_markers.html)
-in higher-performance but non-backwards-compatible modes.
-
-Most deployments do not use this setting -it is ony used in deployments where
-specific parts of a bucket (e.g. Apache Hive managed tables) are known to
-have exclusive access by a single application (Hive) and other
tools/applications
-from exactly the same Hadoop release.
-
-The _authoritative_ expression in S3Guard is present in two different layers,
for
-two different reasons:
-
-* Authoritative S3Guard
- * S3Guard can be set as authoritative, which means that an S3A client will
- avoid round-trips to S3 when **getting file metadata**, and **getting
- directory listings** if there is a fully cached version of the directory
- stored in metadata store.
- * This mode can be set as a configuration property
- `fs.s3a.metadatastore.authoritative`
- * It can also be set only on specific directories by setting
- `fs.s3a.authoritative.path` to one or more prefixes, for example
- `s3a://bucket/path` or "/auth1,/auth2".
- * All interactions with the S3 bucket(s) must be through S3A clients
sharing
- the same metadata store.
- * This is independent from which metadata store implementation is used.
- * In authoritative mode the metadata TTL metadata expiry is not effective.
- This means that the metadata entries won't expire on authoritative paths.
-
-* Authoritative directory listings (isAuthoritative bit)
- * Tells if the stored directory listing metadata is complete.
- * This is set by the FileSystem client (e.g. s3a) via the
`DirListingMetadata`
- class (`org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata`).
- (The MetadataStore only knows what the FS client tells it.)
- * If set to `TRUE`, we know that the directory listing
- (`DirListingMetadata`) is full, and complete.
- * If set to `FALSE` the listing may not be complete.
- * Metadata store may persist the isAuthoritative bit on the metadata store.
- * Currently `org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore` and
- `org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore` implementation
- supports authoritative bit.
-
-More on Authoritative S3Guard:
-
-* This setting is about treating the MetadataStore (e.g. dynamodb) as the
source
- of truth in general, and also to short-circuit S3 list objects and serve
- listings from the MetadataStore in some circumstances.
-* For S3A to skip S3's get object metadata, and serve it directly from the
-MetadataStore, the following things must all be true:
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a file metadata (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has the file metadata for the path stored in it.
-* For S3A to skip S3's list objects on some path, and serve it directly from
-the MetadataStore, the following things must all be true:
- 1. The MetadataStore implementation persists the bit
- `DirListingMetadata.isAuthorititative` set when calling
- `MetadataStore#put` (`DirListingMetadata`)
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a directory listing (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has a **full listing for path** stored in it. This
only
- happens if the FS client (s3a) explicitly has stored a full directory
- listing with `DirListingMetadata.isAuthorititative=true` before the said
- listing request happens.
-
-This configuration only enables authoritative mode in the client layer. It is
-recommended that you leave the default setting here:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.authoritative</name>
- <value>false</value>
-</property>
-```
-
-Note that a MetadataStore MAY persist this bit in the directory listings. (Not
-MUST).
-
-Note that if this is set to true, it may exacerbate or persist existing race
-conditions around multiple concurrent modifications and listings of a given
-directory tree.
-
-In particular: **If the Metadata Store is declared as authoritative,
-all interactions with the S3 bucket(s) must be through S3A clients sharing
-the same Metadata Store**
-
-#### TTL metadata expiry
-
-It can be configured how long an entry is valid in the MetadataStore
-**if the authoritative mode is turned off**, or the path is not
-configured to be authoritative.
-If `((lastUpdated + ttl) <= now)` is false for an entry, the entry will
-be expired, so the S3 bucket will be queried for fresh metadata.
-The time for expiry of metadata can be set as the following:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.metadata.ttl</name>
- <value>15m</value>
-</property>
-```
-
-#### Fail on Error
-
-By default, S3AFileSystem write operations will fail when updates to
-S3Guard metadata fail. S3AFileSystem first writes the file to S3 and then
-updates the metadata in S3Guard. If the metadata write fails,
-`MetadataPersistenceException` is thrown. The file in S3 **is not** rolled
-back.
-
-If the write operation cannot be programmatically retried, the S3Guard metadata
-for the given file can be corrected with a command like the following:
-
-```bash
-hadoop s3guard import [-meta URI] s3a://my-bucket/file-with-bad-metadata
-```
-
-Programmatic retries of the original operation would require overwrite=true.
-Suppose the original operation was `FileSystem.create(myFile,
overwrite=false)`.
-If this operation failed with `MetadataPersistenceException` a repeat of the
-same operation would result in `FileAlreadyExistsException` since the original
-operation successfully created the file in S3 and only failed in writing the
-metadata to S3Guard.
-
-Metadata update failures can be downgraded to ERROR logging instead of
exception
-by setting the following configuration:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.fail.on.write.error</name>
- <value>false</value>
-</property>
-```
-
-Setting this false is dangerous as it could result in the type of issue S3Guard
-is designed to avoid. For example, a reader may see an inconsistent listing
-after a recent write since S3Guard may not contain metadata about the recently
-written file due to a metadata write error.
-
-As with the default setting, the new/updated file is still in S3 and **is not**
-rolled back. The S3Guard metadata is likely to be out of sync.
-
-### 3. Configure the Metadata Store.
-
-Here are the `DynamoDBMetadataStore` settings. Other Metadata Store
-implementations will have their own configuration parameters.
-
-
-### 4. Name Your Table
-
-First, choose the name of the table you wish to use for the S3Guard metadata
-storage in your DynamoDB instance. If you leave it unset/empty, a
-separate table will be created for each S3 bucket you access, and that
-bucket's name will be used for the name of the DynamoDB table. For example,
-this sets the table name to `my-ddb-table-name`
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table</name>
- <value>my-ddb-table-name</value>
- <description>
- The DynamoDB table name to operate. Without this property, the respective
- S3 bucket names will be used.
- </description>
-</property>
-```
-
-It is good to share a table across multiple buckets for multiple reasons,
-especially if you are *not* using on-demand DynamoDB tables, and instead
-prepaying for provisioned I/O capacity.
-
-1. You are billed for the provisioned I/O capacity allocated to the table,
-*even when the table is not used*. Sharing capacity can reduce costs.
-
-1. You can share the "provision burden" across the buckets. That is, rather
-than allocating for the peak load on a single bucket, you can allocate for
-the peak load *across all the buckets*, which is likely to be significantly
-lower.
-
-1. It's easier to measure and tune the load requirements and cost of
-S3Guard, because there is only one table to review and configure in the
-AWS management console.
-
-1. When you don't grant the permission to create DynamoDB tables to users.
-A single pre-created table for all buckets avoids the needs for an
administrator
-to create one for every bucket.
-
-When wouldn't you want to share a table?
-
-1. When you are using on-demand DynamoDB and want to keep each table isolated.
-1. When you do explicitly want to provision I/O capacity to a specific bucket
-and table, isolated from others.
-
-1. When you are using separate billing for specific buckets allocated
-to specific projects.
-
-1. When different users/roles have different access rights to different
buckets.
-As S3Guard requires all users to have R/W access to the table, all users will
-be able to list the metadata in all buckets, even those to which they lack
-read access.
-
-### 5. Locate your Table
-
-You may also wish to specify the region to use for DynamoDB. If a region
-is not configured, S3A will assume that it is in the same region as the S3
-bucket. A list of regions for the DynamoDB service can be found in
-[Amazon's
documentation](http://docs.aws.amazon.com/general/latest/gr/rande.html#ddb_region).
-In this example, to use the US West 2 region:
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.region</name>
- <value>us-west-2</value>
-</property>
-```
-
-When working with S3Guard-managed buckets from EC2 VMs running in AWS
-infrastructure, using a local DynamoDB region ensures the lowest latency
-and highest reliability, as well as avoiding all long-haul network charges.
-The S3Guard tables, and indeed, the S3 buckets, should all be in the same
-region as the VMs.
-
-### 6. Optional: Create your Table
-
-Next, you can choose whether or not the table will be automatically created
-(if it doesn't already exist). If you want this feature, set the
-`fs.s3a.s3guard.ddb.table.create` option to `true`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.create</name>
- <value>true</value>
- <description>
- If true, the S3A client will create the table if it does not already exist.
- </description>
-</property>
-```
-
-### 7. If creating a table: Choose your billing mode (and perhaps I/O Capacity)
-
-Next, you need to decide whether to use On-Demand DynamoDB and its
-pay-per-request billing (recommended), or to explicitly request a
-provisioned IO capacity.
-
-Before AWS offered pay-per-request billing, the sole billing mechanism,
-was "provisioned capacity". This mechanism requires you to choose
-the DynamoDB read and write throughput requirements you
-expect to need for your expected uses of the S3Guard table.
-Setting higher values cost you more money -*even when the table was idle*
- *Note* that these settings only affect table creation when
-`fs.s3a.s3guard.ddb.table.create` is enabled. To change the throughput for
-an existing table, use the AWS console or CLI tool.
-
-For more details on DynamoDB capacity units, see the AWS page on [Capacity
-Unit
Calculations](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html#CapacityUnitCalculations).
-
-Provisioned IO capacity is billed per hour for the life of the table, *even
when the
-table and the underlying S3 buckets are not being used*.
-
-There are also charges incurred for data storage and for data I/O outside of
the
-region of the DynamoDB instance. S3Guard only stores metadata in DynamoDB:
path names
-and summary details of objects —the actual data is stored in S3, so billed at
S3
-rates.
-
-With provisioned I/O capacity, attempting to perform more I/O than the capacity
-requested throttles the operation and may result in operations failing.
-Larger I/O capacities cost more.
-
-With the introduction of On-Demand DynamoDB, you can now avoid paying for
-provisioned capacity by creating an on-demand table.
-With an on-demand table you are not throttled if your DynamoDB requests exceed
-any pre-provisioned limit, nor do you pay per hour even when a table is idle.
-
-You do, however, pay more per DynamoDB operation.
-Even so, the ability to cope with sudden bursts of read or write requests,
combined
-with the elimination of charges for idle tables, suit the use patterns made of
-S3Guard tables by applications interacting with S3. That is: periods when the
table
-is rarely used, with intermittent high-load operations when directory trees
-are scanned (query planning and similar), or updated (rename and delete
operations).
-
-
-We recommending using On-Demand DynamoDB for maximum performance in operations
-such as query planning, and lowest cost when S3 buckets are not being accessed.
-
-This is the default, as configured in the default configuration options.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.read</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for read operations in terms of
capacity
- units for the DynamoDB table. This config value will only be used when
- creating a new DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request"
capacity.
- If a positive integer is provided for this and the write capacity, then
- a table with "provisioned capacity" will be created.
- You can change the capacity of an existing provisioned-capacity table
- through the "s3guard set-capacity" command.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.write</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for write operations in terms of
- capacity units for the DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request"
capacity.
- Refer to related configuration option
fs.s3a.s3guard.ddb.table.capacity.read
- </description>
-</property>
-```
-
-### 8. If creating a table: Enable server side encryption (SSE)
-
-Encryption at rest can help you protect sensitive data in your DynamoDB table.
-When creating a new table, you can set server side encryption on the table
-using the default AWS owned customer master key (CMK), AWS managed CMK, or
-customer managed CMK. S3Guard code accessing the table is all the same whether
-SSE is enabled or not. For more details on DynamoDB table server side
-encryption, see the AWS page on [Encryption at Rest: How It
Works](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.howitworks.html).
-
-These are the default configuration options, as configured in
`core-default.xml`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.enabled</name>
- <value>false</value>
- <description>
- Whether server-side encryption (SSE) is enabled or disabled on the table.
- By default it's disabled, meaning SSE is set to AWS owned CMK.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.cmk</name>
- <value/>
- <description>
- The KMS Customer Master Key (CMK) used for the KMS encryption on the table.
- To specify a CMK, this config value can be its key ID, Amazon Resource Name
- (ARN), alias name, or alias ARN. Users only need to provide this config if
- the key is different from the default DynamoDB KMS Master Key, which is
- alias/aws/dynamodb.
- </description>
-</property>
-```
-
-## Authenticating with S3Guard
-
-The DynamoDB metadata store takes advantage of the fact that the DynamoDB
-service uses the same authentication mechanisms as S3. S3Guard
-gets all its credentials from the S3A client that is using it.
-
-All existing S3 authentication mechanisms can be used.
-
-## Per-bucket S3Guard configuration
-
-In production, it is likely only some buckets will have S3Guard enabled;
-those which are read-only may have disabled, for example. Equally importantly,
-buckets in different regions should have different tables, each
-in the relevant region.
-
-These options can be managed through S3A's [per-bucket configuration
-mechanism](./index.html#Configuring_different_S3_buckets).
-All options with the under `fs.s3a.bucket.BUCKETNAME.KEY` are propagated
-to the options `fs.s3a.KEY` *for that bucket only*.
-
-As an example, here is a configuration to use different metadata stores
-and tables for different buckets
-
-First, we define shortcuts for the metadata store classnames:
-
-
-```xml
-<property>
- <name>s3guard.null</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
-</property>
-
-<property>
- <name>s3guard.dynamo</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
-</property>
-```
-
-Next, Amazon's public landsat database is configured with no
-metadata store:
-
-```xml
-<property>
- <name>fs.s3a.bucket.landsat-pds.metadatastore.impl</name>
- <value>${s3guard.null}</value>
- <description>The read-only landsat-pds repository isn't
- managed by S3Guard</description>
-</property>
-```
-
-Next the `ireland-2` and `ireland-offline` buckets are configured with
-DynamoDB as the store, and a shared table `production-table`:
-
-
-```xml
-<property>
- <name>fs.s3a.bucket.ireland-2.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-offline.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-2.s3guard.ddb.table</name>
- <value>production-table</value>
-</property>
-```
-
-The region of this table is automatically set to be that of the buckets,
-here `eu-west-1`; the same table name may actually be used in different
-regions.
-
-Together then, this configuration enables the DynamoDB Metadata Store
-for two buckets with a shared table, while disabling it for the public
-bucket.
-
-
-### Out-of-band operations with S3Guard
-
-We call an operation out-of-band (OOB) when a bucket is used by a client with
- S3Guard, and another client runs a write (e.g delete, move, rename,
- overwrite) operation on an object in the same bucket without S3Guard.
-
-The definition of behaviour in S3AFileSystem/MetadataStore in case of OOBs:
-* A client with S3Guard
-* B client without S3Guard (Directly to S3)
-
-
-* OOB OVERWRITE, authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F1 metadata
-
-* OOB OVERWRITE, NOT authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F2 metadata. In not authoritative mode we
- check S3 for the file. If the modification time of the file in S3 is greater
- than in S3Guard, we can safely return the S3 file metadata and update the
- cache.
-
-* OOB DELETE, authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-* OOB DELETE, NOT authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-Note: authoritative and NOT authoritative mode behaves the same at
-OOB DELETE case.
-
-The behaviour in case of getting directory listings:
-* File status in metadata store gets updated during the listing the same way
-as in getFileStatus.
-
## S3Guard Command Line Interface (CLI)
-Note that in some cases an AWS region or `s3a://` URI can be provided.
-
-Metadata store URIs include a scheme that designates the backing store. For
-example (e.g. `dynamodb://table_name`;). As documented above, the
-AWS region can be inferred if the URI to an existing bucket is provided.
-
-
-The S3A URI must also be provided for per-bucket configuration options
-to be picked up. That is: when an s3a URL is provided on the command line,
-all its "resolved" per-bucket settings are used to connect to, authenticate
-with and configure the S3Guard table. If no such URL is provided, then
-the base settings are picked up.
-
-
-### Create a table: `s3guard init`
-
-```bash
-hadoop s3guard init -meta URI ( -region REGION | s3a://BUCKET )
-```
-
-Creates and initializes an empty metadata store.
-
-A DynamoDB metadata store can be initialized with additional parameters
-pertaining to capacity.
-
-If these values are both zero, then an on-demand DynamoDB table is created;
-if positive values then they set the
-[Provisioned
Throughput](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html)
-of the table.
-
-
-```bash
-[-write PROVISIONED_WRITES] [-read PROVISIONED_READS]
-```
-
-Server side encryption (SSE) can be enabled with AWS managed customer master
key
-(CMK), or customer managed CMK. By default the DynamoDB table will be encrypted
-with AWS owned CMK. To use a customer managed CMK, you can specify its KMS key
-ID, ARN, alias name, or alias ARN. If not specified, the default AWS managed
CMK
-for DynamoDB "alias/aws/dynamodb" will be used.
-
-```bash
-[-sse [-cmk KMS_CMK_ID]]
-```
-
-Tag argument can be added with a key=value list of tags. The table for the
-metadata store will be created with these tags in DynamoDB.
-
-```bash
-[-tag key=value;]
-```
-
-
-Example 1
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -write 0 -read 0
s3a://ireland-1
-```
-
-Creates an on-demand table "ireland-team",
-in the same location as the S3 bucket "ireland-1".
-
-
-Example 2
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -region eu-west-1 --read 0
--write 0
-```
-
-Creates a table "ireland-team" in the region "eu-west-1.amazonaws.com"
-
-
-Example 3
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -tag tag1=first;tag2=second;
-```
-
-Creates a table "ireland-team" with tags "first" and "second". The read and
-write capacity will be those of the site configuration's values of
-`fs.s3a.s3guard.ddb.table.capacity.read` and
`fs.s3a.s3guard.ddb.table.capacity.write`;
-if these are both zero then it will be an on-demand table.
-
-
-Example 4
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -sse
-```
-
-Creates a table "ireland-team" with server side encryption enabled. The CMK
will
-be using the default AWS managed "alias/aws/dynamodb".
-
-
-### Import a bucket: `s3guard import`
-
-```bash
-hadoop s3guard import [-meta URI] [-authoritative] [-verbose] s3a://PATH
-```
-
-Pre-populates a metadata store according to the current contents of an S3
-bucket/path. If the `-meta` option is omitted, the binding information is taken
-from the `core-site.xml` configuration.
-
-Usage
+The `h`
Review comment:
Continuation missing?
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##########
@@ -494,27 +497,25 @@ private Constants() {
public static final String CUSTOM_SIGNERS = "fs.s3a.custom.signers";
/**
- * There's 3 parameters that can be used to specify a non-default signing
+ * Multiple can be used to specify a non-default signing
* algorithm.<br>
* fs.s3a.signing-algorithm - This property has existed for the longest time.
- * If specified, without either of the other 2 properties being specified,
- * this signing algorithm will be used for S3 and DDB (S3Guard). <br>
- * The other 2 properties override this value for S3 or DDB. <br>
+ * If specified, without other properties being specified,
+ * this signing algorithm will be used for all services. <br>
+ * Another property overridea this value for S3. <br>
Review comment:
nit: `overridea` -> `overrides`
##########
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##########
@@ -456,21 +349,13 @@ public void checkGlobOperations() throws Throwable {
// baseline: the real filesystem on a subdir
globFS(getFileSystem(), subdirFile, null, false, 1);
// a file fails if not in auth mode
- globFS(readonlyFS, subdirFile, null, !guardedInAuthMode, 1);
+ globFS(readonlyFS, subdirFile, null, !false, 1);
Review comment:
nit: `!false` => `true`
##########
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##########
@@ -422,7 +315,7 @@ public void checkBasicFileOperations() throws Throwable {
readonlyFS.getFileStatus(emptyDir);
// now look at a file; the outcome depends on the mode.
- accessDeniedIf(!guardedInAuthMode, () ->
+ accessDeniedIf(!false, () ->
Review comment:
nit: `!false` => `true`
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java
##########
@@ -18,1003 +18,85 @@
package org.apache.hadoop.fs.s3a.s3guard;
-import java.io.FileNotFoundException;
-import java.io.IOException;
import java.net.URI;
import java.util.ArrayList;
import java.util.Collection;
-import java.util.Collections;
-import java.util.List;
-import java.util.Locale;
-import java.util.Map;
-import java.util.Objects;
-import java.util.Set;
-import java.util.concurrent.TimeUnit;
import java.util.function.Function;
-import java.util.stream.Collectors;
-
-import javax.annotation.Nullable;
import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.thirdparty.com.google.common.base.Preconditions;
-import org.apache.hadoop.fs.RemoteIterator;
-import org.apache.hadoop.fs.s3a.S3AFileSystem;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.s3a.Retries;
-import org.apache.hadoop.fs.s3a.Retries.RetryTranslated;
-import org.apache.hadoop.fs.s3a.S3AFileStatus;
-import org.apache.hadoop.io.IOUtils;
-import org.apache.hadoop.util.ExitUtil;
-import org.apache.hadoop.util.ReflectionUtils;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.s3a.Constants;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
-import static org.apache.hadoop.fs.s3a.Constants.*;
+import static org.apache.hadoop.fs.s3a.Constants.AUTHORITATIVE_PATH;
import static org.apache.hadoop.fs.s3a.Constants.DEFAULT_AUTHORITATIVE_PATH;
-import static org.apache.hadoop.fs.s3a.S3AUtils.createUploadFileStatus;
-import static
org.apache.hadoop.fs.s3a.s3guard.PathMetadataDynamoDBTranslation.authoritativeEmptyDirectoryMarker;
-import static
org.apache.hadoop.service.launcher.LauncherExitCodes.EXIT_BAD_CONFIGURATION;
/**
* Logic for integrating MetadataStore with S3A.
+ * Now that S3Guard is removed it is retained for the final
+ * end-of-S3Guard calls and when classes may be needed for
+ * compatibilty in external tests.
*/
+@SuppressWarnings("deprecation")
@InterfaceAudience.Private
@InterfaceStability.Unstable
public final class S3Guard {
private static final Logger LOG = LoggerFactory.getLogger(S3Guard.class);
- @InterfaceAudience.Private
- @InterfaceStability.Unstable
- @VisibleForTesting
- public static final String S3GUARD_DDB_CLIENT_FACTORY_IMPL =
- "fs.s3a.s3guard.ddb.client.factory.impl";
-
- static final Class<? extends DynamoDBClientFactory>
- S3GUARD_DDB_CLIENT_FACTORY_IMPL_DEFAULT =
- DynamoDBClientFactory.DefaultDynamoDBClientFactory.class;
- private static final S3AFileStatus[] EMPTY_LISTING = new S3AFileStatus[0];
-
- /**
- * Hard-coded policy : {@value}.
- * If true, when merging an S3 LIST with S3Guard in non-auth mode,
- * only updated entries are added; new entries are left out.
- * This policy choice reduces the amount of data stored in Dynamo,
- * and hence the complexity of the merge in a non-auth listing.
- */
- @VisibleForTesting
- public static final boolean DIR_MERGE_UPDATES_ALL_RECORDS_NONAUTH = false;
-
// Utility class. All static functions.
- private S3Guard() { }
-
- /* Utility functions. */
-
- /**
- * Create a new instance of the configured MetadataStore.
- * The returned MetadataStore will have been initialized via
- * {@link MetadataStore#initialize(FileSystem, ITtlTimeProvider)}
- * by this function before returning it. Callers must clean up by calling
- * {@link MetadataStore#close()} when done using the MetadataStore.
- *
- * @param fs FileSystem whose Configuration specifies which
- * implementation to use.
- * @param ttlTimeProvider
- * @return Reference to new MetadataStore.
- * @throws IOException if the metadata store cannot be instantiated
- */
- @Retries.OnceTranslated
- public static MetadataStore getMetadataStore(FileSystem fs,
- ITtlTimeProvider ttlTimeProvider)
- throws IOException {
- Preconditions.checkNotNull(fs);
- Configuration conf = fs.getConf();
- Preconditions.checkNotNull(conf);
- MetadataStore msInstance;
- try {
- Class<? extends MetadataStore> msClass = getMetadataStoreClass(conf);
- msInstance = ReflectionUtils.newInstance(msClass, conf);
- LOG.debug("Using {} metadata store for {} filesystem",
- msClass.getSimpleName(), fs.getScheme());
- msInstance.initialize(fs, ttlTimeProvider);
- return msInstance;
- } catch (FileNotFoundException e) {
- // Don't log this exception as it means the table doesn't exist yet;
- // rely on callers to catch and treat specially
- throw e;
- } catch (RuntimeException | IOException e) {
- String message = "Failed to instantiate metadata store " +
- conf.get(S3_METADATA_STORE_IMPL)
- + " defined in " + S3_METADATA_STORE_IMPL
- + ": " + e;
- LOG.error(message, e);
- if (e instanceof IOException) {
- throw e;
- } else {
- throw new IOException(message, e);
- }
- }
- }
-
- static Class<? extends MetadataStore> getMetadataStoreClass(
- Configuration conf) {
- if (conf == null) {
- return NullMetadataStore.class;
- }
- if (conf.get(S3_METADATA_STORE_IMPL) != null && LOG.isDebugEnabled()) {
- LOG.debug("Metastore option source {}",
- (Object)conf.getPropertySources(S3_METADATA_STORE_IMPL));
- }
-
- Class<? extends MetadataStore> aClass = conf.getClass(
- S3_METADATA_STORE_IMPL, NullMetadataStore.class,
- MetadataStore.class);
- return aClass;
- }
-
-
- /**
- * We update the metastore for the specific case of S3 value == S3Guard value
- * so as to place a more recent modtime in the store.
- * because if not, we will continue to probe S3 whenever we look for this
- * object, even we only do this if confident the S3 status is the same
- * as the one in the store (i.e. it is not an older version)
- * @param metadataStore MetadataStore to {@code put()} into.
- * @param pm current data
- * @param s3AFileStatus status to store
- * @param timeProvider Time provider to use when writing entries
- * @return true if the entry was updated.
- * @throws IOException if metadata store update failed
- */
- @RetryTranslated
- public static boolean refreshEntry(
- MetadataStore metadataStore,
- PathMetadata pm,
- S3AFileStatus s3AFileStatus,
- ITtlTimeProvider timeProvider) throws IOException {
- // the modtime of the data is the same as/older than the s3guard value
- // either an old object has been found, or the existing one was retrieved
- // in both cases -return s3guard value
- S3AFileStatus msStatus = pm.getFileStatus();
-
- // first check: size
- boolean sizeMatch = msStatus.getLen() == s3AFileStatus.getLen();
-
- // etags are expected on all objects, but handle the situation
- // that a third party store doesn't serve them.
- String s3Etag = s3AFileStatus.getETag();
- String pmEtag = msStatus.getETag();
- boolean etagsMatch = s3Etag != null && s3Etag.equals(pmEtag);
-
- // version ID: only in some stores, and will be missing in the metastore
- // if the entry was created through a list operation.
- String s3VersionId = s3AFileStatus.getVersionId();
- String pmVersionId = msStatus.getVersionId();
- boolean versionsMatchOrMissingInMetastore =
- pmVersionId == null || pmVersionId.equals(s3VersionId);
- if (sizeMatch && etagsMatch && versionsMatchOrMissingInMetastore) {
- // update the store, return the new value
- LOG.debug("Refreshing the metastore entry/timestamp");
- putAndReturn(metadataStore, s3AFileStatus, timeProvider);
- return true;
- }
- return false;
- }
-
- /**
- * Helper function which puts a given S3AFileStatus into the MetadataStore
and
- * returns the same S3AFileStatus. Instrumentation monitors the put
operation.
- * @param ms MetadataStore to {@code put()} into.
- * @param status status to store
- * @param timeProvider Time provider to use when writing entries
- * @return The same status as passed in
- * @throws IOException if metadata store update failed
- */
- @RetryTranslated
- public static S3AFileStatus putAndReturn(MetadataStore ms,
- S3AFileStatus status,
- ITtlTimeProvider timeProvider) throws IOException {
- return putAndReturn(ms, status, timeProvider, null);
- }
-
- /**
- * Helper function which puts a given S3AFileStatus into the MetadataStore
and
- * returns the same S3AFileStatus. Instrumentation monitors the put
operation.
- * @param ms MetadataStore to {@code put()} into.
- * @param status status to store
- * @param timeProvider Time provider to use when writing entries
- * @param operationState possibly-null metastore state tracker.
- * @return The same status as passed in
- * @throws IOException if metadata store update failed
- */
- @RetryTranslated
- public static S3AFileStatus putAndReturn(
- final MetadataStore ms,
- final S3AFileStatus status,
- final ITtlTimeProvider timeProvider,
- @Nullable final BulkOperationState operationState) throws IOException {
- long startTimeNano = System.nanoTime();
- try {
- putWithTtl(ms, new PathMetadata(status), timeProvider, operationState);
- } finally {
- ms.getInstrumentation().entryAdded((System.nanoTime() - startTimeNano));
- }
- return status;
- }
-
- /**
- * Creates an authoritative directory marker for the store.
- * @param ms MetadataStore to {@code put()} into.
- * @param status status to store
- * @param timeProvider Time provider to use when writing entries
- * @param operationState possibly-null metastore state tracker.
- * @throws IOException if metadata store update failed
- */
- @RetryTranslated
- public static void putAuthDirectoryMarker(
- final MetadataStore ms,
- final S3AFileStatus status,
- final ITtlTimeProvider timeProvider,
- @Nullable final BulkOperationState operationState) throws IOException {
- long startTimeNano = System.nanoTime();
- try {
- final PathMetadata fileMeta = authoritativeEmptyDirectoryMarker(status);
- putWithTtl(ms, fileMeta, timeProvider, operationState);
- } finally {
- ms.getInstrumentation().directoryMarkedAuthoritative();
- ms.getInstrumentation().entryAdded((System.nanoTime() - startTimeNano));
- }
- }
-
- /**
- * Initiate a bulk write and create an operation state for it.
- * This may then be passed into put operations.
- * @param metastore store
- * @param operation the type of the operation.
- * @param path path under which updates will be explicitly put.
- * @return a store-specific state to pass into the put operations, or null
- * @throws IOException failure
- */
- public static BulkOperationState initiateBulkWrite(
- @Nullable final MetadataStore metastore,
- final BulkOperationState.OperationType operation,
- final Path path) throws IOException {
- Preconditions.checkArgument(
- operation != BulkOperationState.OperationType.Rename,
- "Rename operations cannot be started through initiateBulkWrite");
- if (metastore == null || isNullMetadataStore(metastore)) {
- return null;
- } else {
- return metastore.initiateBulkWrite(operation, path);
- }
- }
-
- /**
- * Convert the data of a directory listing to an array of {@link FileStatus}
- * entries. Tombstones are filtered out at this point. If the listing is null
- * an empty array is returned.
- * @param dirMeta directory listing -may be null
- * @return a possibly-empty array of file status entries
- */
- public static S3AFileStatus[] dirMetaToStatuses(DirListingMetadata dirMeta)
{
- if (dirMeta == null) {
- return EMPTY_LISTING;
- }
-
- Collection<PathMetadata> listing = dirMeta.getListing();
- List<FileStatus> statuses = new ArrayList<>();
-
- for (PathMetadata pm : listing) {
- if (!pm.isDeleted()) {
- statuses.add(pm.getFileStatus());
- }
- }
-
- return statuses.toArray(new S3AFileStatus[0]);
- }
-
- /**
- * Given directory listing metadata from both the backing store and the
- * MetadataStore, merge the two sources of truth to create a consistent
- * view of the current directory contents, which can be returned to clients.
- *
- * Also update the MetadataStore to reflect the resulting directory listing.
- *
- * In not authoritative case: update file metadata if mod_time in listing
- * of a file is greater then what is currently in the ms
- *
- * @param ms MetadataStore to use.
- * @param path path to directory
- * @param backingStatuses Directory listing from the backing store.
- * @param dirMeta Directory listing from MetadataStore. May be null.
- * @param isAuthoritative State of authoritative mode
- * @param timeProvider Time provider to use when updating entries
- * @param toStatusItr function to convert array of file status to
- * RemoteIterator.
- * @return Final result of directory listing.
- * @throws IOException if metadata store update failed
- */
- public static RemoteIterator<S3AFileStatus> dirListingUnion(
- MetadataStore ms, Path path,
- RemoteIterator<S3AFileStatus> backingStatuses,
- DirListingMetadata dirMeta, boolean isAuthoritative,
- ITtlTimeProvider timeProvider,
- Function<S3AFileStatus[], RemoteIterator<S3AFileStatus>> toStatusItr)
- throws IOException {
-
- // Fast-path for NullMetadataStore
- if (isNullMetadataStore(ms)) {
- return backingStatuses;
- }
-
- assertQualified(path);
-
- if (dirMeta == null) {
- // The metadataStore had zero state for this directory
- dirMeta = new DirListingMetadata(path, DirListingMetadata.EMPTY_DIR,
- false);
- }
-
- // Since we treat the MetadataStore as a "fresher" or "consistent" view
- // of metadata, we always use its metadata first.
-
- // Since the authoritative case is already handled outside this function,
- // we will basically start with the set of directory entries in the
- // DirListingMetadata, and add any that only exist in the backingStatuses.
- //
- // We try to avoid writing any more child entries than need be to :-
- // (a) save time and money.
- // (b) avoid overwriting the authoritative bit of children (HADOOP-16746).
- // For auth mode updates, we supply the full listing and a list of which
- // child entries have not been changed; the store gets to optimize its
- // update however it chooses.
- //
- // for non-auth-mode S3Guard, we just build a list of entries to add and
- // submit them in a batch; this is more efficient than trickling out the
- // updates one-by-one.
-
- BulkOperationState operationState = ms.initiateBulkWrite(
- BulkOperationState.OperationType.Listing,
- path);
- if (isAuthoritative) {
- authoritativeUnion(ms, path, backingStatuses, dirMeta,
- timeProvider, operationState);
- } else {
- nonAuthoritativeUnion(ms, path, backingStatuses, dirMeta,
- timeProvider, operationState);
- }
- IOUtils.cleanupWithLogger(LOG, operationState);
-
- return toStatusItr.apply(dirMetaToStatuses(dirMeta));
+ private S3Guard() {
}
/**
- * Perform the authoritative union operation.
- * Here all updated/missing entries are added back; we take care
- * not to overwrite unchanged entries as that will lose their
- * isAuthoritative bit (HADOOP-16746).
- * @param ms MetadataStore to use.
- * @param path path to directory
- * @param backingStatuses Directory listing from the backing store.
- * @param dirMeta Directory listing from MetadataStore. May be null.
- * @param timeProvider Time provider to use when updating entries
- * @param operationState ongoing operation
- * @throws IOException if metadata store update failed
- */
- private static void authoritativeUnion(
- final MetadataStore ms,
- final Path path,
- final RemoteIterator<S3AFileStatus> backingStatuses,
- final DirListingMetadata dirMeta,
- final ITtlTimeProvider timeProvider,
- final BulkOperationState operationState) throws IOException {
- // track all unchanged entries; used so the metastore can identify entries
- // it doesn't need to update
- List<Path> unchangedEntries = new ArrayList<>(dirMeta.getListing().size());
- boolean changed = !dirMeta.isAuthoritative();
- Set<Path> deleted = dirMeta.listTombstones();
- final Map<Path, PathMetadata> dirMetaMap = dirMeta.getListing().stream()
- .collect(Collectors.toMap(pm -> pm.getFileStatus().getPath(), pm ->
pm));
- while (backingStatuses.hasNext()) {
- S3AFileStatus s = backingStatuses.next();
- final Path statusPath = s.getPath();
- if (deleted.contains(statusPath)) {
- continue;
- }
-
- // this is built up to be whatever entry is to be added to the dirMeta
- // collection
- PathMetadata pathMetadata = dirMetaMap.get(statusPath);
-
- if (pathMetadata == null) {
- // there's no entry in the listing, so create one.
- pathMetadata = new PathMetadata(s);
- } else {
- // no change -add the path to the list of unchangedEntries
- unchangedEntries.add(statusPath);
- }
-
- // Minor race condition here. Multiple threads could add to this
- // mutable DirListingMetadata. Since it is backed by a
- // ConcurrentHashMap, the last put() wins.
- // More concerning is two threads racing on listStatus() and delete().
- // Any FileSystem has similar race conditions, but we could persist
- // a stale entry longer. We could expose an atomic
- // DirListingMetadata#putIfNotPresent()
- changed |= dirMeta.put(pathMetadata);
- }
-
- if (changed) {
- // in an authoritative update, we pass in the full list of entries,
- // but do declare which have not changed to avoid needless and
potentially
- // destructive overwrites.
- LOG.debug("Marking the directory {} as authoritative", path);
- ms.getInstrumentation().directoryMarkedAuthoritative();
- dirMeta.setAuthoritative(true); // This is the full directory contents
- // write the updated dir entry and any changed children.
- S3Guard.putWithTtl(ms, dirMeta, unchangedEntries, timeProvider,
operationState);
- }
- }
-
- /**
- * Perform the authoritative union operation.
- * @param ms MetadataStore to use.
- * @param path path to directory
- * @param backingStatuses Directory listing from the backing store.
- * @param dirMeta Directory listing from MetadataStore. May be null.
- * @param timeProvider Time provider to use when updating entries
- * @param operationState ongoing operation
- * @throws IOException if metadata store update failed
- */
- private static void nonAuthoritativeUnion(
- final MetadataStore ms,
- final Path path,
- final RemoteIterator<S3AFileStatus> backingStatuses,
- final DirListingMetadata dirMeta,
- final ITtlTimeProvider timeProvider,
- final BulkOperationState operationState) throws IOException {
- List<PathMetadata> entriesToAdd = new ArrayList<>();
- Set<Path> deleted = dirMeta.listTombstones();
-
- final Map<Path, PathMetadata> dirMetaMap = dirMeta.getListing().stream()
- .collect(Collectors.toMap(pm -> pm.getFileStatus().getPath(), pm ->
pm));
- while (backingStatuses.hasNext()) {
- S3AFileStatus s = backingStatuses.next();
- final Path statusPath = s.getPath();
- if (deleted.contains(statusPath)) {
- continue;
- }
-
- // this is the record in dynamo
- PathMetadata pathMetadata = dirMetaMap.get(statusPath);
-
- // in non-auth listings, we compare the file status of the metastore
- // list with those in the FS, and overwrite the MS entry if
- // either of two conditions are met
- // - there is no entry in the metastore and
- // DIR_MERGE_UPDATES_ALL_RECORDS_NONAUTH is compiled to true
- // - there is an entry in the metastore the FS entry is newer.
- boolean shouldUpdate;
- if (pathMetadata != null) {
- // entry is in DDB; check modification time
- shouldUpdate = s.getModificationTime() > (pathMetadata.getFileStatus())
- .getModificationTime();
- // create an updated record.
- pathMetadata = new PathMetadata(s);
- } else {
- // entry is not present. Create for insertion into dirMeta
- pathMetadata = new PathMetadata(s);
- // use hard-coded policy about updating
- shouldUpdate = DIR_MERGE_UPDATES_ALL_RECORDS_NONAUTH;
- }
- if (shouldUpdate) {
- // we do want to update DDB and the listing with a new entry.
- LOG.debug("Update ms with newer metadata of: {}", s);
- // ensure it gets into the dirListing
- // add to the list of entries to add later,
- entriesToAdd.add(pathMetadata);
- }
- // add the entry to the union; no-op if it was already there.
- dirMeta.put(pathMetadata);
- }
-
- if (!entriesToAdd.isEmpty()) {
- // non-auth, just push out the updated entry list
- LOG.debug("Adding {} entries under directory {}", entriesToAdd.size(),
path);
- putWithTtl(ms, entriesToAdd, timeProvider, operationState);
- }
- }
-
- /**
- * Although NullMetadataStore does nothing, callers may wish to avoid work
- * (fast path) when the NullMetadataStore is in use.
- * @param ms The MetadataStore to test
- * @return true iff the MetadataStore is the null, or no-op, implementation.
- */
- public static boolean isNullMetadataStore(MetadataStore ms) {
- return (ms instanceof NullMetadataStore);
- }
-
- /**
- * Update MetadataStore to reflect creation of the given directories.
- *
- * If an IOException is raised while trying to update the entry, this
- * operation catches the exception, swallows it and returns.
- *
- * @deprecated this is no longer called by {@code S3AFilesystem.innerMkDirs}.
- * See: HADOOP-15079 (January 2018).
- * It is currently retained because of its discussion in the method on
- * atomicity and in case we need to reinstate it or adapt the current
- * process of directory marker creation.
- * But it is not being tested and so may age with time...consider
- * deleting it in future if it's clear there's no need for it.
- * @param ms MetadataStore to update.
- * @param dirs null, or an ordered list of directories from leaf to root.
- * E.g. if /a/ exists, and mkdirs(/a/b/c/d) is called, this
- * list will contain [/a/b/c/d, /a/b/c, /a/b]. /a/b/c/d is
- * an empty, dir, and the other dirs only contain their child
- * dir.
- * @param owner Hadoop user name.
- * @param authoritative Whether to mark new directories as authoritative.
- * @param timeProvider Time provider.
- */
- @Deprecated
- @Retries.OnceExceptionsSwallowed
- public static void makeDirsOrdered(MetadataStore ms, List<Path> dirs,
- String owner, boolean authoritative, ITtlTimeProvider timeProvider) {
- if (dirs == null) {
- return;
- }
-
- /* We discussed atomicity of this implementation.
- * The concern is that multiple clients could race to write different
- * cached directories to the MetadataStore. Two solutions are proposed:
- * 1. Move mkdirs() into MetadataStore interface and let implementations
- * ensure they are atomic.
- * 2. Specify that the semantics of MetadataStore#putListStatus() is
- * always additive, That is, if MetadataStore has listStatus() state
- * for /a/b that contains [/a/b/file0, /a/b/file1], and we then call
- * putListStatus(/a/b -> [/a/b/file2, /a/b/file3],
isAuthoritative=true),
- * then we will end up with final state of
- * [/a/b/file0, /a/b/file1, /a/b/file2, /a/b/file3], isAuthoritative =
- * true
- */
- S3AFileStatus prevStatus = null;
-
- // Use new batched put to reduce round trips.
- List<PathMetadata> pathMetas = new ArrayList<>(dirs.size());
-
- try {
- // Iterate from leaf to root
- for (int i = 0; i < dirs.size(); i++) {
- boolean isLeaf = (prevStatus == null);
- Path f = dirs.get(i);
- assertQualified(f);
- S3AFileStatus status =
- createUploadFileStatus(f, true, 0, 0, owner, null, null);
-
- // We only need to put a DirListingMetadata if we are setting
- // authoritative bit
- DirListingMetadata dirMeta = null;
- if (authoritative) {
- Collection<PathMetadata> children;
- if (isLeaf) {
- children = DirListingMetadata.EMPTY_DIR;
- } else {
- children = new ArrayList<>(1);
- children.add(new PathMetadata(prevStatus));
- }
- dirMeta = new DirListingMetadata(f, children, authoritative);
- S3Guard.putWithTtl(ms, dirMeta, Collections.emptyList(),
timeProvider, null);
- }
-
- pathMetas.add(new PathMetadata(status));
- prevStatus = status;
- }
-
- // Batched put
- S3Guard.putWithTtl(ms, pathMetas, timeProvider, null);
- } catch (IOException ioe) {
- LOG.error("MetadataStore#put() failure:", ioe);
- }
- }
-
- /**
- * Helper function that records the move of directory paths, adding
- * resulting metadata to the supplied lists.
- * Does not store in MetadataStore.
- * @param ms MetadataStore, used to make this a no-op, when it is
- * NullMetadataStore.
- * @param srcPaths stores the source path here
- * @param dstMetas stores destination metadata here
- * @param srcPath source path to store
- * @param dstPath destination path to store
- * @param owner file owner to use in created records
- */
- public static void addMoveDir(MetadataStore ms, Collection<Path> srcPaths,
- Collection<PathMetadata> dstMetas, Path srcPath, Path dstPath,
- String owner) {
- if (isNullMetadataStore(ms)) {
- return;
- }
- assertQualified(srcPath, dstPath);
-
- S3AFileStatus dstStatus = createUploadFileStatus(dstPath, true, 0,
- 0, owner, null, null);
- addMoveStatus(srcPaths, dstMetas, srcPath, dstStatus);
- }
-
- /**
- * Like {@link #addMoveDir(MetadataStore, Collection, Collection, Path,
- * Path, String)} (), but for files.
- * @param ms MetadataStore, used to make this a no-op, when it is
- * NullMetadataStore.
- * @param srcPaths stores the source path here
- * @param dstMetas stores destination metadata here
- * @param srcPath source path to store
- * @param dstPath destination path to store
- * @param size length of file moved
- * @param blockSize blocksize to associate with destination file
- * @param owner file owner to use in created records
- * @param eTag the s3 object eTag of file moved
- * @param versionId the s3 object versionId of file moved
- */
- public static void addMoveFile(MetadataStore ms, Collection<Path> srcPaths,
- Collection<PathMetadata> dstMetas, Path srcPath, Path dstPath,
- long size, long blockSize, String owner, String eTag, String versionId) {
- if (isNullMetadataStore(ms)) {
- return;
- }
- assertQualified(srcPath, dstPath);
- S3AFileStatus dstStatus = createUploadFileStatus(dstPath, false,
- size, blockSize, owner, eTag, versionId);
- addMoveStatus(srcPaths, dstMetas, srcPath, dstStatus);
- }
-
- /**
- * Helper method that records the move of all ancestors of a path.
- *
- * In S3A, an optimization is to delete unnecessary fake directory objects if
- * the directory is non-empty. In that case, for a nested child to move, S3A
- * is not listing and thus moving all its ancestors (up to source root). So
we
- * take care of those inferred directories of this path explicitly.
- *
- * As {@link #addMoveFile} and {@link #addMoveDir}, this method adds
resulting
- * metadata to the supplied lists. It does not update the MetadataStore.
- *
- * @param ms MetadataStore, no-op if it is NullMetadataStore
- * @param srcPaths stores the source path here
- * @param dstMetas stores destination metadata here
- * @param srcRoot source root up to which (exclusive) should we add ancestors
- * @param srcPath source path of the child to add ancestors
- * @param dstPath destination path of the child to add ancestors
- * @param owner Hadoop user name
- */
- public static void addMoveAncestors(MetadataStore ms,
- Collection<Path> srcPaths, Collection<PathMetadata> dstMetas,
- Path srcRoot, Path srcPath, Path dstPath, String owner) {
- if (isNullMetadataStore(ms)) {
- return;
- }
-
- assertQualified(srcRoot, srcPath, dstPath);
-
- if (srcPath.equals(srcRoot)) {
- LOG.debug("Skip moving ancestors of source root directory {}", srcRoot);
- return;
- }
-
- Path parentSrc = srcPath.getParent();
- Path parentDst = dstPath.getParent();
- while (parentSrc != null
- && !parentSrc.isRoot()
- && !parentSrc.equals(srcRoot)
- && !srcPaths.contains(parentSrc)) {
- LOG.debug("Renaming non-listed parent {} to {}", parentSrc, parentDst);
- S3Guard.addMoveDir(ms, srcPaths, dstMetas, parentSrc, parentDst, owner);
- parentSrc = parentSrc.getParent();
- parentDst = parentDst.getParent();
- }
- }
-
- /**
- * This adds all new ancestors of a path as directories.
- * This forwards to
- * {@link MetadataStore#addAncestors(Path, BulkOperationState)}.
- * <p>
- * Originally it implemented the logic to probe for an add ancestors,
- * but with the addition of a store-specific bulk operation state
- * it became unworkable.
- *
- * @param metadataStore store
- * @param qualifiedPath path to update
- * @param operationState (nullable) operational state for a bulk update
- * @throws IOException failure
- */
- @Retries.RetryTranslated
- public static void addAncestors(
- final MetadataStore metadataStore,
- final Path qualifiedPath,
- final ITtlTimeProvider timeProvider,
- @Nullable final BulkOperationState operationState) throws IOException {
- metadataStore.addAncestors(qualifiedPath, operationState);
- }
-
- /**
- * Add the fact that a file was moved from a source path to a destination.
- * @param srcPaths collection of source paths to update
- * @param dstMetas collection of destination meta data entries to update.
- * @param srcPath path of the source file.
- * @param dstStatus status of the source file after it was copied.
- */
- private static void addMoveStatus(Collection<Path> srcPaths,
- Collection<PathMetadata> dstMetas,
- Path srcPath,
- S3AFileStatus dstStatus) {
- srcPaths.add(srcPath);
- dstMetas.add(new PathMetadata(dstStatus));
- }
-
- /**
- * Assert that the path is qualified with a host and scheme.
- * @param p path to check
- * @throws NullPointerException if either argument does not hold
- */
- public static void assertQualified(Path p) {
- URI uri = p.toUri();
- // Paths must include bucket in case MetadataStore is shared between
- // multiple S3AFileSystem instances
- Preconditions.checkNotNull(uri.getHost(), "Null host in " + uri);
-
- // This should never fail, but is retained for completeness.
- Preconditions.checkNotNull(uri.getScheme(), "Null scheme in " + uri);
- }
-
- /**
- * Assert that all paths are valid.
- * @param paths path to check
- * @throws NullPointerException if either argument does not hold
+ * Assert that there is no S3Guard.
+ * @param fsURI FileSystem URI
+ * @param conf configuration
*/
- public static void assertQualified(Path...paths) {
- for (Path path : paths) {
- assertQualified(path);
+ public static void checkNoS3Guard(URI fsURI, Configuration conf) throws
PathIOException {
+ final String classname = conf.getTrimmed(Constants.S3_METADATA_STORE_IMPL,
"");
+ if (!"org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore".equals(classname)
+ && !classname.isEmpty()) {
+ LOG.error("Metastore option source {}",
+ (Object) conf.getPropertySources(Constants.S3_METADATA_STORE_IMPL));
+ throw new PathIOException(fsURI.toString(),
+ "Filesystem is configured to use Unsupported S3Guard store " +
classname);
}
}
/**
* Runtime implementation for TTL Time Provider interface.
+ * This is deprecated; retained pnly because the StoreContext builder takes
it
Review comment:
nit: `pnly` -> `only`
##########
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
##########
@@ -437,7 +330,7 @@ public void checkBasicFileOperations() throws Throwable {
// This means that permissions on the file do not get checked.
// See: HADOOP-16464.
Optional<FSDataInputStream> optIn = accessDeniedIf(
- !guardedInAuthMode, () -> readonlyFS.open(emptyFile));
+ !false, () -> readonlyFS.open(emptyFile));
Review comment:
nit: `!false` => `true`
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
##########
@@ -84,705 +89,31 @@ Once you are confident that all applications have been
restarted, _Delete the Dy
This is to avoid paying for a database you no longer need.
This is best done from the AWS GUI.
-## Setting up S3Guard
-
-### S3A to warn or fail if S3Guard is disabled
-A seemingly recurrent problem with S3Guard is that people think S3Guard is
-turned on but it isn't.
-You can set `org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level`
-to avoid this. The property sets what to do when an S3A FS is instantiated
-without S3Guard. The following values are available:
-
-* `SILENT`: Do nothing.
-* `INFORM`: Log at info level that FS is instantiated without S3Guard.
-* `WARN`: Warn that data may be at risk in workflows.
-* `FAIL`: S3AFileSystem instantiation will fail.
-
-The default setting is `SILENT`. The setting is case insensitive.
-The required level can be set in the `core-site.xml`.
-
----
-The latest configuration parameters are defined in `core-default.xml`. You
-should consult that file for full information, but a summary is provided here.
-
+## Removing S3Guard Configurations
-### 1. Choose the Database
+The `fs.s3a.metadatastore.impl` option must be deleted, set to the empty
string "",
+or to the "Null" Metadata store
`org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore`.
-A core concept of S3Guard is that the directory listing data of the object
-store, *the metadata* is replicated in a higher-performance, consistent,
-database. In S3Guard, this database is called *The Metadata Store*
-
-By default, S3Guard is not enabled.
-
-The Metadata Store to use in production is bonded to Amazon's DynamoDB
-database service. The following setting will enable this Metadata Store:
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
+ <value></value>
</property>
```
-Note that the `NullMetadataStore` store can be explicitly requested if desired.
-This offers no metadata storage, and effectively disables S3Guard.
-
```xml
<property>
<name>fs.s3a.metadatastore.impl</name>
<value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
</property>
```
-### 2. Configure S3Guard Settings
-
-More settings will may be added in the future.
-Currently the only Metadata Store-independent setting, besides the
-implementation class above, are the *allow authoritative* and *fail-on-error*
-flags.
-
-#### <a name="authoritative"></a> Authoritative S3Guard
-
-Authoritative S3Guard is a complicated configuration which delivers performance
-at the expense of being unsafe for other applications to use the same directory
-tree/bucket unless configured consistently.
-
-It can also be used to support [directory marker
retention](directory_markers.html)
-in higher-performance but non-backwards-compatible modes.
-
-Most deployments do not use this setting -it is ony used in deployments where
-specific parts of a bucket (e.g. Apache Hive managed tables) are known to
-have exclusive access by a single application (Hive) and other
tools/applications
-from exactly the same Hadoop release.
-
-The _authoritative_ expression in S3Guard is present in two different layers,
for
-two different reasons:
-
-* Authoritative S3Guard
- * S3Guard can be set as authoritative, which means that an S3A client will
- avoid round-trips to S3 when **getting file metadata**, and **getting
- directory listings** if there is a fully cached version of the directory
- stored in metadata store.
- * This mode can be set as a configuration property
- `fs.s3a.metadatastore.authoritative`
- * It can also be set only on specific directories by setting
- `fs.s3a.authoritative.path` to one or more prefixes, for example
- `s3a://bucket/path` or "/auth1,/auth2".
- * All interactions with the S3 bucket(s) must be through S3A clients
sharing
- the same metadata store.
- * This is independent from which metadata store implementation is used.
- * In authoritative mode the metadata TTL metadata expiry is not effective.
- This means that the metadata entries won't expire on authoritative paths.
-
-* Authoritative directory listings (isAuthoritative bit)
- * Tells if the stored directory listing metadata is complete.
- * This is set by the FileSystem client (e.g. s3a) via the
`DirListingMetadata`
- class (`org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata`).
- (The MetadataStore only knows what the FS client tells it.)
- * If set to `TRUE`, we know that the directory listing
- (`DirListingMetadata`) is full, and complete.
- * If set to `FALSE` the listing may not be complete.
- * Metadata store may persist the isAuthoritative bit on the metadata store.
- * Currently `org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore` and
- `org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore` implementation
- supports authoritative bit.
-
-More on Authoritative S3Guard:
-
-* This setting is about treating the MetadataStore (e.g. dynamodb) as the
source
- of truth in general, and also to short-circuit S3 list objects and serve
- listings from the MetadataStore in some circumstances.
-* For S3A to skip S3's get object metadata, and serve it directly from the
-MetadataStore, the following things must all be true:
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a file metadata (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has the file metadata for the path stored in it.
-* For S3A to skip S3's list objects on some path, and serve it directly from
-the MetadataStore, the following things must all be true:
- 1. The MetadataStore implementation persists the bit
- `DirListingMetadata.isAuthorititative` set when calling
- `MetadataStore#put` (`DirListingMetadata`)
- 1. The S3A client is configured to allow MetadataStore to be authoritative
- source of a directory listing (`fs.s3a.metadatastore.authoritative=true`).
- 1. The MetadataStore has a **full listing for path** stored in it. This
only
- happens if the FS client (s3a) explicitly has stored a full directory
- listing with `DirListingMetadata.isAuthorititative=true` before the said
- listing request happens.
-
-This configuration only enables authoritative mode in the client layer. It is
-recommended that you leave the default setting here:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.authoritative</name>
- <value>false</value>
-</property>
-```
-
-Note that a MetadataStore MAY persist this bit in the directory listings. (Not
-MUST).
-
-Note that if this is set to true, it may exacerbate or persist existing race
-conditions around multiple concurrent modifications and listings of a given
-directory tree.
-
-In particular: **If the Metadata Store is declared as authoritative,
-all interactions with the S3 bucket(s) must be through S3A clients sharing
-the same Metadata Store**
-
-#### TTL metadata expiry
-
-It can be configured how long an entry is valid in the MetadataStore
-**if the authoritative mode is turned off**, or the path is not
-configured to be authoritative.
-If `((lastUpdated + ttl) <= now)` is false for an entry, the entry will
-be expired, so the S3 bucket will be queried for fresh metadata.
-The time for expiry of metadata can be set as the following:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.metadata.ttl</name>
- <value>15m</value>
-</property>
-```
-
-#### Fail on Error
-
-By default, S3AFileSystem write operations will fail when updates to
-S3Guard metadata fail. S3AFileSystem first writes the file to S3 and then
-updates the metadata in S3Guard. If the metadata write fails,
-`MetadataPersistenceException` is thrown. The file in S3 **is not** rolled
-back.
-
-If the write operation cannot be programmatically retried, the S3Guard metadata
-for the given file can be corrected with a command like the following:
-
-```bash
-hadoop s3guard import [-meta URI] s3a://my-bucket/file-with-bad-metadata
-```
-
-Programmatic retries of the original operation would require overwrite=true.
-Suppose the original operation was `FileSystem.create(myFile,
overwrite=false)`.
-If this operation failed with `MetadataPersistenceException` a repeat of the
-same operation would result in `FileAlreadyExistsException` since the original
-operation successfully created the file in S3 and only failed in writing the
-metadata to S3Guard.
-
-Metadata update failures can be downgraded to ERROR logging instead of
exception
-by setting the following configuration:
-
-```xml
-<property>
- <name>fs.s3a.metadatastore.fail.on.write.error</name>
- <value>false</value>
-</property>
-```
-
-Setting this false is dangerous as it could result in the type of issue S3Guard
-is designed to avoid. For example, a reader may see an inconsistent listing
-after a recent write since S3Guard may not contain metadata about the recently
-written file due to a metadata write error.
-
-As with the default setting, the new/updated file is still in S3 and **is not**
-rolled back. The S3Guard metadata is likely to be out of sync.
-
-### 3. Configure the Metadata Store.
-
-Here are the `DynamoDBMetadataStore` settings. Other Metadata Store
-implementations will have their own configuration parameters.
-
-
-### 4. Name Your Table
-
-First, choose the name of the table you wish to use for the S3Guard metadata
-storage in your DynamoDB instance. If you leave it unset/empty, a
-separate table will be created for each S3 bucket you access, and that
-bucket's name will be used for the name of the DynamoDB table. For example,
-this sets the table name to `my-ddb-table-name`
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table</name>
- <value>my-ddb-table-name</value>
- <description>
- The DynamoDB table name to operate. Without this property, the respective
- S3 bucket names will be used.
- </description>
-</property>
-```
-
-It is good to share a table across multiple buckets for multiple reasons,
-especially if you are *not* using on-demand DynamoDB tables, and instead
-prepaying for provisioned I/O capacity.
-
-1. You are billed for the provisioned I/O capacity allocated to the table,
-*even when the table is not used*. Sharing capacity can reduce costs.
-
-1. You can share the "provision burden" across the buckets. That is, rather
-than allocating for the peak load on a single bucket, you can allocate for
-the peak load *across all the buckets*, which is likely to be significantly
-lower.
-
-1. It's easier to measure and tune the load requirements and cost of
-S3Guard, because there is only one table to review and configure in the
-AWS management console.
-
-1. When you don't grant the permission to create DynamoDB tables to users.
-A single pre-created table for all buckets avoids the needs for an
administrator
-to create one for every bucket.
-
-When wouldn't you want to share a table?
-
-1. When you are using on-demand DynamoDB and want to keep each table isolated.
-1. When you do explicitly want to provision I/O capacity to a specific bucket
-and table, isolated from others.
-
-1. When you are using separate billing for specific buckets allocated
-to specific projects.
-
-1. When different users/roles have different access rights to different
buckets.
-As S3Guard requires all users to have R/W access to the table, all users will
-be able to list the metadata in all buckets, even those to which they lack
-read access.
-
-### 5. Locate your Table
-
-You may also wish to specify the region to use for DynamoDB. If a region
-is not configured, S3A will assume that it is in the same region as the S3
-bucket. A list of regions for the DynamoDB service can be found in
-[Amazon's
documentation](http://docs.aws.amazon.com/general/latest/gr/rande.html#ddb_region).
-In this example, to use the US West 2 region:
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.region</name>
- <value>us-west-2</value>
-</property>
-```
-
-When working with S3Guard-managed buckets from EC2 VMs running in AWS
-infrastructure, using a local DynamoDB region ensures the lowest latency
-and highest reliability, as well as avoiding all long-haul network charges.
-The S3Guard tables, and indeed, the S3 buckets, should all be in the same
-region as the VMs.
-
-### 6. Optional: Create your Table
-
-Next, you can choose whether or not the table will be automatically created
-(if it doesn't already exist). If you want this feature, set the
-`fs.s3a.s3guard.ddb.table.create` option to `true`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.create</name>
- <value>true</value>
- <description>
- If true, the S3A client will create the table if it does not already exist.
- </description>
-</property>
-```
-
-### 7. If creating a table: Choose your billing mode (and perhaps I/O Capacity)
-
-Next, you need to decide whether to use On-Demand DynamoDB and its
-pay-per-request billing (recommended), or to explicitly request a
-provisioned IO capacity.
-
-Before AWS offered pay-per-request billing, the sole billing mechanism,
-was "provisioned capacity". This mechanism requires you to choose
-the DynamoDB read and write throughput requirements you
-expect to need for your expected uses of the S3Guard table.
-Setting higher values cost you more money -*even when the table was idle*
- *Note* that these settings only affect table creation when
-`fs.s3a.s3guard.ddb.table.create` is enabled. To change the throughput for
-an existing table, use the AWS console or CLI tool.
-
-For more details on DynamoDB capacity units, see the AWS page on [Capacity
-Unit
Calculations](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html#CapacityUnitCalculations).
-
-Provisioned IO capacity is billed per hour for the life of the table, *even
when the
-table and the underlying S3 buckets are not being used*.
-
-There are also charges incurred for data storage and for data I/O outside of
the
-region of the DynamoDB instance. S3Guard only stores metadata in DynamoDB:
path names
-and summary details of objects —the actual data is stored in S3, so billed at
S3
-rates.
-
-With provisioned I/O capacity, attempting to perform more I/O than the capacity
-requested throttles the operation and may result in operations failing.
-Larger I/O capacities cost more.
-
-With the introduction of On-Demand DynamoDB, you can now avoid paying for
-provisioned capacity by creating an on-demand table.
-With an on-demand table you are not throttled if your DynamoDB requests exceed
-any pre-provisioned limit, nor do you pay per hour even when a table is idle.
-
-You do, however, pay more per DynamoDB operation.
-Even so, the ability to cope with sudden bursts of read or write requests,
combined
-with the elimination of charges for idle tables, suit the use patterns made of
-S3Guard tables by applications interacting with S3. That is: periods when the
table
-is rarely used, with intermittent high-load operations when directory trees
-are scanned (query planning and similar), or updated (rename and delete
operations).
-
-
-We recommending using On-Demand DynamoDB for maximum performance in operations
-such as query planning, and lowest cost when S3 buckets are not being accessed.
-
-This is the default, as configured in the default configuration options.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.read</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for read operations in terms of
capacity
- units for the DynamoDB table. This config value will only be used when
- creating a new DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request"
capacity.
- If a positive integer is provided for this and the write capacity, then
- a table with "provisioned capacity" will be created.
- You can change the capacity of an existing provisioned-capacity table
- through the "s3guard set-capacity" command.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.capacity.write</name>
- <value>0</value>
- <description>
- Provisioned throughput requirements for write operations in terms of
- capacity units for the DynamoDB table.
- If set to 0 (the default), new tables are created with "per-request"
capacity.
- Refer to related configuration option
fs.s3a.s3guard.ddb.table.capacity.read
- </description>
-</property>
-```
-
-### 8. If creating a table: Enable server side encryption (SSE)
-
-Encryption at rest can help you protect sensitive data in your DynamoDB table.
-When creating a new table, you can set server side encryption on the table
-using the default AWS owned customer master key (CMK), AWS managed CMK, or
-customer managed CMK. S3Guard code accessing the table is all the same whether
-SSE is enabled or not. For more details on DynamoDB table server side
-encryption, see the AWS page on [Encryption at Rest: How It
Works](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.howitworks.html).
-
-These are the default configuration options, as configured in
`core-default.xml`.
-
-```xml
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.enabled</name>
- <value>false</value>
- <description>
- Whether server-side encryption (SSE) is enabled or disabled on the table.
- By default it's disabled, meaning SSE is set to AWS owned CMK.
- </description>
-</property>
-
-<property>
- <name>fs.s3a.s3guard.ddb.table.sse.cmk</name>
- <value/>
- <description>
- The KMS Customer Master Key (CMK) used for the KMS encryption on the table.
- To specify a CMK, this config value can be its key ID, Amazon Resource Name
- (ARN), alias name, or alias ARN. Users only need to provide this config if
- the key is different from the default DynamoDB KMS Master Key, which is
- alias/aws/dynamodb.
- </description>
-</property>
-```
-
-## Authenticating with S3Guard
-
-The DynamoDB metadata store takes advantage of the fact that the DynamoDB
-service uses the same authentication mechanisms as S3. S3Guard
-gets all its credentials from the S3A client that is using it.
-
-All existing S3 authentication mechanisms can be used.
-
-## Per-bucket S3Guard configuration
-
-In production, it is likely only some buckets will have S3Guard enabled;
-those which are read-only may have disabled, for example. Equally importantly,
-buckets in different regions should have different tables, each
-in the relevant region.
-
-These options can be managed through S3A's [per-bucket configuration
-mechanism](./index.html#Configuring_different_S3_buckets).
-All options with the under `fs.s3a.bucket.BUCKETNAME.KEY` are propagated
-to the options `fs.s3a.KEY` *for that bucket only*.
-
-As an example, here is a configuration to use different metadata stores
-and tables for different buckets
-
-First, we define shortcuts for the metadata store classnames:
-
-
-```xml
-<property>
- <name>s3guard.null</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
-</property>
-
-<property>
- <name>s3guard.dynamo</name>
- <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
-</property>
-```
-
-Next, Amazon's public landsat database is configured with no
-metadata store:
-
-```xml
-<property>
- <name>fs.s3a.bucket.landsat-pds.metadatastore.impl</name>
- <value>${s3guard.null}</value>
- <description>The read-only landsat-pds repository isn't
- managed by S3Guard</description>
-</property>
-```
-
-Next the `ireland-2` and `ireland-offline` buckets are configured with
-DynamoDB as the store, and a shared table `production-table`:
-
-
-```xml
-<property>
- <name>fs.s3a.bucket.ireland-2.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-offline.metadatastore.impl</name>
- <value>${s3guard.dynamo}</value>
-</property>
-
-<property>
- <name>fs.s3a.bucket.ireland-2.s3guard.ddb.table</name>
- <value>production-table</value>
-</property>
-```
-
-The region of this table is automatically set to be that of the buckets,
-here `eu-west-1`; the same table name may actually be used in different
-regions.
-
-Together then, this configuration enables the DynamoDB Metadata Store
-for two buckets with a shared table, while disabling it for the public
-bucket.
-
-
-### Out-of-band operations with S3Guard
-
-We call an operation out-of-band (OOB) when a bucket is used by a client with
- S3Guard, and another client runs a write (e.g delete, move, rename,
- overwrite) operation on an object in the same bucket without S3Guard.
-
-The definition of behaviour in S3AFileSystem/MetadataStore in case of OOBs:
-* A client with S3Guard
-* B client without S3Guard (Directly to S3)
-
-
-* OOB OVERWRITE, authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F1 metadata
-
-* OOB OVERWRITE, NOT authoritative mode:
- * A client creates F1 file
- * B client overwrites F1 file with F2 (Same, or different file size)
- * A client's getFileStatus returns F2 metadata. In not authoritative mode we
- check S3 for the file. If the modification time of the file in S3 is greater
- than in S3Guard, we can safely return the S3 file metadata and update the
- cache.
-
-* OOB DELETE, authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-* OOB DELETE, NOT authoritative mode:
- * A client creates F file
- * B client deletes F file
- * A client's getFileStatus returns that the file is still there
-
-Note: authoritative and NOT authoritative mode behaves the same at
-OOB DELETE case.
-
-The behaviour in case of getting directory listings:
-* File status in metadata store gets updated during the listing the same way
-as in getFileStatus.
-
## S3Guard Command Line Interface (CLI)
-Note that in some cases an AWS region or `s3a://` URI can be provided.
-
-Metadata store URIs include a scheme that designates the backing store. For
-example (e.g. `dynamodb://table_name`;). As documented above, the
-AWS region can be inferred if the URI to an existing bucket is provided.
-
-
-The S3A URI must also be provided for per-bucket configuration options
-to be picked up. That is: when an s3a URL is provided on the command line,
-all its "resolved" per-bucket settings are used to connect to, authenticate
-with and configure the S3Guard table. If no such URL is provided, then
-the base settings are picked up.
-
-
-### Create a table: `s3guard init`
-
-```bash
-hadoop s3guard init -meta URI ( -region REGION | s3a://BUCKET )
-```
-
-Creates and initializes an empty metadata store.
-
-A DynamoDB metadata store can be initialized with additional parameters
-pertaining to capacity.
-
-If these values are both zero, then an on-demand DynamoDB table is created;
-if positive values then they set the
-[Provisioned
Throughput](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html)
-of the table.
-
-
-```bash
-[-write PROVISIONED_WRITES] [-read PROVISIONED_READS]
-```
-
-Server side encryption (SSE) can be enabled with AWS managed customer master
key
-(CMK), or customer managed CMK. By default the DynamoDB table will be encrypted
-with AWS owned CMK. To use a customer managed CMK, you can specify its KMS key
-ID, ARN, alias name, or alias ARN. If not specified, the default AWS managed
CMK
-for DynamoDB "alias/aws/dynamodb" will be used.
-
-```bash
-[-sse [-cmk KMS_CMK_ID]]
-```
-
-Tag argument can be added with a key=value list of tags. The table for the
-metadata store will be created with these tags in DynamoDB.
-
-```bash
-[-tag key=value;]
-```
-
-
-Example 1
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -write 0 -read 0
s3a://ireland-1
-```
-
-Creates an on-demand table "ireland-team",
-in the same location as the S3 bucket "ireland-1".
-
-
-Example 2
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -region eu-west-1 --read 0
--write 0
-```
-
-Creates a table "ireland-team" in the region "eu-west-1.amazonaws.com"
-
-
-Example 3
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -tag tag1=first;tag2=second;
-```
-
-Creates a table "ireland-team" with tags "first" and "second". The read and
-write capacity will be those of the site configuration's values of
-`fs.s3a.s3guard.ddb.table.capacity.read` and
`fs.s3a.s3guard.ddb.table.capacity.write`;
-if these are both zero then it will be an on-demand table.
-
-
-Example 4
-
-```bash
-hadoop s3guard init -meta dynamodb://ireland-team -sse
-```
-
-Creates a table "ireland-team" with server side encryption enabled. The CMK
will
-be using the default AWS managed "alias/aws/dynamodb".
-
-
-### Import a bucket: `s3guard import`
-
-```bash
-hadoop s3guard import [-meta URI] [-authoritative] [-verbose] s3a://PATH
-```
-
-Pre-populates a metadata store according to the current contents of an S3
-bucket/path. If the `-meta` option is omitted, the binding information is taken
-from the `core-site.xml` configuration.
-
-Usage
+The `h`
-```
-hadoop s3guard import
-
-import [OPTIONS] [s3a://PATH]
- import metadata from existing S3 data
-
-Common options:
- -authoritative - Mark imported directory data as authoritative.
- -verbose - Verbose Output.
- -meta URL - Metadata repository details (implementation-specific)
-
-Amazon DynamoDB-specific options:
- -region REGION - Service region for connections
-
- URLs for Amazon DynamoDB are of the form dynamodb://TABLE_NAME.
- Specifying both the -region option and an S3A path
- is not supported.
-```
-
-Example
-
-Import all files and directories in a bucket into the S3Guard table.
-
-```bash
-hadoop s3guard import s3a://ireland-1
-```
-
-Import a directory tree, marking directories as authoritative.
-
-```bash
-hadoop s3guard import -authoritative -verbose s3a://ireland-1/fork-0008
-
-2020-01-03 12:05:18,321 [main] INFO - Metadata store
DynamoDBMetadataStore{region=eu-west-1,
- tableName=s3guard-metadata,
tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/s3guard-metadata} is
initialized.
-2020-01-03 12:05:18,324 [main] INFO - Starting: Importing
s3a://ireland-1/fork-0008
-2020-01-03 12:05:18,324 [main] INFO - Importing directory
s3a://ireland-1/fork-0008
-2020-01-03 12:05:18,537 [main] INFO - Dir
s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-0-false
-2020-01-03 12:05:18,630 [main] INFO - Dir
s3a://ireland-1/fork-0008/test/doTestListFiles-0-0-0-true
-2020-01-03 12:05:19,142 [main] INFO - Dir
s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false/dir-0
-2020-01-03 12:05:19,191 [main] INFO - Dir
s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-false/dir-1
-2020-01-03 12:05:19,240 [main] INFO - Dir
s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true/dir-0
-2020-01-03 12:05:19,289 [main] INFO - Dir
s3a://ireland-1/fork-0008/test/doTestListFiles-2-0-0-true/dir-1
-2020-01-03 12:05:19,314 [main] INFO - Updated S3Guard with 0 files and 6
directory entries
-2020-01-03 12:05:19,315 [main] INFO - Marking directory tree
s3a://ireland-1/fork-0008 as authoritative
-2020-01-03 12:05:19,342 [main] INFO - Importing s3a://ireland-1/fork-0008:
duration 0:01.018s
-Inserted 6 items into Metadata Store
-```
-
-### Compare a S3Guard table and the S3 Store: `s3guard diff`
-
-```bash
-hadoop s3guard diff [-meta URI] s3a://BUCKET
-```
-
-Lists discrepancies between a metadata store and bucket. Note that depending on
-how S3Guard is used, certain discrepancies are to be expected.
-
-Example
-
-```bash
-hadoop s3guard diff s3a://ireland-1
```
Review comment:
Needs to be deleted too probably
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentS3Object.java
##########
@@ -173,7 +173,7 @@ private void skipFailpoint(long len) throws IOException {
}
private boolean shouldInjectFailure(String key) {
- if (policy.shouldDelay(key) &&
+ if (false &&
readFailureCounter < MAX_READ_FAILURES) {
readFailureCounter++;
return true;
Review comment:
Always returns false so probably remove?
##########
File path:
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/delegation_tokens.md
##########
@@ -39,8 +39,7 @@ Clients with this token have the full permissions of the user.
*Role Delegation Tokens:* These contain an "STS Session Token" requested by by
the
STS "Assume Role" API, so grant the caller to interact with S3 as specific AWS
-role, *with permissions restricted to purely accessing the S3 bucket
-and associated S3Guard data*.
+role, *with permissions restricted to purely accessing the S3 bucket.
Review comment:
nit: missing `*` at the end
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 667779)
Time Spent: 50m (was: 40m)
> Remove S3Guard - no longer needed
> ---------------------------------
>
> Key: HADOOP-17409
> URL: https://issues.apache.org/jira/browse/HADOOP-17409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> With Consistent S3, S3Guard is superfluous.
> stop developing it and wean people off it as soon as they can.
> Then we can worry about what to do in the code. It has gradually insinuated
> its way through the layers, especially things like multi-object delete
> handling (see HADOOP-17244). Things would be a lot simpler without it
> This work is being done in the feature branch HADOOP-17409-remove-s3guard
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
