This is an automated email from the ASF dual-hosted git repository. wenningd pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git
commit d89c0ce798f3842ab325ad2c7b879e554cf18214 Author: atharvai <[email protected]> AuthorDate: Thu Jul 21 11:18:29 2022 +0100 update versioned docs for dynamodb lock provider docs to include iam and additional dependencies --- website/docs/concurrency_control.md | 1 - .../version-0.10.0/concurrency_control.md | 32 +++++++++++++++++++++ .../version-0.10.1/concurrency_control.md | 33 +++++++++++++++++++++- .../version-0.11.0/concurrency_control.md | 33 +++++++++++++++++++++- .../version-0.11.1/concurrency_control.md | 33 +++++++++++++++++++++- 5 files changed, 128 insertions(+), 4 deletions(-) diff --git a/website/docs/concurrency_control.md b/website/docs/concurrency_control.md index 689b7632f7..25a523ee7c 100644 --- a/website/docs/concurrency_control.md +++ b/website/docs/concurrency_control.md @@ -90,7 +90,6 @@ hoodie.aws.session.token ``` If not configured, Hudi falls back to use [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html). - IAM policy for your service instance will need to add the following permissions: ```json diff --git a/website/versioned_docs/version-0.10.0/concurrency_control.md b/website/versioned_docs/version-0.10.0/concurrency_control.md index a9a0d5860c..fe38f102cd 100644 --- a/website/versioned_docs/version-0.10.0/concurrency_control.md +++ b/website/versioned_docs/version-0.10.0/concurrency_control.md @@ -78,7 +78,10 @@ hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLoc hoodie.write.lock.dynamodb.table hoodie.write.lock.dynamodb.partition_key hoodie.write.lock.dynamodb.region +hoodie.write.lock.dynamodb.endpoint_url +hoodie.write.lock.dynamodb.billing_mode ``` + Also, to set up the credentials for accessing AWS resources, customers can pass the following props to Hudi jobs: ``` hoodie.aws.access.key @@ -87,6 +90,35 @@ hoodie.aws.session.token ``` If not configured, Hudi falls back to use [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html). +IAM policy for your service instance will need to add the following permissions: + +```json +{ + "Sid":"DynamoDBLocksTable", + "Effect": "Allow", + "Action": [ + "dynamodb:CreateTable", + "dynamodb:DeleteItem", + "dynamodb:DescribeTable", + "dynamodb:GetItem", + "dynamodb:PutItem", + "dynamodb:Scan", + "dynamodb:UpdateItem" + ], + "Resource": "arn:${Partition}:dynamodb:${Region}:${Account}:table/${TableName}" +} +``` +- `TableName` : same as `hoodie.write.lock.dynamodb.partition_key` +- `Region`: same as `hoodie.write.lock.dynamodb.region` + +AWS SDK dependencies are not bundled with Hudi from v0.10.x and will need to be added to your classpath. +Add the following Maven packages (check the latest versions at time of install): +``` +com.amazonaws:dynamodb-lock-client +com.amazonaws:aws-java-sdk-dynamodb +com.amazonaws:aws-java-sdk-core +``` + ## Datasource Writer The `hudi-spark` module offers the DataSource API to write (and read) a Spark DataFrame into a Hudi table. diff --git a/website/versioned_docs/version-0.10.1/concurrency_control.md b/website/versioned_docs/version-0.10.1/concurrency_control.md index a9a0d5860c..6377c762bd 100644 --- a/website/versioned_docs/version-0.10.1/concurrency_control.md +++ b/website/versioned_docs/version-0.10.1/concurrency_control.md @@ -70,7 +70,6 @@ hoodie.write.lock.hivemetastore.table `The HiveMetastore URI's are picked up from the hadoop configuration file loaded during runtime.` **`Amazon DynamoDB`** based lock provider - Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters ``` @@ -78,7 +77,10 @@ hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLoc hoodie.write.lock.dynamodb.table hoodie.write.lock.dynamodb.partition_key hoodie.write.lock.dynamodb.region +hoodie.write.lock.dynamodb.endpoint_url +hoodie.write.lock.dynamodb.billing_mode ``` + Also, to set up the credentials for accessing AWS resources, customers can pass the following props to Hudi jobs: ``` hoodie.aws.access.key @@ -87,6 +89,35 @@ hoodie.aws.session.token ``` If not configured, Hudi falls back to use [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html). +IAM policy for your service instance will need to add the following permissions: + +```json +{ + "Sid":"DynamoDBLocksTable", + "Effect": "Allow", + "Action": [ + "dynamodb:CreateTable", + "dynamodb:DeleteItem", + "dynamodb:DescribeTable", + "dynamodb:GetItem", + "dynamodb:PutItem", + "dynamodb:Scan", + "dynamodb:UpdateItem" + ], + "Resource": "arn:${Partition}:dynamodb:${Region}:${Account}:table/${TableName}" +} +``` +- `TableName` : same as `hoodie.write.lock.dynamodb.partition_key` +- `Region`: same as `hoodie.write.lock.dynamodb.region` + +AWS SDK dependencies are not bundled with Hudi starting from v0.10.0 and will need to be added to your classpath. +Add the following Maven packages (check the latest versions at time of install): +``` +com.amazonaws:dynamodb-lock-client +com.amazonaws:aws-java-sdk-dynamodb +com.amazonaws:aws-java-sdk-core +``` + ## Datasource Writer The `hudi-spark` module offers the DataSource API to write (and read) a Spark DataFrame into a Hudi table. diff --git a/website/versioned_docs/version-0.11.0/concurrency_control.md b/website/versioned_docs/version-0.11.0/concurrency_control.md index e71cb4a8f2..aec626f9d2 100644 --- a/website/versioned_docs/version-0.11.0/concurrency_control.md +++ b/website/versioned_docs/version-0.11.0/concurrency_control.md @@ -70,7 +70,6 @@ hoodie.write.lock.hivemetastore.table `The HiveMetastore URI's are picked up from the hadoop configuration file loaded during runtime.` **`Amazon DynamoDB`** based lock provider - Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters ``` @@ -78,7 +77,10 @@ hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLoc hoodie.write.lock.dynamodb.table hoodie.write.lock.dynamodb.partition_key hoodie.write.lock.dynamodb.region +hoodie.write.lock.dynamodb.endpoint_url +hoodie.write.lock.dynamodb.billing_mode ``` + Also, to set up the credentials for accessing AWS resources, customers can pass the following props to Hudi jobs: ``` hoodie.aws.access.key @@ -87,6 +89,35 @@ hoodie.aws.session.token ``` If not configured, Hudi falls back to use [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html). +IAM policy for your service instance will need to add the following permissions: + +```json +{ + "Sid":"DynamoDBLocksTable", + "Effect": "Allow", + "Action": [ + "dynamodb:CreateTable", + "dynamodb:DeleteItem", + "dynamodb:DescribeTable", + "dynamodb:GetItem", + "dynamodb:PutItem", + "dynamodb:Scan", + "dynamodb:UpdateItem" + ], + "Resource": "arn:${Partition}:dynamodb:${Region}:${Account}:table/${TableName}" +} +``` +- `TableName` : same as `hoodie.write.lock.dynamodb.partition_key` +- `Region`: same as `hoodie.write.lock.dynamodb.region` + +AWS SDK dependencies are not bundled with Hudi from v0.10.x and will need to be added to your classpath. +Add the following Maven packages (check the latest versions at time of install): +``` +com.amazonaws:dynamodb-lock-client +com.amazonaws:aws-java-sdk-dynamodb +com.amazonaws:aws-java-sdk-core +``` + ## Datasource Writer The `hudi-spark` module offers the DataSource API to write (and read) a Spark DataFrame into a Hudi table. diff --git a/website/versioned_docs/version-0.11.1/concurrency_control.md b/website/versioned_docs/version-0.11.1/concurrency_control.md index e71cb4a8f2..aec626f9d2 100644 --- a/website/versioned_docs/version-0.11.1/concurrency_control.md +++ b/website/versioned_docs/version-0.11.1/concurrency_control.md @@ -70,7 +70,6 @@ hoodie.write.lock.hivemetastore.table `The HiveMetastore URI's are picked up from the hadoop configuration file loaded during runtime.` **`Amazon DynamoDB`** based lock provider - Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters ``` @@ -78,7 +77,10 @@ hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLoc hoodie.write.lock.dynamodb.table hoodie.write.lock.dynamodb.partition_key hoodie.write.lock.dynamodb.region +hoodie.write.lock.dynamodb.endpoint_url +hoodie.write.lock.dynamodb.billing_mode ``` + Also, to set up the credentials for accessing AWS resources, customers can pass the following props to Hudi jobs: ``` hoodie.aws.access.key @@ -87,6 +89,35 @@ hoodie.aws.session.token ``` If not configured, Hudi falls back to use [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html). +IAM policy for your service instance will need to add the following permissions: + +```json +{ + "Sid":"DynamoDBLocksTable", + "Effect": "Allow", + "Action": [ + "dynamodb:CreateTable", + "dynamodb:DeleteItem", + "dynamodb:DescribeTable", + "dynamodb:GetItem", + "dynamodb:PutItem", + "dynamodb:Scan", + "dynamodb:UpdateItem" + ], + "Resource": "arn:${Partition}:dynamodb:${Region}:${Account}:table/${TableName}" +} +``` +- `TableName` : same as `hoodie.write.lock.dynamodb.partition_key` +- `Region`: same as `hoodie.write.lock.dynamodb.region` + +AWS SDK dependencies are not bundled with Hudi from v0.10.x and will need to be added to your classpath. +Add the following Maven packages (check the latest versions at time of install): +``` +com.amazonaws:dynamodb-lock-client +com.amazonaws:aws-java-sdk-dynamodb +com.amazonaws:aws-java-sdk-core +``` + ## Datasource Writer The `hudi-spark` module offers the DataSource API to write (and read) a Spark DataFrame into a Hudi table.
