This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new a0e0138cfcd2 docs(concurrency): document the implicit partition key
DynamoDB lock provider (#19591)
a0e0138cfcd2 is described below
commit a0e0138cfcd24a1c7dfd73970cfcfbe69cbc5dbd
Author: deepakpanda93 <[email protected]>
AuthorDate: Wed Aug 19 14:30:57 2026 +0530
docs(concurrency): document the implicit partition key DynamoDB lock
provider (#19591)
* docs(concurrency): document the implicit partition key DynamoDB lock
provider
The Distributed Locking section covers the storage-based, ZooKeeper,
HiveMetastore, DynamoDB and FileSystem providers but never mentions
DynamoDBBasedImplicitPartitionKeyLockProvider. Confirmed the gap:
"ImplicitPartitionKey" appears nowhere under website/docs, versioned_docs or
learn. This is what HUDI-8964 asks for, and matches the backlog triage note
on
the issue.
What it does, from the class at release-1.2.0. It extends
DynamoDBBasedLockProviderBase and overrides getDynamoDBPartitionKey to
derive
the key from the table base path rather than read it from config:
HashID.generateXXHashAsString(s3aToS3(basePath), BITS_64). The s3aToS3 call
means a table addressed as s3a:// and s3:// maps to one lock.
Why it matters. DynamoDBBasedLockProvider requires
hoodie.write.lock.dynamodb.partition_key -- it throws
"Config key is not found" when absent -- and the config has no default
value,
only an infer function falling back to HoodieTableConfig.NAME. So two tables
sharing a name resolve to the same lock and serialize writers that never
touch
the same data. Deriving from base path removes that without per-table
config.
Note this is inference rather than a literal default, which is why the
section
says "infers it from the table name" rather than "defaults to".
Applied to next and every 1.x versioned copy. Checked applicability per
release rather than assuming: the class is present at release-1.0.0 through
release-1.2.0, and the implementation is behaviourally identical in all five
-- the only diff between the two source variants is an import moving from
S3Utils.s3aToS3 to FSUtils.s3aToS3. No open issue reports it broken on any
release; the only issue referencing the class is this documentation task.
The page structure differs by version, so the section is placed to match:
next, 1.1.1 and 1.2.0 use "### DynamoDB-Based Lock Provider" and get an h3,
while 1.0.0, 1.0.1 and 1.0.2 use "#### Amazon DynamoDB based" and get an h4,
each inserted before that version's FileSystem provider heading. Verified in
the rendered output that the new heading sits at the same level as its
siblings on both layouts.
Closes #16814.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* docs(concurrency): drop the "requires but infers" ambiguity on partition
key
Review feedback on apache/hudi#19591: saying the standard provider
"requires"
hoodie.write.lock.dynamodb.partition_key and then that Hudi "infers it from
the table name" when unset reads as self-contradictory, leaving the reader
unsure whether it fails or falls back.
Traced it, and the answer is neither phrasing on its own.
DynamoDbBasedLockConfig.from
copies the caller props and then calls setDefaults, and
HoodieConfig.setDefaultValue
materializes an inferred value into the props when the property has an infer
function. Only after that does
DynamoDBBasedLockProvider.getDynamoDBPartitionKey
run checkArgument(config.contains(DYNAMODB_LOCK_PARTITION_KEY)).
HoodieConfig.contains
looks only at the props map, so the outcome is:
hoodie.table.name present -> inference writes partition_key into props,
contains passes, key is the table name
neither present -> contains fails, "Config key is not found"
So it does fall back in every normal write, and it can still throw in the
corner case where the table name is absent too. The paragraph now avoids
asserting either extreme:
The standard provider takes its partition key from
hoodie.write.lock.dynamodb.partition_key, which you rarely set: when it is
absent, Hudi fills it in from the table name.
The reviewer also noted this sat in tension with the pre-existing sentence
in
the section above, which called the table name a "default". The config is
declared noDefaultValue with an infer function, so "default" was loose;
changed that parenthetical to "inferred from the table name when unset" in
the
three copies that carry that wording. The 1.0.x copies word the same point
differently and needed no change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---
website/docs/concurrency_control.md | 28 +++++++++++++++++++++-
.../version-1.0.0/concurrency_control.md | 26 ++++++++++++++++++++
.../version-1.0.1/concurrency_control.md | 26 ++++++++++++++++++++
.../version-1.0.2/concurrency_control.md | 26 ++++++++++++++++++++
.../version-1.1.1/concurrency_control.md | 28 +++++++++++++++++++++-
.../version-1.2.0/concurrency_control.md | 28 +++++++++++++++++++++-
6 files changed, 159 insertions(+), 3 deletions(-)
diff --git a/website/docs/concurrency_control.md
b/website/docs/concurrency_control.md
index 85c3cf697d98..acb8eea302a6 100644
--- a/website/docs/concurrency_control.md
+++ b/website/docs/concurrency_control.md
@@ -145,7 +145,7 @@ The Amazon DynamoDB–based lock provider supports
multi-writing across clusters
Further configurations: [DynamoDB-Based Locks
Configurations](configurations.md#DynamoDB-based-Locks-Configurations)
-Table creation: Hudi auto-creates the DynamoDB table specified by
`hoodie.write.lock.dynamodb.table`. If using an existing table, ensure a `key`
attribute (as partition key) exists. `hoodie.write.lock.dynamodb.partition_key`
is the value written for the partition key (default: table name), ensuring
multiple writers share the same lock.
+Table creation: Hudi auto-creates the DynamoDB table specified by
`hoodie.write.lock.dynamodb.table`. If using an existing table, ensure a `key`
attribute (as partition key) exists. `hoodie.write.lock.dynamodb.partition_key`
is the value written for the partition key (inferred from the table name when
unset), ensuring multiple writers share the same lock.
Credential props (if not using the default provider chain):
@@ -188,6 +188,32 @@ com.amazonaws:aws-java-sdk-dynamodb
com.amazonaws:aws-java-sdk-core
```
+### DynamoDB-Based Lock Provider with Implicit Partition Key
+
+```properties
+hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedImplicitPartitionKeyLockProvider
+```
+
+This variant behaves like the DynamoDB-based lock provider above, except in
how it determines the DynamoDB partition
+key. Rather than reading `hoodie.write.lock.dynamodb.partition_key`, it
derives the key from the table's base path: the
+64-bit xxHash of that path, with `s3a://` normalized to `s3://` so that
writers reaching the same table through either
+scheme take the same lock.
+
+Prefer it when many tables share one lock table. The standard provider takes
its partition key from
+`hoodie.write.lock.dynamodb.partition_key`, which you rarely set: when it is
absent, Hudi fills it in from the table
+name. Two tables that happen to share a name, in different databases or under
different paths, therefore resolve to the
+same lock and serialize writers that never touch the same data. Deriving the
key from the base path keeps it unique per
+table with no per-table configuration.
+
+Everything else is unchanged: lock table, region, billing mode, endpoint URL,
credentials and IAM permissions all work
+as described above, and `hoodie.write.lock.dynamodb.partition_key` is not read.
+
+:::note
+Because rows are keyed by a hash rather than the table name, entries in the
DynamoDB lock table are not recognizable at
+a glance. The provider logs the base path together with its derived key when
acquiring a lock, which is how to map one
+to the other.
+:::
+
### FileSystem based lock provider
FileSystem based lock provider supports multiple writers across different
jobs/applications based on atomic create/delete operations of the underlying
filesystem.
diff --git a/website/versioned_docs/version-1.0.0/concurrency_control.md
b/website/versioned_docs/version-1.0.0/concurrency_control.md
index 7a149329e978..d0b5fb473966 100644
--- a/website/versioned_docs/version-1.0.0/concurrency_control.md
+++ b/website/versioned_docs/version-1.0.0/concurrency_control.md
@@ -128,6 +128,32 @@ com.amazonaws:aws-java-sdk-dynamodb
com.amazonaws:aws-java-sdk-core
```
+#### DynamoDB-Based Lock Provider with Implicit Partition Key
+
+```properties
+hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedImplicitPartitionKeyLockProvider
+```
+
+This variant behaves like the DynamoDB-based lock provider above, except in
how it determines the DynamoDB partition
+key. Rather than reading `hoodie.write.lock.dynamodb.partition_key`, it
derives the key from the table's base path: the
+64-bit xxHash of that path, with `s3a://` normalized to `s3://` so that
writers reaching the same table through either
+scheme take the same lock.
+
+Prefer it when many tables share one lock table. The standard provider takes
its partition key from
+`hoodie.write.lock.dynamodb.partition_key`, which you rarely set: when it is
absent, Hudi fills it in from the table
+name. Two tables that happen to share a name, in different databases or under
different paths, therefore resolve to the
+same lock and serialize writers that never touch the same data. Deriving the
key from the base path keeps it unique per
+table with no per-table configuration.
+
+Everything else is unchanged: lock table, region, billing mode, endpoint URL,
credentials and IAM permissions all work
+as described above, and `hoodie.write.lock.dynamodb.partition_key` is not read.
+
+:::note
+Because rows are keyed by a hash rather than the table name, entries in the
DynamoDB lock table are not recognizable at
+a glance. The provider logs the base path together with its derived key when
acquiring a lock, which is how to map one
+to the other.
+:::
+
#### FileSystem based (not for production use)
FileSystem based lock provider supports multiple writers cross different
jobs/applications based on atomic create/delete operations of the underlying
filesystem.
diff --git a/website/versioned_docs/version-1.0.1/concurrency_control.md
b/website/versioned_docs/version-1.0.1/concurrency_control.md
index 1e233b5fc69e..7f753c27c764 100644
--- a/website/versioned_docs/version-1.0.1/concurrency_control.md
+++ b/website/versioned_docs/version-1.0.1/concurrency_control.md
@@ -143,6 +143,32 @@ com.amazonaws:aws-java-sdk-dynamodb
com.amazonaws:aws-java-sdk-core
```
+#### DynamoDB-Based Lock Provider with Implicit Partition Key
+
+```properties
+hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedImplicitPartitionKeyLockProvider
+```
+
+This variant behaves like the DynamoDB-based lock provider above, except in
how it determines the DynamoDB partition
+key. Rather than reading `hoodie.write.lock.dynamodb.partition_key`, it
derives the key from the table's base path: the
+64-bit xxHash of that path, with `s3a://` normalized to `s3://` so that
writers reaching the same table through either
+scheme take the same lock.
+
+Prefer it when many tables share one lock table. The standard provider takes
its partition key from
+`hoodie.write.lock.dynamodb.partition_key`, which you rarely set: when it is
absent, Hudi fills it in from the table
+name. Two tables that happen to share a name, in different databases or under
different paths, therefore resolve to the
+same lock and serialize writers that never touch the same data. Deriving the
key from the base path keeps it unique per
+table with no per-table configuration.
+
+Everything else is unchanged: lock table, region, billing mode, endpoint URL,
credentials and IAM permissions all work
+as described above, and `hoodie.write.lock.dynamodb.partition_key` is not read.
+
+:::note
+Because rows are keyed by a hash rather than the table name, entries in the
DynamoDB lock table are not recognizable at
+a glance. The provider logs the base path together with its derived key when
acquiring a lock, which is how to map one
+to the other.
+:::
+
#### FileSystem based (not for production use)
FileSystem based lock provider supports multiple writers cross different
jobs/applications based on atomic create/delete operations of the underlying
filesystem.
diff --git a/website/versioned_docs/version-1.0.2/concurrency_control.md
b/website/versioned_docs/version-1.0.2/concurrency_control.md
index 76e965429c9f..fa848e542018 100644
--- a/website/versioned_docs/version-1.0.2/concurrency_control.md
+++ b/website/versioned_docs/version-1.0.2/concurrency_control.md
@@ -150,6 +150,32 @@ com.amazonaws:aws-java-sdk-dynamodb
com.amazonaws:aws-java-sdk-core
```
+#### DynamoDB-Based Lock Provider with Implicit Partition Key
+
+```properties
+hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedImplicitPartitionKeyLockProvider
+```
+
+This variant behaves like the DynamoDB-based lock provider above, except in
how it determines the DynamoDB partition
+key. Rather than reading `hoodie.write.lock.dynamodb.partition_key`, it
derives the key from the table's base path: the
+64-bit xxHash of that path, with `s3a://` normalized to `s3://` so that
writers reaching the same table through either
+scheme take the same lock.
+
+Prefer it when many tables share one lock table. The standard provider takes
its partition key from
+`hoodie.write.lock.dynamodb.partition_key`, which you rarely set: when it is
absent, Hudi fills it in from the table
+name. Two tables that happen to share a name, in different databases or under
different paths, therefore resolve to the
+same lock and serialize writers that never touch the same data. Deriving the
key from the base path keeps it unique per
+table with no per-table configuration.
+
+Everything else is unchanged: lock table, region, billing mode, endpoint URL,
credentials and IAM permissions all work
+as described above, and `hoodie.write.lock.dynamodb.partition_key` is not read.
+
+:::note
+Because rows are keyed by a hash rather than the table name, entries in the
DynamoDB lock table are not recognizable at
+a glance. The provider logs the base path together with its derived key when
acquiring a lock, which is how to map one
+to the other.
+:::
+
#### FileSystem based (not for production use)
FileSystem based lock provider supports multiple writers cross different
jobs/applications based on atomic create/delete operations of the underlying
filesystem.
diff --git a/website/versioned_docs/version-1.1.1/concurrency_control.md
b/website/versioned_docs/version-1.1.1/concurrency_control.md
index 8fb5aa1cca93..284f522092c1 100644
--- a/website/versioned_docs/version-1.1.1/concurrency_control.md
+++ b/website/versioned_docs/version-1.1.1/concurrency_control.md
@@ -123,7 +123,7 @@ The Amazon DynamoDB–based lock provider supports
multi-writing across clusters
Further configurations: [DynamoDB-Based Locks
Configurations](configurations.md#DynamoDB-based-Locks-Configurations)
-Table creation: Hudi auto-creates the DynamoDB table specified by
`hoodie.write.lock.dynamodb.table`. If using an existing table, ensure a `key`
attribute (as partition key) exists. `hoodie.write.lock.dynamodb.partition_key`
is the value written for the partition key (default: table name), ensuring
multiple writers share the same lock.
+Table creation: Hudi auto-creates the DynamoDB table specified by
`hoodie.write.lock.dynamodb.table`. If using an existing table, ensure a `key`
attribute (as partition key) exists. `hoodie.write.lock.dynamodb.partition_key`
is the value written for the partition key (inferred from the table name when
unset), ensuring multiple writers share the same lock.
Credential props (if not using the default provider chain):
@@ -166,6 +166,32 @@ com.amazonaws:aws-java-sdk-dynamodb
com.amazonaws:aws-java-sdk-core
```
+### DynamoDB-Based Lock Provider with Implicit Partition Key
+
+```properties
+hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedImplicitPartitionKeyLockProvider
+```
+
+This variant behaves like the DynamoDB-based lock provider above, except in
how it determines the DynamoDB partition
+key. Rather than reading `hoodie.write.lock.dynamodb.partition_key`, it
derives the key from the table's base path: the
+64-bit xxHash of that path, with `s3a://` normalized to `s3://` so that
writers reaching the same table through either
+scheme take the same lock.
+
+Prefer it when many tables share one lock table. The standard provider takes
its partition key from
+`hoodie.write.lock.dynamodb.partition_key`, which you rarely set: when it is
absent, Hudi fills it in from the table
+name. Two tables that happen to share a name, in different databases or under
different paths, therefore resolve to the
+same lock and serialize writers that never touch the same data. Deriving the
key from the base path keeps it unique per
+table with no per-table configuration.
+
+Everything else is unchanged: lock table, region, billing mode, endpoint URL,
credentials and IAM permissions all work
+as described above, and `hoodie.write.lock.dynamodb.partition_key` is not read.
+
+:::note
+Because rows are keyed by a hash rather than the table name, entries in the
DynamoDB lock table are not recognizable at
+a glance. The provider logs the base path together with its derived key when
acquiring a lock, which is how to map one
+to the other.
+:::
+
### FileSystem based lock provider
FileSystem based lock provider supports multiple writers across different
jobs/applications based on atomic create/delete operations of the underlying
filesystem.
diff --git a/website/versioned_docs/version-1.2.0/concurrency_control.md
b/website/versioned_docs/version-1.2.0/concurrency_control.md
index f58d14524726..997cf30b1c20 100644
--- a/website/versioned_docs/version-1.2.0/concurrency_control.md
+++ b/website/versioned_docs/version-1.2.0/concurrency_control.md
@@ -145,7 +145,7 @@ The Amazon DynamoDB–based lock provider supports
multi-writing across clusters
Further configurations: [DynamoDB-Based Locks
Configurations](configurations.md#DynamoDB-based-Locks-Configurations)
-Table creation: Hudi auto-creates the DynamoDB table specified by
`hoodie.write.lock.dynamodb.table`. If using an existing table, ensure a `key`
attribute (as partition key) exists. `hoodie.write.lock.dynamodb.partition_key`
is the value written for the partition key (default: table name), ensuring
multiple writers share the same lock.
+Table creation: Hudi auto-creates the DynamoDB table specified by
`hoodie.write.lock.dynamodb.table`. If using an existing table, ensure a `key`
attribute (as partition key) exists. `hoodie.write.lock.dynamodb.partition_key`
is the value written for the partition key (inferred from the table name when
unset), ensuring multiple writers share the same lock.
Credential props (if not using the default provider chain):
@@ -188,6 +188,32 @@ com.amazonaws:aws-java-sdk-dynamodb
com.amazonaws:aws-java-sdk-core
```
+### DynamoDB-Based Lock Provider with Implicit Partition Key
+
+```properties
+hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedImplicitPartitionKeyLockProvider
+```
+
+This variant behaves like the DynamoDB-based lock provider above, except in
how it determines the DynamoDB partition
+key. Rather than reading `hoodie.write.lock.dynamodb.partition_key`, it
derives the key from the table's base path: the
+64-bit xxHash of that path, with `s3a://` normalized to `s3://` so that
writers reaching the same table through either
+scheme take the same lock.
+
+Prefer it when many tables share one lock table. The standard provider takes
its partition key from
+`hoodie.write.lock.dynamodb.partition_key`, which you rarely set: when it is
absent, Hudi fills it in from the table
+name. Two tables that happen to share a name, in different databases or under
different paths, therefore resolve to the
+same lock and serialize writers that never touch the same data. Deriving the
key from the base path keeps it unique per
+table with no per-table configuration.
+
+Everything else is unchanged: lock table, region, billing mode, endpoint URL,
credentials and IAM permissions all work
+as described above, and `hoodie.write.lock.dynamodb.partition_key` is not read.
+
+:::note
+Because rows are keyed by a hash rather than the table name, entries in the
DynamoDB lock table are not recognizable at
+a glance. The provider logs the base path together with its derived key when
acquiring a lock, which is how to map one
+to the other.
+:::
+
### FileSystem based lock provider
FileSystem based lock provider supports multiple writers across different
jobs/applications based on atomic create/delete operations of the underlying
filesystem.
