[
https://issues.apache.org/jira/browse/HUDI-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382962#comment-17382962
]
Dave Hagman edited comment on HUDI-2173 at 7/19/21, 2:57 AM:
-------------------------------------------------------------
Very basic pseudocode for an optimistic locking implementation:
* Fetch lock object for partition
- If lock object does not exist:
-- Issue a Put request for the partition, locked = true
-- If the Put request succeeds, return the Lock
-- If the Put request fails
---
---- Issue GET request for the partition lock
---- If locked = true then start monitor to wait for lock release
---- If locked = false, submit PUT request to set locked = true with latest
version number
***** If PUT request fails again, start monitor to wait for lock release
- If the lock object exists:
-- If locked = false, submit PUT request to set locked = true with latest
version number
*** If PUT request fails, start monitor to wait for lock release
-- If locked = true, start monitor to wait for lock release
The two main entry scenarios depend on the existence of a lock object.
While an object is locked by another client we need to monitor the lock for
release, accounting for the possibility of failure to release the lock by the
lock owner. I will dive deeper into the failure recovery semantics already in
use by the other lock providers here.
was (Author: dave_hagman):
Very basic pseudocode for an optimistic locking implementation:
* Fetch lock object for partition
- If lock object does not exist:
-- Issue a Put request for the partition, locked = true
-- If the Put request succeeds, return the Lock
-- If the Put request fails
---- Issue GET request for the partition lock
---- If locked = true then start monitor to wait for lock release
---- If locked = false, submit PUT request to set locked = true with latest
version number
------ If PUT request fails again, start monitor to wait for lock release
- If the lock object exists:
-- If locked = false, submit PUT request to set locked = true with latest
version number
---- If PUT request fails again, start monitor to wait for lock release
-- If locked = true, start monitor to wait for lock release
The two main entry scenarios depend on the existence of a lock object.
While an object is locked by another client we need to monitor the lock for
release, accounting for the possibility of failure to release the lock by the
lock owner. I will dive deeper into the failure recovery semantics already in
use by the other lock providers here.
> Implement a DynamoDB based LockProvider
> ---------------------------------------
>
> Key: HUDI-2173
> URL: https://issues.apache.org/jira/browse/HUDI-2173
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Writer Core
> Reporter: Vinoth Chandar
> Assignee: Dave Hagman
> Priority: Major
> Fix For: 0.10.0
>
>
> Currently, we have ZK and HMS based Lock providers, which can be limited to
> co-ordinating across a single EMR or Hadoop cluster.
> For aws users, DynamoDB is a readily available , fully managed , geo
> replicated datastore, that can actually be used to hold locks, that can now
> span across EMR/hadoop clusters.
> This effort involves supporting a new `DynamoDB` lock provider that
> implements org.apache.hudi.common.lock.LockProvider. We can place the
> implementation itself in hudi-client-common, so it can be used across Spark,
> Flink, Deltastreamer etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
