Lin Liu created HUDI-7740:
-----------------------------
Summary: Sharing Lock between Concurrent Hudi Writers
Key: HUDI-7740
URL: https://issues.apache.org/jira/browse/HUDI-7740
Project: Apache Hudi
Issue Type: Improvement
Components: metadata
Reporter: Lin Liu
Assignee: Ethan Guo
One Hudi table can be ingested by multiple writers concurrently. Without proper
synchronization, data corruptions can happen easily, and the most straight
forward method is to use locks. In-process locks can be used for writers in the
same JVM. For distributed writers, a distributed lock should be utilized.
Given a distributed lock has been generated for these writers, our immediate
question is to share the lock across all these writers reliably. In this
effort, we aim to utilize the `.hoodie` folder as the central place to share
lock information.
The goal of this effort is to make sure in this concurrent scenario,
# Each write operation should be guarded with a lock.
# All writers utilize the same lock at any moment.
# When there is an lock update, the lock update operation should satisfy the
above two conditions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
