n3nash commented on a change in pull request #2698:
URL: https://github.com/apache/hudi/pull/2698#discussion_r598354123
##########
File path: docs/_docs/2_9_concurrency_control.md
##########
@@ -0,0 +1,140 @@
+---
+title: "Concurrent Writes to Hudi Tables"
+permalink: /docs/concurrency_control.html
+summary: In this page, we will discuss how to perform concurrent writes to
Hudi Tables.
+toc: true
+last_modified_at: 2021-03-19T15:59:57-04:00
+---
+
+In this section, we will cover Hudi's concurrency model and describe ways to
ingest data into a Hudi Table from multiple writers; using the
[DeltaStreamer](#deltastreamer) tool as well as
+using the [Hudi datasource](#datasource-writer).
+
+## Supported Concurrency Controls
+
+- **MVCC** : Hudi table services such as compaction, cleaning, clustering
leverage Multi Version Concurrency Control to provide snapshot isolation
+between multiple table service writers and readers. Additionally, using MVCC,
Hudi provides snapshot isolation between an ingestion writer and multiple
concurrent readers.
+ With this model, Hudi supports running any number of table service jobs
concurrently, without any concurrency conflict.
+ This is made possible by ensuring that scheduling plans of such table
services always happens in a single writer mode to ensure no conflict and
avoids race conditions.
+
+- **[NEW] OPTIMISTIC CONCURRENCY** : Write operations such as the ones
described above (UPSERT, INSERT) etc, leverage optimistic concurrency control
to enable multiple ingestion writers to
+the same Hudi Table. Hudi supports `file level OCC`, i.e., for any 2 commits
(or writers) happening to the same table, if they do not have writes to
overlapping files being changed, both writers are allowed to succeed.
+ This feature is currently *experimental* and requires either Zookeeper or
HiveMetastore to acquire locks.
+
+It may be helpful to understand the different guarantees provided by [write
operations](/docs/writing_data.html#write-operations) via Hudi datasource or
the delta streamer.
+
+## Single Writer Guarantees
+
+ - *UPSERT Guarantee*: The target table will NEVER show duplicates.
+ - *INSERT Guarantee*: The target table wilL NEVER have duplicates if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER
out of order.
+
+## Multi Writer Guarantees
+
+With multiple writers using OCC, some of the above guarantees change as follows
+
+- *UPSERT Guarantee*: The target table will NEVER show duplicates.
+- *INSERT Guarantee*: The target table MIGHT have duplicates even if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if
[dedup](/docs/configurations.html#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints MIGHT be out
of order due to multiple writer jobs finishing at different times.
+
+## Configurations
+
+The following properties are needed to be set properly to turn on optimistic
concurrency control.
+
+```
+hoodie.write.concurrency.mode=optimistic_concurrency_control
+hoodie.failed.writes.cleaner.policy=LAZY
+hoodie.writer.lock.provider=<lock-provider-classname>
Review comment:
This is mentioned right after this section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
