[
https://issues.apache.org/jira/browse/HUDI-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated HUDI-6123:
-----------------------------
Description:
Currently, the `hoodie.auto.adjust.lock.configs` opiton is by default false,
while MDT is by default enabled,
for single writer with any async table services enabled, the MDT commit is not
protected by any lock providers,
this could incur inconsistentcy between dataset and metadata.
At least by default, we should make it work. Imagine a simple use case: MOR
single writer + async compaction.
Change list:
1. Change option `hoodie.auto.adjust.lock.configs` by default as true.
2. Restrict the option `hoodie.auto.adjust.lock.configs` to take effect only
for single writer, because in multi-writer scenarios, the
`InProcessLockProvider` can not work as expected for multiple processes.
was:
Currently, the `hoodie.auto.adjust.lock.configs` opiton is by default false,
while MDT is by default enabled,
for single writer with any async table services enabled, that means the MDT
commit is not proteced by any lock providers,
this could cause inconsistentcy between data set and metadata.
At least by default, we should make it work. Imagine a simple use case: MOR
single writer + async compaction.
Change list:
1. Change option `hoodie.auto.adjust.lock.configs` by default as true.
2. Restrict the option `hoodie.auto.adjust.lock.configs` to take effect only
for single writer, because for multi-writer, the
`InProcessLockProvider` can not work for multiple processes.
> Should add lock guard by default for single table with async table services
> and MDT
> -----------------------------------------------------------------------------------
>
> Key: HUDI-6123
> URL: https://issues.apache.org/jira/browse/HUDI-6123
> Project: Apache Hudi
> Issue Type: Bug
> Components: writer-core
> Affects Versions: 0.13.0
> Reporter: Danny Chen
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.13.1, 0.14.0
>
>
> Currently, the `hoodie.auto.adjust.lock.configs` opiton is by default false,
> while MDT is by default enabled,
> for single writer with any async table services enabled, the MDT commit is
> not protected by any lock providers,
> this could incur inconsistentcy between dataset and metadata.
> At least by default, we should make it work. Imagine a simple use case: MOR
> single writer + async compaction.
>
> Change list:
> 1. Change option `hoodie.auto.adjust.lock.configs` by default as true.
> 2. Restrict the option `hoodie.auto.adjust.lock.configs` to take effect only
> for single writer, because in multi-writer scenarios, the
> `InProcessLockProvider` can not work as expected for multiple processes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
