[
https://issues.apache.org/jira/browse/HUDI-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jian Feng updated HUDI-4065:
----------------------------
Description:
when using HiveMetastoreBasedLockProvider or ZookeeperBasedLockProvider, if the
writer with locker crash before releasing the locker, no writers can obtain
the locker anymore, hence can not cleaning/expiring the lock after a failed
write.
Instead of those lockers depending on external components, FIleBaseLocker only
relies on FileSystem's atomic file creation behavior. an instant will be
written to the file, so it will clean the lock when failing writer rollback.
also can set expire strategy on it.
was:
when using HiveMetastoreBasedLockProvider or ZookeeperBasedLockProvider, if the
writer with locker crash before releasing the locker, no writers can obtain
the locker anymore, hence no cleaning/expiring the lock after a failed write.
Instead of those lockers depending on external components, FIleBaseLocker only
relies on FileSystem's atomic file creation behavior. an instant will be
written to the file, so it will clean the lock when failing writer rollback.
also can set expire strategy on it.
> Implement a self-releasing FileLock for multiple writers scenario
> -----------------------------------------------------------------
>
> Key: HUDI-4065
> URL: https://issues.apache.org/jira/browse/HUDI-4065
> Project: Apache Hudi
> Issue Type: Improvement
> Components: core
> Reporter: Jian Feng
> Assignee: Jian Feng
> Priority: Major
> Labels: multi-writer
>
> when using HiveMetastoreBasedLockProvider or ZookeeperBasedLockProvider, if
> the writer with locker crash before releasing the locker, no writers can
> obtain the locker anymore, hence can not cleaning/expiring the lock after a
> failed write.
> Instead of those lockers depending on external components, FIleBaseLocker
> only relies on FileSystem's atomic file creation behavior. an instant will
> be written to the file, so it will clean the lock when failing writer
> rollback. also can set expire strategy on it.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
