[
https://issues.apache.org/jira/browse/HUDI-9254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lokesh Jain updated HUDI-9254:
------------------------------
Description:
Only InProcessLockProvider is thread safe as of now. The other lock providers
are not thread safe. The Jira aims to ensure all the lock providers which
support thread safety are made thread safe.
- InProcessLockProvider uses ReentrantReadWriteLock which inherently is thread
safe.
- DynamoDBBasedLockProvider uses AmazonDynamoDBLockClient which is thread safe
but it stores the lock item as an instance variable which can cause issues with
multiple threads.
- Similarly ZookeeperBasedLockProvider uses CuratorFramework which is thread
safe but it stores InterProcessMutex as instance variable. The
InterProcessMutex is then released during unlock.
- HiveMetastoreBasedLockProvider uses IMetaStoreClient which inherently is not
thread safe.
- FileSystemBasedLockProvider is also not thread safe since it stores LockInfo
and currentOwnerLockInfo as instance variables.
Currently lock provider is closed during unlock and if we are sharing the lock
provider amongst multiple threads, we would need to fix the close semantics for
lock provider as well.
was:
Only InProcessLockProvider is thread safe as of now. The other lock providers
are not thread safe. The Jira aims to ensure all the lock providers which
support thread safety are made thread safe.
- InProcessLockProvider uses ReentrantReadWriteLock which inherently is thread
safe.
- DynamoDBBasedLockProvider uses AmazonDynamoDBLockClient which is thread safe
but it stores the lock item as an instance variable which can cause issues with
multiple threads.
- Similarly ZookeeperBasedLockProvider uses CuratorFramework which is thread
safe but it stores InterProcessMutex as instance variable. The
InterProcessMutex is then released during unlock.
- HiveMetastoreBasedLockProvider uses IMetaStoreClient which inherently is not
thread safe.
- FileSystemBasedLockProvider is also not thread safe since it stores LockInfo
and currentOwnerLockInfo as instance variables.
> Ensure lock providers are thread safe
> -------------------------------------
>
> Key: HUDI-9254
> URL: https://issues.apache.org/jira/browse/HUDI-9254
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Lokesh Jain
> Priority: Critical
> Fix For: 1.1.0
>
>
> Only InProcessLockProvider is thread safe as of now. The other lock providers
> are not thread safe. The Jira aims to ensure all the lock providers which
> support thread safety are made thread safe.
> - InProcessLockProvider uses ReentrantReadWriteLock which inherently is
> thread safe.
> - DynamoDBBasedLockProvider uses AmazonDynamoDBLockClient which is thread
> safe but it stores the lock item as an instance variable which can cause
> issues with multiple threads.
> - Similarly ZookeeperBasedLockProvider uses CuratorFramework which is thread
> safe but it stores InterProcessMutex as instance variable. The
> InterProcessMutex is then released during unlock.
> - HiveMetastoreBasedLockProvider uses IMetaStoreClient which inherently is
> not thread safe.
> - FileSystemBasedLockProvider is also not thread safe since it stores
> LockInfo and currentOwnerLockInfo as instance variables.
>
> Currently lock provider is closed during unlock and if we are sharing the
> lock provider amongst multiple threads, we would need to fix the close
> semantics for lock provider as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
