[
https://issues.apache.org/jira/browse/HBASE-29634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huginn updated HBASE-29634:
---------------------------
Description:
In CleanerChore, a thread is assigned to each subdirectory under the archive
directory to identify deletable files. In determining files that are not
referenced by snapshots, all threads share a single SnapshotHFileCleaner and
SnapshotHFileCache.
Since taking snapshots changes the set of files referenced by snapshots, taking
snapshots and the SnapshotHFileCleaner are mutually exclusive operations. Thus,
taking snapshots requires acquiring the takingSnapshot.readLock, while the
SnapshotHFileCleaner, when selecting unreferenced files, must acquire the
takingSnapshot.writeLock. Moreover, because the SnapshotHFileCache is shared
among threads, different threads must acquire the takingSnapshot.writeLock
before refreshing the cache.
With the current implementation, the coarse granularity of the lock resulted in
a serialized process for selecting unreferenced files in multi-threaded
environments, meaning that increasing the number of cleaning threads did not
significantly improve efficiency. To reduce lock granularity and enhance
cleaning efficiency, we introduced the following optimizations:
1. Optimistic locking based on version: Since taking snapshot is not a
high-frequency operation, this allows getting unreferenced file to proceed
without requiring locks throughout the entire process.
2. Fine-grained locking: When multiple threads concurrently determine
unreferenced files, locks are only required when updating shared objects — the
SnapshotHFileCache and the snapshot cache version marker.
> Snapshot file cache version based optimization
> ----------------------------------------------
>
> Key: HBASE-29634
> URL: https://issues.apache.org/jira/browse/HBASE-29634
> Project: HBase
> Issue Type: Improvement
> Reporter: huginn
> Assignee: huginn
> Priority: Minor
> Labels: pull-request-available
>
> In CleanerChore, a thread is assigned to each subdirectory under the archive
> directory to identify deletable files. In determining files that are not
> referenced by snapshots, all threads share a single SnapshotHFileCleaner and
> SnapshotHFileCache.
> Since taking snapshots changes the set of files referenced by snapshots,
> taking snapshots and the SnapshotHFileCleaner are mutually exclusive
> operations. Thus, taking snapshots requires acquiring the
> takingSnapshot.readLock, while the SnapshotHFileCleaner, when selecting
> unreferenced files, must acquire the takingSnapshot.writeLock. Moreover,
> because the SnapshotHFileCache is shared among threads, different threads
> must acquire the takingSnapshot.writeLock before refreshing the cache.
> With the current implementation, the coarse granularity of the lock resulted
> in a serialized process for selecting unreferenced files in multi-threaded
> environments, meaning that increasing the number of cleaning threads did not
> significantly improve efficiency. To reduce lock granularity and enhance
> cleaning efficiency, we introduced the following optimizations:
> 1. Optimistic locking based on version: Since taking snapshot is not a
> high-frequency operation, this allows getting unreferenced file to proceed
> without requiring locks throughout the entire process.
> 2. Fine-grained locking: When multiple threads concurrently determine
> unreferenced files, locks are only required when updating shared objects —
> the SnapshotHFileCache and the snapshot cache version marker.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
