[
https://issues.apache.org/jira/browse/HDDS-16289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-16289:
-------------------------------------
Assignee: Andrey Yarovoy
> Narrow the FSO create/commit apply-path bucket write-lock to the
> cache-mutation tail so it does not gate readers of a hot bucket
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16289
> URL: https://issues.apache.org/jira/browse/HDDS-16289
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> *Problem*
> On the FSO write-apply path, {{validateAndUpdateCache}} acquires the
> {{BUCKET_LOCK}} *write* lock and holds it across a block of work that is
> mostly {_}reads of already-committed state{_}, not cache mutation. Because a
> hot bucket hashes to a single lock stripe and the OM applies transactions on
> a *single* Ratis apply thread, the duration of this write-lock hold is
> exactly the window during which every read RPC on that bucket
> ({{{}getBucketInfo{}}}, {{{}getBucketOwner{}}}, {{{}readKeyInfo{}}}) is
> blocked — Java's {{ReentrantReadWriteLock}} blocks incoming readers whenever
> a writer is queued at the head, in both fair and non-fair (default) mode.
> OM {{/stacks}} analysis showed this as recurring flares in handler threads
> sit in {{OzoneManagerLock.acquireLock}} (read side) behind the single apply
> thread's {{{}acquireWriteLock{}}}, with only a handful of readers actually
> inside the critical section. The critical section is not slow because of a
> contended mutation — it is slow because read-heavy work runs under the write
> lock.
> *Where the time goes*
> ({{{}OMFileCreateRequestWithFSO.validateAndUpdateCache{}}}, same shape in
> {{{}OMKeyCommitRequestWithFSO{}}})
> Held under the write lock today:
> * {{verifyDirectoryKeysInPath(...)}} — the FSO path walk: one RocksDB point
> lookup per path segment ({{{}directoryTable.get{}}}) plus a {{fileTable.get}}
> for the leaf. This is the dominant cost and it only *reads* committed/cache
> state.
> * {{getOmKeyInfoFromFileTable(...)}} — another RocksDB read.
> * {{{}checkDirectoryResult{}}}, {{{}checkAllParentsExist{}}},
> {{{}getAllMissingParentDirInfo{}}}, quota checks — validation over the values
> just read.
> * *Only the tail* genuinely needs write exclusivity:
> {{{}addOpenFileTableCacheEntry{}}}, {{{}addDirectoryTableCacheEntries{}}},
> and the in-place bucket quota update ({{{}incrUsedNamespace{}}}).
> The code already hints at this — the comment above
> {{addOpenFileTableCacheEntry}} notes the open-key cache add "can be done
> outside of lock."
> *Proposed change (investigation + implementation)*
> Reorder so the exclusive section covers only the cache mutations that require
> it:
> # Resolve the path and read existing entries / build {{missingParentInfos}}
> and the prepared {{OmKeyInfo}} _before_ taking the write lock (or under the
> shared read lock).
> # Acquire the bucket write lock and, under it, *re-validate the invariants
> that could have changed* (existence/overwrite, quota) against the current
> cache, then apply the cache entries and quota update.
> # Release.
> Because writes are applied by a single thread, this does not change write
> throughput — the prior finding that write-lock _scope_ reduction is low-value
> for write p99 (single apply thread bounds write commit rate) still holds. The
> value here is on a {*}different axis: read availability/latency on hot
> buckets{*}, which the current wide critical section directly harms.
> *Testing*
> * Concurrency test: drive many read RPCs at one bucket while a create/commit
> is applied to that bucket; assert readers block only for the (short) mutation
> tail, not the path-resolution phase.
> * Correctness: existing {{TestOMFileCreateRequestWithFSO}} /
> {{TestOMKeyCommitRequestWithFSO}} plus new cases for the
> re-validation-under-lock path (concurrent overwrite, quota-exceeded
> discovered at re-check).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
