Sammi Chen created HDDS-16998:
---------------------------------
Summary: Skip set the lastScanned fields for Lifecycle task during
cache scan
Key: HDDS-16998
URL: https://issues.apache.org/jira/browse/HDDS-16998
Project: Apache Ozone
Issue Type: Sub-task
Reporter: Sammi Chen
In evaluateKeyAndDirTable (FSO), each directory first iterates
keyTable.cacheIterator() and then the RocksDB iterator, and both set
lastScannedKey. The file table uses PartialTableCache, which is a
ConcurrentHashMap, so the cache pass runs in hash order and lastScannedKey can
end up at an arbitrary key. If scan state is saved at that point (either the
expired list fills inside the cache loop and handleAndClearFullList sends the
state, or the task aborts on shouldRun() at the start of the DB loop), we
persist that cache key. On resume we seek(lastScannedKey) in the DB iterator,
which skips every key in that directory sorting before it. Those keys aren't
evaluated until the next full scan cycle.
Example: directory has DB keys a, b, c, x, and key m is only in cache. The
cache pass sets lastScannedKey = m, the leader changes, state is saved with m,
and on resume we seek to m and skip a, b, c.
Since the cache pass doesn't seek and is fully re-evaluated on every resume
anyway, the simple fix is to not update lastScannedKey in the cache loop, so
the saved position only comes from the sorted DB iteration. OBS
(evaluateBucket) only iterates the DB, so it isn't affected
It's found by Chandolu Rajesh Kumar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
