rajeshkumarchandolu opened a new pull request, #11533: URL: https://github.com/apache/ozone/pull/11533
## What changes were proposed in this pull request? In `KeyLifecycleService#evaluateKeyAndDirTable` (FSO buckets), each directory is evaluated in two passes: first over `keyTable.cacheIterator()`, then over the RocksDB table iterator. Both passes updated `lastScannedKey`. The file table uses `PartialTableCache`, which is backed by a `ConcurrentHashMap`, so the cache pass runs in hash order and `lastScannedKey` can end up at an arbitrary key. If the scan state is saved at that point (either the expired list fills during the cache pass and `handleAndClearFullList` sends the state, or the task aborts on `shouldRun()` at the start of the table loop), that cache key is persisted. On resume, the table iterator does `seek(lastScannedKey)`, which skips every key in that directory that sorts before it. Those keys are not evaluated until the next full scan cycle. Example: a directory has table keys `a, b, c, x`, and key `m` exists only in the cache. The cache pass sets `lastScannedKey = m`, the leader changes, state is saved with `m`, and on resume the scan seeks to `m` and skips `a, b, c`. The cache pass does not seek and is fully re-evaluated on every resume, so it never needs a saved position. This PR stops setting `lastScannedKey` in the cache loop, so the saved position only comes from the sorted table iteration. OBS buckets (`evaluateBucket`) only iterate the table and are not affected. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-16998 ## How was this patch tested? - Added `TestKeyLifecycleService#testCacheOnlyKeyNotSavedAsLastScannedKey`. It commits two keys under `dir1` to the table, adds an expired key `dir1/key9` that exists only in the file table cache, and uses a delete batch size of 1 so the scan state is saved right after the cache pass. It verifies that no saved scan state has `lastScannedKey` set to the cache-only key, and that all expired keys are deleted. - Verified the new test fails without the fix and passes with it, for both state-save configurations. - `TestKeyLifecycleService` passes locally (370 tests, 0 failures). - `checkstyle.sh` passes. ## build : -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
