priyeshkaratha commented on code in PR #11241:
URL: https://github.com/apache/ozone/pull/11241#discussion_r4068673996
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java:
##########
@@ -1779,25 +1779,39 @@ public Table<String, OmLifecycleScanState>
getLifecycleScanStateTable() {
* @return list all LifecycleConfigurations.
*/
@Override
- public List<OmLifecycleConfiguration> listLifecycleConfigurations() {
+ public List<OmLifecycleConfiguration> listLifecycleConfigurations() throws
IOException {
List<OmLifecycleConfiguration> result = Lists.newArrayList();
+ Set<String> cachedKeys = new HashSet<>();
- /* lifecycleConfigurationTable is full-cache, so we use cacheIterator. */
+ // lifecycleConfigurationTable uses partial cache, so cacheIterator() only
returns
+ // entries that are currently in memory. Process cache entries first to
handle
+ // any pending writes or pending deletes that have not yet been flushed to
RocksDB.
Iterator<Map.Entry<CacheKey<String>, CacheValue<OmLifecycleConfiguration>>>
Review Comment:
This merges the cache scan and the RocksDB scan without holding any lock.
Since lifecycleConfigurationTable is now partial-cache, there's a window
between the cacheIterator() pass and the double-buffer flush landing in RocksDB
where an entry is in neither snapshot and it's been evicted from the cache but
not yet visible to the RocksDB iterator (or vice versa). That entry would
silently disappear from the result, which for KeyLifecycleService.getTasks()
means a bucket just doesn't get scheduled for a lifecycle scan that cycle, with
no error or log to indicate it happened.
ListIterator.MinHeapIterator solves this same cache/RocksDB merge problem
elsewhere in this file's neighborhood by acquiring a bucket read lock around
the iterator setup (omMetadataManager.getLock().acquireReadLock(BUCKET_LOCK,
...)).
Can we do the same here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
