swamirishi commented on PR #10876: URL: https://github.com/apache/ozone/pull/10876#issuecomment-5101524609
> > No that should not happen as long as you decide to do the iteration and delete range inside validateAndUpdateCache method of > > @swamirishi This will kill performance, we cannot iterate every single key in validateAndUpdateCache and split them into multiple deleteRange groups since it will block subsequent writes. The lifecycle configuration scanning should not be run in a critical path. > > > I see you already have a ticket for this https://issues.apache.org/jira/browse/HDDS-14613 . Let us work on this particular change. > > I closed that for the reason above. Not exactly IMO the iteration that we are doing is very unoptimal. The actual algo would be something like this: Say we want to delete k01, k02, k04, k05, k10, k11 and the key table has keys from k00..k99 1) Init rocksdb iterator. Create a tmpList to keep track of OmKeyInfos and deleteRangeList 2) The first key in iterator is k00 and delete keys is k01. If pointer at deleted key is greater than iterator then iterator seeks to deleted key by adding the current deleteRange to the deleteRangeList. So the iterator is at k01 now. 3) Both iterator is k01 and delete key is k01 so add k01 to OmKeyInfoList and move pointer of delete Key list to k02 and itr.next(). Similarly K02 will match and would be added as well and we proceed to next. 4) Now DeleteKey is at K04 and rocksdbIterator is at k03. So we add deleteKeyRange [k01, k03) in deleteKeyRangeList and seek to k04. The algo proceeds on like this till we don't iterate through all keys in the deleteKeyList. This would be much more optimal than whatever we are doing right now. Every rocksdb get is going to be an equivalent of randomized get in the LSM which would be very unoptimal to sequential gets. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
