ivandika3 commented on PR #10876: URL: https://github.com/apache/ozone/pull/10876#issuecomment-5087700779
Thanks @ashishkumar50 for checking this patch. > Thanks for the patch, whether you are using kSkip or kForce for the compaction? As in below PR now compaction is using kSkip by default which will not compact bottommost layer and should be fast enough. We are using `kForce`, https://github.com/apache/ozone/pull/10419 is a very recent patch so we are not using this. However, I don't see any good reason why we have to `setExclusiveManualCompaction` rather than just prematurely preventing hypothetical overhead of concurrent compactions (which I don't think is well understood by the Ozone community). For example, we already have a previous `DatanodeStoreSchemaThreeImpl#compactionIfNeeded` in datanode which does not `setExclusiveManualCompaction` and we have no write stall issues. So unless there is a stress test proving that the concurrent manual and automatic compaction actually has any meaningful overhead, we should remove it since now we know that there is a very real production risk (not just hypotheticals). From my understanding the `kForce` is the entire point of the compaction service. If there are a lot of tombstones in the bottommost layer, we want to clean them up. Additionally, is the difference between kSkip and automatic compaction well-understood? > should be fast enough If the N-1 LSM level is very big, the issue might still happen. Unless we have a strong guarantee or data of no write stall, IMO this exclusive compaction adds another unnecessary surprise in OM stability. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
