bbeaudreault opened a new pull request, #5561: URL: https://github.com/apache/hbase/pull/5561
We routinely see JVM crashes originating from our RegionServerMetricsWrapperRunnable in jdk17 on aarch64. Since that method is so large, it's hard to know exactly where or what the problem is. There is an [issue submitted](https://github.com/adoptium/adoptium-support/issues/951), but no indication of whether or when a fix will come. The core dump indicates its a problem with On Stack Replacement (OSR). Reading up on that, it can kick in when long methods are suddenly deemed hot and attempt to be optimized in place. Taking a stab in the dark based on that, I saw two potential issues in RegionServerMetricsWrapperRunnable: 1. The method is exceptionally long, especially when considering the loops 2. There are over 50 volatile variables updated in the parent class This PR attempts to solve both issues: 1. Most of the volatile variables are replaced by a single volatile RegionMetricAggregate. 2. I tried to break down the giant method into organized units: a. regionserver-level calculations are left in the run() method b. the main `aggregate()` method of RegionMetricAggregate loops and collects region-level metrics c. for each region, an `aggregateStores()` method is called which loops and collects store-level metrics. This may not be perfect, but I think it's an improvement for three reasons: 1. I've been running it internally for a couple weeks and have not seen a crash (previously multiple per day) 2. It attempts to organize the code a little bit so its easier to manage. 3. It's a net-reduction in production code In order to ensure this refactor did not break any metrics, I wrote an exhaustive unit test. It validates each of the aggregated getters, and succeeds against both the old and new implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
