[
https://issues.apache.org/jira/browse/PHOENIX-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17530109#comment-17530109
]
ASF GitHub Bot commented on PHOENIX-6699:
-----------------------------------------
stoty commented on code in PR #1431:
URL: https://github.com/apache/phoenix/pull/1431#discussion_r861979379
##########
phoenix-core/src/main/java/org/apache/phoenix/monitoring/MetricUtil.java:
##########
@@ -38,4 +46,22 @@ public static MetricsStopWatch getMetricsStopWatch(boolean
isRequestMetricsEnabl
return new MetricsStopWatch(true);
}
+ // We need to cover the case when JmxCacheBuster has just stopped the
HBase metrics
+ // system, and not accidentally overwrite the DefaultMetricsSystem
singleton.
+ // See PHOENIX-6699
+ public static boolean isDefaultMetricsInitialized() {
+ try {
+ MetricsSystemImpl metrics = (MetricsSystemImpl)
DefaultMetricsSystem.instance();
+ Field prefixField =
MetricsSystemImpl.class.getDeclaredField("prefix");
Review Comment:
_MetricsSystemImpl.monitoring_ is not useful for this purpose, as it is set
to false when the metrics system is stopped.
This is why we can overwrite the stopped Default Metrics System:
https://github.com/apache/hadoop/blob/7bd7725532fd139d2e0e1662df7700f7ab95067a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java#L156
_MetricsSystemImpl.prefix_ is only written by the init() method, it is not
touched otherwise, so it can be used to check for the codition when the
MetricsSystemImpl is initialized, but in a stopped state:
https://github.com/apache/hadoop/blob/7bd7725532fd139d2e0e1662df7700f7ab95067a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java#L154
I don't think we have to worry about a race around prefix, AFAICT by the
time the Phoenix classes are loaded, HBase has long initialized DefaultMetrics,
and prefix has been set.
> Phoenix metrics overwriting DefaultMetricsSystem in RegionServers
> -----------------------------------------------------------------
>
> Key: PHOENIX-6699
> URL: https://issues.apache.org/jira/browse/PHOENIX-6699
> Project: Phoenix
> Issue Type: Bug
> Components: core
> Affects Versions: 5.2.0
> Reporter: Istvan Toth
> Assignee: Istvan Toth
> Priority: Major
>
> There is a race condition in the Hbase and Phoenix metrics handling.
> Hbase metrics is supposed to run in the daemon processes, while Phoenix
> metrics is supposed to run on the phoenix client side.
> On an RS both HBase and Phoenix tries to register its metrics implementation
> into DefaultMetricsSystem.
>
> On the happy path, when Phoenix tries to register its metrics on the RS, it
> won't have any effect, because HBase has already registered an active Metrics
> implementation.
> However, JmxCacheBuster stops the Hbase metrics system for ~500ms every five
> seconds, and if Phoenix tries to initialize its metrics in this period, then
> it will succeed in initializing its Metrics object, and will overwrite
> theDefaultMetricsSystem object initialized by HBase.
> This disables the HBase metrics system for the RS process.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
