[
https://issues.apache.org/jira/browse/PHOENIX-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17529205#comment-17529205
]
Istvan Toth commented on PHOENIX-6699:
--------------------------------------
To elaborate:
We saw this (filtered) log on a Region server:
{noformat}
2022-04-26 15:09:18,233 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
HBase metrics system started
2022-04-26 15:09:22,619 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Stopping HBase metrics system...
2022-04-26 15:09:22,624 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
HBase metrics system stopped.
2022-04-26 15:09:23,136 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
HBase metrics system started
2022-04-26 15:09:27,863 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Stopping HBase metrics system...
2022-04-26 15:09:27,880 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
HBase metrics system stopped.
2022-04-26 15:09:27,891 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Phoenix metrics system started
2022-04-26 15:09:28,380 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Phoenix metrics system already started!
2022-04-26 15:09:32,786 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Stopping Phoenix metrics system...
2022-04-26 15:09:32,840 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Phoenix metrics system stopped.
2022-04-26 15:09:33,343 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Phoenix metrics system started
2022-04-26 15:09:37,904 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Stopping Phoenix metrics system...
2022-04-26 15:09:37,909 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Phoenix metrics system stopped.
2022-04-26 15:09:38,411 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Phoenix metrics system started{noformat}
The HBase metrics messages are caused by JmxCacheBuster.
then we see that 10 ms after JMXCacheBuster has stopped the
DefaultMetricsSystem, the "Phoenix" metrics system gets registered.
This happens, because while DefaultMetricsSystem.init() will not overwrite the
instance while it's running, init() WILL overwrite the instance when it's
stopped (by JMXCachecBuster).
> Phoenix metrics overwriting DefaultMetricsSystem in RegionServers
> -----------------------------------------------------------------
>
> Key: PHOENIX-6699
> URL: https://issues.apache.org/jira/browse/PHOENIX-6699
> Project: Phoenix
> Issue Type: Bug
> Components: core
> Affects Versions: 5.2.0
> Reporter: Istvan Toth
> Assignee: Istvan Toth
> Priority: Major
>
> There is a race condition in the Hbase and Phoenix metrics handling.
> Hbase metrics is supposed to run in the daemon processes, while Phoenix
> metrics is supposed to run on the phoenix client side.
> On an RS both HBase and Phoenix tries to register its metrics implementation
> into DefaultMetricsSystem.
>
> On the happy path, when Phoenix tries to register its metrics on the RS, it
> won't have any effect, because HBase has already registered an active Metrics
> implementation.
> However, JmxCacheBuster stops the Hbase metrics system for ~500ms every five
> seconds, and if Phoenix tries to initialize its metrics in this period, then
> it will succeed in initializing its Metrics object, and will overwrite
> theDefaultMetricsSystem object initialized by HBase.
> This disables the HBase metrics system for the RS process.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
