[
https://issues.apache.org/jira/browse/PHOENIX-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531060#comment-17531060
]
ASF GitHub Bot commented on PHOENIX-6699:
-----------------------------------------
stoty commented on code in PR #1431:
URL: https://github.com/apache/phoenix/pull/1431#discussion_r863505525
##########
phoenix-core/src/main/java/org/apache/phoenix/monitoring/MetricUtil.java:
##########
@@ -38,4 +46,22 @@ public static MetricsStopWatch getMetricsStopWatch(boolean
isRequestMetricsEnabl
return new MetricsStopWatch(true);
}
+ // We need to cover the case when JmxCacheBuster has just stopped the
HBase metrics
+ // system, and not accidentally overwrite the DefaultMetricsSystem
singleton.
+ // See PHOENIX-6699
+ public static boolean isDefaultMetricsInitialized() {
+ try {
+ MetricsSystemImpl metrics = (MetricsSystemImpl)
DefaultMetricsSystem.instance();
+ Field prefixField =
MetricsSystemImpl.class.getDeclaredField("prefix");
Review Comment:
> Could we somehow separate the Phoenix client-side-only metrics from the
execution pathway of the Phoenix server-side metrics? I imagine this is no
easier to do.
We're veering into territories that are beyond my knowledge of the metrics
system.
AFAICT the Hadoop metrics system always uses the DefaultMetrics object, I
haven't seen anything that points to using multiple MetricsSystem objects at
the same time.
This is mostly fine, as we can add/remove metrics from this default
MetricsSystem objects, and this doesn't cause any problems, and this means that
we add the additional Phoenix metrics in the RS dynamically, and they can be
accessed in a unified way.
Pre HBase 1.4, Phoenix didn't use the metrics2 framework for its client
metrics, and it had to be accessed programmatically (no JMX). I don't think we
want that.
The only problem comes from the fact that SOMETHING has to initialize a
DefaultMetrics object on the Phoenix client side once before the Metrics System
can be used, and
1. HBase (server side) and Phoenix (anywhere) attempt this
2. While DefaultMetrics is protected from re-initialization, the stop/start
sequence preformed by JmxCacheBuster defeats this, hence this bug.
I have also tried to simply disable the DefaultMetrics initialization from
Phoenix, but in that case the Metrics System wasn't initialized at all on the
client side.
All in all, I think that my current solution is good enough, and I'm not
convinced that expliring alternatives is worth the time (even though I asked
for them from you)
> Phoenix metrics overwriting DefaultMetricsSystem in RegionServers
> -----------------------------------------------------------------
>
> Key: PHOENIX-6699
> URL: https://issues.apache.org/jira/browse/PHOENIX-6699
> Project: Phoenix
> Issue Type: Bug
> Components: core
> Affects Versions: 5.2.0
> Reporter: Istvan Toth
> Assignee: Istvan Toth
> Priority: Major
>
> There is a race condition in the Hbase and Phoenix metrics handling.
> Hbase metrics is supposed to run in the daemon processes, while Phoenix
> metrics is supposed to -run on the phoenix client side.- create its own
> DefaultMetrics object on the client side, and add its metrics to the HBase
> DefaultMetrics object on the RS side.
> On an RS both HBase and Phoenix tries to register its metrics implementation
> into DefaultMetricsSystem.
>
> On the happy path, when Phoenix tries to register its metrics on the RS, it
> won't have any effect, because HBase has already registered an active Metrics
> implementation.
> However, JmxCacheBuster stops the Hbase metrics system for ~500ms every five
> seconds, and if Phoenix tries to initialize its metrics in this period, then
> it will succeed in initializing its Metrics object, and will overwrite
> theDefaultMetricsSystem object initialized by HBase.
> This disables the HBase metrics system for the RS process.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
