[
https://issues.apache.org/jira/browse/SOLR-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15960052#comment-15960052
]
Hoss Man commented on SOLR-9959:
--------------------------------
{quote}
On the change to ObjectNames: "instance" is gone in the latest patch. I think
it's more confusing to use reporter's name as JMX domain than to always use
"solr" for mbeans reported by Solr. "Registry name" that I referred to is the
name of a metric registry where metrics are maintained, and each SolrCore (and
each CoreContainer) gets its own unique registry eg.
"solr.core.techproducts.shard1.replica1", and the "solr." prefix is enforced.
This is already a hierarchical name, so it makes sense to use it at the top
level of hierarchy. Then, since you can configure multiple reporters for each
registry, it stands to reason that the next element in the hierarchy should be
the reporter's name, and finally followed by the metric name.
{quote}
I'm sorry -- but that still just doesn't make any sense to me.
For an "inside looking out" perspective, as someone who really understands the
metrics code and the reporting internals, it might make a lot of sense to you
to put the "registry" hierarchy first, and then put the reporter name after
that -- but from "outside looking in" perspective of an end user who is
monitoring solr via a JMX client, I don't see how that helps them at all.
Here's some use cases I'm imagining...
Assume a basic setup, w/o any explicitly configured metrics reporters, and JMX
is enabled at the JVM level so the implicit {{SolrJmxReporter}} starts up: If
I'm using a JMX tool like jconsole, and I'm drilling looking at MBeans and
browsing the info that's availale. As I drill down evetually -- For *every*
MBean I encounter -- there is at some seemingly arbitrary (to me) point in the
hierarchy a {{reporter=default}} aspect of it's name, making the hierarchy one
level deeper ... for no aparent reason.
Now let's asume that there are multiple {{SolrJmxReporter}} configurations
defined in solr.xml (that I as the outside user may not know/care about that)
that are filtering/limiting what metrics are reported to the local
MBeanServer....
* {{name="CACHE_STATS"}} is only exposing (all) the metrics from all the
internal {{CACHE}} s
* {{name="QUERY_TIMES"}} is only exposing the {{requestTime}} metrics from all
{{QUERY}} handlers
...so now ss i'm drilling down into the hierarchy of MBeans, how does it help
me that at some (again seemingly arbitrary) point in the hierarchy, every
object has either {{reporter=CACHE_STATS}} in it's name, or
{{reporter=QUERY_TIMES}} in it's name? Even if there was overlap in filtering,
such that some metrics were being redundently exposed by both reporters, how
does it help me if the disambuiation of those (otherwise identical) MBeans
happens at some point in the middle of the hierarchical ObjectName. If those
portions of the name were at the *begining* then at least I (as an outside
user) would have some visibility into the "sets" of metrics that this Solr
instance was making available, so right off the bat i could choose to drill
down into "QUERY_TIMES" or "CACHE_STATS" depending on what I'm interested in.
That idea -- grouping the metrics together at the top level / begining of the
ObjectName -- is really the only use case I can think of where including the
"reporter" in the ObjectName is particularly beneficial to the end users.
Given that you've added back the {{rootName}} config option, that use case has
another solution, and I'm starting to think that we should just eliminate the
"reporter" part of the name completely. Any {{ObjectName}} collision that
might result from multiple (overlapping) {{SolrJmxReporter}} should still
ultimately be pointing to the same MBean ... so what's the upside of including
the "reporter" portion of the name?
Can you help me understand your position by giving me an example usecase where
it would be helpful to the person monitoring Solr (via JMX) that these reporter
names are in the "middle" of the ObjectName instead of at the begining?
----
FWIW: I actually tried to cobble together that situation I described above,
with two {{SolrJmxReporter}} configurations exposing a diff subset of metrics,
based on what I saw in some test configurations. This is what I had in my
solr.xml...
{code}
<metrics>
<reporter class="org.apache.solr.metrics.reporters.SolrJmxReporter"
name="FOO">
<str name="filter">QUERY\./select.*requests</str>
</reporter>
<reporter class="org.apache.solr.metrics.reporters.SolrJmxReporter"
name="BAR">
<str name="filter">CACHE.*</str>
</reporter>
</metrics>
{code}
...but it failed because {{SolrJmxReporter}} doesn't have a {{setFilter}}
method, and neither does the base class -- it seems like only some special case
reporters support filtering? Shouldn't that be a general purpose feature of
all reporters? As is i'm not sure how to do the type of filtering ab
previously described to limit what can be exposed via JMX?
Which leads me to another concern / problem...
----
{quote}
bq. The config can specify multiple groups or registry names, or registry name
prefixes that it applies to, and for each matching unique registry name a
separate reporter instance will be created.
ok ... seems weird, but i'll take your word for it that it's expected ... my
main concern was that we weren't excessively and unneccessarily
"re-registering" equivilent MBeans with the MBean server every time a
(seemingly duplicate) SolrJmxReporter was created ... if you're saying that's
not happening then i'll take your word for it.
{quote}
Setting aside my above comment about not being clear how someone would
configure the diff prefixes for (multiple) {{SolrJmxReporter}} declarations,
this notion that multiple {{SolrJmxReporter}} instances will be created from a
single _config_ as needed for each "registry" didn't sit will with me yesterday
... and today I finally realized why: {{serviceUrl}}
For the default behavior of connecting to the first existing MBeanServer, or if
an explicit {{agentId}} is configured, then creating multiple
{{SolrJmxReporter}} instances for a single {{<reporter>}} config is fine --
because ultimately the same {{MbeanServer}} is still going to be used by each
of them. But as things stand today you can't use the {{serviceUrl}} option
because then each {{SolrJmxReporter}} _instance_ will (attempt) to create it's
own {{MBeanServer}} instance bound to the same service url -- and only the
first one will win.
On master, if you've got {{rmiregistry 9999}} running in the background, then
you can replace {{<jmx />}} in the techproducts sample {{solrconfig.xml}} and
everything works fine...
{noformat}
<jmx serviceUrl="service:jmx:rmi:///jndi/rmi://localhost:9999/solrjmx" />
{noformat}
...Solr binds to that URL and you can then run {{jconsole
service:jmx:rmi:///jndi/rmi://localhost:9999/solrjmx}} and it will happily
connect to that service URL and you've got stats about the techproduct core.
But if you try the same (effective) config on this branch, then you get all
sorts of errors on startup because multiple {{JMXConnectorServer}} instances
are trying to bind to the same RMI URL...
{code}
<metrics>
<reporter class="org.apache.solr.metrics.reporters.SolrJmxReporter"
name="BAR">
<str
name="serviceUrl">service:jmx:rmi:///jndi/rmi://localhost:9999/solrjmx</str>
</reporter>
</metrics>
{code}
{noformat}
WARN - 2017-04-06 23:57:08.411; [ ]
org.apache.solr.metrics.reporters.SolrJmxReporter;
findMBeanServerForServiceUrl(%s) exception: %s
java.io.IOException: Cannot bind to URL [rmi://localhost:9999/solrjmx]:
javax.naming.NameAlreadyBoundException: solrjmx [Root exception is
java.rmi.AlreadyBoundException: solrjmx]
at
javax.management.remote.rmi.RMIConnectorServer.newIOException(RMIConnectorServer.java:827)
at
javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:432)
at
org.apache.solr.util.JmxUtil.findMBeanServerForServiceUrl(JmxUtil.java:56)
at
org.apache.solr.metrics.reporters.SolrJmxReporter.init(SolrJmxReporter.java:94)
at
org.apache.solr.metrics.SolrMetricManager.loadReporter(SolrMetricManager.java:773)
at
org.apache.solr.metrics.SolrMetricManager.loadReporters(SolrMetricManager.java:741)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:529)
...
WARN - 2017-04-06 23:57:08.424; [ ]
org.apache.solr.metrics.reporters.SolrJmxReporter;
findMBeanServerForServiceUrl(%s) exception: %s
java.io.IOException: Cannot bind to URL [rmi://localhost:9999/solrjmx]:
javax.naming.NameAlreadyBoundException: solrjmx [Root exception is
java.rmi.AlreadyBoundException: solrjmx]
at
javax.management.remote.rmi.RMIConnectorServer.newIOException(RMIConnectorServer.java:827)
at
javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:432)
at
org.apache.solr.util.JmxUtil.findMBeanServerForServiceUrl(JmxUtil.java:56)
at
org.apache.solr.metrics.reporters.SolrJmxReporter.init(SolrJmxReporter.java:94)
at
org.apache.solr.metrics.SolrMetricManager.loadReporter(SolrMetricManager.java:773)
at
org.apache.solr.metrics.SolrMetricManager.loadReporters(SolrMetricManager.java:741)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:530)
...
WARN - 2017-04-06 23:57:08.430; [ ]
org.apache.solr.metrics.reporters.SolrJmxReporter;
findMBeanServerForServiceUrl(%s) exception: %s
java.io.IOException: Cannot bind to URL [rmi://localhost:9999/solrjmx]:
javax.naming.NameAlreadyBoundException: solrjmx [Root exception is
java.rmi.AlreadyBoundException: solrjmx]
at
javax.management.remote.rmi.RMIConnectorServer.newIOException(RMIConnectorServer.java:827)
at
javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:432)
at
org.apache.solr.util.JmxUtil.findMBeanServerForServiceUrl(JmxUtil.java:56)
at
org.apache.solr.metrics.reporters.SolrJmxReporter.init(SolrJmxReporter.java:94)
at
org.apache.solr.metrics.SolrMetricManager.loadReporter(SolrMetricManager.java:773)
at
org.apache.solr.metrics.SolrMetricManager.loadReporters(SolrMetricManager.java:741)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:531)
...
WARN - 2017-04-06 23:57:10.636; [ x:techproducts]
org.apache.solr.metrics.reporters.SolrJmxReporter;
findMBeanServerForServiceUrl(%s) exception: %s
java.io.IOException: Cannot bind to URL [rmi://localhost:9999/solrjmx]:
javax.naming.NameAlreadyBoundException: solrjmx [Root exception is
java.rmi.AlreadyBoundException: solrjmx]
at
javax.management.remote.rmi.RMIConnectorServer.newIOException(RMIConnectorServer.java:827)
at
javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:432)
at
org.apache.solr.util.JmxUtil.findMBeanServerForServiceUrl(JmxUtil.java:56)
at
org.apache.solr.metrics.reporters.SolrJmxReporter.init(SolrJmxReporter.java:94)
at
org.apache.solr.metrics.SolrMetricManager.loadReporter(SolrMetricManager.java:773)
at
org.apache.solr.metrics.SolrMetricManager.loadReporters(SolrMetricManager.java:741)
at
org.apache.solr.metrics.SolrCoreMetricManager.loadReporters(SolrCoreMetricManager.java:86)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:889)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:855)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:975)
...
{noformat}
So either the {{MBeanServer}} "creation" needs to be refactored to ensure it
only happens once for a given {{<reporter>}} configuration, or we need to
rethink the question of if/how we're going to support {{serviceUrl}} at all?
(Off the top of my head: maybe if you want to specify a {{serviceUrl}}, you
must also conifgure an {{agentId}} and JmxUtil can first search for and
existing {{MBeanServer}} with that {{agentId}}, if one exists it will use it,
if not then it will create a new {{MBeanServer}} it binds to that
{{serviceUrl}} and sets the {{agentId}} on so future "instances" of the same
{{SolrJmxReporter}} config will find it. ... ????)
----
bq. Regarding the disabling of JMX reporting from Solr but not from JVM - I
think a more elegant approach is a boolean flag "enabled" in a SolrJmxReporter
config - if the config is present in solr.xml but the flag is false then this
turns off the reporter, no need for yet another class.
+1
We should be able to have a test that's working, right? Something like...
{code}
// force at least one local MBeanServer,
// this would normally cause implicit SolrJmxReporter
ManagementFactory.getPlatformMBeanServer();
initSolrWithSolrXmlThatHasSolrJmxReporterUsingEnableEqualsFalse();
for(MBeanServer server : MBeanServerFactory.findMBeanServer(null)) {
assertNoMBeansWithPrefix("solr", server);
}
{code}
> SolrInfoMBean-s category and hierarchy cleanup
> ----------------------------------------------
>
> Key: SOLR-9959
> URL: https://issues.apache.org/jira/browse/SOLR-9959
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: metrics
> Affects Versions: master (7.0)
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Priority: Blocker
> Fix For: master (7.0)
>
> Attachments: SOLR-9959.patch, SOLR-9959.patch
>
>
> SOLR-9947 changed categories of some of {{SolrInfoMBean-s}}, and it also
> added an alternative view in JMX, similar to the one produced by
> {{SolrJmxReporter}}.
> Some changes were left out from that issue because they would break the
> back-compatibility in 6.x, but they should be done before 7.0:
> * remove the old JMX view of {{SolrInfoMBean}}-s and improve the new one so
> that it's more readable and useful.
> * in many cases {{SolrInfoMBean.getName()}} just returns a FQCN, but it could
> be more informative - eg. for highlighter or query plugins this could be the
> symbolic name of a plugin that users know and use in configuration.
> * top-level categories need more thought. On one hand it's best to minimize
> their number, on the other hand they need to meaningfully represent the
> functionality of components that use them. SOLR-9947 made some cosmetic
> changes, but more discussion is necessary (eg. QUERY vs. SEARCHHANDLER)
> * we should consider removing some of the methods in {{SolrInfoMBean}} that
> usually don't return any useful information, eg. {{getDocs}}, {{getSource()}}
> and {{getVersion()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
