Andrzej Bialecki  commented on SOLR-11221:

Hmm, yet another more serious bug is present: 
{{SolrJmxReporterTest.testReloadCore}} doesn't really test the exact sequence 
of reporters starting and closing, so it doesn't catch the following problem.

On reload the old core is closed after the new core has been created - and 
consequently the old instance of {{SolrJmxReporter}} removes the mbeans right 
after they were registered by a new reporter started by the new core. End 
result is that only a handful of mbeans related to Searcher are registered, 
other mbeans are gone.

In the past {{JmxMonitoredMap}} used to track what mbean belongs to what core 
by adding a "coreHashCode" attribute to all DynamicMBean-s, and removing only 
those with matching coreHashCode attribute. Unfortunately, it's not possible to 
add additional attributes to MBeans created by Dropwizard's {{JmxReporter}}, 
and on {{close()}} it indiscriminately closes all MBeans with known names.

One possible solution would be to insert reporter's hashCode into the 
ObjectName hierarchy, but this would make the MBean names variable and 
unpredictable. Another (more complicated but more elegant) option is to 
reimplement {{JmxReporter}} to re-introduce "coreHashCore" attribute.

> SolrJmxReporter may not report all metrics
> ------------------------------------------
>                 Key: SOLR-11221
>                 URL: https://issues.apache.org/jira/browse/SOLR-11221
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: metrics
>    Affects Versions: 6.6, 7.0, 6.7, master (8.0)
>            Reporter: Andrzej Bialecki 
>            Assignee: Andrzej Bialecki 
>            Priority: Critical
> {{SolrJmxReporter}} uses Dropwizard's {{JmxReporter}}, which in turn uses a 
> {{MetricRegistry}} listener to report newly added metrics as MBeans, and to 
> unregister them from MBeanServer when the reporter is closed, which happens 
> when core is closed.
> The metrics API keeps around existing metrics in {{solr.core.*}} registries 
> to help maintain continuous metrics in presence of core reloads. However, 
> this means that some of these metric instances are not registered anew when a 
> core is reloaded - and for these metrics the listener won't fire, so the 
> MBeans won't be registered.
> This limitation is a result of the use of {{MetricRegistryListener}} in 
> {{JmxReporter}} and can't be fixed without reimplementing this class. Another 
> possible approach would be to configure the {{JmxReporter}} to use a 
> "mirroring" registry instead, which will be populated with existing metrics 
> from the original registry (thus generating "metric added" events) and then 
> kept in sync with the main registry via a listener.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to