[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15780615#comment-15780615 ] ASF subversion and git services commented on SOLR-4735: --- Commit 9dde8a30303de4bce5da45189219dd6150252b29 in lucene-solr's branch refs/heads/branch_6x from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9dde8a3 ] Cumulative patch from master, originally developed on branch 'feature/metrics', which brings the following issues: * SOLR-4735: Improve Solr metrics reporting * SOLR-9812: Implement /admin/metrics API * SOLR-9805: Use metrics-jvm library to instrument jvm internals * SOLR-9788: Use instrumented jetty classes > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Fix For: master (7.0), 6.4 > > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, screenshot-2.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15764246#comment-15764246 ] ASF subversion and git services commented on SOLR-4735: --- Commit b37a72d941dba705b43c8868584bd9df775a1537 in lucene-solr's branch refs/heads/master from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b37a72d ] SOLR-4735 Use overridableRegistryName also for predefined shared registries. Cleanup + javadocs. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Fix For: master (7.0) > > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, screenshot-2.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763710#comment-15763710 ] ASF subversion and git services commented on SOLR-4735: --- Commit 8bbdb6248c5de3f3bd61501ba42a50aeec29c78b in lucene-solr's branch refs/heads/master from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8bbdb62 ] Squashed commit of branch 'feature/metrics', containing: SOLR-4735: Improve Solr metrics reporting SOLR-9812: Implement /admin/metrics API SOLR-9805: Use metrics-jvm library to instrument jvm internals SOLR-9788: Use instrumented jetty classes > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, SOLR-4735.patch, screenshot-2.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761261#comment-15761261 ] Shalin Shekhar Mangar commented on SOLR-4735: - [~dsmiley] - I didn't know that commits to "jira/*" branches are skipped by the commit bot before I created that branch. I figured that since multiple jira issues will be committed to this branch before we merge it to master, I should use a more descriptive name such as "feature/metrics". I have opened a ticket with ASF INFRA to skip notifications for commits on all branches beginning with "feature/*" for Lucene/Solr project. See https://issues.apache.org/jira/servicedesk/customer/portal/1/INFRA-13133 > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, SOLR-4735.patch, screenshot-2.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761133#comment-15761133 ] Andrzej Bialecki commented on SOLR-4735: - Wilco. Sorry about that, I was wondering why this is so chatty... > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, SOLR-4735.patch, screenshot-2.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761126#comment-15761126 ] David Smiley commented on SOLR-4735: In the future, please create branches named SOLR- (or LUCENE-) which is the pattern expected by ASF's git JIRA notifier. Since this "metrics" feature-branch does not follow this convention, merge commits here trigger a bunch of notifications to other issues that were merged in. Alternatively if this is going to keep occurring... then maybe we can ask ASF Infra to modify the regexp that has this rule. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, SOLR-4735.patch, screenshot-2.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761083#comment-15761083 ] ASF subversion and git services commented on SOLR-4735: --- Commit f56da1df6e92da5f1ab524caf62d30ea3a3ceede in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f56da1d ] SOLR-4735 Update CHANGES.txt > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, SOLR-4735.patch, screenshot-2.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761049#comment-15761049 ] ASF subversion and git services commented on SOLR-4735: --- Commit 92ef10dbdea6d8764d2a1ecba3d53abee542542d in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=92ef10d ] SOLR-4735 Add gauges that report the current number of cores. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, SOLR-4735.patch, screenshot-2.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754834#comment-15754834 ] ASF subversion and git services commented on SOLR-4735: --- Commit 5f0637cc8569768ac9ce2a38cef5405163a974c0 in lucene-solr's branch refs/heads/feature/metrics from [~shalinmangar] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5f0637c ] SOLR-4735: Use method in java.util.Objects instead of the forbidden methods in Guava's Preconditions class > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, SOLR-4735.patch, screenshot-2.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745982#comment-15745982 ] ASF subversion and git services commented on SOLR-4735: --- Commit 3ba5d7db0971eb638a5e1dae17fb3c9ea36440d4 in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3ba5d7d ] SOLR-4735 Fix javadoc errors. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, screenshot-2.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745981#comment-15745981 ] ASF subversion and git services commented on SOLR-4735: --- Commit 98a04b6cca32d7b2a3da8b30291e1df1edd885cd in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=98a04b6 ] SOLR-4735 Add unit test for SolrMetricManager. Other changes: * add metrics to SolrCore related to newSearcher events. * build a dot-separated hierarchical registry name for cores that are a part of SolrCloud collection. * don't split metrics by class in SolrJmxReporter - this only obscured the hierarchy. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, screenshot-2.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743325#comment-15743325 ] Jeff Wartes commented on SOLR-4735: --- Understood, and not all cores are part of a collection. But if it matches the solrcloud convention, it would be pretty nice to use it. (and the node name if it doesn't) I could've sworn I saw an existing function for picking a node name apart somewhere, but I can't seem to find it now - maybe it was in a patch I read or something. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, screenshot-1.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743223#comment-15743223
]
Andrzej Bialecki commented on SOLR-4735:
-
Hmm, not blindly, you could have a collection named {{my_collection}} - but we
could make check the collection name and split replica names into proper
hierarchy, eg. {{my_collection_shard1_replica1 ->
my_collection.shard1.replica1}}
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
> Components: metrics
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch, screenshot-1.png
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743207#comment-15743207 ] Jeff Wartes commented on SOLR-4735: --- That's almost perfect. Can we replace those underscores with dots? That would mean the dashboard doesn't need to regex the "name" in order to group similar metrics. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, screenshot-1.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743201#comment-15743201 ] Jeff Wartes commented on SOLR-4735: --- Oh, one thing just occurred to me though. There are essentially two classes of request to a collection - the top-level request, and the per-shard fan-out requests. I guess you can sort of derive the metrics of the top-level request from the per-core metrics, but it requires you know the number of shards, and still only works if the two classes of request are not mixed together. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, screenshot-1.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743199#comment-15743199
]
Andrzej Bialecki commented on SOLR-4735:
-
Metric names already contain collection name in their namespace - eg. in
SolrCloud a core name would be {{collection1_shard1_replica1}}. The full
metrics name would be then something like
{{solr.core.collection1_shard1_replica1.myReporter.QUERYHANDLER./admin/ping.counters.requests.Count}},
which seems awfully long to a human but is well-structured and easy to process
in code.
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
> Components: metrics
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch, screenshot-1.png
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743174#comment-15743174 ] ASF subversion and git services commented on SOLR-4735: --- Commit d1227af722ef95494efc6c977a1c29f94108b4bf in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d1227af ] SOLR-4735 Javadoc plus test improvements. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, screenshot-1.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742740#comment-15742740
]
Jeff Wartes commented on SOLR-4735:
---
I've fallen behind keeping up with your changes, but for what it's worth, I
agree with this. Collection-level metrics are at the cluster level, in
aggregate. It's up to the thing you're reporting the metrics into to do the
aggregation. For example, what I really want on my dashboard in grafana is a
line, something like:
AVG(solr.{all nodes}.{all cores belonging to a particular
collection}.latency.p95)
Then I can drill into a particular node, or core, in my reporting tool if I
want. There's a requirement that the metrics namespaces being reported allows
for aggregation like this, which might mean a core needs to know the collection
to which it belongs, but I don't think the node itself should needs to report
collection metrics.
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
> Components: metrics
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch, screenshot-1.png
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742676#comment-15742676 ] ASF subversion and git services commented on SOLR-4735: --- Commit 99799cafe57cc32fa2a6bf6c25de3acd18cea248 in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=99799ca ] Revert "SOLR-4735 WIP. Initial design for propagating updates to linked registries." This reverts commit c2291001356e6fa1bb72e0543c3e035b5f320862. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, screenshot-1.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742673#comment-15742673 ] Andrzej Bialecki commented on SOLR-4735: - After discussing this with Shalin I came to conclusion that core-level metrics should be sufficient, and the cost of this added complexity is too high, so I'm going to revert that last commit. Collection-level metrics wouldn't be complete anyway in general case - shards and their replicas would live on different nodes anyway, so the collection-level metrics from a single node would never represent the whole picture for a collection, just a partial one for the parts of a collection that live on a single node. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, screenshot-1.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741835#comment-15741835 ] ASF subversion and git services commented on SOLR-4735: --- Commit c2291001356e6fa1bb72e0543c3e035b5f320862 in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c229100 ] SOLR-4735 WIP. Initial design for propagating updates to linked registries. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, screenshot-1.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15737567#comment-15737567
]
Andrzej Bialecki commented on SOLR-4735:
-
There is one important issue that is still unsolved in the current patch -
collecting per-collection metrics at the same time as we collect per-core
metrics. A single instance of Solr can hold multiple shards and / or replicas
that belong to the same logical collection and it would be useful to get
combined metrics at a collection level.
If they weren't needed at the same time, we could probably set up per-core
aliases (using {{overridableRegistryName}} or a similar mechanism) so that a
single per-collection registry would be used for all shards/replicas. However,
since all shards would then modify the same underlying {{Metric}} objects it
would not be possible anymore to separate per-shard metrics from these
aggregates. This is however the simple fallback solution of the problem - allow
either per-core or per-collection metrics and never both.
Codahale API doesn't support aggregation of child metrics - if it were possible
then we could fake the aggregated metrics on the fly when they are needed.
So far the only solution that comes to my mind that allows us to keep both
levels of reporting is to extend {{Metric}} subclasses so that they delegate
their updates to the parent instance(s), something like the following:
{code}
public class ChildCounter extends Counter {
public ChildCounter(Counter... parents) { ... }
public void inc(long n) {
super.inc(n);
for (Counter c : parents) {
c.inc(n);
}
}
}
{code}
I.e. all updates to the child instances would be applied at (nearly) the same
time to parent instances - and parent instances will be referenced by several
child instances from different shards. For example, the {{ChildCounter}}
instance would be registered in "solr.core.collection1_shard1" registry, and
the aggregate counter would be registered in "solr.core.collection1" registry,
and the same aggregate counter would be used by {{ChildCounter}} from
"solr.core.collection1_shard2".
In order to maintain this delegation Solr components would have to always use
{{SolrMetricRegsitry.counter(...), .timer(...), .meter(...), .histogram(...)}}
methods that would set up this delegation, by obtaining metric instances from
the parent registries and returning eg. {{ChildCounter}} instead of a regular
{{Counter}}, {{ChildTimer}} instead of regular {{Timer}} etc.
This should work and it meets the criteria, but it feels clunky and
complicated. Any other suggestions?
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch, screenshot-1.png
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733233#comment-15733233 ] ASF subversion and git services commented on SOLR-4735: --- Commit ab52041c9bfea8285446b79f39ddfbf02eebc845 in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ab52041 ] SOLR-4735 Test also core rename. Fix expectations when registries are reused across tests in the same JVM. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, screenshot-1.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731828#comment-15731828 ] Shalin Shekhar Mangar commented on SOLR-4735: - bq. Didn't Shalin already add a /admin/metrics endpoint for reporting metrics? In SOLR-9812, I've added the metrics servlet supplied by the library but it is not longer useful for the flexible registry scheme implemented here. I'm writing a custom handler for it now. I will upload a patch soon. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, screenshot-1.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731809#comment-15731809 ] Ramkumar Aiyengar commented on SOLR-4735: - Didn't Shalin already add a /admin/metrics endpoint for reporting metrics? > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, screenshot-1.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729770#comment-15729770
]
Andrzej Bialecki commented on SOLR-4735:
-
Brief summary of the current state on the branch:
* {{SolrMetricManager}} is now the single central location responsible for
managing instances of {{MetricRegistry}} and {{SolrMetricReporter}}-s.
* plugin configurations are declared in {{solr.xml}}:
{code}
...
...
{code}
* each reporter plugin can be configured so that it's automatically
instantiated:
** for a specific "group" or "registry". These attributes may contain multiple
comma- or space-separated values, in which case the reporter will be
instantiated for each matching group / registry.
** for every registry, when neither "group" or "registry" attribute is
specified.
* reporters are instantiated on component initialization and closed when a
component is closed / shut down. Currently this happens in: {{CoreContainer}}
for {{node}} group, {{SolrCore}} for {{core}} group and in
{{SolrDispatchFilter}} for {{jvm}} group.
* instances of metrics are maintained across core reloads and renames (or
swaps). They are only permanently deleted on core delete.
* {{SolrJmxReporter}} now maintains hierarchy of {{ObjectName}}-s that consists
of the following:
** registry name (split on dots into sub-paths)
** reporter name
** category
** scope
** metric type
** metric name
See the attached JConsole screenshot.
To do, possibly in another issues:
* add metrics for "jetty" and "http" groups.
* add a handler to report all this wealth of information :)
Comments, suggestions, review and patches are welcome!
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729588#comment-15729588 ] ASF subversion and git services commented on SOLR-4735: --- Commit cff7097f40488bb97fb8c38a76ad28d2e1ae50d2 in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=cff7097 ] SOLR-4735 Fix SolrMetricsIntegrationTest. Fix registration of "solr.jvm" metrics. Add a few helper methods to SolrMetricManager. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728672#comment-15728672 ] ASF subversion and git services commented on SOLR-4735: --- Commit 6f14f41044f6ec5b58c3328fd7186e3f8e1a9d33 in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6f14f41 ] SOLR-4735 Move metric reporter configuration to solr.xml: * add metric reporting at "node" level, * introduce reporters applicable to multiple groups, multiple registries or all groups, based on the value of "group" and "registry" attributes, * handle core rename and swap operations so that metrics are preserved but renamed. * modify SolrJmxReporter to split the hierarchy based also on the reporter's name. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722085#comment-15722085 ] ASF subversion and git services commented on SOLR-4735: --- Commit 46c662fcab56906f3fa6fde09d3789d1d2fc6aed in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=46c662f ] SOLR-4735 WIP: move metric reporter config to CoreContainer level. Manage reporters in SolrMetricManager. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15716772#comment-15716772 ] Jeff Wartes commented on SOLR-4735: --- That seems pretty viable too. As I mentioned, the memory overhead of a registry is pretty low, just a concurrent map and a list. Plus, the actual metric objects in the map would be shared by both registries, so I'd be more concerned about the work involved keeping them synchronized then with just having multiple registries. I confess though, I don't have a clear idea whether that's more or less overhead than multiple identically-configured reporters. It feels like most of the possible performance issues here are linear, so it may not matter. Two reporters iterating through 10 metrics each sounds pretty much the same as one reporter iterating over 20 to me, all else being equal. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15716698#comment-15716698 ] Kelvin Wong commented on SOLR-4735: --- Hmm wouldn't these aggregate registries defeat the point of keeping them separate in the first place (from a performance perspective)? For example, if a user configures a JMXReporter and a GraphiteReporter on all registries, Solr would have to basically make two copies of all of its registries. Perhaps we can just "fake" an aggregate reporter? There can be configuration logic so that one reporter is instantiated for each registry that the user configured. This might be a bit wasteful but we won't have to deal with maintaining an aggregate registry or writing reporters that do the aggregation. And to the user, it seems as though they only needed to configure one reporter. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15716247#comment-15716247 ] Jeff Wartes commented on SOLR-4735: --- Yeah, I get that. I like this line of thought because it means we can create as many registries as make sense, (cores, collections, logical code sections, etc) without worrying about how to get everything reported. We only have to pick some names. What about a class that extends MetricRegistry and also implements MetricRegistryListener? Call that a ListeningMetricRegistry or something. When the configuration asks for a reporter on some set of (registry) names, we create a new, perhaps non-shared ListeningMetricRegistry, use registerAll to scoop the metrics in the desired registries into it, and then call addListener on all the desired registries with the ListeningMetricRegistry so everything stays in sync? So that could still mean a single registry with a ton of metrics, but only in cases where there's been an explicit request for a reporter on a ton of metrics. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15715972#comment-15715972
]
Andrzej Bialecki commented on SOLR-4735:
-
bq. `MetricRegistry` is really just a bunch of convenience methods and
thread-safety around a `MetricSet`
Well, my comment referred to the fact that {{MetricRegistry}} actually uses
{{ConcurrentHashMap}} for keeping metrics, and having a map with 100k+ keys is
never good. But I agree the API could have been more flexible - if reporters
were taking {{MetricSet}} we could fake one either from multiple registries or
from a subset of metrics from one registry, or a combination thereof.
We can implement an aggregating franken-registry by overriding all methods in
{{MetricRegistry}} to always delegate operations to sub-registries. It's a
little bit hackish but doable. We could create these as non-shared registries
only for the purpose of reporting.
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15715931#comment-15715931 ] Andrzej Bialecki commented on SOLR-4735: - bq. Will we be instantiating separate reporters for each `Group` then? Part of the refactoring I'm working on is moving reporter configs to `solr.xml` under ` ...` . Then appropriate reporters would be created for each group at the time when the component that manages this group of metrics is initialized (eg. "core" on SolrCore creation, "node" when `CoreContainer` is loaded etc). Regarding reporters that could take multiple registries ... yeah, it seems a waste to create separate reporters for each core if they have identical configs. I'm not sure yet how to solve this - eg. for JMX reporting any sort of aggregate reporter would have to create multiple `JMXReporter`s anyway, one per registry, because that's how the API is implemented. bq. it would be nice if we could just specify which registry we'd like a reporter to attach to Hmm, we could perhaps use either `group` or `registry` attribute in the reporter config. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15715880#comment-15715880 ] Jeff Wartes commented on SOLR-4735: --- `MetricRegistry` is really just a bunch of convenience methods and thread-safety around a `MetricSet`. There isn't much overhead difference between the two. But really, when I think of a MetricRegistry, I think of it as "a set of metrics I want to attach a reporter to", nothing more. It's a bit disappointing that reporters take a Registry instead of a MetricSet, since a Registry isa MetricSet. With that in mind, one strategy would be have every logical grouping of metrics use its own dedicated (probably shared) registry, and then bind the reporter-registry concept together at reporter definition time. That is, create a non-shared registry explicitly for the purpose of attaching a reporter to it, and only when asked to define a reporter. The reporter definition would then include the names of the registries to be reported. Under the hood, a new registry would be created as the union of the requested registries, and the reporter instantiated and attached to that. We'd have to make sure the namespace of all the metrics in the metric groups is unique, so that arbitrary groups can be combined without conflict, but that sounds desirable regardless. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15715107#comment-15715107
]
Kelvin Wong commented on SOLR-4735:
---
Hi Andrzej,
{quote}
I added a notion of "group" of metrics, which corresponds to a top-level
subsystem in a Solr node
{quote}
* Nice! I really like this concept. Will we be instantiating separate reporters
for each `Group` then? That way, reporting can be more flexibly configured.
(ex. Jetty goes to JMX and Graphite, JVM goes to only JMX, etc...)
{quote}
I'll look into reusing single global-level reporters when possible, and
creating new instances only if there are per-collection overrides.
{quote}
* It looks like
[JmxReporter|https://github.com/dropwizard/metrics/blob/3.2-development/metrics-core/src/main/java/com/codahale/metrics/JmxReporter.java#L701]
takes only one `MetricRegistry` at a time (and
[GraphiteReporter|https://github.com/dropwizard/metrics/blob/3.2-development/metrics-graphite/src/main/java/com/codahale/metrics/graphite/GraphiteReporter.java#L145],
etc. for that matter). Will we need to build some sort of
`AggregateMetricRegistry` to join each core's registries? Or do you have
something else in mind?
* On a separate note, it would be nice if we could just specify which registry
we'd like a reporter to attach to. So for example, we can attach one reporter
to `collection1`, another to `zookeeper`, and one more to `jvm`. These are at
different levels in the metrics hierarchy but perhaps we can just pass in the
registry's name as part of the config for a reporter?
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15712011#comment-15712011
]
Andrzej Bialecki commented on SOLR-4735:
-
Thanks Kelvin for creating the new tests! You beat me to it... :)
After some discussion with Shalin it looks like we need to be able to manage
{{SolrMetricReporter}}-s outside the {{SolrCore}} context - after all they need
to also report metrics that are not related to {{SolrCore}}:
* I added a notion of "group" of metrics, which corresponds to a top-level
subsystem in a Solr node (I'm not attached to the name, naming is hard, so if
you have a better suggestion please speak up):
{code}
enum Group { jvm, jetty, http, node, core }
{code}
These correspond to JVM, Jetty, HttpClient, SolrCore, and node-level metrics
such as Zookeeper stats, Overseer queues etc. The idea is that if you fire
JConsole you should see something like this:
{code}
* solr
- jvm
- jetty
- http
- node
- zookeeper
- overseer
...
- core
- collection1
- CORE
- QUERYHANDLER
…
{code}
* Since we need reporters for these top-level groups we need to be able to
configure them outside {{SolrCore}} config. On the other hand we probably don't
want to always specify the same reporters in each collection config, too. I'll
try to refactor the code so that we can specify the base reporter configs in
{{CoreContainer}} configuration, and then override them per collection if
needed.
* Current implementation instantiates separate reporters for each collection.
I'll look into reusing single global-level reporters when possible, and
creating new instances only if there are per-collection overrides.
* Currently we use separate instances of {{MetricRegistry}} per core. This may
seem wasteful but if you consider deployments with thousands of cores, and
multiply this by hundreds of metrics per core then any lookups in a HashMap
with 100k keys, or filtering by prefix, will cause performance issues. So I
think the current design is more scalable than using a single registry for all
cores.
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711969#comment-15711969 ] ASF subversion and git services commented on SOLR-4735: --- Commit 1bc01d25ef2fa8fa015ba2fdb5b2ffc8ae3cec0a in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1bc01d2 ] SOLR-4735 Introduce a concept of top-level group of metrics for things that happen outside SolrCore. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709559#comment-15709559 ] ASF subversion and git services commented on SOLR-4735: --- Commit f489bb8566985174111d4e91df2d6ec03ffcb01e in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f489bb8 ] SOLR-4735 This method may actually remove several metrics. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709129#comment-15709129 ] ASF subversion and git services commented on SOLR-4735: --- Commit fea0e200a8083ebd86d8e522939e4977d072bbe7 in lucene-solr's branch refs/heads/feature/metrics from [~kwong494] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fea0e20 ] SOLR-4735: SolrMetricsIntegrationTest (Kelvin Wong via Christine Poerschke) Adds SolrMetricsIntegrationTest which uses solrconfig-metricreporter.xml which configures MockMetricReporter instances. also: * JmxUtil and SolrJmxReporter tweaks * SolrMetricReporterTest.MockReporter turned into MockMetricReporter * changes in SolrCoreMetricManagerTest and SolrJmxReporterTest: ** moved initCore from BeforeClass to Before(Test) so that After(Test) can do deleteCore ** TODO: verify interaction between tests (SolrCoreMetricManagerTest and SolrMetricsIntegrationTest and SolrJmxReporterTest) * SolrCoreMetricManagerTest instead of SolrJmxReporter use MockMetricReporter > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709123#comment-15709123
]
Andrzej Bialecki commented on SOLR-4735:
-
Ad 1. Indeed, there is some inconsistency here. I think we should deprecate the
{{}}, and turn on SolrJmxReporter by default in cloud mode (in the example
non-cloud config {{}} is turned off).
Ad 2. currently it's independent. Yes, I think we should eventually remove the
{{}} section.
Ad 3. Good point, I'll create one.
Ad 4. Right, I'll add this.
Ad 5. Not sure how to do that, let's discuss this offline.
Ad 6. This would be easy to add but it brings several additional dependencies
from {{metrics-graphite}} and {{metrics-ganglia}} artifacts. Are we ok with
that?
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708742#comment-15708742
]
Shalin Shekhar Mangar commented on SOLR-4735:
-
I built Solr from the feature/metric branch and tried it out. I have a few
questions/comments:
# I expected that the SolrJmxReporter would be enabled by default but it is
not. Should it be? Eventually we should get rid of our current JMX integration
(maybe in 7.0?) so it makes sense to have the alternative enabled by default.
# If or how does the SolrJmxReporter work with the {{}} tag in
solrconfig.xml? Does that get deprecated eventually?
# There is no test solrconfig.xml which has a reporter section in it. There
should be at least one with the jmx reporter configured that we test instead of
just relying on code to create a new metric manager and add a reporter to it.
# The example solrconfig.xml should have a sample reporter section even if it
is commented out if the default jmx reporter is not enabled by default.
# The metric reporter should be configurable via the Config API
# Do we want to support Graphite or Ganglia reporters as well?
The last two can be worked upon in separate issues.
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708517#comment-15708517 ] ASF subversion and git services commented on SOLR-4735: --- Commit 38887c4a446990b8c37195c08d9bb63108dad31e in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=38887c4 ] SOLR-4735 Clean up the API. Remove metrics on core unload. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708347#comment-15708347 ] ASF subversion and git services commented on SOLR-4735: --- Commit 14ff79175f524b38ec85f0542753e28dc8b1b2c6 in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=14ff791 ] SOLR-4735 Simplify method names. Allow removing multiple metrics with a prefix. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708101#comment-15708101 ] ASF subversion and git services commented on SOLR-4735: --- Commit 7cedc4acb26c23d11218037bfcad0737c3c566b6 in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7cedc4a ] SOLR-4735 Simplify the code by using registryFor(String). > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707955#comment-15707955
]
Andrzej Bialecki commented on SOLR-4735:
-
Ouch! Yeah, let's keep the discussion here. Unfortunately, the way Github
integration is set up now it makes pull requests painful to work with.
So... originally I wasn't sure how often that method would be called and I
wanted to save the cost of an additional method call - but it's called so
infrequently that elegance indeed seems more important.
Also, I merged the latest code from the PR into branch {{feature/metrics}}
because we need to coordinate this work with [~shalinmangar], so I'll continue
working there. I'll update the PR anyway for completeness.
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707000#comment-15707000 ] Jeff Wartes commented on SOLR-4735: --- Heh, I wondered whether something like that would happen if I commented on github. Should I constrain myself to talking in Jira? > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706995#comment-15706995
]
ASF GitHub Bot commented on SOLR-4735:
--
Github user randomstatistic commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/120#discussion_r90144072
--- Diff: solr/core/src/java/org/apache/solr/metrics/SolrMetricManager.java
---
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.metrics;
+
+import java.util.Set;
+
+import com.codahale.metrics.Counter;
+import com.codahale.metrics.Histogram;
+import com.codahale.metrics.Meter;
+import com.codahale.metrics.MetricFilter;
+import com.codahale.metrics.MetricRegistry;
+import com.codahale.metrics.SharedMetricRegistries;
+import com.codahale.metrics.Snapshot;
+import com.codahale.metrics.Timer;
+import org.apache.solr.common.util.NamedList;
+
+/**
+ *
+ */
+public class SolrMetricManager {
+
+ public static final String REGISTRY_NAME_PREFIX = "solr";
+ public static final String DEFAULT_REGISTRY =
MetricRegistry.name(REGISTRY_NAME_PREFIX, "default");
+
+ // don't create instances of this class
+ private SolrMetricManager() { }
+
+
+ /**
+ * Return a set of existing registry names.
+ */
+ public static Set registryNames() {
+return SharedMetricRegistries.names();
+ }
+
+ /**
+ * Get (or create if not present) a named registry
+ * @param registry name of the registry
+ * @return existing or newly created registry
+ */
+ public static MetricRegistry registryFor(String registry) {
+return
SharedMetricRegistries.getOrCreate(overridableRegistryName(registry));
+ }
+
+ /**
+ * Remove all metrics from a specified registry.
+ * @param registry registry name
+ */
+ public static void clearRegistryFor(String registry) {
+
SharedMetricRegistries.getOrCreate(overridableRegistryName(registry)).removeMatching(MetricFilter.ALL);
--- End diff --
This, and several other places below could delegate to
`registryFor(registry)`
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselv
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705963#comment-15705963 ] ASF subversion and git services commented on SOLR-4735: --- Commit f027098b4c0a27c70c7cb33dc80492a199bc44cf in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f027098 ] SOLR-4735 Cleanup and fix lint errors. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705871#comment-15705871 ] ASF subversion and git services commented on SOLR-4735: --- Commit db1339ff622cc73871897f8b345a9be19134a95e in lucene-solr's branch refs/heads/feature/metrics from [~ab] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=db1339f ] SOLR-4735 Improve Solr metric reporting (Alan Woodward, Kelvin Wong, Christine Poerschke, Jeff Wartes, ab) > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705370#comment-15705370 ] ASF GitHub Bot commented on SOLR-4735: -- GitHub user sigram opened a pull request: https://github.com/apache/lucene-solr/pull/120 SOLR-4735 Improve Solr metrics reporting You can merge this pull request into a Git repository by running: $ git pull https://github.com/sigram/lucene-solr jira/solr-4735 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/120.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #120 commit dba0663c79f7b27d4626152d36f8d6d4c62a878d Author: Andrzej Bialecki Date: 2016-11-23T12:48:26Z Initial patch from Jira. commit 1ade9c443dbd5b9eae2ec5208b233d28fb20a8cb Author: Andrzej Bialecki Date: 2016-11-24T10:45:20Z Merge branch 'master' into jira/solr-4735 commit ba2a94fb52d21ed05053a098c8fb9919a469e5b3 Author: Andrzej Bialecki Date: 2016-11-24T15:32:04Z Use SharedMetricRegistries for managing per-core metrics. commit 7199e818da503bc0e8a40fed7d6a1f742ecbae55 Author: Andrzej Bialecki Date: 2016-11-24T15:44:55Z Revert accidental changes to this file. commit a4514638df8a2528f339107869b03fe568856fd9 Author: Andrzej Bialecki Date: 2016-11-28T14:01:42Z SOLR-4735 Use separate registries for core and other stuff. Use collapsible and configurable namespace. commit bf424d1ec1602dffeb33ab0acc8f470e351a6959 Author: Kevin Risden Date: 2016-11-22T19:22:16Z SOLR-9728: Ability to specify Key Store type in solr.in file for SSL commit 5b2594350df11ef54d52f417b34c6d082ad85e89 Author: Noble Paul Date: 2016-11-29T02:35:47Z SOLR-9784: added deprecation javadocs commit 32c4bd7cc0ac2e93e833f5fe84be4ff69f0b7aeb Author: Noble Paul Date: 2016-11-29T02:36:26Z Merge remote-tracking branch 'origin/master' commit 9b4f99c1b351b1401e2dd5922a06d79a809332fb Author: Andrzej Bialecki Date: 2016-11-29T10:37:14Z SOLR-4735 Reorder args from top level to bottom. commit 70b358960dfe8a6da35991b2a84c93cc9370c3d8 Author: Noble Paul Date: 2016-11-29T12:32:59Z SOLR-9546: remove unnecessary boxing commit 2af8ec2689610f4dfb1f5d87b069b0eb54f72155 Author: Andrzej Bialecki Date: 2016-11-29T13:48:44Z SOLR-4735 More cleanup and generalization of JMX reporter. commit 4b7002eae75839d2f56a17a65d7e789cb71a9b26 Author: Andrzej Bialecki Date: 2016-11-29T13:56:24Z Merge branch 'master' into jira/solr-4735 > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705358#comment-15705358 ] ASF GitHub Bot commented on SOLR-4735: -- Github user sigram commented on the issue: https://github.com/apache/lucene-solr/pull/119 This work will be merged and continued in the `features/metrics` branch at ASF. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705357#comment-15705357 ] ASF GitHub Bot commented on SOLR-4735: -- Github user sigram closed the pull request at: https://github.com/apache/lucene-solr/pull/119 > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15702044#comment-15702044
]
Andrzej Bialecki commented on SOLR-4735:
-
[~jwartes] thanks for the pointer to your PR, I borrowed parts of your code and
updated my PR:
* simplified and renamed {{SolrMetricManager}} to {{SolrCoreMetricManager}} as
it really is specific to managing metrics related to {{SolrCore}}.
* added a global component for registry management {{SolrMetricManager}}, which
mostly offers useful syntactic sugar for working with {{SharedMetricRegistries}}
Next step: I'm going to merge my work into [~shalinmangar]'s branch.
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15701901#comment-15701901 ] Shalin Shekhar Mangar commented on SOLR-4735: - I have created a new branch called "feature/metrics" for SOLR-9788, this and other future metrics enhancements -- http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/497212e0 Let's use this for integration between different patches. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15696479#comment-15696479 ] Jeff Wartes commented on SOLR-4735: --- I had a scheme for collapsable namespaced registries in my original PR for SOLR-8785. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15695829#comment-15695829
]
Andrzej Bialecki commented on SOLR-4735:
-
bq. Right now, each producer creates/manages its own set of metrics.
Correct, but if the producer's life-cycle is tied to that of SolrCore then the
lifecycle of that metric will be the same, ie it will vanish when that instance
of core is closed.
bq. My only concern here is how to namespace the metrics so that different
producers clash on metric names. Perhaps give each producer access to the core
name?
I think that the registration mechanism where SolrCore registers these metrics
for each plugin is ok - it would be similar to how it works in your patch,
except we would {{getOrCreate}} these metric instances instead of creating new
ones for each registration.
bq. Sounds good. One suggestion I have is to namespace the registries.
+1. In the pull request I used a core name, but that would be insufficient - it
needs to be made into a hierarchy, eg. "/core/" for SolrCore metrics,
"/jetty/..." for Jetty metrics, etc.
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15695483#comment-15695483
]
Kelvin Wong commented on SOLR-4735:
---
{quote}
Having a reload reset stats is a bad idea. We can provide an explicit API to
reset stats for a node or a core if required.
{quote}
Right now, each producer creates/manages its own set of metrics. It might make
sense to have some global object creating/managing all our metrics instead.
Each producer can then call {{getOrCreate}} so that the same set of metrics are
used across core reloads. My only concern here is how to namespace the metrics
so that different producers clash on metric names. Perhaps give each producer
access to the core name?
{quote}
Perhaps SolrMetricManager should use long-lived MetricRegistry instances that
are managed in SharedMetricsRegistries?
{quote}
Sounds good. One suggestion I have is to namespace the registries. For example,
we can have each {{SolrCore}} report on a registry, Jetty report on another,
JVM on another, etc... Then we can configure reporters for each such registry
by just specifying its name. This keeps the different sets of metrics nicely
isolated and gives us flexibility as to how to report each set of metrics?
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Andrzej Bialecki
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693874#comment-15693874 ] Jeff Wartes commented on SOLR-4735: --- >From what I see, SolrMetricManager only needs the SolrCore for the >config-based reporter instantiation, but that's a pretty nice thing to have. How about SolrMetricManager takes, as an optional second parameter to the constructor, the name of a SharedMetricRegistry. If absent, then it creates a new, isolated registry. With a name though, that means the config-based reporters you attach are actually being attached to the shared registry, pulling whatever happens to be in there too. Of course, then the core unregister action needs to be careful to only replace/reset those metrics that it'd added to the registry, instead of all of them as currently written. It could remove/replace the reporters with no real issue on every core reload (aside from possibly a blip in the reporting interval) though. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693809#comment-15693809 ] Shalin Shekhar Mangar commented on SOLR-4735: - bq. I share Jeff's concern though that we need to consider how to maintain metrics that outlive any particular core instance (even core reload events ) Having a reload reset stats is a bad idea. We can provide an explicit API to reset stats for a node or a core if required. bq. Perhaps SolrMetricManager should use long-lived MetricRegistry instances that are managed in SharedMetricsRegistries? +1 Also see the patch on SOLR-9788 which adds instrumented classes to Jetty that are managed by SharedMetricsRegistries. The JVM metrics can also be exposed in a similar way. It currently adds all jetty statistics to a metric registry named "solr" but we can split them out into multiple ones if needed. I'll rebase my patch over this pull request. As far as the pull request is concerned I'd suggest that we rename SolrMetricManager to SolrCoreMetricManager because it is tied to a single core. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693560#comment-15693560 ] ASF GitHub Bot commented on SOLR-4735: -- GitHub user sigram opened a pull request: https://github.com/apache/lucene-solr/pull/119 SOLR-4735 Improve Solr metrics Branch created from the patch in Jira by Kelvin Wong. Changes include: * using shared instances of `MetricRegistry` per core. * unit test modifications. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sigram/lucene-solr jira/solr-4735 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/119.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #119 commit dba0663c79f7b27d4626152d36f8d6d4c62a878d Author: Andrzej Bialecki Date: 2016-11-23T12:48:26Z Initial patch from Jira. commit 1ade9c443dbd5b9eae2ec5208b233d28fb20a8cb Author: Andrzej Bialecki Date: 2016-11-24T10:45:20Z Merge branch 'master' into jira/solr-4735 commit ba2a94fb52d21ed05053a098c8fb9919a469e5b3 Author: Andrzej Bialecki Date: 2016-11-24T15:32:04Z Use SharedMetricRegistries for managing per-core metrics. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693048#comment-15693048
]
Andrzej Bialecki commented on SOLR-4735:
-
I'm interested in getting this issue resolved, so I'd be happy to work on
committing this (I asked Alan and he doesn't mind :) ).
[~cpoerschke] & Kelvin: great stuff, I really like the abstractions. I share
Jeff's concern though that we need to consider how to maintain metrics that
outlive any particular core instance (even core reload events :) ). Core
reloads may be caused by several reasons (explicit action, config change,
replication). I'm not sure under which scenario I'd prefer to reset metrics
from a previous version of the core...
Eventually we will want to instrument also other aspects of Solr, things that
happen outside SolrCore (eg. SolrCloud operations, replication, leader metrics
per replica, replica recovery stats, Jetty connections, heap, etc). For these
using {{SharedMetricsRegistries}} would make more sense, so the question is
whether we should use two different mechanisms for managing {{MetricRegistry}}
instances, the other one being {{SolrMetricManager}}. Perhaps
{{SolrMetricManager}} should use long-lived {{MetricRegistry}} instances that
are managed in {{SharedMetricsRegistries}}?
Also, from the point of view of monitoring the overall "load" of a particular
node it would make sense to also track some really low-level Lucene stuff, such
as major merges and read/write IO, but this can come later - let's first get
the design right.
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch,
> SOLR-4735.patch
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693036#comment-15693036 ] Kelvin Wong commented on SOLR-4735: --- Thanks Jeff. The attached patch really just piggybacks off of Alan's work and tries to flesh out the design. For now, only RequestHandlerBase exposes metrics through this framework. The idea is to eventually convert other SolrInfoMBeans into SolrMetricProducers so they can start providing reportable metrics too. This seems to be fairly doable. Re SharedMetricsRegistries: that's something we can definitely do. My rationale for not using it is so that logical groups of metrics can be nicely isolated at a per-core level. This ensures that any metric in a MetricManager's registry must have been registered through MetricManager::registerMetrics. A nice side-effect is that we can also store meta-information about each metric and pass that on to the reporters. I realize that using SharedMetricsRegistries provides a level of flexibility that this patch's approach does not. For example, if we wanted to share the registry on a CoreContainer level. I think there are ways around this and my personal preference is still for this logical grouping of metrics. But perhaps there may be use cases I'm neglecting to consider? Would be interested to hear your thoughts on this. Thanks! > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15690992#comment-15690992 ] Jeff Wartes commented on SOLR-4735: --- For what it's worth, this looks like really great stuff to me. I'm still unconvinced that metrics should always get reset on core reload, which is a source of some complexity, but doing so is certainly consistent with the prior behavior, so I can hardly complain. I think I can see a path to providing reportable metrics outside of the RequestHandler. I'd be interested in Kelvin's thoughts on that subject though, since he chose not to use SharedMetricsRegistries. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15626442#comment-15626442 ] Jeff Wartes commented on SOLR-4735: --- I have, and am, by instantiating a SharedMetricRegistry and GraphiteReporter directly in the jetty.xml. (Which is hacky, but in lieu of SOLR-8785, does work fine) I'm also using the logging and JVM metrics plugins quite happily. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15626331#comment-15626331 ] Walter Underwood commented on SOLR-4735: Anybody using the CodaHale metrics.jetty9.InstrumentedHandler? It looks a lot like something we built for our own use with Solr 4. http://metrics.dropwizard.io/3.1.0/manual/jetty/ http://metrics.dropwizard.io/3.1.0/apidocs/com/codahale/metrics/jetty9/InstrumentedHandler.html wunder > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1459#comment-1459 ] Ramkumar Aiyengar commented on SOLR-4735: - [~romseygeek], still quite interested in this, would you happen to have the current state in a patch and an idea of things which are still left? > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223572#comment-14223572 ] Alan Woodward commented on SOLR-4735: - re introspection and discovery, each MetricProvider class will register metrics with a common prefix, and it should be easy to write a handler that returns all metrics with a given prefix. Merging stats isn't directly supported by the Metrics library, as far as I know. I'm halfway through a patch on this - LazyRequestHandlers are causing me a headache at the moment, but I think I can see a way through. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14222847#comment-14222847 ] Shalin Shekhar Mangar commented on SOLR-4735: - That sounds like a reasonable start, Alan. Do you have any ideas on how we can introspect/discover metrics supported by a plugin? Also, does the metrics API have any support for merging stats from different servers? The last two questions are important to build aggregate stats API for SolrCloud (see SOLR-6325) and to easily build the next generation of Solr's Admin UI. We should also add stats support to the SearchComponents (maybe as a separate issue) and hopefully have the debug component be a wrapper around the metrics API. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14222807#comment-14222807 ] Alan Woodward commented on SOLR-4735: - I did some thinking on this over the weekend. Here's what I think the API should look like: * we use dropwizard metrics: https://dropwizard.github.io/metrics/3.1.0/. This is the same library I used in the initial patch, moved on a bit. * each core has its own MetricRegistry * SolrInfoMBean is replaced by an interface called SolrMetricsProducer, which has just three methods: ** String getName() ** Category getCategory() ** void registerMetrics(MetricsCollector registry) * when core.inform() is called with the SolrMetricsProducer, it creates a MetricsCollector using the name and category, and then passes it to registerMetrics(). The producer can then add its own metrics as individual Counter, Gauge, Histogram, etc, instances to the collector. The Core then adds all these to its MetricRegistry, with appropriate names. * getStatistics() calls are replaced by a NamedListMetricReporter, as in the initial patch * the metrics JmxReporter will have to be extended slightly so that it knows about corehash values, to make sure that we deal with core reloads correctly. * CoreContainer could have its own MetricRegistry for any node-wide stats (things like System/Memory/JVM info - there's a metrics/jvm subproject which has a bunch of useful stats that we could add here). This way, all the individual stats are independent, and reporting and collection are nicely isolated. I also suggest we bin getSource(), getVersion() and getDocs() as they're generally left unimplemented, and probably getDescription() as well. And the Category enum could be replaced with a plain String, to allow plugins to add their own types. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14216504#comment-14216504 ] Mark Miller commented on SOLR-4735: --- bq. feel free to gut what we have in Solr5 We don't have a lot of time, but it would be great to solve SOLR-6586 - it really requires a different stats API to be sensible I think. It's a little tricky to make nice, but really the API calls for each individual attribute should be able to be calculated independently. Otherwise, there is just so much recalculation that it's hard to have everything be live and fast and even if you only want to fetch a single fast attribute, you will be penalized by the slowest. If you currently use a tool to enumerate and look at each attribute for monitoring, because of the duplicate bean issue and SOLR-6586, you can check the size of a directory like 40 times or something crazy when it really only had to be checked once. There is an API mismatch. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13959290#comment-13959290 ] Hoss Man commented on SOLR-4735: Tim: see also SOLR-5095 which points out some anoying problems we have with the lack of namespacing of plugins/mbeans right now that make it confusing/impossible to get stats from certain things. I think Alan's earlier point about how SolrInfoMBean cleanup is needed in general is spot on. We should keep an open mind to what the ideal APIs look like for monitoring/metrics and feel free to gut what we have in Solr5 to move towards those APIs > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13959253#comment-13959253 ] Alan Woodward commented on SOLR-4735: - The patch that's here is pretty out-of-date, and seriously incomplete, but I still think it's a workable idea. I just don't have time to work on it... feel free to assign this one to yourself and have a go. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13959230#comment-13959230 ] Timothy Potter commented on SOLR-4735: -- I have some interest in working on this idea and wanted to get a sense where things sit / current thinking on this topic. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13642969#comment-13642969 ] Alan Woodward commented on SOLR-4735: - That's a nice idea. The MetricsReporters would have to be able to build a filter that meant they only reported on Metrics set up for their core, but that should be doable. Will work on a patch over the weekend. > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[
https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13642148#comment-13642148
]
Ryan McKinley commented on SOLR-4735:
-
ideally CoreContainer could have a function like:
{code:java}
MetricsRegistry createMetricsRegistry( ?? config ) {
return new MetricsRegistry();
}
{code}
This would let other applications slip in their own registry -- that already
has reporting hooked up!
> Improve Solr metrics reporting
> --
>
> Key: SOLR-4735
> URL: https://issues.apache.org/jira/browse/SOLR-4735
> Project: Solr
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Minor
> Attachments: SOLR-4735.patch, SOLR-4735.patch
>
>
> Following on from a discussion on the mailing list:
> http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+
> It would be good to make Solr play more nicely with existing devops
> monitoring systems, such as Graphite or Ganglia. Stats monitoring at the
> moment is poll-only, either via JMX or through the admin stats page. I'd
> like to refactor things a bit to make this more pluggable.
> This patch is a start. It adds a new interface, InstrumentedBean, which
> extends SolrInfoMBean to return a
> [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a
> couple of MetricReporters (which basically just duplicate the JMX and admin
> page reporting that's there at the moment, but which should be more
> extensible). The patch includes a change to RequestHandlerBase showing how
> this could work. The idea would be to eventually replace the getStatistics()
> call on SolrInfoMBean with this instead.
> The next step would be to allow more MetricReporters to be defined in
> solrconfig.xml. The Metrics library comes with ganglia and graphite
> reporting modules, and we can add contrib plugins for both of those.
> There's some more general cleanup that could be done around SolrInfoMBean
> (we've got two plugin handlers at /mbeans and /plugins that basically do the
> same thing, and the beans themselves have some weirdly inconsistent data on
> them - getVersion() returns different things for different impls, and
> getSource() seems pretty useless), but maybe that's for another issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13642139#comment-13642139 ] Ryan McKinley commented on SOLR-4735: - This looks like it creates a new registry for every core (am I reading that wrong?) If so, I think sharing one registry would be best. Can the registry be in the CoreContainer rather then the core? I guess that would involve some cleanup when a core is unloaded, but it would let us share a single registry across cores and other apps (the case I am actually concerned with) > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
