[
https://issues.apache.org/jira/browse/SOLR-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185978#comment-16185978
]
Erik Persson edited comment on SOLR-11413 at 9/29/17 3:26 PM:
--------------------------------------------------------------
Example error messages using Solr 6.6.0.
{quote}2017-09-28 19:09:16.226 ERROR (metrics-graphite-reporter-2-thread-1) [
] c.c.m.ScheduledReporter Exception thrown from GraphiteReporter#report.
Exception was suppressed.
java.lang.IllegalStateException: Already connected
at com.codahale.metrics.graphite.Graphite.connect(Graphite.java:111)
at
com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:240)
at
com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:235)
at
com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:174)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-09-28 19:09:16.226 WARN (metrics-graphite-reporter-1-thread-1) [ ]
c.c.m.g.GraphiteReporter Unable to report to Graphite
java.io.IOException: Stream closed
at java.io.BufferedWriter.ensureOpen(BufferedWriter.java:116)
at java.io.BufferedWriter.write(BufferedWriter.java:221)
at java.io.Writer.write(Writer.java:157)
at com.codahale.metrics.graphite.Graphite.send(Graphite.java:138)
at
com.codahale.metrics.graphite.GraphiteReporter.sendIfEnabled(GraphiteReporter.java:328)
at
com.codahale.metrics.graphite.GraphiteReporter.reportMetered(GraphiteReporter.java:305)
at
com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:255)
at
com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:235)
at
com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:174)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-09-28 19:09:16.234 ERROR (metrics-graphite-reporter-3-thread-1) [ ]
c.c.m.ScheduledReporter Exception thrown from GraphiteReporter#report.
Exception was suppressed.
java.lang.NullPointerException{quote}
was (Author: erikpersson):
Example error messages using Solr 6.6.0.
{{2017-09-28 19:09:16.226 ERROR (metrics-graphite-reporter-2-thread-1) [ ]
c.c.m.ScheduledReporter Exception thrown from GraphiteReporter#report.
Exception was suppressed.
java.lang.IllegalStateException: Already connected
at com.codahale.metrics.graphite.Graphite.connect(Graphite.java:111)
at
com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:240)
at
com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:235)
at
com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:174)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-09-28 19:09:16.226 WARN (metrics-graphite-reporter-1-thread-1) [ ]
c.c.m.g.GraphiteReporter Unable to report to Graphite
java.io.IOException: Stream closed
at java.io.BufferedWriter.ensureOpen(BufferedWriter.java:116)
at java.io.BufferedWriter.write(BufferedWriter.java:221)
at java.io.Writer.write(Writer.java:157)
at com.codahale.metrics.graphite.Graphite.send(Graphite.java:138)
at
com.codahale.metrics.graphite.GraphiteReporter.sendIfEnabled(GraphiteReporter.java:328)
at
com.codahale.metrics.graphite.GraphiteReporter.reportMetered(GraphiteReporter.java:305)
at
com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:255)
at
com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:235)
at
com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:174)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-09-28 19:09:16.234 ERROR (metrics-graphite-reporter-3-thread-1) [ ]
c.c.m.ScheduledReporter Exception thrown from GraphiteReporter#report.
Exception was suppressed.
java.lang.NullPointerException}}
> SolrGraphiteReporter fails to report metrics due to non-thread safe code
> ------------------------------------------------------------------------
>
> Key: SOLR-11413
> URL: https://issues.apache.org/jira/browse/SOLR-11413
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: metrics
> Affects Versions: 6.6, 7.0
> Reporter: Erik Persson
> Attachments: SOLR-11413.patch
>
>
> Symptom:
> Intermittent errors writing graphite metrics. Errors indicate use of sockets
> which have already been closed.
> Cause:
> SolrGraphiteReporter caches and shares dropwizard Graphite instances. These
> reporters are not thread safe as they open and close an instance variable of
> type GraphiteSender. On modern bare metal hardware this problem was observed
> consistently, and resulted in the majority of metrics failing to be delivered
> to graphite.
> Proposed Fix:
> Graphite (and PickledGraphite) are not designed to be cached, and should not
> be.
> Test:
> Patch file includes test which forces error.
> Alternative Fixes Considered:
> * Totally change solr metrics architecture to use a single metrics registry -
> seems undesirable and impractical
> * Create a synchronized or otherwise thread-safe implementation of dropwizard
> graphite reporter - should be fixed upstream in dropwizard and not obviously
> preferred to current model
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
