[jira] [Updated] (HADOOP-13730) After 5 connection failures, yarn stops sending metrics graphite until restarted
[ https://issues.apache.org/jira/browse/HADOOP-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Young updated HADOOP-13730: Fix Version/s: 2.8.0 Status: Patch Available (was: Open) Keep on trying forever, graphite might come back. > After 5 connection failures, yarn stops sending metrics graphite until > restarted > > > Key: HADOOP-13730 > URL: https://issues.apache.org/jira/browse/HADOOP-13730 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.7.2 >Reporter: Sean Young >Priority: Minor > Fix For: 2.8.0 > > Attachments: > 0001-Graphite-can-be-unreachable-for-some-time-and-come-b.patch > > > We've had issues in production where metrics stopped. We found the following > in the log files: > 2016-09-02 21:44:32,493 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: > Error sending metrics to Graphite > java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > at > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120) > at java.net.SocketOutputStream.write(SocketOutputStream.java:164) > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233) > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147) > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270) > at java.io.Writer.write(Writer.java:154) > at > org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170) > at > org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43) > at > org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88) > 2016-09-03 00:03:04,335 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: > Error sending metrics to Graphite > java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > at > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120) > at java.net.SocketOutputStream.write(SocketOutputStream.java:164) > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233) > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147) > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270) > at java.io.Writer.write(Writer.java:154) > at > org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170) > at > org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43) > at > org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88) > 2016-09-03 00:20:35,436 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: > Error sending metrics to Graphite > java.net.SocketException: Connection timed out > at java.net.SocketOutputStream.socketWrite0(Native Method) > at > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120) > at java.net.SocketOutputStream.write(SocketOutputStream.java:164) > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233) > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147) > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270) > at java.io.Writer.write(Writer.java:154) > at > org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170) > at >
[jira] [Updated] (HADOOP-13730) After 5 connection failures, yarn stops sending metrics graphite until restarted
[ https://issues.apache.org/jira/browse/HADOOP-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Young updated HADOOP-13730: Attachment: 0001-Graphite-can-be-unreachable-for-some-time-and-come-b.patch > After 5 connection failures, yarn stops sending metrics graphite until > restarted > > > Key: HADOOP-13730 > URL: https://issues.apache.org/jira/browse/HADOOP-13730 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.7.2 >Reporter: Sean Young >Priority: Minor > Attachments: > 0001-Graphite-can-be-unreachable-for-some-time-and-come-b.patch > > > We've had issues in production where metrics stopped. We found the following > in the log files: > 2016-09-02 21:44:32,493 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: > Error sending metrics to Graphite > java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > at > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120) > at java.net.SocketOutputStream.write(SocketOutputStream.java:164) > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233) > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147) > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270) > at java.io.Writer.write(Writer.java:154) > at > org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170) > at > org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43) > at > org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88) > 2016-09-03 00:03:04,335 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: > Error sending metrics to Graphite > java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > at > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120) > at java.net.SocketOutputStream.write(SocketOutputStream.java:164) > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233) > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147) > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270) > at java.io.Writer.write(Writer.java:154) > at > org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170) > at > org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43) > at > org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88) > 2016-09-03 00:20:35,436 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: > Error sending metrics to Graphite > java.net.SocketException: Connection timed out > at java.net.SocketOutputStream.socketWrite0(Native Method) > at > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120) > at java.net.SocketOutputStream.write(SocketOutputStream.java:164) > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233) > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147) > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270) > at java.io.Writer.write(Writer.java:154) > at > org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170) > at > org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98) > at >
[jira] [Created] (HADOOP-13730) After 5 connection failures, yarn stops sending metrics graphite until restarted
Sean Young created HADOOP-13730: --- Summary: After 5 connection failures, yarn stops sending metrics graphite until restarted Key: HADOOP-13730 URL: https://issues.apache.org/jira/browse/HADOOP-13730 Project: Hadoop Common Issue Type: Bug Components: common Affects Versions: 2.7.2 Reporter: Sean Young Priority: Minor We've had issues in production where metrics stopped. We found the following in the log files: 2016-09-02 21:44:32,493 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: Error sending metrics to Graphite java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120) at java.net.SocketOutputStream.write(SocketOutputStream.java:164) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233) at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294) at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137) at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147) at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270) at java.io.Writer.write(Writer.java:154) at org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170) at org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43) at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88) 2016-09-03 00:03:04,335 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: Error sending metrics to Graphite java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120) at java.net.SocketOutputStream.write(SocketOutputStream.java:164) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233) at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294) at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137) at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147) at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270) at java.io.Writer.write(Writer.java:154) at org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170) at org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43) at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88) 2016-09-03 00:20:35,436 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: Error sending metrics to Graphite java.net.SocketException: Connection timed out at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120) at java.net.SocketOutputStream.write(SocketOutputStream.java:164) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233) at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294) at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137) at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147) at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270) at java.io.Writer.write(Writer.java:154) at org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170) at org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43) at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88) 2016-09-03 00:22:48,862 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: Error sending
