[ 
https://issues.apache.org/jira/browse/HADOOP-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Young updated HADOOP-13730:
--------------------------------
    Fix Version/s: 2.8.0
           Status: Patch Available  (was: Open)

Keep on trying forever, graphite might come back.

> After 5 connection failures, yarn stops sending metrics graphite until 
> restarted
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-13730
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13730
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: common
>    Affects Versions: 2.7.2
>            Reporter: Sean Young
>            Priority: Minor
>             Fix For: 2.8.0
>
>         Attachments: 
> 0001-Graphite-can-be-unreachable-for-some-time-and-come-b.patch
>
>
> We've had issues in production where metrics stopped. We found the following 
> in the log files:
> 2016-09-02 21:44:32,493 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: 
> Error sending metrics to Graphite
> java.net.SocketException: Broken pipe
>         at java.net.SocketOutputStream.socketWrite0(Native Method)
>         at 
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120)
>         at java.net.SocketOutputStream.write(SocketOutputStream.java:164)
>         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
>         at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147)
>         at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270)
>         at java.io.Writer.write(Writer.java:154)
>         at 
> org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170)
>         at 
> org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
>         at 
> org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> 2016-09-03 00:03:04,335 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: 
> Error sending metrics to Graphite
> java.net.SocketException: Broken pipe
>         at java.net.SocketOutputStream.socketWrite0(Native Method)
>         at 
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120)
>         at java.net.SocketOutputStream.write(SocketOutputStream.java:164)
>         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
>         at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147)
>         at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270)
>         at java.io.Writer.write(Writer.java:154)
>         at 
> org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170)
>         at 
> org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
>         at 
> org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> 2016-09-03 00:20:35,436 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: 
> Error sending metrics to Graphite
> java.net.SocketException: Connection timed out
>         at java.net.SocketOutputStream.socketWrite0(Native Method)
>         at 
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120)
>         at java.net.SocketOutputStream.write(SocketOutputStream.java:164)
>         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
>         at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147)
>         at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270)
>         at java.io.Writer.write(Writer.java:154)
>         at 
> org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170)
>         at 
> org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
>         at 
> org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> 2016-09-03 00:22:48,862 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: 
> Error sending metrics to Graphite
> java.net.SocketException: Broken pipe
>         at java.net.SocketOutputStream.socketWrite0(Native Method)
>         at 
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120)
>         at java.net.SocketOutputStream.write(SocketOutputStream.java:164)
>         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
>         at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147)
>         at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270)
>         at java.io.Writer.write(Writer.java:154)
>         at 
> org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170)
>         at 
> org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
>         at 
> org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> 2016-09-03 00:24:00,270 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: 
> Error sending metrics to Graphite
> java.net.SocketException: Broken pipe
>         at java.net.SocketOutputStream.socketWrite0(Native Method)
>         at 
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120)
>         at java.net.SocketOutputStream.write(SocketOutputStream.java:164)
>         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
>         at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147)
>         at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270)
>         at java.io.Writer.write(Writer.java:154)
>         at 
> org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170)
>         at 
> org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
>         at 
> org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> 2016-09-03 00:24:41,987 INFO 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 482 
> Total time for transactions(ms): 9 Number of transactions batched in Syncs: 3 
> Number of syncs: 355 SyncTimes(ms): 342
> 2016-09-03 00:25:39,772 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: 
> Error sending metrics to Graphite
> java.net.SocketException: Connection reset
>         at 
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:124)
>         at java.net.SocketOutputStream.write(SocketOutputStream.java:164)
>         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
>         at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147)
>         at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270)
>         at java.io.Writer.write(Writer.java:154)
>         at 
> org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170)
>         at 
> org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
>         at 
> org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> The last error was:
> 2016-09-03 01:13:51,619 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: 
> Error sending metrics to Graphite
> java.net.SocketException: Broken pipe
>         at java.net.SocketOutputStream.socketWrite0(Native Method)
>         at 
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120)
>         at java.net.SocketOutputStream.write(SocketOutputStream.java:164)
>         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
>         at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147)
>         at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270)
>         at java.io.Writer.write(Writer.java:154)
>         at 
> org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170)
>         at 
> org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
>         at 
> org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
>         at 
> org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> 2016-09-03 01:15:58,828 ERROR org.apache.hadoop.metrics2.sink.GraphiteSink: 
> Too many connection failures, would not try to connect again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to