No firewall between them. Infact, I checked on another server of the same cluster, it does not throw any error. My Prometheus server and target server both reside under the same Datacenter. No error logs whatsoever for that time period on Prometheus side. On Tuesday, December 22, 2020 at 6:59:11 PM UTC+5:30 Stuart Clark wrote:
> On 22/12/2020 13:25, [email protected] wrote: > > Hi. I am running node_exporter version 0.18.1 on around 2500 servers. > > On some of the servers I am observing that node_exporter is throwing > > error intermittently for sending the metrics to Prometheus server. > > > > Dec 22 17:10:29 hostname node_exporter[10807]: > > time="2020-12-22T17:10:29+05:30" level=error msg="error encoding and > > sending metric family: write tcp ServerIP:9100->PrometheusIP:40984: > > write: broken pipe\n" source="log.go:172" > > Dec 22 17:10:29 hostname node_exporter[10807]: > > time="2020-12-22T17:10:29+05:30" level=error msg="error encoding and > > sending metric family: write tcp ServerIP:9100->PrometheusIP:40984: > > write: broken pipe\n" source="log.go:172" > > Dec 22 17:10:29 hostname node_exporter[10807]: 2020/12/22 17:10:29 > > http: superfluous response.WriteHeader call from > > > github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader > > > (delegator.go:59) > > > That suggests the Prometheus server disconnected before all the data was > returned. What do the Prometheus logs at that time say? Are there any > firewalls between the two that could have broken the connection? > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ec4c1243-c10a-438c-aefa-98830a4c439cn%40googlegroups.com.
