Kostas, Thanks for the help, nc showed I wasn't actually getting traffic. Turned out I failed to properly unload my firewall configuration and it was still blocking traffic. Tcpdump was just capturing the traffic before the firewall. PEBCAK error! Regards, -Michael
On Friday, July 1, 2016 at 6:59:53 AM UTC-4, [email protected] wrote: > > On Thursday, June 30, 2016 at 8:26:50 PM UTC+3, Michael G. wrote: > > Recently I've been revisiting influxdb and trying to setup some toy > deployments. As part of this process we've set up a handful of systems with > collectd pulling metrics and pushing to my instance of influxdb over the > collectd network plugin. There are several examples of this setup available > but it seems I can't get data from the remote instances of collectd to > populate in influx. The catch here is that a local collectd instance that > monitors the influxdb system is able to populate but it's the only one. > Below are the debug steps I've taken, any advice or something I might be > overlooking is appreciated. > > > > > > system information: > > CentOS Linux release 7.2.1511 > > collectd-5.5.0 > > influxdb-0.13.0.x86_64 > > > > > > influxdb configuration: > > > > > > > > [[collectd]] > > enabled = true > > database = "collectd" > > typesdb = "/usr/share/collectd/types.db" > > bind-address = ":25826" > > collectd network plugin configuration: (same on all remote/local) > > > > > > > > <Plugin network> > > Server "metrics.tst" "25826" > > </Plugin> > > > > I've verified the following for debug purposes as per various guides > online: > > > > > > UDP buffer size: > > > > > > > > [root@metrics ~]# sysctl net.core.rmem_max > > net.core.rmem_max = 8388608 > > Verify receiving packets: > > > > > > > > [root@metrics ~]# tcpdump -v -n dst port 25826 > > tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size > 65535 bytes > > 13:06:47.283486 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto > UDP (17), length 1339) > > 10.0.0.100.46339 > 10.0.0.10.25826: UDP, length 1311 > > 13:06:49.283651 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto > UDP (17), length 1336) > > 10.0.0.100.46339 > 10.0.0.10.25826: UDP, length 1308 > > Verify that I'm listening: > > > > > > > > > > [root@metrics ~]# ss -anu > > State Recv-Q Send-Q Local Address:Port Peer Address:Port > > UNCONN 0 0 :::25826 :::* > > > > > > > > [root@metrics ~]# netstat -anlp | grep 25826 > > udp6 0 0 :::25826 :::* > 9387/influxd > > > > influxdb log output: > > > > > > > > [root@metrics ~]# tail -F /var/log/influxdb/influxd.log | grep collectd > > [collectd] 2016/06/30 13:15:01 Starting collectd service > > [collectd] 2016/06/30 13:15:01 Listening on UDP: [::]:25826 > > > > Every time when I try to dump the series information: > > > > > > > > [root@metrics ~]# influx > > Visit https://enterprise.influxdata.com to register for updates, > InfluxDB server management, and monitoring. > > Connected to http://localhost:8086 version 0.13.0 > > InfluxDB shell version: 0.13.0 > > > use collectd > > Using database collectd > > > show series > > > > > > > However when I use a local collectd instance I get several populated > series.I've tried looking for more debugging output from influx but what > the collectd service provides isn't much. Additionally the I haven't found > anything in stats or diagnostics that helps. Any ideas anyone? > > Regards, > > -Michael > > Hi Michael, > > I suggest opening up a dummy port on your "metrics" server to see how the > incoming udp traffic looks like. > > An easy way to do this is using netcat. > After stopping influxdb, start listening on the UDP port 25826 with nc and > examine your UDP input: `nc -u -l 25826` > This would be exactly what the InfluxDB collectd plugin receives. Does the > input look right? > > Thanks, > Kostas > -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/ee234206-2036-4710-b172-8e2e83785415%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
