On Tuesday, July 26, 2016 at 10:21:32 AM UTC-4, [email protected] wrote: > On Monday, July 25, 2016 at 5:57:53 PM UTC-4, Sean Beckett wrote: > > It is expected behavior that pointsRx will be larger than pointsTx, but not > > for more than 1s or 5000 points. pointsRx is a count of the points received > > by the UDP listener. Those points are fed to a batcher running in the > > background which batches up points every 1s or 5000 points, whichever comes > > first. Those batches are then written to the database, which increments > > pointsTx. > > > > > > Are you certain none of your points are duplicates? (Same measurement, tag > > set, and timestamp) > > > > > > On Fri, Jul 22, 2016 at 3:01 PM, Hiro <[email protected]> wrote: > > I have been sending UDP data over to one of my databases, and some points > > are not coming through. While investigating where this is happening (I am > > aware of the nature of UDP but the volume of data is quite small), I was > > looking at the UDP measurement in the _internal database. I can see that at > > some times, pointsTx (points transmitted) is lower than pointsRx (points > > received), and was curious as to what is meant by points transmitted.. are > > these points that made it into the database? I am just curious about the > > relationship between the two and if there is any way to observe that > > behavior further? Any help would be greatly appreciated! > > > > > > > > Thanks, > > > > Hiro > > > > > > > > -- > > > > Remember to include the InfluxDB version number with all issue reports > > > > --- > > > > You received this message because you are subscribed to the Google Groups > > "InfluxDB" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > > > To post to this group, send email to [email protected]. > > > > Visit this group at https://groups.google.com/group/influxdb. > > > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/influxdb/8438915b-1c5a-4e10-99a9-75cd716d5807%40googlegroups.com. > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > -- > > > > > > Sean Beckett > > Director of Support and Professional Services > > InfluxDB > > Thanks for the explanation! Our system seems to far worse than expected, as > we are often (basically always) farther ahead with pointsRx than with > pointsTx. Here is a graph that shows that relationship a few times, as you > can see pointsTx lags behind by a lot... > > https://i.sli.mg/kL67PV.png > > And this is how our data looks coming in (it seems to be fine for a while > initially, but then points stop arriving consistently and our data is chunky > > https://i.sli.mg/NCfKRE.png > > As for duplicates, I am sure that none of the points are the same, they all > differ. I generated some text files with all our data points and ran "sort > <file> | uniq -d" on our ubuntu server, and all of the files had unique lines > (Tested this with 25532 points from 26 seconds of data and 68680 points for > 98 seconds of data) > > Is it possible that I just configured UDP wrong and it can't handle this > large of an amount of data? > > In the influxdb.conf file it is set up as follows: > > [[udp]] > enabled = true > bind-address = ":8089" > database = "udp" > batch-size = 5000 > batch-timeout = "2s" > read-buffer = 8388608
So I ended up ramping up the influxdb.conf settings and now we are not losing data anymore. The config seems a bit ridiculous but this is what I changed it to: [[udp]] enabled = true bind-address = ":8089" database = "udp" batch-size = 100000 batch-pending = 25 batch-timeout = "10s" -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/d26783ac-1717-4cee-814b-49f25d3ae785%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
