On Tuesday, July 26, 2016 at 12:59:51 PM UTC-4, [email protected] wrote: > On Tuesday, July 26, 2016 at 10:21:32 AM UTC-4, [email protected] wrote: > > On Monday, July 25, 2016 at 5:57:53 PM UTC-4, Sean Beckett wrote: > > > It is expected behavior that pointsRx will be larger than pointsTx, but > > > not for more than 1s or 5000 points. pointsRx is a count of the points > > > received by the UDP listener. Those points are fed to a batcher running > > > in the background which batches up points every 1s or 5000 points, > > > whichever comes first. Those batches are then written to the database, > > > which increments pointsTx. > > > > > > > > > Are you certain none of your points are duplicates? (Same measurement, > > > tag set, and timestamp) > > > > > > > > > On Fri, Jul 22, 2016 at 3:01 PM, Hiro <[email protected]> wrote: > > > I have been sending UDP data over to one of my databases, and some points > > > are not coming through. While investigating where this is happening (I am > > > aware of the nature of UDP but the volume of data is quite small), I was > > > looking at the UDP measurement in the _internal database. I can see that > > > at some times, pointsTx (points transmitted) is lower than pointsRx > > > (points received), and was curious as to what is meant by points > > > transmitted.. are these points that made it into the database? I am just > > > curious about the relationship between the two and if there is any way to > > > observe that behavior further? Any help would be greatly appreciated! > > > > > > > > > > > > Thanks, > > > > > > Hiro > > > > > > > > > > > > -- > > > > > > Remember to include the InfluxDB version number with all issue reports > > > > > > --- > > > > > > You received this message because you are subscribed to the Google Groups > > > "InfluxDB" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [email protected]. > > > > > > To post to this group, send email to [email protected]. > > > > > > Visit this group at https://groups.google.com/group/influxdb. > > > > > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/influxdb/8438915b-1c5a-4e10-99a9-75cd716d5807%40googlegroups.com. > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Sean Beckett > > > Director of Support and Professional Services > > > InfluxDB > > > > Thanks for the explanation! Our system seems to far worse than expected, as > > we are often (basically always) farther ahead with pointsRx than with > > pointsTx. Here is a graph that shows that relationship a few times, as you > > can see pointsTx lags behind by a lot... > > > > https://i.sli.mg/kL67PV.png > > > > And this is how our data looks coming in (it seems to be fine for a while > > initially, but then points stop arriving consistently and our data is chunky > > > > https://i.sli.mg/NCfKRE.png > > > > As for duplicates, I am sure that none of the points are the same, they all > > differ. I generated some text files with all our data points and ran "sort > > <file> | uniq -d" on our ubuntu server, and all of the files had unique > > lines (Tested this with 25532 points from 26 seconds of data and 68680 > > points for 98 seconds of data) > > > > Is it possible that I just configured UDP wrong and it can't handle this > > large of an amount of data? > > > > In the influxdb.conf file it is set up as follows: > > > > [[udp]] > > enabled = true > > bind-address = ":8089" > > database = "udp" > > batch-size = 5000 > > batch-timeout = "2s" > > read-buffer = 8388608 > > So I ended up ramping up the influxdb.conf settings and now we are not losing > data anymore. The config seems a bit ridiculous but this is what I changed it > to: > > [[udp]] > enabled = true > bind-address = ":8089" > database = "udp" > batch-size = 100000 > batch-pending = 25 > batch-timeout = "10s"
And additionally: read-buffer = 8388608 udp-payload-size = 65536 -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/ae774721-0472-424e-a9a7-0ecd5d6349fe%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
