On Tuesday, July 26, 2016 at 1:01:35 PM UTC-4, [email protected] wrote: > On Tuesday, July 26, 2016 at 12:59:51 PM UTC-4, [email protected] wrote: > > On Tuesday, July 26, 2016 at 10:21:32 AM UTC-4, [email protected] wrote: > > > On Monday, July 25, 2016 at 5:57:53 PM UTC-4, Sean Beckett wrote: > > > > It is expected behavior that pointsRx will be larger than pointsTx, but > > > > not for more than 1s or 5000 points. pointsRx is a count of the points > > > > received by the UDP listener. Those points are fed to a batcher running > > > > in the background which batches up points every 1s or 5000 points, > > > > whichever comes first. Those batches are then written to the database, > > > > which increments pointsTx. > > > > > > > > > > > > Are you certain none of your points are duplicates? (Same measurement, > > > > tag set, and timestamp) > > > > > > > > > > > > On Fri, Jul 22, 2016 at 3:01 PM, Hiro <[email protected]> wrote: > > > > I have been sending UDP data over to one of my databases, and some > > > > points are not coming through. While investigating where this is > > > > happening (I am aware of the nature of UDP but the volume of data is > > > > quite small), I was looking at the UDP measurement in the _internal > > > > database. I can see that at some times, pointsTx (points transmitted) > > > > is lower than pointsRx (points received), and was curious as to what is > > > > meant by points transmitted.. are these points that made it into the > > > > database? I am just curious about the relationship between the two and > > > > if there is any way to observe that behavior further? Any help would be > > > > greatly appreciated! > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Hiro > > > > > > > > > > > > > > > > -- > > > > > > > > Remember to include the InfluxDB version number with all issue reports > > > > > > > > --- > > > > > > > > You received this message because you are subscribed to the Google > > > > Groups "InfluxDB" group. > > > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [email protected]. > > > > > > > > To post to this group, send email to [email protected]. > > > > > > > > Visit this group at https://groups.google.com/group/influxdb. > > > > > > > > To view this discussion on the web visit > > > > https://groups.google.com/d/msgid/influxdb/8438915b-1c5a-4e10-99a9-75cd716d5807%40googlegroups.com. > > > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Sean Beckett > > > > Director of Support and Professional Services > > > > InfluxDB > > > > > > Thanks for the explanation! Our system seems to far worse than expected, > > > as we are often (basically always) farther ahead with pointsRx than with > > > pointsTx. Here is a graph that shows that relationship a few times, as > > > you can see pointsTx lags behind by a lot... > > > > > > https://i.sli.mg/kL67PV.png > > > > > > And this is how our data looks coming in (it seems to be fine for a while > > > initially, but then points stop arriving consistently and our data is > > > chunky > > > > > > https://i.sli.mg/NCfKRE.png > > > > > > As for duplicates, I am sure that none of the points are the same, they > > > all differ. I generated some text files with all our data points and ran > > > "sort <file> | uniq -d" on our ubuntu server, and all of the files had > > > unique lines (Tested this with 25532 points from 26 seconds of data and > > > 68680 points for 98 seconds of data) > > > > > > Is it possible that I just configured UDP wrong and it can't handle this > > > large of an amount of data? > > > > > > In the influxdb.conf file it is set up as follows: > > > > > > [[udp]] > > > enabled = true > > > bind-address = ":8089" > > > database = "udp" > > > batch-size = 5000 > > > batch-timeout = "2s" > > > read-buffer = 8388608 > > > > So I ended up ramping up the influxdb.conf settings and now we are not > > losing data anymore. The config seems a bit ridiculous but this is what I > > changed it to: > > > > [[udp]] > > enabled = true > > bind-address = ":8089" > > database = "udp" > > batch-size = 100000 > > batch-pending = 25 > > batch-timeout = "10s" > > And additionally: > > read-buffer = 8388608 > udp-payload-size = 65536
Nevermind, lost a few chunks towards the end of our data collection, will keep looking into it -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/5b8b1dd6-820d-4ef0-8cd5-4369bd779dfc%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
