Embarrassingly enough, we found that pointsTx was double-counting each point. There's a fix already merged: https://github.com/influxdata/influxdb/pull/7088
The lead developer also mentioned: "Adjusting the OS max buffer size may help the root problem of some points going missing. The Linux default of 13kb is so small it might not be able to handle 10,000 msgs/s" On Wed, Jul 27, 2016 at 1:50 PM, <[email protected]> wrote: > On Tuesday, July 26, 2016 at 4:06:44 PM UTC-4, Sean Beckett wrote: > > What is your write volume? From your graphs it seems to be 15-20k points > per second. > > > > > > What does the CPU and RAM use on the machine look like as it ramps up > and the points begin to fail? > > > > > > On Tue, Jul 26, 2016 at 11:06 AM, <[email protected]> wrote: > > > > > > On Tuesday, July 26, 2016 at 1:01:35 PM UTC-4, [email protected] wrote: > > > > > On Tuesday, July 26, 2016 at 12:59:51 PM UTC-4, [email protected] > wrote: > > > > > > On Tuesday, July 26, 2016 at 10:21:32 AM UTC-4, [email protected] > wrote: > > > > > > > On Monday, July 25, 2016 at 5:57:53 PM UTC-4, Sean Beckett wrote: > > > > > > > > It is expected behavior that pointsRx will be larger than > pointsTx, but not for more than 1s or 5000 points. pointsRx is a count of > the points received by the UDP listener. Those points are fed to a batcher > running in the background which batches up points every 1s or 5000 points, > whichever comes first. Those batches are then written to the database, > which increments pointsTx. > > > > > > > > > > > > > > > > > > > > > > > > Are you certain none of your points are duplicates? (Same > measurement, tag set, and timestamp) > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 22, 2016 at 3:01 PM, Hiro <[email protected]> wrote: > > > > > > > > I have been sending UDP data over to one of my databases, and > some points are not coming through. While investigating where this is > happening (I am aware of the nature of UDP but the volume of data is quite > small), I was looking at the UDP measurement in the _internal database. I > can see that at some times, pointsTx (points transmitted) is lower than > pointsRx (points received), and was curious as to what is meant by points > transmitted.. are these points that made it into the database? I am just > curious about the relationship between the two and if there is any way to > observe that behavior further? Any help would be greatly appreciated! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Hiro > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Remember to include the InfluxDB version number with all issue > reports > > > > > > > > > > > > > > > > --- > > > > > > > > > > > > > > > > You received this message because you are subscribed to the > Google Groups "InfluxDB" group. > > > > > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from > it, send an email to [email protected]. > > > > > > > > > > > > > > > > To post to this group, send email to [email protected]. > > > > > > > > > > > > > > > > Visit this group at https://groups.google.com/group/influxdb. > > > > > > > > > > > > > > > > To view this discussion on the web visit > https://groups.google.com/d/msgid/influxdb/8438915b-1c5a-4e10-99a9-75cd716d5807%40googlegroups.com > . > > > > > > > > > > > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > Sean Beckett > > > > > > > > Director of Support and Professional Services > > > > > > > > InfluxDB > > > > > > > > > > > > > > Thanks for the explanation! Our system seems to far worse than > expected, as we are often (basically always) farther ahead with pointsRx > than with pointsTx. Here is a graph that shows that relationship a few > times, as you can see pointsTx lags behind by a lot... > > > > > > > > > > > > > > https://i.sli.mg/kL67PV.png > > > > > > > > > > > > > > And this is how our data looks coming in (it seems to be fine for > a while initially, but then points stop arriving consistently and our data > is chunky > > > > > > > > > > > > > > https://i.sli.mg/NCfKRE.png > > > > > > > > > > > > > > As for duplicates, I am sure that none of the points are the same, > they all differ. I generated some text files with all our data points and > ran "sort <file> | uniq -d" on our ubuntu server, and all of the files had > unique lines (Tested this with 25532 points from 26 seconds of data and > 68680 points for 98 seconds of data) > > > > > > > > > > > > > > Is it possible that I just configured UDP wrong and it can't > handle this large of an amount of data? > > > > > > > > > > > > > > In the influxdb.conf file it is set up as follows: > > > > > > > > > > > > > > [[udp]] > > > > > > > enabled = true > > > > > > > bind-address = ":8089" > > > > > > > database = "udp" > > > > > > > batch-size = 5000 > > > > > > > batch-timeout = "2s" > > > > > > > read-buffer = 8388608 > > > > > > > > > > > > So I ended up ramping up the influxdb.conf settings and now we are > not losing data anymore. The config seems a bit ridiculous but this is what > I changed it to: > > > > > > > > > > > > [[udp]] > > > > > > enabled = true > > > > > > bind-address = ":8089" > > > > > > database = "udp" > > > > > > batch-size = 100000 > > > > > > batch-pending = 25 > > > > > > batch-timeout = "10s" > > > > > > > > > > And additionally: > > > > > > > > > > read-buffer = 8388608 > > > > > udp-payload-size = 65536 > > > > > > > > Nevermind, lost a few chunks towards the end of our data collection, > will keep looking into it > > > > > > > > -- > > > > Remember to include the InfluxDB version number with all issue reports > > > > --- > > > > You received this message because you are subscribed to the Google > Groups "InfluxDB" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > > > > To post to this group, send email to [email protected]. > > > > Visit this group at https://groups.google.com/group/influxdb. > > > > To view this discussion on the web visit > https://groups.google.com/d/msgid/influxdb/5b8b1dd6-820d-4ef0-8cd5-4369bd779dfc%40googlegroups.com > . > > > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > -- > > > > > > Sean Beckett > > Director of Support and Professional Services > > InfluxDB > > Right now all our tags are the same, but for each measurement there is > only one instance per timestamp. None of the points should end up being the > same! > > We are writing around 10k points per second from one machine, but ideally > are hoping to scale this up to multiple machines with 20-25k from each of > them.. > > I am in the process of changing up some of our code, so I will take a look > at the CPU/RAM usage tomorrow if things still haven't improved! > > -- > Remember to include the InfluxDB version number with all issue reports > --- > You received this message because you are subscribed to the Google Groups > "InfluxDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/influxdb. > To view this discussion on the web visit > https://groups.google.com/d/msgid/influxdb/d248f00c-a321-4a36-8faf-c13b0b547262%40googlegroups.com > . > For more options, visit https://groups.google.com/d/optout. > -- Sean Beckett Director of Support and Professional Services InfluxDB -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CALGqCvOUGJh7P6w4HTzyqGP9zV3PkiWK8j5s%2Bpj5fX8c%3D3d3rg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
