On Tuesday, July 26, 2016 at 4:06:44 PM UTC-4, Sean Beckett wrote: > What is your write volume? From your graphs it seems to be 15-20k points per > second. > > > What does the CPU and RAM use on the machine look like as it ramps up and the > points begin to fail? > > > On Tue, Jul 26, 2016 at 11:06 AM, <[email protected]> wrote: > > > On Tuesday, July 26, 2016 at 1:01:35 PM UTC-4, [email protected] wrote: > > > On Tuesday, July 26, 2016 at 12:59:51 PM UTC-4, [email protected] wrote: > > > > On Tuesday, July 26, 2016 at 10:21:32 AM UTC-4, [email protected] wrote: > > > > > On Monday, July 25, 2016 at 5:57:53 PM UTC-4, Sean Beckett wrote: > > > > > > It is expected behavior that pointsRx will be larger than pointsTx, > > > > > but not for more than 1s or 5000 points. pointsRx is a count of the > > > > > points received by the UDP listener. Those points are fed to a > > > > > batcher running in the background which batches up points every 1s or > > > > > 5000 points, whichever comes first. Those batches are then written to > > > > > the database, which increments pointsTx. > > > > > > > > > > > > > > > > > > Are you certain none of your points are duplicates? (Same > > > > > measurement, tag set, and timestamp) > > > > > > > > > > > > > > > > > > On Fri, Jul 22, 2016 at 3:01 PM, Hiro <[email protected]> wrote: > > > > > > I have been sending UDP data over to one of my databases, and some > > > > > points are not coming through. While investigating where this is > > > > > happening (I am aware of the nature of UDP but the volume of data is > > > > > quite small), I was looking at the UDP measurement in the _internal > > > > > database. I can see that at some times, pointsTx (points transmitted) > > > > > is lower than pointsRx (points received), and was curious as to what > > > > > is meant by points transmitted.. are these points that made it into > > > > > the database? I am just curious about the relationship between the > > > > > two and if there is any way to observe that behavior further? Any > > > > > help would be greatly appreciated! > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Hiro > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Remember to include the InfluxDB version number with all issue reports > > > > > > > > > > > > --- > > > > > > > > > > > > You received this message because you are subscribed to the Google > > > > > Groups "InfluxDB" group. > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > send an email to [email protected]. > > > > > > > > > > > > To post to this group, send email to [email protected]. > > > > > > > > > > > > Visit this group at https://groups.google.com/group/influxdb. > > > > > > > > > > > > To view this discussion on the web visit > > > > > https://groups.google.com/d/msgid/influxdb/8438915b-1c5a-4e10-99a9-75cd716d5807%40googlegroups.com. > > > > > > > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > Sean Beckett > > > > > > Director of Support and Professional Services > > > > > > InfluxDB > > > > > > > > > > Thanks for the explanation! Our system seems to far worse than > > > > expected, as we are often (basically always) farther ahead with > > > > pointsRx than with pointsTx. Here is a graph that shows that > > > > relationship a few times, as you can see pointsTx lags behind by a > > > > lot... > > > > > > > > > > https://i.sli.mg/kL67PV.png > > > > > > > > > > And this is how our data looks coming in (it seems to be fine for a > > > > while initially, but then points stop arriving consistently and our > > > > data is chunky > > > > > > > > > > https://i.sli.mg/NCfKRE.png > > > > > > > > > > As for duplicates, I am sure that none of the points are the same, they > > > > all differ. I generated some text files with all our data points and > > > > ran "sort <file> | uniq -d" on our ubuntu server, and all of the files > > > > had unique lines (Tested this with 25532 points from 26 seconds of data > > > > and 68680 points for 98 seconds of data) > > > > > > > > > > Is it possible that I just configured UDP wrong and it can't handle > > > > this large of an amount of data? > > > > > > > > > > In the influxdb.conf file it is set up as follows: > > > > > > > > > > [[udp]] > > > > > enabled = true > > > > > bind-address = ":8089" > > > > > database = "udp" > > > > > batch-size = 5000 > > > > > batch-timeout = "2s" > > > > > read-buffer = 8388608 > > > > > > > > So I ended up ramping up the influxdb.conf settings and now we are not > > > losing data anymore. The config seems a bit ridiculous but this is what I > > > changed it to: > > > > > > > > [[udp]] > > > > enabled = true > > > > bind-address = ":8089" > > > > database = "udp" > > > > batch-size = 100000 > > > > batch-pending = 25 > > > > batch-timeout = "10s" > > > > > > And additionally: > > > > > > read-buffer = 8388608 > > > udp-payload-size = 65536 > > > > Nevermind, lost a few chunks towards the end of our data collection, will > keep looking into it > > > > -- > > Remember to include the InfluxDB version number with all issue reports > > --- > > You received this message because you are subscribed to the Google Groups > "InfluxDB" group. > > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > To post to this group, send email to [email protected]. > > Visit this group at https://groups.google.com/group/influxdb. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/influxdb/5b8b1dd6-820d-4ef0-8cd5-4369bd779dfc%40googlegroups.com. > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > > Sean Beckett > Director of Support and Professional Services > InfluxDB
Right now all our tags are the same, but for each measurement there is only one instance per timestamp. None of the points should end up being the same! We are writing around 10k points per second from one machine, but ideally are hoping to scale this up to multiple machines with 20-25k from each of them.. I am in the process of changing up some of our code, so I will take a look at the CPU/RAM usage tomorrow if things still haven't improved! -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/d248f00c-a321-4a36-8faf-c13b0b547262%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
