On further consideration, an unbounded query on 1.6 billion points is a lot to sample. Presumably if you put a time boundary on that query it doesn't OOM?
On Fri, Aug 19, 2016 at 11:12 AM, Sean Beckett <[email protected]> wrote: > That is not expected behavior. 5000 points per second is a light workload, > unless each of those points has 10-100 fields. Even 500k values per second > is a sustainable workload on a multi-core machine. > > A series cardinality less than 10k is also fairly trivial. That shouldn't > require more than a gig or two of RAM. > > Do you have long strings in your database? Is there something else running > on the system that needs RAM? > > Do you have many many databases or measurements? > > On Fri, Aug 19, 2016 at 10:45 AM, John Jelinek <[email protected]> > wrote: > >> I have a cardinality of `9876` from this query: `SELECT sum(numSeries) AS >> "total_series" FROM "_internal".."database" WHERE time > now() - 10s` and >> when I query one of my measurements with something like `SELECT * FROM bars >> LIMIT 1` the RAM instantly spikes up to 32GB, maxes out swap, and the >> influxdb service restarts. Note, this measurement is getting writes of 5000 >> points per second. Total number of points are about 1.6GB. Is this to be >> expected? >> >> >> On Wednesday, August 10, 2016 at 8:04:16 AM UTC-5, whille zg wrote: >>> >>> I'm having OOM issue, post at https://github.com/influxda >>> ta/influxdb/issues/7134 >>> It seems RAM will drop slowly to small amount if no query continues, but >>> i need to read recent data several times continuously. >>> I'm try ing v1.0beta on 32G machine, but it's been killed, will try 256G >>> RAM. >>> Or should v0.12 ok with the RAM problem? >>> >>> 在 2016年7月13日星期三 UTC+8上午12:19:25，Sean Beckett写道： >>>> >>>> Currently InfluxDB must load the entire series index into RAM. We're >>>> working on a caching mechanism so that only recently written or queries >>>> series need to be kept in RAM. It's a complex feature to implement while >>>> maintaining performance, but we hope to have a first version in some >>>> months. >>>> >>>> On Tue, Jul 12, 2016 at 3:36 AM, Jan Kis <[email protected]> wrote: >>>> >>>>> Hi Sean, nice guess, we have 91 786 506 series :) To understand this >>>>> a bit better. Does the high memory consumption come from the fact that >>>>> influx loads the index into memory for faster writes and querying? >>>>> >>>>> I will dive into the individual measurements to see where exactly do >>>>> we have such a large tag cardinality, so that we can reduce the number of >>>>> series. >>>>> >>>>> Thank you >>>>> >>>>> On Monday, July 11, 2016 at 6:51:52 PM UTC+2, Sean Beckett wrote: >>>>>> >>>>>> High RAM usage usually correlates with high series cardinality >>>>>> <https://docs.influxdata.com/influxdb/v0.13/concepts/glossary/#series-cardinality> >>>>>> . >>>>>> >>>>>> You can run "SELECT sum(numSeries) AS "total_series" FROM >>>>>> "_internal".."database" WHERE time > now() - 10s" to determine your >>>>>> series >>>>>> cardinality, assuming you haven't altered the default sample rate for the >>>>>> _internal database. If you have, change the WHERE time clause to grab >>>>>> only >>>>>> one sample, or use "SELECT last(numSeries) FROM "_internal".."database" >>>>>> GROUP BY "database"" and sum the results. >>>>>> >>>>>> With 100GB of RAM in use, I'm going to guess you have 5+ million >>>>>> series. >>>>>> >>>>>> On Mon, Jul 11, 2016 at 10:21 AM, Jan Kis <[email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> we are using influxdb 0.13 on Fedora 23. We see influx consuming >>>>>>> more than 100GB of ram. At some point it eventually runs out of memory >>>>>>> and >>>>>>> dies. There are no errors in the logs. Our configuration is below. >>>>>>> >>>>>>> Is there a way to control how much memory influx is consuming? >>>>>>> What can we do to figure out why is influx consuming so much memory? >>>>>>> >>>>>>> Thank you >>>>>>> >>>>>>> reporting-disabled = false >>>>>>> bind-address = ":8088" >>>>>>> hostname = "" >>>>>>> join = "" >>>>>>> >>>>>>> [meta] >>>>>>> dir = "/data/influxdb/meta" >>>>>>> retention-autocreate = true >>>>>>> logging-enabled = true >>>>>>> pprof-enabled = false >>>>>>> lease-duration = "1m0s" >>>>>>> >>>>>>> [data] >>>>>>> dir = "/data/influxdb/data" >>>>>>> engine = "tsm1" >>>>>>> wal-dir = "/data/influxdb/wal" >>>>>>> wal-logging-enabled = true >>>>>>> query-log-enabled = true >>>>>>> cache-max-memory-size = 524288000 >>>>>>> cache-snapshot-memory-size = 26214400 >>>>>>> cache-snapshot-write-cold-duration = "1h0m0s" >>>>>>> compact-full-write-cold-duration = "24h0m0s" >>>>>>> max-points-per-block = 0 >>>>>>> data-logging-enabled = true >>>>>>> >>>>>>> [cluster] >>>>>>> force-remote-mapping = false >>>>>>> write-timeout = "10s" >>>>>>> shard-writer-timeout = "5s" >>>>>>> max-remote-write-connections = 3 >>>>>>> shard-mapper-timeout = "5s" >>>>>>> max-concurrent-queries = 0 >>>>>>> query-timeout = "0" >>>>>>> log-queries-after = "0" >>>>>>> max-select-point = 0 >>>>>>> max-select-series = 0 >>>>>>> max-select-buckets = 0 >>>>>>> >>>>>>> [retention] >>>>>>> enabled = true >>>>>>> check-interval = "30m0s" >>>>>>> >>>>>>> [shard-precreation] >>>>>>> enabled = true >>>>>>> check-interval = "10m0s" >>>>>>> advance-period = "30m0s" >>>>>>> >>>>>>> [admin] >>>>>>> enabled = true >>>>>>> bind-address = ":8083" >>>>>>> https-enabled = false >>>>>>> https-certificate = "/etc/ssl/influxdb.pem" >>>>>>> Version = "" >>>>>>> >>>>>>> [monitor] >>>>>>> store-enabled = true >>>>>>> store-database = "_internal" >>>>>>> store-interval = "10s" >>>>>>> >>>>>>> [subscriber] >>>>>>> enabled = true >>>>>>> >>>>>>> [http] >>>>>>> enabled = true >>>>>>> bind-address = ":8086" >>>>>>> auth-enabled = false >>>>>>> log-enabled = true >>>>>>> write-tracing = false >>>>>>> pprof-enabled = false >>>>>>> https-enabled = false >>>>>>> https-certificate = "/etc/ssl/influxdb.pem" >>>>>>> max-row-limit = 10000 >>>>>>> >>>>>>> [[graphite]] >>>>>>> enabled = true >>>>>>> bind-address = ":2003" >>>>>>> database = "graphite" >>>>>>> protocol = "udp" >>>>>>> batch-size = 5000 >>>>>>> batch-pending = 10 >>>>>>> batch-timeout = "1s" >>>>>>> consistency-level = "one" >>>>>>> separator = "." >>>>>>> udp-read-buffer = 0 >>>>>>> >>>>>>> [[collectd]] >>>>>>> enabled = false >>>>>>> bind-address = ":25826" >>>>>>> database = "collectd" >>>>>>> retention-policy = "" >>>>>>> batch-size = 5000 >>>>>>> batch-pending = 10 >>>>>>> batch-timeout = "10s" >>>>>>> read-buffer = 0 >>>>>>> typesdb = "/usr/share/collectd/types.db" >>>>>>> >>>>>>> [[opentsdb]] >>>>>>> enabled = false >>>>>>> bind-address = ":4242" >>>>>>> database = "opentsdb" >>>>>>> retention-policy = "" >>>>>>> consistency-level = "one" >>>>>>> tls-enabled = false >>>>>>> certificate = "/etc/ssl/influxdb.pem" >>>>>>> batch-size = 1000 >>>>>>> batch-pending = 5 >>>>>>> batch-timeout = "1s" >>>>>>> log-point-errors = true >>>>>>> >>>>>>> [[udp]] >>>>>>> enabled = false >>>>>>> bind-address = ":8089" >>>>>>> database = "udp" >>>>>>> retention-policy = "" >>>>>>> batch-size = 5000 >>>>>>> batch-pending = 10 >>>>>>> read-buffer = 0 >>>>>>> batch-timeout = "1s" >>>>>>> precision = "" >>>>>>> >>>>>>> [continuous_queries] >>>>>>> log-enabled = true >>>>>>> enabled = true >>>>>>> run-interval = "1s" >>>>>>> >>>>>>> -- >>>>>>> Remember to include the InfluxDB version number with all issue >>>>>>> reports >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "InfluxDB" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/influxdb/770d4dc6-8a9b-449 >>>>>>> e-ad43-fa558e53a16d%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/influxdb/770d4dc6-8a9b-449e-ad43-fa558e53a16d%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Sean Beckett >>>>>> Director of Support and Professional Services >>>>>> InfluxDB >>>>>> >>>>> -- >>>>> Remember to include the InfluxDB version number with all issue reports >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "InfluxDB" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/influxdb/eaa4d5ef-1e81-409 >>>>> b-89e1-867c83ef3939%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/influxdb/eaa4d5ef-1e81-409b-89e1-867c83ef3939%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Sean Beckett >>>> Director of Support and Professional Services >>>> InfluxDB >>>> >>> -- >> Remember to include the InfluxDB version number with all issue reports >> --- >> You received this message because you are subscribed to the Google Groups >> "InfluxDB" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/influxdb. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/influxdb/194493ab-664a-46e5-9336-9bfd18a82416%40googlegroups.com >> <https://groups.google.com/d/msgid/influxdb/194493ab-664a-46e5-9336-9bfd18a82416%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Sean Beckett > Director of Support and Professional Services > InfluxDB > -- Sean Beckett Director of Support and Professional Services InfluxDB -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CALGqCvN3w0ctzhPGOSjjdq9byaQA24Bao-WAj0NfHi4ZEzzXPA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
