That is not expected behavior. 5000 points per second is a light workload, unless each of those points has 10-100 fields. Even 500k values per second is a sustainable workload on a multi-core machine.
A series cardinality less than 10k is also fairly trivial. That shouldn't require more than a gig or two of RAM. Do you have long strings in your database? Is there something else running on the system that needs RAM? Do you have many many databases or measurements? On Fri, Aug 19, 2016 at 10:45 AM, John Jelinek <[email protected]> wrote: > I have a cardinality of `9876` from this query: `SELECT sum(numSeries) AS > "total_series" FROM "_internal".."database" WHERE time > now() - 10s` and > when I query one of my measurements with something like `SELECT * FROM bars > LIMIT 1` the RAM instantly spikes up to 32GB, maxes out swap, and the > influxdb service restarts. Note, this measurement is getting writes of 5000 > points per second. Total number of points are about 1.6GB. Is this to be > expected? > > > On Wednesday, August 10, 2016 at 8:04:16 AM UTC-5, whille zg wrote: >> >> I'm having OOM issue, post at https://github.com/influxda >> ta/influxdb/issues/7134 >> It seems RAM will drop slowly to small amount if no query continues, but >> i need to read recent data several times continuously. >> I'm try ing v1.0beta on 32G machine, but it's been killed, will try 256G >> RAM. >> Or should v0.12 ok with the RAM problem? >> >> 在 2016年7月13日星期三 UTC+8上午12:19:25，Sean Beckett写道： >>> >>> Currently InfluxDB must load the entire series index into RAM. We're >>> working on a caching mechanism so that only recently written or queries >>> series need to be kept in RAM. It's a complex feature to implement while >>> maintaining performance, but we hope to have a first version in some months. >>> >>> On Tue, Jul 12, 2016 at 3:36 AM, Jan Kis <[email protected]> wrote: >>> >>>> Hi Sean, nice guess, we have 91 786 506 series :) To understand this a >>>> bit better. Does the high memory consumption come from the fact that influx >>>> loads the index into memory for faster writes and querying? >>>> >>>> I will dive into the individual measurements to see where exactly do we >>>> have such a large tag cardinality, so that we can reduce the number of >>>> series. >>>> >>>> Thank you >>>> >>>> On Monday, July 11, 2016 at 6:51:52 PM UTC+2, Sean Beckett wrote: >>>>> >>>>> High RAM usage usually correlates with high series cardinality >>>>> <https://docs.influxdata.com/influxdb/v0.13/concepts/glossary/#series-cardinality> >>>>> . >>>>> >>>>> You can run "SELECT sum(numSeries) AS "total_series" FROM >>>>> "_internal".."database" WHERE time > now() - 10s" to determine your series >>>>> cardinality, assuming you haven't altered the default sample rate for the >>>>> _internal database. If you have, change the WHERE time clause to grab only >>>>> one sample, or use "SELECT last(numSeries) FROM "_internal".."database" >>>>> GROUP BY "database"" and sum the results. >>>>> >>>>> With 100GB of RAM in use, I'm going to guess you have 5+ million >>>>> series. >>>>> >>>>> On Mon, Jul 11, 2016 at 10:21 AM, Jan Kis <[email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> we are using influxdb 0.13 on Fedora 23. We see influx consuming more >>>>>> than 100GB of ram. At some point it eventually runs out of memory and >>>>>> dies. >>>>>> There are no errors in the logs. Our configuration is below. >>>>>> >>>>>> Is there a way to control how much memory influx is consuming? >>>>>> What can we do to figure out why is influx consuming so much memory? >>>>>> >>>>>> Thank you >>>>>> >>>>>> reporting-disabled = false >>>>>> bind-address = ":8088" >>>>>> hostname = "" >>>>>> join = "" >>>>>> >>>>>> [meta] >>>>>> dir = "/data/influxdb/meta" >>>>>> retention-autocreate = true >>>>>> logging-enabled = true >>>>>> pprof-enabled = false >>>>>> lease-duration = "1m0s" >>>>>> >>>>>> [data] >>>>>> dir = "/data/influxdb/data" >>>>>> engine = "tsm1" >>>>>> wal-dir = "/data/influxdb/wal" >>>>>> wal-logging-enabled = true >>>>>> query-log-enabled = true >>>>>> cache-max-memory-size = 524288000 >>>>>> cache-snapshot-memory-size = 26214400 >>>>>> cache-snapshot-write-cold-duration = "1h0m0s" >>>>>> compact-full-write-cold-duration = "24h0m0s" >>>>>> max-points-per-block = 0 >>>>>> data-logging-enabled = true >>>>>> >>>>>> [cluster] >>>>>> force-remote-mapping = false >>>>>> write-timeout = "10s" >>>>>> shard-writer-timeout = "5s" >>>>>> max-remote-write-connections = 3 >>>>>> shard-mapper-timeout = "5s" >>>>>> max-concurrent-queries = 0 >>>>>> query-timeout = "0" >>>>>> log-queries-after = "0" >>>>>> max-select-point = 0 >>>>>> max-select-series = 0 >>>>>> max-select-buckets = 0 >>>>>> >>>>>> [retention] >>>>>> enabled = true >>>>>> check-interval = "30m0s" >>>>>> >>>>>> [shard-precreation] >>>>>> enabled = true >>>>>> check-interval = "10m0s" >>>>>> advance-period = "30m0s" >>>>>> >>>>>> [admin] >>>>>> enabled = true >>>>>> bind-address = ":8083" >>>>>> https-enabled = false >>>>>> https-certificate = "/etc/ssl/influxdb.pem" >>>>>> Version = "" >>>>>> >>>>>> [monitor] >>>>>> store-enabled = true >>>>>> store-database = "_internal" >>>>>> store-interval = "10s" >>>>>> >>>>>> [subscriber] >>>>>> enabled = true >>>>>> >>>>>> [http] >>>>>> enabled = true >>>>>> bind-address = ":8086" >>>>>> auth-enabled = false >>>>>> log-enabled = true >>>>>> write-tracing = false >>>>>> pprof-enabled = false >>>>>> https-enabled = false >>>>>> https-certificate = "/etc/ssl/influxdb.pem" >>>>>> max-row-limit = 10000 >>>>>> >>>>>> [[graphite]] >>>>>> enabled = true >>>>>> bind-address = ":2003" >>>>>> database = "graphite" >>>>>> protocol = "udp" >>>>>> batch-size = 5000 >>>>>> batch-pending = 10 >>>>>> batch-timeout = "1s" >>>>>> consistency-level = "one" >>>>>> separator = "." >>>>>> udp-read-buffer = 0 >>>>>> >>>>>> [[collectd]] >>>>>> enabled = false >>>>>> bind-address = ":25826" >>>>>> database = "collectd" >>>>>> retention-policy = "" >>>>>> batch-size = 5000 >>>>>> batch-pending = 10 >>>>>> batch-timeout = "10s" >>>>>> read-buffer = 0 >>>>>> typesdb = "/usr/share/collectd/types.db" >>>>>> >>>>>> [[opentsdb]] >>>>>> enabled = false >>>>>> bind-address = ":4242" >>>>>> database = "opentsdb" >>>>>> retention-policy = "" >>>>>> consistency-level = "one" >>>>>> tls-enabled = false >>>>>> certificate = "/etc/ssl/influxdb.pem" >>>>>> batch-size = 1000 >>>>>> batch-pending = 5 >>>>>> batch-timeout = "1s" >>>>>> log-point-errors = true >>>>>> >>>>>> [[udp]] >>>>>> enabled = false >>>>>> bind-address = ":8089" >>>>>> database = "udp" >>>>>> retention-policy = "" >>>>>> batch-size = 5000 >>>>>> batch-pending = 10 >>>>>> read-buffer = 0 >>>>>> batch-timeout = "1s" >>>>>> precision = "" >>>>>> >>>>>> [continuous_queries] >>>>>> log-enabled = true >>>>>> enabled = true >>>>>> run-interval = "1s" >>>>>> >>>>>> -- >>>>>> Remember to include the InfluxDB version number with all issue reports >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "InfluxDB" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/influxdb/770d4dc6-8a9b-449 >>>>>> e-ad43-fa558e53a16d%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/influxdb/770d4dc6-8a9b-449e-ad43-fa558e53a16d%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Sean Beckett >>>>> Director of Support and Professional Services >>>>> InfluxDB >>>>> >>>> -- >>>> Remember to include the InfluxDB version number with all issue reports >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "InfluxDB" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/influxdb. >>>> To view this discussion on the web visit https://groups.google.com/d/ms >>>> gid/influxdb/eaa4d5ef-1e81-409b-89e1-867c83ef3939%40googlegroups.com >>>> <https://groups.google.com/d/msgid/influxdb/eaa4d5ef-1e81-409b-89e1-867c83ef3939%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Sean Beckett >>> Director of Support and Professional Services >>> InfluxDB >>> >> -- > Remember to include the InfluxDB version number with all issue reports > --- > You received this message because you are subscribed to the Google Groups > "InfluxDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/influxdb. > To view this discussion on the web visit https://groups.google.com/d/ms > gid/influxdb/194493ab-664a-46e5-9336-9bfd18a82416%40googlegroups.com > <https://groups.google.com/d/msgid/influxdb/194493ab-664a-46e5-9336-9bfd18a82416%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Sean Beckett Director of Support and Professional Services InfluxDB -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CALGqCvM1KBsF1zrqfBnGTs1d70pY%2BLHWXPBMpSqPRLK2Ax7LjQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
