Cardinality has jumped to 14041 after all the points were uploaded, but `SELECT * FROM bars LIMIT 1` still maxes out the RAM and reboots the influx service.
On Friday, August 19, 2016 at 1:41:45 PM UTC-5, John Jelinek wrote: > > Also, I don't have any other processes running on the latest environment > (ubuntu w/ 32GB of RAM), I built that box just for this test. > > On Friday, August 19, 2016 at 1:38:59 PM UTC-5, John Jelinek wrote: >> >> I did the query while uploading 14288591 points (at 5000 points/second). >> Here's a CSV sample of the kind of data: >> >> ``` >> "Symbol","Date","Open","High","Low","Close","Volume","Ex-Dividend","Split >> Ratio","Adj. Open","Adj. High","Adj. Low","Adj. Close","Adj. Volume" >> >> A,1999-11-18,45.5,50.0,40.0,44.0,44739900.0,0.0,1.0,43.471809559155,47.771219295775,38.21697543662,42.038672980282,44739900.0 >> >> A,1999-11-19,42.94,43.0,39.81,40.38,10897100.0,0.0,1.0,41.025923131212,41.083248594367,38.035444803296,38.580036703268,10897100.0 >> >> A,1999-11-22,41.31,44.0,40.06,44.0,4705200.0,0.0,1.0,39.468581382169,42.038672980282,38.274300899775,42.038672980282,4705200.0 >> >> A,1999-11-23,42.5,43.63,40.25,40.25,4274400.0,0.0,1.0,40.605536401409,41.685165957493,38.455831533099,38.455831533099,4274400.0 >> >> A,1999-11-24,40.13,41.94,40.0,41.06,3464400.0,0.0,1.0,38.341180606789,40.070498745296,38.21697543662,39.22972528569,3464400.0 >> >> A,1999-11-26,40.88,41.5,40.75,41.19,1237100.0,0.0,1.0,39.057748896226,39.650112015493,38.933543726057,39.353930455859,1237100.0 >> >> A,1999-11-29,41.0,42.44,40.56,42.13,2914700.0,0.0,1.0,39.172399822536,40.548210938254,38.752013092733,40.25202937862,2914700.0 >> >> A,1999-11-30,42.0,42.94,40.94,42.19,3083000.0,0.0,1.0,40.127824208451,41.025923131212,39.115074359381,40.309354841775,3083000.0 >> >> A,1999-12-01,42.19,43.44,41.88,42.94,2115400.0,0.0,1.0,40.309354841775,41.503635324169,40.013173282141,41.025923131212,2115400.0 >> ``` >> >> I only have the one database created for this measurement and just this >> measurement. I've tested this in 3 different environments, docker on 8GB >> RAM, running influx directly on a macbook pro w/ 16GB RAM, and running >> influx directly on an ubuntu 16.04 server with 32GB of RAM. On all >> environments, the RAM has maxed out and swap is then maxed out. I'm using >> this process to upload the CSV into influx 0.13 >> https://github.com/jpillora/csv-to-influxdb. This is the dataset I'm >> uploading into influx: https://www.quandl.com/data/WIKI. This is the >> command I'm using to get it into influx: `csv-to-influxdb -m bars -t Symbol >> -ts Date -tf 2006-01-02 -d eodbars WIKI_20160818.csv`. Let me know if you >> need to know any other details. >> >> On Friday, August 19, 2016 at 12:13:59 PM UTC-5, Sean Beckett wrote: >>> >>> On further consideration, an unbounded query on 1.6 billion points is a >>> lot to sample. Presumably if you put a time boundary on that query it >>> doesn't OOM? >>> >>> On Fri, Aug 19, 2016 at 11:12 AM, Sean Beckett <[email protected]> >>> wrote: >>> >>>> That is not expected behavior. 5000 points per second is a light >>>> workload, unless each of those points has 10-100 fields. Even 500k values >>>> per second is a sustainable workload on a multi-core machine. >>>> >>>> A series cardinality less than 10k is also fairly trivial. That >>>> shouldn't require more than a gig or two of RAM. >>>> >>>> Do you have long strings in your database? Is there something else >>>> running on the system that needs RAM? >>>> >>>> Do you have many many databases or measurements? >>>> >>>> On Fri, Aug 19, 2016 at 10:45 AM, John Jelinek <[email protected]> >>>> wrote: >>>> >>>>> I have a cardinality of `9876` from this query: `SELECT sum(numSeries) >>>>> AS "total_series" FROM "_internal".."database" WHERE time > now() - 10s` >>>>> and when I query one of my measurements with something like `SELECT * >>>>> FROM >>>>> bars LIMIT 1` the RAM instantly spikes up to 32GB, maxes out swap, and >>>>> the >>>>> influxdb service restarts. Note, this measurement is getting writes of >>>>> 5000 >>>>> points per second. Total number of points are about 1.6GB. Is this to be >>>>> expected? >>>>> >>>>> >>>>> On Wednesday, August 10, 2016 at 8:04:16 AM UTC-5, whille zg wrote: >>>>>> >>>>>> I'm having OOM issue, post at >>>>>> https://github.com/influxdata/influxdb/issues/7134 >>>>>> It seems RAM will drop slowly to small amount if no query continues, >>>>>> but i need to read recent data several times continuously. >>>>>> I'm try ing v1.0beta on 32G machine, but it's been killed, will try >>>>>> 256G RAM. >>>>>> Or should v0.12 ok with the RAM problem? >>>>>> >>>>>> 在 2016年7月13日星期三 UTC+8上午12:19:25，Sean Beckett写道： >>>>>>> >>>>>>> Currently InfluxDB must load the entire series index into RAM. We're >>>>>>> working on a caching mechanism so that only recently written or queries >>>>>>> series need to be kept in RAM. It's a complex feature to implement >>>>>>> while >>>>>>> maintaining performance, but we hope to have a first version in some >>>>>>> months. >>>>>>> >>>>>>> On Tue, Jul 12, 2016 at 3:36 AM, Jan Kis <[email protected]> wrote: >>>>>>> >>>>>>>> Hi Sean, nice guess, we have 91 786 506 series :) To understand >>>>>>>> this a bit better. Does the high memory consumption come from the fact >>>>>>>> that >>>>>>>> influx loads the index into memory for faster writes and querying? >>>>>>>> >>>>>>>> >>>>>>>> I will dive into the individual measurements to see where exactly >>>>>>>> do we have such a large tag cardinality, so that we can reduce the >>>>>>>> number >>>>>>>> of series. >>>>>>>> >>>>>>>> Thank you >>>>>>>> >>>>>>>> On Monday, July 11, 2016 at 6:51:52 PM UTC+2, Sean Beckett wrote: >>>>>>>>> >>>>>>>>> High RAM usage usually correlates with high series cardinality >>>>>>>>> <https://docs.influxdata.com/influxdb/v0.13/concepts/glossary/#series-cardinality> >>>>>>>>> . >>>>>>>>> >>>>>>>>> You can run "SELECT sum(numSeries) AS "total_series" FROM >>>>>>>>> "_internal".."database" WHERE time > now() - 10s" to determine your >>>>>>>>> series >>>>>>>>> cardinality, assuming you haven't altered the default sample rate for >>>>>>>>> the >>>>>>>>> _internal database. If you have, change the WHERE time clause to grab >>>>>>>>> only >>>>>>>>> one sample, or use "SELECT last(numSeries) FROM >>>>>>>>> "_internal".."database" >>>>>>>>> GROUP BY "database"" and sum the results. >>>>>>>>> >>>>>>>>> With 100GB of RAM in use, I'm going to guess you have 5+ million >>>>>>>>> series. >>>>>>>>> >>>>>>>>> On Mon, Jul 11, 2016 at 10:21 AM, Jan Kis <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> we are using influxdb 0.13 on Fedora 23. We see influx consuming >>>>>>>>>> more than 100GB of ram. At some point it eventually runs out of >>>>>>>>>> memory and >>>>>>>>>> dies. There are no errors in the logs. Our configuration is below. >>>>>>>>>> >>>>>>>>>> Is there a way to control how much memory influx is consuming? >>>>>>>>>> What can we do to figure out why is influx consuming so much >>>>>>>>>> memory? >>>>>>>>>> >>>>>>>>>> Thank you >>>>>>>>>> >>>>>>>>>> reporting-disabled = false >>>>>>>>>> bind-address = ":8088" >>>>>>>>>> hostname = "" >>>>>>>>>> join = "" >>>>>>>>>> >>>>>>>>>> [meta] >>>>>>>>>> dir = "/data/influxdb/meta" >>>>>>>>>> retention-autocreate = true >>>>>>>>>> logging-enabled = true >>>>>>>>>> pprof-enabled = false >>>>>>>>>> lease-duration = "1m0s" >>>>>>>>>> >>>>>>>>>> [data] >>>>>>>>>> dir = "/data/influxdb/data" >>>>>>>>>> engine = "tsm1" >>>>>>>>>> wal-dir = "/data/influxdb/wal" >>>>>>>>>> wal-logging-enabled = true >>>>>>>>>> query-log-enabled = true >>>>>>>>>> cache-max-memory-size = 524288000 >>>>>>>>>> cache-snapshot-memory-size = 26214400 >>>>>>>>>> cache-snapshot-write-cold-duration = "1h0m0s" >>>>>>>>>> compact-full-write-cold-duration = "24h0m0s" >>>>>>>>>> max-points-per-block = 0 >>>>>>>>>> data-logging-enabled = true >>>>>>>>>> >>>>>>>>>> [cluster] >>>>>>>>>> force-remote-mapping = false >>>>>>>>>> write-timeout = "10s" >>>>>>>>>> shard-writer-timeout = "5s" >>>>>>>>>> max-remote-write-connections = 3 >>>>>>>>>> shard-mapper-timeout = "5s" >>>>>>>>>> max-concurrent-queries = 0 >>>>>>>>>> query-timeout = "0" >>>>>>>>>> log-queries-after = "0" >>>>>>>>>> max-select-point = 0 >>>>>>>>>> max-select-series = 0 >>>>>>>>>> max-select-buckets = 0 >>>>>>>>>> >>>>>>>>>> [retention] >>>>>>>>>> enabled = true >>>>>>>>>> check-interval = "30m0s" >>>>>>>>>> >>>>>>>>>> [shard-precreation] >>>>>>>>>> enabled = true >>>>>>>>>> check-interval = "10m0s" >>>>>>>>>> advance-period = "30m0s" >>>>>>>>>> >>>>>>>>>> [admin] >>>>>>>>>> enabled = true >>>>>>>>>> bind-address = ":8083" >>>>>>>>>> https-enabled = false >>>>>>>>>> https-certificate = "/etc/ssl/influxdb.pem" >>>>>>>>>> Version = "" >>>>>>>>>> >>>>>>>>>> [monitor] >>>>>>>>>> store-enabled = true >>>>>>>>>> store-database = "_internal" >>>>>>>>>> store-interval = "10s" >>>>>>>>>> >>>>>>>>>> [subscriber] >>>>>>>>>> enabled = true >>>>>>>>>> >>>>>>>>>> [http] >>>>>>>>>> enabled = true >>>>>>>>>> bind-address = ":8086" >>>>>>>>>> auth-enabled = false >>>>>>>>>> log-enabled = true >>>>>>>>>> write-tracing = false >>>>>>>>>> pprof-enabled = false >>>>>>>>>> https-enabled = false >>>>>>>>>> https-certificate = "/etc/ssl/influxdb.pem" >>>>>>>>>> max-row-limit = 10000 >>>>>>>>>> >>>>>>>>>> [[graphite]] >>>>>>>>>> enabled = true >>>>>>>>>> bind-address = ":2003" >>>>>>>>>> database = "graphite" >>>>>>>>>> protocol = "udp" >>>>>>>>>> batch-size = 5000 >>>>>>>>>> batch-pending = 10 >>>>>>>>>> batch-timeout = "1s" >>>>>>>>>> consistency-level = "one" >>>>>>>>>> separator = "." >>>>>>>>>> udp-read-buffer = 0 >>>>>>>>>> >>>>>>>>>> [[collectd]] >>>>>>>>>> enabled = false >>>>>>>>>> bind-address = ":25826" >>>>>>>>>> database = "collectd" >>>>>>>>>> retention-policy = "" >>>>>>>>>> batch-size = 5000 >>>>>>>>>> batch-pending = 10 >>>>>>>>>> batch-timeout = "10s" >>>>>>>>>> read-buffer = 0 >>>>>>>>>> typesdb = "/usr/share/collectd/types.db" >>>>>>>>>> >>>>>>>>>> [[opentsdb]] >>>>>>>>>> enabled = false >>>>>>>>>> bind-address = ":4242" >>>>>>>>>> database = "opentsdb" >>>>>>>>>> retention-policy = "" >>>>>>>>>> consistency-level = "one" >>>>>>>>>> tls-enabled = false >>>>>>>>>> certificate = "/etc/ssl/influxdb.pem" >>>>>>>>>> batch-size = 1000 >>>>>>>>>> batch-pending = 5 >>>>>>>>>> batch-timeout = "1s" >>>>>>>>>> log-point-errors = true >>>>>>>>>> >>>>>>>>>> [[udp]] >>>>>>>>>> enabled = false >>>>>>>>>> bind-address = ":8089" >>>>>>>>>> database = "udp" >>>>>>>>>> retention-policy = "" >>>>>>>>>> batch-size = 5000 >>>>>>>>>> batch-pending = 10 >>>>>>>>>> read-buffer = 0 >>>>>>>>>> batch-timeout = "1s" >>>>>>>>>> precision = "" >>>>>>>>>> >>>>>>>>>> [continuous_queries] >>>>>>>>>> log-enabled = true >>>>>>>>>> enabled = true >>>>>>>>>> run-interval = "1s" >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Remember to include the InfluxDB version number with all issue >>>>>>>>>> reports >>>>>>>>>> --- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "InfluxDB" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/influxdb/770d4dc6-8a9b-449e-ad43-fa558e53a16d%40googlegroups.com >>>>>>>>>> >>>>>>>>>> <https://groups.google.com/d/msgid/influxdb/770d4dc6-8a9b-449e-ad43-fa558e53a16d%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Sean Beckett >>>>>>>>> Director of Support and Professional Services >>>>>>>>> InfluxDB >>>>>>>>> >>>>>>>> -- >>>>>>>> Remember to include the InfluxDB version number with all issue >>>>>>>> reports >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "InfluxDB" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To post to this group, send email to [email protected]. >>>>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/influxdb/eaa4d5ef-1e81-409b-89e1-867c83ef3939%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/influxdb/eaa4d5ef-1e81-409b-89e1-867c83ef3939%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Sean Beckett >>>>>>> Director of Support and Professional Services >>>>>>> InfluxDB >>>>>>> >>>>>> -- >>>>> Remember to include the InfluxDB version number with all issue reports >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "InfluxDB" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/influxdb/194493ab-664a-46e5-9336-9bfd18a82416%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/influxdb/194493ab-664a-46e5-9336-9bfd18a82416%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Sean Beckett >>>> Director of Support and Professional Services >>>> InfluxDB >>>> >>> >>> >>> >>> -- >>> Sean Beckett >>> Director of Support and Professional Services >>> InfluxDB >>> >> -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/f968a4b6-0849-4b22-bf93-fa032acd72e2%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
