Hi Sean Thanks for the prompt reply No I am writing points every 15 minutes and are not a wide time range. There should be no overwrites.
Steve On Tuesday, November 8, 2016 at 9:26:35 PM UTC+2, Sean Beckett wrote: > > If you are writing points across a wide time range, change the default > shard duration: > http://docs.influxdata.com/influxdb/v1.0/concepts/schema_and_data_layout/#shard-group-duration-recommendations > > Are you overwriting points? > > On Tue, Nov 8, 2016 at 12:24 PM, <[email protected] <javascript:>> > wrote: > >> On Tuesday, November 8, 2016 at 9:18:08 PM UTC+2, [email protected] >> wrote: >> > Hi All >> > >> > I am inserting a low volume of data (less than 2000 points every 15 >> minutes). I do this via a Java program (using a batch). What I have noticed >> is inserting the same amount of data takes longer and longer and over time >> and the CPU usage constantly increases (almost linearly) over 24 hours (It >> is not swapping as far as I can see). >> > >> > If I restart InfluxDB the CPU usage goes back to normal for a while but >> it slowly and steadily climbs with each insert. The only thing that reduces >> the CPU load for writes (roughly the same amount of data every time) is if >> I restart InfluxDB (I have tried stopping and starting the Java program and >> it has no effect (I thought it might be holding onto ports etc). >> > >> > I am using a very small server (1G / 1 CPU) to test it (there is very >> little data) but I was wondering if I am configuring InfluxDB properly. >> > >> > I am using version 1.0.2 of InfluxDB on Ubuntu Linux (on DigitalOcean) >> and the "recommended" Java library. >> > >> > Here is my conf file >> > >> > reporting-disabled = false >> > bind-address = ":8088" >> > >> > [meta] >> > dir = "/var/lib/influxdb/meta" >> > retention-autocreate = true >> > logging-enabled = true >> > >> > [data] >> > dir = "/var/lib/influxdb/data" >> > engine = "tsm1" >> > wal-dir = "/var/lib/influxdb/wal" >> > wal-logging-enabled = true >> > query-log-enabled = false >> > cache-max-memory-size = 524288000 >> > cache-snapshot-memory-size = 26214400 >> > cache-snapshot-write-cold-duration = "1h0m0s" >> > compact-full-write-cold-duration = "24h0m0s" >> > max-points-per-block = 0 >> > max-series-per-database = 1000000 >> > trace-logging-enabled = false >> > >> > [coordinator] >> > write-timeout = "10s" >> > max-concurrent-queries = 0 >> > query-timeout = "0" >> > log-queries-after = "0" >> > max-select-point = 0 >> > max-select-series = 0 >> > max-select-buckets = 0 >> > >> > [retention] >> > enabled = true >> > check-interval = "30m0s" >> > >> > [shard-precreation] >> > enabled = true >> > check-interval = "10m0s" >> > advance-period = "30m0s" >> > >> > [admin] >> > enabled = true >> > bind-address = ":8083" >> > https-enabled = false >> > https-certificate = "/etc/ssl/influxdb.pem" >> > >> > [monitor] >> > store-enabled = true >> > store-database = "_internal" >> > store-interval = "10s" >> > >> > [subscriber] >> > enabled = true >> > http-timeout = "30s" >> > write-concurrency = 40 >> > write-buffer-size = 1000 >> > >> > [http] >> > enabled = true >> > bind-address = ":8086" >> > auth-enabled = false >> > log-enabled = false >> > write-tracing = false >> > https-enabled = false >> > https-certificate = "/etc/ssl/influxdb.pem" >> > https-private-key = "" >> > max-row-limit = 10000 >> > max-connection-limit = 0 >> > shared-secret = "" >> > realm = "InfluxDB" >> > >> > [continuous_queries] >> > log-enabled = true >> > enabled = true >> > run-interval = "1s" >> > >> > Any pointers will be appreciated. >> > >> > Regards >> > Steve van der Merwe >> >> My apologies here is a log snippet: >> >> Nov 8 19:06:35 db influxd[17415]: [tsm1] 2016/11/08 19:06:35 compacted >> level 2 group 0 of 2 files into 1 files in 452.912177ms >> Nov 8 19:06:36 db influxd[17415]: [tsm1] 2016/11/08 19:06:36 beginning >> level 3 compaction of group 0, 4 TSM files >> Nov 8 19:06:36 db influxd[17415]: [tsm1] 2016/11/08 19:06:36 compacting >> level 3 group (0) >> /var/lib/influxdb/data/db/autogen/50/000000551-000000003.tsm (#0) >> Nov 8 19:06:36 db influxd[17415]: [tsm1] 2016/11/08 19:06:36 compacting >> level 3 group (0) >> /var/lib/influxdb/data/db/autogen/50/000000555-000000003.tsm (#1) >> Nov 8 19:06:36 db influxd[17415]: [tsm1] 2016/11/08 19:06:36 compacting >> level 3 group (0) >> /var/lib/influxdb/data/db/autogen/50/000000559-000000003.tsm (#2) >> Nov 8 19:06:36 db influxd[17415]: [tsm1] 2016/11/08 19:06:36 compacting >> level 3 group (0) >> /var/lib/influxdb/data/db/autogen/50/000000563-000000003.tsm (#3) >> Nov 8 19:06:37 db influxd[17415]: [tsm1] 2016/11/08 19:06:37 compacted >> level 3 group (0) into >> /var/lib/influxdb/data/db/autogen/50/000000563-000000004.tsm.tmp (#0) >> Nov 8 19:06:37 db influxd[17415]: [tsm1] 2016/11/08 19:06:37 compacted >> level 3 group 0 of 4 files into 1 files in 798.550193ms >> Nov 8 19:07:05 db influxd[17415]: [tsm1] 2016/11/08 19:07:05 Snapshot >> for path /var/lib/influxdb/data/db/autogen/50 written in 2.163763999s >> Nov 8 19:07:37 db influxd[17415]: [tsm1] 2016/11/08 19:07:37 Snapshot >> for path /var/lib/influxdb/data/db/autogen/50 written in 2.6193556s >> Nov 8 19:07:37 db influxd[17415]: [tsm1] 2016/11/08 19:07:37 beginning >> level 1 compaction of group 0, 2 TSM files >> Nov 8 19:07:37 db influxd[17415]: [tsm1] 2016/11/08 19:07:37 compacting >> level 1 group (0) >> /var/lib/influxdb/data/db/autogen/50/000000564-000000001.tsm (#0) >> Nov 8 19:07:37 db influxd[17415]: [tsm1] 2016/11/08 19:07:37 compacting >> level 1 group (0) >> /var/lib/influxdb/data/db/autogen/50/000000565-000000001.tsm (#1) >> Nov 8 19:07:38 db influxd[17415]: [tsm1] 2016/11/08 19:07:38 compacted >> level 1 group (0) into >> /var/lib/influxdb/data/db/autogen/50/000000565-000000002.tsm.tmp (#0) >> Nov 8 19:07:38 db influxd[17415]: [tsm1] 2016/11/08 19:07:38 compacted >> level 1 group 0 of 2 files into 1 files in 490.727946ms >> Nov 8 19:12:16 db influxd[17415]: [tsm1] 2016/11/08 19:12:16 Snapshot >> for path /var/lib/influxdb/data/db/autogen/50 written in 3.521008625s >> Nov 8 19:12:51 db influxd[17415]: [tsm1] 2016/11/08 19:12:51 Snapshot >> for path /var/lib/influxdb/data/db/autogen/50 written in 2.915170587s >> Nov 8 19:12:51 db influxd[17415]: [tsm1] 2016/11/08 19:12:51 beginning >> level 1 compaction of group 0, 2 TSM files >> Nov 8 19:12:51 db influxd[17415]: [tsm1] 2016/11/08 19:12:51 compacting >> level 1 group (0) >> /var/lib/influxdb/data/db/autogen/50/000000566-000000001.tsm (#0) >> Nov 8 19:12:51 db influxd[17415]: [tsm1] 2016/11/08 19:12:51 compacting >> level 1 group (0) >> /var/lib/influxdb/data/db/autogen/50/000000567-000000001.tsm (#1) >> Nov 8 19:12:52 db influxd[17415]: [tsm1] 2016/11/08 19:12:52 compacted >> level 1 group (0) into >> /var/lib/influxdb/data/db/autogen/50/000000567-000000002.tsm.tmp (#0) >> Nov 8 19:12:52 db influxd[17415]: [tsm1] 2016/11/08 19:12:52 compacted >> level 1 group 0 of 2 files into 1 files in 421.739431ms >> Nov 8 19:12:52 db influxd[17415]: [tsm1] 2016/11/08 19:12:52 beginning >> level 2 compaction of group 0, 2 TSM files >> Nov 8 19:12:52 db influxd[17415]: [tsm1] 2016/11/08 19:12:52 compacting >> level 2 group (0) >> /var/lib/influxdb/data/db/autogen/50/000000565-000000002.tsm (#0) >> Nov 8 19:12:52 db influxd[17415]: [tsm1] 2016/11/08 19:12:52 compacting >> level 2 group (0) >> /var/lib/influxdb/data/db/autogen/50/000000567-000000002.tsm (#1) >> Nov 8 19:12:53 db influxd[17415]: [tsm1] 2016/11/08 19:12:53 compacted >> level 2 group (0) into >> /var/lib/influxdb/data/db/autogen/50/000000567-000000003.tsm.tmp (#0) >> Nov 8 19:12:53 db influxd[17415]: [tsm1] 2016/11/08 19:12:53 compacted >> level 2 group 0 of 2 files into 1 files in 573.971276ms >> Nov 8 19:13:24 db influxd[17415]: [tsm1] 2016/11/08 19:13:24 Snapshot >> for path /var/lib/influxdb/data/db/autogen/50 written in 1.842949453s >> >> -- >> Remember to include the version number! >> --- >> You received this message because you are subscribed to the Google Groups >> "InfluxData" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at https://groups.google.com/group/influxdb. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/influxdb/57d726a8-fcc7-488e-8fce-365ab39e467c%40googlegroups.com >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Sean Beckett > Director of Support and Professional Services > InfluxDB > -- Remember to include the version number! --- You received this message because you are subscribed to the Google Groups "InfluxData" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/1a96503c-7663-490d-8a16-f9049f2d8d0d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
