Hi Sean, our CQ runs every 30 minutes and yes I guess it's complex (I can post it here) but our box CPU and RAM has been set according to official recommendations. Also I can't find a way to reschedule TSM compactions which I think could solve the issue.
On Mon, Oct 17, 2016 at 11:30 PM, Sean Beckett <[email protected]> wrote: > It looks like the combination of the CQ running at 20:00 and the TSM > compactions also kicking off at 20:00 are consuming the available RAM on > the box. As Mathias mentioned, a CQ that takes 3.5 minutes to complete > sounds very RAM intensive. Either add RAM or reduce the complexity of that > CQ so it can complete with less RAM. > > On Mon, Oct 17, 2016 at 2:50 PM, Mathias Herberts < > [email protected]> wrote: > >> Your CQ completed in 3m27s, does it manipulate a very large amount of >> data? >> >> >> On Monday, October 17, 2016 at 10:43:22 PM UTC+2, [email protected] >> wrote: >>> >>> Heh believe it or not once again I got an OOM error! And it's becomes >>> really 'funny' that it happens at the same time? Look at this: >>> >>> Oct 17 20:04:11 node1 kernel: kthreadd invoked oom-killer: >>> gfp_mask=0x3000d0, order=2, oom_score_adj=0 >>> >>> When I look at InfluxDB log I see this: >>> >>> Oct 17 20:03:29 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:03:29 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 >>> "-" "-" 0252f60c-9494-11e6-b227-000000000000 239168 >>> Oct 17 20:03:33 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:03:32 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 >>> "-" "-" 044496f0-9494-11e6-b228-000000000000 198616 >>> Oct 17 20:03:36 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:03:36 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 >>> "-" "-" 0655bab6-9494-11e6-b229-000000000000 141794 >>> Oct 17 20:03:36 node1 influxd: [tsm1] 2016/10/17 20:03:36 Compacting >>> cache for /var/lib/influxdb/data/macdb/seven_days/579 >>> Oct 17 20:03:37 node1 influxd: [continuous_querier] 2016/10/17 20:03:37 >>> finished continuous query cq_30m (2016-10-17 19:30:00 +0200 CEST to >>> 2016-10-17 20 >>> :00:00 +0200 CEST) in 3m37.079684067s >>> Oct 17 20:03:38 node1 influxd: [tsm1] 2016/10/17 20:03:38 Compacting >>> cache for /var/lib/influxdb/data/macdb/three_months/575 >>> Oct 17 20:03:38 node1 influxd: [tsm1] 2016/10/17 20:03:38 Snapshot for >>> path /var/lib/influxdb/data/macdb/seven_days/579 deduplicated in >>> 368.123875m >>> s >>> Oct 17 20:03:40 node1 influxd: [tsm1] 2016/10/17 20:03:40 Snapshot for >>> path /var/lib/influxdb/data/macdb/three_months/575 deduplicated in >>> 485.55333 >>> 4ms >>> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing >>> /var/lib/influxdb/wal/macdb/seven_days/579/_01024.wal >>> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing >>> /var/lib/influxdb/wal/macdb/seven_days/579/_01025.wal >>> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing >>> /var/lib/influxdb/wal/macdb/seven_days/579/_01026.wal >>> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 Snapshot for >>> path /var/lib/influxdb/data/macdb/seven_days/579 written in >>> 25.602303689s >>> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 beginning >>> level 1 compaction of group 0, 2 TSM files >>> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 compacting >>> level 1 group (0) /var/lib/influxdb/data/macdb/s >>> even_days/579/000000359-000000 >>> 001.tsm (#0) >>> Oct 17 20:04:03 node1 influxd: [tsm1] 2016/10/17 20:04:02 compacting >>> level 1 group (0) /var/lib/influxdb/data/macdb/s >>> even_days/579/000000360-000000 >>> 001.tsm (#1) >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 >>> "-" "-" 1616a065-9494-11e6-b22f-000000000000 152030 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 >>> "-" "-" 161979a7-9494-11e6-b238-000000000000 133364 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 >>> "-" "-" 16169ee0-9494-11e6-b22b-000000000000 152118 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 >>> "-" "-" 1616a01c-9494-11e6-b22e-000000000000 153616 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 >>> "-" "-" 1616a070-9494-11e6-b230-000000000000 153639 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 "-" "-" 16169f73-9494-11e6-b22c-000000000000 153690 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 "-" "-" 16169f73-9494-11e6-b22c-000000000000 153690 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 "-" "-" 1616d89c-9494-11e6-b237-000000000000 152262 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 "-" "-" 16169fb1-9494-11e6-b22d-000000000000 153706 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 "-" "-" 1616c37e-9494-11e6-b234-000000000000 152707 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 "-" "-" 1616cbfb-9494-11e6-b235-000000000000 152636 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 "-" "-" 1616cc3e-9494-11e6-b236-000000000000 152641 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 "-" "-" 1616c2eb-9494-11e6-b233-000000000000 153000 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 "-" "-" 1616a080-9494-11e6-b231-000000000000 153896 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 "-" "-" 1616a0a7-9494-11e6-b232-000000000000 153927 >>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >>> 204 0 "-" "-" 16169e92-9494-11e6-b22a-000000000000 153946 >>> Oct 17 20:04:11 node1 kernel: kthreadd invoked oom-killer: >>> gfp_mask=0x3000d0, order=2, oom_score_adj=0 >>> Oct 17 20:04:11 node1 kernel: kthreadd cpuset=/ mems_allowed=0 >>> >>> Why it happens always at the same time? I don't have any cron jobs nor >>> any strange traffic patterns at this time. >>> >>> -- >> Remember to include the version number! >> --- >> You received this message because you are subscribed to the Google Groups >> "InfluxData" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/influxdb. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/influxdb/fa5b6da3-c176-41e5-a984-eb5678cffa21%40googlegroups.com >> <https://groups.google.com/d/msgid/influxdb/fa5b6da3-c176-41e5-a984-eb5678cffa21%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Sean Beckett > Director of Support and Professional Services > InfluxDB > -- Remember to include the version number! --- You received this message because you are subscribed to the Google Groups "InfluxData" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CAKb_Nuoh3JKWrC8DM-6qdgpzOebfYeOnYUhgRyTYFGUpmx_b2g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
