It looks like the combination of the CQ running at 20:00 and the TSM compactions also kicking off at 20:00 are consuming the available RAM on the box. As Mathias mentioned, a CQ that takes 3.5 minutes to complete sounds very RAM intensive. Either add RAM or reduce the complexity of that CQ so it can complete with less RAM.
On Mon, Oct 17, 2016 at 2:50 PM, Mathias Herberts < [email protected]> wrote: > Your CQ completed in 3m27s, does it manipulate a very large amount of data? > > > On Monday, October 17, 2016 at 10:43:22 PM UTC+2, [email protected] > wrote: >> >> Heh believe it or not once again I got an OOM error! And it's becomes >> really 'funny' that it happens at the same time? Look at this: >> >> Oct 17 20:04:11 node1 kernel: kthreadd invoked oom-killer: >> gfp_mask=0x3000d0, order=2, oom_score_adj=0 >> >> When I look at InfluxDB log I see this: >> >> Oct 17 20:03:29 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:03:29 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 0252f60c-9494-11e6-b227-000000000000 239168 >> Oct 17 20:03:33 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:03:32 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 044496f0-9494-11e6-b228-000000000000 198616 >> Oct 17 20:03:36 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:03:36 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 0655bab6-9494-11e6-b229-000000000000 141794 >> Oct 17 20:03:36 node1 influxd: [tsm1] 2016/10/17 20:03:36 Compacting >> cache for /var/lib/influxdb/data/macdb/seven_days/579 >> Oct 17 20:03:37 node1 influxd: [continuous_querier] 2016/10/17 20:03:37 >> finished continuous query cq_30m (2016-10-17 19:30:00 +0200 CEST to >> 2016-10-17 20 >> :00:00 +0200 CEST) in 3m37.079684067s >> Oct 17 20:03:38 node1 influxd: [tsm1] 2016/10/17 20:03:38 Compacting >> cache for /var/lib/influxdb/data/macdb/three_months/575 >> Oct 17 20:03:38 node1 influxd: [tsm1] 2016/10/17 20:03:38 Snapshot for >> path /var/lib/influxdb/data/macdb/seven_days/579 deduplicated in >> 368.123875m >> s >> Oct 17 20:03:40 node1 influxd: [tsm1] 2016/10/17 20:03:40 Snapshot for >> path /var/lib/influxdb/data/macdb/three_months/575 deduplicated in >> 485.55333 >> 4ms >> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing >> /var/lib/influxdb/wal/macdb/seven_days/579/_01024.wal >> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing >> /var/lib/influxdb/wal/macdb/seven_days/579/_01025.wal >> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing >> /var/lib/influxdb/wal/macdb/seven_days/579/_01026.wal >> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 Snapshot for >> path /var/lib/influxdb/data/macdb/seven_days/579 written in 25.602303689s >> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 beginning level >> 1 compaction of group 0, 2 TSM files >> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 compacting >> level 1 group (0) /var/lib/influxdb/data/macdb/s >> even_days/579/000000359-000000 >> 001.tsm (#0) >> Oct 17 20:04:03 node1 influxd: [tsm1] 2016/10/17 20:04:02 compacting >> level 1 group (0) /var/lib/influxdb/data/macdb/s >> even_days/579/000000360-000000 >> 001.tsm (#1) >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 1616a065-9494-11e6-b22f-000000000000 152030 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 161979a7-9494-11e6-b238-000000000000 133364 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 16169ee0-9494-11e6-b22b-000000000000 152118 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 1616a01c-9494-11e6-b22e-000000000000 153616 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 1616a070-9494-11e6-b230-000000000000 153639 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 16169f73-9494-11e6-b22c-000000000000 153690 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 16169f73-9494-11e6-b22c-000000000000 153690 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 1616d89c-9494-11e6-b237-000000000000 152262 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 16169fb1-9494-11e6-b22d-000000000000 153706 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 1616c37e-9494-11e6-b234-000000000000 152707 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 1616cbfb-9494-11e6-b235-000000000000 152636 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 1616cc3e-9494-11e6-b236-000000000000 152641 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 1616c2eb-9494-11e6-b233-000000000000 153000 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 1616a080-9494-11e6-b231-000000000000 153896 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 1616a0a7-9494-11e6-b232-000000000000 153927 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 16169e92-9494-11e6-b22a-000000000000 153946 >> Oct 17 20:04:11 node1 kernel: kthreadd invoked oom-killer: >> gfp_mask=0x3000d0, order=2, oom_score_adj=0 >> Oct 17 20:04:11 node1 kernel: kthreadd cpuset=/ mems_allowed=0 >> >> Why it happens always at the same time? I don't have any cron jobs nor >> any strange traffic patterns at this time. >> >> -- > Remember to include the version number! > --- > You received this message because you are subscribed to the Google Groups > "InfluxData" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/influxdb. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/influxdb/fa5b6da3-c176-41e5-a984-eb5678cffa21%40googlegroups.com > <https://groups.google.com/d/msgid/influxdb/fa5b6da3-c176-41e5-a984-eb5678cffa21%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Sean Beckett Director of Support and Professional Services InfluxDB -- Remember to include the version number! --- You received this message because you are subscribed to the Google Groups "InfluxData" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CALGqCvO_FBYH%3DrzKLhC-BRcyoZPZ7KMKbp0%3DnfwWPLhq%2BN2yOA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
