It looks like the combination of the CQ running at 20:00 and the TSM
compactions also kicking off at 20:00 are consuming the available RAM on
the box. As Mathias mentioned, a CQ that takes 3.5 minutes to complete
sounds very RAM intensive. Either add RAM or reduce the complexity of that
CQ so it can complete with less RAM.

On Mon, Oct 17, 2016 at 2:50 PM, Mathias Herberts <
mathias.herbe...@gmail.com> wrote:

> Your CQ completed in 3m27s, does it manipulate a very large amount of data?
>
>
> On Monday, October 17, 2016 at 10:43:22 PM UTC+2, pavel...@gmail.com
> wrote:
>>
>> Heh believe it or not once again I got an OOM error! And it's becomes
>> really 'funny' that it happens at the same time? Look at this:
>>
>> Oct 17 20:04:11 node1 kernel: kthreadd invoked oom-killer:
>> gfp_mask=0x3000d0, order=2, oom_score_adj=0
>>
>> When I look at InfluxDB log I see this:
>>
>> Oct 17 20:03:29 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:03:29 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0
>> "-" "-" 0252f60c-9494-11e6-b227-000000000000 239168
>> Oct 17 20:03:33 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:03:32 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0
>> "-" "-" 044496f0-9494-11e6-b228-000000000000 198616
>> Oct 17 20:03:36 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:03:36 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0
>> "-" "-" 0655bab6-9494-11e6-b229-000000000000 141794
>> Oct 17 20:03:36 node1 influxd: [tsm1] 2016/10/17 20:03:36 Compacting
>> cache for /var/lib/influxdb/data/macdb/seven_days/579
>> Oct 17 20:03:37 node1 influxd: [continuous_querier] 2016/10/17 20:03:37
>> finished continuous query cq_30m (2016-10-17 19:30:00 +0200 CEST to
>> 2016-10-17 20
>> :00:00 +0200 CEST) in 3m37.079684067s
>> Oct 17 20:03:38 node1 influxd: [tsm1] 2016/10/17 20:03:38 Compacting
>> cache for /var/lib/influxdb/data/macdb/three_months/575
>> Oct 17 20:03:38 node1 influxd: [tsm1] 2016/10/17 20:03:38 Snapshot for
>> path /var/lib/influxdb/data/macdb/seven_days/579 deduplicated in
>> 368.123875m
>> s
>> Oct 17 20:03:40 node1 influxd: [tsm1] 2016/10/17 20:03:40 Snapshot for
>> path /var/lib/influxdb/data/macdb/three_months/575 deduplicated in
>> 485.55333
>> 4ms
>> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing
>> /var/lib/influxdb/wal/macdb/seven_days/579/_01024.wal
>> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing
>> /var/lib/influxdb/wal/macdb/seven_days/579/_01025.wal
>> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing
>> /var/lib/influxdb/wal/macdb/seven_days/579/_01026.wal
>> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 Snapshot for
>> path /var/lib/influxdb/data/macdb/seven_days/579 written in 25.602303689s
>> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 beginning level
>> 1 compaction of group 0, 2 TSM files
>> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 compacting
>> level 1 group (0) /var/lib/influxdb/data/macdb/s
>> even_days/579/000000359-000000
>> 001.tsm (#0)
>> Oct 17 20:04:03 node1 influxd: [tsm1] 2016/10/17 20:04:02 compacting
>> level 1 group (0) /var/lib/influxdb/data/macdb/s
>> even_days/579/000000360-000000
>> 001.tsm (#1)
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0
>> "-" "-" 1616a065-9494-11e6-b22f-000000000000 152030
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0
>> "-" "-" 161979a7-9494-11e6-b238-000000000000 133364
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0
>> "-" "-" 16169ee0-9494-11e6-b22b-000000000000 152118
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0
>> "-" "-" 1616a01c-9494-11e6-b22e-000000000000 153616
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0
>> "-" "-" 1616a070-9494-11e6-b230-000000000000 153639
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0 "-" "-" 16169f73-9494-11e6-b22c-000000000000 153690
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0 "-" "-" 16169f73-9494-11e6-b22c-000000000000 153690
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0 "-" "-" 1616d89c-9494-11e6-b237-000000000000 152262
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0 "-" "-" 16169fb1-9494-11e6-b22d-000000000000 153706
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0 "-" "-" 1616c37e-9494-11e6-b234-000000000000 152707
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0 "-" "-" 1616cbfb-9494-11e6-b235-000000000000 152636
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0 "-" "-" 1616cc3e-9494-11e6-b236-000000000000 152641
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0 "-" "-" 1616c2eb-9494-11e6-b233-000000000000 153000
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0 "-" "-" 1616a080-9494-11e6-b231-000000000000 153896
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0 "-" "-" 1616a0a7-9494-11e6-b232-000000000000 153927
>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>> 204 0 "-" "-" 16169e92-9494-11e6-b22a-000000000000 153946
>> Oct 17 20:04:11 node1 kernel: kthreadd invoked oom-killer:
>> gfp_mask=0x3000d0, order=2, oom_score_adj=0
>> Oct 17 20:04:11 node1 kernel: kthreadd cpuset=/ mems_allowed=0
>>
>> Why it happens always at the same time? I don't have any cron jobs nor
>> any strange traffic patterns at this time.
>>
>> --
> Remember to include the version number!
> ---
> You received this message because you are subscribed to the Google Groups
> "InfluxData" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to influxdb+unsubscr...@googlegroups.com.
> To post to this group, send email to influxdb@googlegroups.com.
> Visit this group at https://groups.google.com/group/influxdb.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/influxdb/fa5b6da3-c176-41e5-a984-eb5678cffa21%40googlegroups.com
> <https://groups.google.com/d/msgid/influxdb/fa5b6da3-c176-41e5-a984-eb5678cffa21%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Sean Beckett
Director of Support and Professional Services
InfluxDB

-- 
Remember to include the version number!
--- 
You received this message because you are subscribed to the Google Groups 
"InfluxData" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to influxdb+unsubscr...@googlegroups.com.
To post to this group, send email to influxdb@googlegroups.com.
Visit this group at https://groups.google.com/group/influxdb.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/influxdb/CALGqCvO_FBYH%3DrzKLhC-BRcyoZPZ7KMKbp0%3DnfwWPLhq%2BN2yOA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to