Hi Mathias, what's to be considered large amount of data in InfluxDB? I do have around 40k metrics every 5 minutes. Every metric has 2 tags and around 70 fields. CQ runs every 30 minutes and downsamples data (I keep 7 days raw data and 3 months downsampled data). Is that too much for a 40GB RAM box?
On Mon, Oct 17, 2016 at 10:50 PM, Mathias Herberts < [email protected]> wrote: > Your CQ completed in 3m27s, does it manipulate a very large amount of data? > > > On Monday, October 17, 2016 at 10:43:22 PM UTC+2, [email protected] > wrote: >> >> Heh believe it or not once again I got an OOM error! And it's becomes >> really 'funny' that it happens at the same time? Look at this: >> >> Oct 17 20:04:11 node1 kernel: kthreadd invoked oom-killer: >> gfp_mask=0x3000d0, order=2, oom_score_adj=0 >> >> When I look at InfluxDB log I see this: >> >> Oct 17 20:03:29 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:03:29 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 0252f60c-9494-11e6-b227-000000000000 239168 >> Oct 17 20:03:33 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:03:32 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 044496f0-9494-11e6-b228-000000000000 198616 >> Oct 17 20:03:36 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:03:36 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 0655bab6-9494-11e6-b229-000000000000 141794 >> Oct 17 20:03:36 node1 influxd: [tsm1] 2016/10/17 20:03:36 Compacting >> cache for /var/lib/influxdb/data/macdb/seven_days/579 >> Oct 17 20:03:37 node1 influxd: [continuous_querier] 2016/10/17 20:03:37 >> finished continuous query cq_30m (2016-10-17 19:30:00 +0200 CEST to >> 2016-10-17 20 >> :00:00 +0200 CEST) in 3m37.079684067s >> Oct 17 20:03:38 node1 influxd: [tsm1] 2016/10/17 20:03:38 Compacting >> cache for /var/lib/influxdb/data/macdb/three_months/575 >> Oct 17 20:03:38 node1 influxd: [tsm1] 2016/10/17 20:03:38 Snapshot for >> path /var/lib/influxdb/data/macdb/seven_days/579 deduplicated in >> 368.123875m >> s >> Oct 17 20:03:40 node1 influxd: [tsm1] 2016/10/17 20:03:40 Snapshot for >> path /var/lib/influxdb/data/macdb/three_months/575 deduplicated in >> 485.55333 >> 4ms >> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing >> /var/lib/influxdb/wal/macdb/seven_days/579/_01024.wal >> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing >> /var/lib/influxdb/wal/macdb/seven_days/579/_01025.wal >> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing >> /var/lib/influxdb/wal/macdb/seven_days/579/_01026.wal >> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 Snapshot for >> path /var/lib/influxdb/data/macdb/seven_days/579 written in 25.602303689s >> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 beginning level >> 1 compaction of group 0, 2 TSM files >> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 compacting >> level 1 group (0) /var/lib/influxdb/data/macdb/s >> even_days/579/000000359-000000 >> 001.tsm (#0) >> Oct 17 20:04:03 node1 influxd: [tsm1] 2016/10/17 20:04:02 compacting >> level 1 group (0) /var/lib/influxdb/data/macdb/s >> even_days/579/000000360-000000 >> 001.tsm (#1) >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 1616a065-9494-11e6-b22f-000000000000 152030 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 161979a7-9494-11e6-b238-000000000000 133364 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 16169ee0-9494-11e6-b22b-000000000000 152118 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 1616a01c-9494-11e6-b22e-000000000000 153616 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 >> "-" "-" 1616a070-9494-11e6-b230-000000000000 153639 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 16169f73-9494-11e6-b22c-000000000000 153690 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 16169f73-9494-11e6-b22c-000000000000 153690 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 1616d89c-9494-11e6-b237-000000000000 152262 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 16169fb1-9494-11e6-b22d-000000000000 153706 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 1616c37e-9494-11e6-b234-000000000000 152707 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 1616cbfb-9494-11e6-b235-000000000000 152636 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 1616cc3e-9494-11e6-b236-000000000000 152641 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 1616c2eb-9494-11e6-b233-000000000000 153000 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 1616a080-9494-11e6-b231-000000000000 153896 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 1616a0a7-9494-11e6-b232-000000000000 153927 >> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter >> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1" >> 204 0 "-" "-" 16169e92-9494-11e6-b22a-000000000000 153946 >> Oct 17 20:04:11 node1 kernel: kthreadd invoked oom-killer: >> gfp_mask=0x3000d0, order=2, oom_score_adj=0 >> Oct 17 20:04:11 node1 kernel: kthreadd cpuset=/ mems_allowed=0 >> >> Why it happens always at the same time? I don't have any cron jobs nor >> any strange traffic patterns at this time. >> >> -- Remember to include the version number! --- You received this message because you are subscribed to the Google Groups "InfluxData" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CAKb_NupzgAgq6NK4ymmWYZSDnAVATAQnyXa89RaHDbQ6e%2B1e8g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
