> 40k metrics every 5 minutes. Every metric has 2 tags and around 70 fields.

40000 points every 300 seconds, each with 70 fields. That's about 10k
values per second. If the load is coming all at once every 300 seconds,
that might cause RAM spikes. Ideally the batches can be evened out to avoid
spikes.

On Tue, Oct 18, 2016 at 3:34 AM, Pavel Dimow <paveldi...@gmail.com> wrote:

> Hi Mathias,
>
> what's to be considered large amount of data in InfluxDB? I do have around
> 40k metrics every 5 minutes. Every metric has 2 tags and around 70 fields.
> CQ runs every 30 minutes and downsamples data (I keep 7 days raw data and 3
> months downsampled data). Is that too much for a 40GB RAM box?
>
> On Mon, Oct 17, 2016 at 10:50 PM, Mathias Herberts <
> mathias.herbe...@gmail.com> wrote:
>
>> Your CQ completed in 3m27s, does it manipulate a very large amount of
>> data?
>>
>>
>> On Monday, October 17, 2016 at 10:43:22 PM UTC+2, pavel...@gmail.com
>> wrote:
>>>
>>> Heh believe it or not once again I got an OOM error! And it's becomes
>>> really 'funny' that it happens at the same time? Look at this:
>>>
>>> Oct 17 20:04:11 node1 kernel: kthreadd invoked oom-killer:
>>> gfp_mask=0x3000d0, order=2, oom_score_adj=0
>>>
>>> When I look at InfluxDB log I see this:
>>>
>>> Oct 17 20:03:29 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:03:29 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0
>>> "-" "-" 0252f60c-9494-11e6-b227-000000000000 239168
>>> Oct 17 20:03:33 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:03:32 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0
>>> "-" "-" 044496f0-9494-11e6-b228-000000000000 198616
>>> Oct 17 20:03:36 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:03:36 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0
>>> "-" "-" 0655bab6-9494-11e6-b229-000000000000 141794
>>> Oct 17 20:03:36 node1 influxd: [tsm1] 2016/10/17 20:03:36 Compacting
>>> cache for /var/lib/influxdb/data/macdb/seven_days/579
>>> Oct 17 20:03:37 node1 influxd: [continuous_querier] 2016/10/17 20:03:37
>>> finished continuous query cq_30m (2016-10-17 19:30:00 +0200 CEST to
>>> 2016-10-17 20
>>> :00:00 +0200 CEST) in 3m37.079684067s
>>> Oct 17 20:03:38 node1 influxd: [tsm1] 2016/10/17 20:03:38 Compacting
>>> cache for /var/lib/influxdb/data/macdb/three_months/575
>>> Oct 17 20:03:38 node1 influxd: [tsm1] 2016/10/17 20:03:38 Snapshot for
>>> path /var/lib/influxdb/data/macdb/seven_days/579 deduplicated in
>>> 368.123875m
>>> s
>>> Oct 17 20:03:40 node1 influxd: [tsm1] 2016/10/17 20:03:40 Snapshot for
>>> path /var/lib/influxdb/data/macdb/three_months/575 deduplicated in
>>> 485.55333
>>> 4ms
>>> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing
>>> /var/lib/influxdb/wal/macdb/seven_days/579/_01024.wal
>>> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing
>>> /var/lib/influxdb/wal/macdb/seven_days/579/_01025.wal
>>> Oct 17 20:04:02 node1 influxd: [tsm1wal] 2016/10/17 20:04:02 Removing
>>> /var/lib/influxdb/wal/macdb/seven_days/579/_01026.wal
>>> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 Snapshot for
>>> path /var/lib/influxdb/data/macdb/seven_days/579 written in
>>> 25.602303689s
>>> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 beginning
>>> level 1 compaction of group 0, 2 TSM files
>>> Oct 17 20:04:02 node1 influxd: [tsm1] 2016/10/17 20:04:02 compacting
>>> level 1 group (0) /var/lib/influxdb/data/macdb/s
>>> even_days/579/000000359-000000
>>> 001.tsm (#0)
>>> Oct 17 20:04:03 node1 influxd: [tsm1] 2016/10/17 20:04:02 compacting
>>> level 1 group (0) /var/lib/influxdb/data/macdb/s
>>> even_days/579/000000360-000000
>>> 001.tsm (#1)
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0
>>> "-" "-" 1616a065-9494-11e6-b22f-000000000000 152030
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0
>>> "-" "-" 161979a7-9494-11e6-b238-000000000000 133364
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0
>>> "-" "-" 16169ee0-9494-11e6-b22b-000000000000 152118
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0
>>> "-" "-" 1616a01c-9494-11e6-b22e-000000000000 153616
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0
>>> "-" "-" 1616a070-9494-11e6-b230-000000000000 153639
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0 "-" "-" 16169f73-9494-11e6-b22c-000000000000 153690
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0 "-" "-" 16169f73-9494-11e6-b22c-000000000000 153690
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0 "-" "-" 1616d89c-9494-11e6-b237-000000000000 152262
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0 "-" "-" 16169fb1-9494-11e6-b22d-000000000000 153706
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0 "-" "-" 1616c37e-9494-11e6-b234-000000000000 152707
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0 "-" "-" 1616cbfb-9494-11e6-b235-000000000000 152636
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0 "-" "-" 1616cc3e-9494-11e6-b236-000000000000 152641
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0 "-" "-" 1616c2eb-9494-11e6-b233-000000000000 153000
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0 "-" "-" 1616a080-9494-11e6-b231-000000000000 153896
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0 "-" "-" 1616a0a7-9494-11e6-b232-000000000000 153927
>>> Oct 17 20:04:03 node1 influxd: [httpd] 192.168.11.24 - writter
>>> [17/Oct/2016:20:04:02 +0200] "POST /write?db=macdb&precision=s HTTP/1.1"
>>> 204 0 "-" "-" 16169e92-9494-11e6-b22a-000000000000 153946
>>> Oct 17 20:04:11 node1 kernel: kthreadd invoked oom-killer:
>>> gfp_mask=0x3000d0, order=2, oom_score_adj=0
>>> Oct 17 20:04:11 node1 kernel: kthreadd cpuset=/ mems_allowed=0
>>>
>>> Why it happens always at the same time? I don't have any cron jobs nor
>>> any strange traffic patterns at this time.
>>>
>>>
> --
> Remember to include the version number!
> ---
> You received this message because you are subscribed to the Google Groups
> "InfluxData" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to influxdb+unsubscr...@googlegroups.com.
> To post to this group, send email to influxdb@googlegroups.com.
> Visit this group at https://groups.google.com/group/influxdb.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/influxdb/CAKb_NupzgAgq6NK4ymmWYZSDnAVATAQnyX
> a89RaHDbQ6e%2B1e8g%40mail.gmail.com
> <https://groups.google.com/d/msgid/influxdb/CAKb_NupzgAgq6NK4ymmWYZSDnAVATAQnyXa89RaHDbQ6e%2B1e8g%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Sean Beckett
Director of Support and Professional Services
InfluxDB

-- 
Remember to include the version number!
--- 
You received this message because you are subscribed to the Google Groups 
"InfluxData" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to influxdb+unsubscr...@googlegroups.com.
To post to this group, send email to influxdb@googlegroups.com.
Visit this group at https://groups.google.com/group/influxdb.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/influxdb/CALGqCvM2bx4w1H8_Ggwx%3DNFSD%2B-nVP7nWjxcWLk5CafAM%2BsFqg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to