Ok so I finally managed to improve performance for overnight peaks (40k/mins to 180k/mins). If it can help someone:
- Moved to SSD and configured ES accordingly. - Changed hard drive from OVA's standard IDE to SCSI with controller. - Increased vCPU to 8 and RAM to 16GB. - Increased VM's IOPS priority. - Increased the ES stream timeout to 5000ms. - Increased journal to 10GB (although 5GB might of been enough) So it's been dealing with the load fine for 48h now and hope for it to continue that way :) On Wednesday, September 28, 2016 at 12:55:30 PM UTC-4, [email protected] wrote: > > So I moved to SSD and raised the journal to 10GB so the journal failures > stopped when logs inputs are spiking. Great! > However I'm still getting the paused stream issue during those spikes. I > don't get why that stream can process those messages fine but falls behind > during spikes. Any way to debug the stream to identify exactly what is > going wrong at that moment? > > > On Tuesday, September 27, 2016 at 12:35:13 PM UTC-4, [email protected] > wrote: >> >> Ok that is quite clear! Now I'll have to figure out why that bottleneck >> is happening. I doubt I can see the metrics later then 15 mins so that will >> be challenging! >> I'm currently moving the VM to SSDs which can't hurt but I doubt it'll >> solve the problem. >> >> >> On Tuesday, 27 September 2016 12:16:47 UTC-4, Jochen Schalanda wrote: >>> >>> Hi, >>> >>> from your screenshot it seems pretty clear that Elasticsearch can't >>> index messages at the same rate that they are ingested and processed by >>> Graylog. >>> >>> >>> On Tuesday, 27 September 2016 17:57:16 UTC+2, [email protected] wrote: >>>> >>>> So process is when be when message is actually parsed and output is >>>> when sent to ES for indexing? >>>> >>> >>> Yes. The process and output buffers filling is basically back pressure >>> from Elasticsearch. >>> >>> >>> Cheers, >>> Jochen >>> >>> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/f237b910-9a19-4a9a-a865-8c89f9970c62%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
