Can you check your ElasticSearch logs for errors? I am pretty sure it is the reason.
On Tue, May 6, 2014 at 5:57 PM, Tyler Bell <[email protected]> wrote: > I'm having an issue with Graylog continuously falling behind with log > processing, and the MasterCache filling up til the 10G of Heap Space maxes > out and crashes. The really weird thing is that a week ago, everything was > processing fine and I was taking between 1500-2000 msg/s. Now I barely get > over 500-750 msg/s. I don't think ElasticSearch is the issue because none of > the OutputCache or Buffer is increasing. > > I'm wondering if it has something to do with this: Number of indices (80) > higher than limit (20). Running retention for 60 indices. It doesn't look > like Graylog is properly rotating indexes and running this retention > instead. > > After restarting graylog2 and emptying cache... > [util][caches][2014-05-06T08:46:04.850-07:00] InputCache size: 5758 > [util][caches][2014-05-06T08:46:04.850-07:00] OutputCache size: 0 > [util][buffers][2014-05-06T08:46:04.850-07:00] OutputBuffer is at 0.0%. > [0/2048] > [util][buffers][2014-05-06T08:46:04.850-07:00] ProcessBuffer is at > 33.251953%. [681/2048] > [util][heap][2014-05-06T08:46:04.850-07:00] Used memory (MB): 1465 > [util][heap][2014-05-06T08:46:04.850-07:00] Free memory (MB): 8330 > [util][heap][2014-05-06T08:46:04.850-07:00] Total memory (MB): 9814 > [util][heap][2014-05-06T08:46:04.850-07:00] Max memory (MB): 9814 > [util][written][2014-05-06T08:46:04.850-07:00] Messages written to all > outputs: 1561 > > > After MasterCache fills up a bit > [util][caches][2014-05-06T08:42:18.109-07:00] InputCache size: 2487587 > [util][caches][2014-05-06T08:42:18.109-07:00] OutputCache size: 0 > [util][buffers][2014-05-06T08:42:18.109-07:00] OutputBuffer is at 0.0%. > [0/2048] > [util][buffers][2014-05-06T08:42:18.109-07:00] ProcessBuffer is at > 40.429688%. [828/2048] > [util][heap][2014-05-06T08:42:18.109-07:00] Used memory (MB): 6392 > [util][heap][2014-05-06T08:42:18.109-07:00] Free memory (MB): 3736 > [util][heap][2014-05-06T08:42:18.109-07:00] Total memory (MB): 10129 > [util][heap][2014-05-06T08:42:18.109-07:00] Max memory (MB): 10129 > [util][written][2014-05-06T08:42:18.109-07:00] Messages written to all > outputs: 3100 > > > ES Node config: (GLNode0 is the Graylog server). I know mlockall is false, > and is configured to be true, but these are virtualized servers and there > are some issues there. > > { > "ok" : true, > "cluster_name" : "Graylog2", > "nodes" : { > "X.X.X.X" : { > "name" : "GLNode1", > "transport_address" : "inet[/X.X.X.X:9300]", > "hostname" : "X.X.X.X", > "version" : "0.90.10", > "http_address" : "inet[/X.X.X.X:9200]", > "attributes" : { > "master" : "true" > }, > "process" : { > "refresh_interval" : 1000, > "id" : 1611, > "max_file_descriptors" : 32000, > "mlockall" : false > } > }, > "X.X.X.X" : { > "name" : "GLNode0", > "transport_address" : "inet[/X.X.X.X:9350]", > "hostname" : "X.X.X.X", > "version" : "0.90.10", > "attributes" : { > "client" : "true", > "data" : "false", > "master" : "false" > }, > "process" : { > "refresh_interval" : 1000, > "id" : 28382, > "max_file_descriptors" : 4096, > "mlockall" : false > } > }, > "X.X.X.X" : { > "name" : "GLNode2", > "transport_address" : "inet[/X.X.X.X:9300]", > "hostname" : "X.X.X.X", > "version" : "0.90.10", > "http_address" : "inet[/X.X.X.X:9200]", > "attributes" : { > "master" : "false" > }, > "process" : { > "refresh_interval" : 1000, > "id" : 4508, > "max_file_descriptors" : 32000, > "mlockall" : false > } > } > } > } > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
