I'm having an issue with Graylog continuously falling behind with log
processing, and the MasterCache filling up til the 10G of Heap Space maxes
out and crashes. The really weird thing is that a week ago, everything was
processing fine and I was taking between 1500-2000 msg/s. Now I barely get
over 500-750 msg/s. I don't think ElasticSearch is the issue because none
of the OutputCache or Buffer is increasing.
I'm wondering if it has something to do with this: Number of indices (80)
higher than limit (20). Running retention for 60 indices. It doesn't look
like Graylog is properly rotating indexes and running this retention
instead.
After restarting graylog2 and emptying cache...
[util][caches][2014-05-06T08:46:04.850-07:00] InputCache size: 5758
[util][caches][2014-05-06T08:46:04.850-07:00] OutputCache size: 0
[util][buffers][2014-05-06T08:46:04.850-07:00] OutputBuffer is at 0.0%.
[0/2048]
[util][buffers][2014-05-06T08:46:04.850-07:00] ProcessBuffer is at
33.251953%. [681/2048]
[util][heap][2014-05-06T08:46:04.850-07:00] Used memory (MB): 1465
[util][heap][2014-05-06T08:46:04.850-07:00] Free memory (MB): 8330
[util][heap][2014-05-06T08:46:04.850-07:00] Total memory (MB): 9814
[util][heap][2014-05-06T08:46:04.850-07:00] Max memory (MB): 9814
[util][written][2014-05-06T08:46:04.850-07:00] Messages written to all
outputs: 1561
After MasterCache fills up a bit
[util][caches][2014-05-06T08:42:18.109-07:00] InputCache size: 2487587
[util][caches][2014-05-06T08:42:18.109-07:00] OutputCache size: 0
[util][buffers][2014-05-06T08:42:18.109-07:00] OutputBuffer is at 0.0%.
[0/2048]
[util][buffers][2014-05-06T08:42:18.109-07:00] ProcessBuffer is at
40.429688%. [828/2048]
[util][heap][2014-05-06T08:42:18.109-07:00] Used memory (MB): 6392
[util][heap][2014-05-06T08:42:18.109-07:00] Free memory (MB): 3736
[util][heap][2014-05-06T08:42:18.109-07:00] Total memory (MB): 10129
[util][heap][2014-05-06T08:42:18.109-07:00] Max memory (MB): 10129
[util][written][2014-05-06T08:42:18.109-07:00] Messages written to all
outputs: 3100
ES Node config: (GLNode0 is the Graylog server). I know mlockall is false,
and is configured to be true, but these are virtualized servers and there
are some issues there.
{
"ok" : true,
"cluster_name" : "Graylog2",
"nodes" : {
"X.X.X.X" : {
"name" : "GLNode1",
"transport_address" : "inet[/X.X.X.X:9300]",
"hostname" : "X.X.X.X",
"version" : "0.90.10",
"http_address" : "inet[/X.X.X.X:9200]",
"attributes" : {
"master" : "true"
},
"process" : {
"refresh_interval" : 1000,
"id" : 1611,
"max_file_descriptors" : 32000,
"mlockall" : false
}
},
"X.X.X.X" : {
"name" : "GLNode0",
"transport_address" : "inet[/X.X.X.X:9350]",
"hostname" : "X.X.X.X",
"version" : "0.90.10",
"attributes" : {
"client" : "true",
"data" : "false",
"master" : "false"
},
"process" : {
"refresh_interval" : 1000,
"id" : 28382,
"max_file_descriptors" : 4096,
"mlockall" : false
}
},
"X.X.X.X" : {
"name" : "GLNode2",
"transport_address" : "inet[/X.X.X.X:9300]",
"hostname" : "X.X.X.X",
"version" : "0.90.10",
"http_address" : "inet[/X.X.X.X:9200]",
"attributes" : {
"master" : "false"
},
"process" : {
"refresh_interval" : 1000,
"id" : 4508,
"max_file_descriptors" : 32000,
"mlockall" : false
}
}
}
}
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
