There are no ES errors. Cluster Health is Green. I see data being added to
my /data partition. Is there a way to see what else ES could be doing that
would force Graylog to only process 1/3 of the logs it was processing a
week ago?
{
"cluster_name" : "XXXXXXXXX",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 2,
"active_primary_shards" : 320,
"active_shards" : 320,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}
On Tuesday, May 6, 2014 12:29:53 PM UTC-6, lennart wrote:
>
> Can you check your ElasticSearch logs for errors? I am pretty sure it
> is the reason.
>
> On Tue, May 6, 2014 at 5:57 PM, Tyler Bell
> <[email protected]<javascript:>>
> wrote:
> > I'm having an issue with Graylog continuously falling behind with log
> > processing, and the MasterCache filling up til the 10G of Heap Space
> maxes
> > out and crashes. The really weird thing is that a week ago, everything
> was
> > processing fine and I was taking between 1500-2000 msg/s. Now I barely
> get
> > over 500-750 msg/s. I don't think ElasticSearch is the issue because
> none of
> > the OutputCache or Buffer is increasing.
> >
> > I'm wondering if it has something to do with this: Number of indices
> (80)
> > higher than limit (20). Running retention for 60 indices. It doesn't
> look
> > like Graylog is properly rotating indexes and running this retention
> > instead.
> >
> > After restarting graylog2 and emptying cache...
> > [util][caches][2014-05-06T08:46:04.850-07:00] InputCache size: 5758
> > [util][caches][2014-05-06T08:46:04.850-07:00] OutputCache size: 0
> > [util][buffers][2014-05-06T08:46:04.850-07:00] OutputBuffer is at 0.0%.
> > [0/2048]
> > [util][buffers][2014-05-06T08:46:04.850-07:00] ProcessBuffer is at
> > 33.251953%. [681/2048]
> > [util][heap][2014-05-06T08:46:04.850-07:00] Used memory (MB): 1465
> > [util][heap][2014-05-06T08:46:04.850-07:00] Free memory (MB): 8330
> > [util][heap][2014-05-06T08:46:04.850-07:00] Total memory (MB): 9814
> > [util][heap][2014-05-06T08:46:04.850-07:00] Max memory (MB): 9814
> > [util][written][2014-05-06T08:46:04.850-07:00] Messages written to all
> > outputs: 1561
> >
> >
> > After MasterCache fills up a bit
> > [util][caches][2014-05-06T08:42:18.109-07:00] InputCache size: 2487587
> > [util][caches][2014-05-06T08:42:18.109-07:00] OutputCache size: 0
> > [util][buffers][2014-05-06T08:42:18.109-07:00] OutputBuffer is at 0.0%.
> > [0/2048]
> > [util][buffers][2014-05-06T08:42:18.109-07:00] ProcessBuffer is at
> > 40.429688%. [828/2048]
> > [util][heap][2014-05-06T08:42:18.109-07:00] Used memory (MB): 6392
> > [util][heap][2014-05-06T08:42:18.109-07:00] Free memory (MB): 3736
> > [util][heap][2014-05-06T08:42:18.109-07:00] Total memory (MB): 10129
> > [util][heap][2014-05-06T08:42:18.109-07:00] Max memory (MB): 10129
> > [util][written][2014-05-06T08:42:18.109-07:00] Messages written to all
> > outputs: 3100
> >
> >
> > ES Node config: (GLNode0 is the Graylog server). I know mlockall is
> false,
> > and is configured to be true, but these are virtualized servers and
> there
> > are some issues there.
> >
> > {
> > "ok" : true,
> > "cluster_name" : "Graylog2",
> > "nodes" : {
> > "X.X.X.X" : {
> > "name" : "GLNode1",
> > "transport_address" : "inet[/X.X.X.X:9300]",
> > "hostname" : "X.X.X.X",
> > "version" : "0.90.10",
> > "http_address" : "inet[/X.X.X.X:9200]",
> > "attributes" : {
> > "master" : "true"
> > },
> > "process" : {
> > "refresh_interval" : 1000,
> > "id" : 1611,
> > "max_file_descriptors" : 32000,
> > "mlockall" : false
> > }
> > },
> > "X.X.X.X" : {
> > "name" : "GLNode0",
> > "transport_address" : "inet[/X.X.X.X:9350]",
> > "hostname" : "X.X.X.X",
> > "version" : "0.90.10",
> > "attributes" : {
> > "client" : "true",
> > "data" : "false",
> > "master" : "false"
> > },
> > "process" : {
> > "refresh_interval" : 1000,
> > "id" : 28382,
> > "max_file_descriptors" : 4096,
> > "mlockall" : false
> > }
> > },
> > "X.X.X.X" : {
> > "name" : "GLNode2",
> > "transport_address" : "inet[/X.X.X.X:9300]",
> > "hostname" : "X.X.X.X",
> > "version" : "0.90.10",
> > "http_address" : "inet[/X.X.X.X:9200]",
> > "attributes" : {
> > "master" : "false"
> > },
> > "process" : {
> > "refresh_interval" : 1000,
> > "id" : 4508,
> > "max_file_descriptors" : 32000,
> > "mlockall" : false
> > }
> > }
> > }
> > }
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups
> > "graylog2" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an
> > email to [email protected] <javascript:>.
> > For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
