Hi, this all boils down to an unstable Elasticsearch instance. When Graylog is not able to forward log messages to ES it buffers them on disk and tries to send them later. This is called journal. So when your ES service is not running properly the journal fills up with messages. Please take a look into the ES logs to figure out why it has problems with message ingestion. You can find them in /var/log/graylog/elasticsearch/current
Cheers, Marius On 27 June 2016 at 13:39, John <[email protected]> wrote: > 1 and 4 > and the graylog server node is not sending data to elasticsearch > I deleted the journal but it doesn't help > the problems began few days after I upgraded from 1.3 to 2.0.2 > > בתאריך יום שני, 27 ביוני 2016 בשעה 14:30:28 UTC+3, מאת Joe K: > >> Which problem out of 4? >> >> >> On Monday, June 27, 2016 at 2:00:14 PM UTC+3, John wrote: >>> >>> Hi Joe >>> I have exactly the same problem few days after I upgraded from 1.3 to >>> 2.0.2 >>> Did you managed to fix this issue? >>> >>> בתאריך יום חמישי, 26 במאי 2016 בשעה 14:02:19 UTC+3, מאת Joe K: >>>> >>>> >>>> - We run it on t2.medium. (4GB RAM, 2 cores) >>>> - About 1 incoming message per second. >>>> - tried 2.0.0 and now running 2.0.1 >>>> >>>> Anyone use Image in real world application? Graylog 2.0 image fails >>>> after few days. Is this Image problem or Graylog in general? >>>> >>>> It runs fine for about a week. After that there's errors and search >>>> stop working. Search requests timeout. >>>> There's many errors and they are very cryptic, google search does not >>>> give any solutions how to manage them: >>>> >>>> >>>> *1. After about a week we have error "Uncommited messages deleted from >>>> journal"* >>>> >>>>> Uncommited messages deleted from journal (triggered 9 days ago) >>>>> Some messages were deleted from the Graylog journal before they could >>>>> be written to Elasticsearch. Please verify that your Elasticsearch cluster >>>>> is healthy and fast enough. You may also want to review your Graylog >>>>> journal settings and set a higher limit. (Node: f12... >>>> >>>> >>>> What to do about this? What is "journal"? Google search produce no >>>> answers. >>>> >>>> *2. After about 4 days of clean install it always trigger "Cluster >>>> unhealthy"* >>>> >>>>> "Elasticsearch cluster unhealthy (RED)" >>>>> "The Elasticsearch cluster state is RED which means shards are >>>>> unassigned. This usually indicates a crashed and corrupt cluster and needs >>>>> to be investigated. Graylog will write into the local disk journal. Read >>>>> how to fix this in the Elasticsearch setup documentation." >>>> >>>> >>>> When you go to that documentation link it says "The red status >>>> indicates that some or all of the primary shards are not available. In this >>>> state, no searches can be performed until all primary shards are restored." >>>> That's it. what are you supposed to do? >>>> After long search finally found one solution: this was cured once with >>>> *curl >>>> -XPUT 'localhost:9200/_settings' -d '{ "index" : { >>>> "number_of_replicas" : 0}}'* >>>> Next time it happened, we tried the solution again, but response was >>>> *{"acknowledged":false}* >>>> So what now??? >>>> >>>> *3. Every time we perform graylog-ctl restart four more unassigled >>>> shards appear:* >>>> Elasticsearch cluster is yellow. Shards: 20 active, 0 initializing, 0 >>>> relocating, 8 unassigned >>>> graylog-ctl restart >>>> Elasticsearch cluster is yellow. Shards: 20 active, 0 initializing, 0 >>>> relocating, 12 unassigned >>>> Etc. >>>> >>>> >>>> >>>> *4. Journal utilization is too high without any hint on how to set it >>>> to higher.* >>>> >>>>> Journal utilization is too high (triggered 11 days ago) >>>>> Journal utilization is too high and may go over the limit soon. Please >>>>> verify that your Elasticsearch cluster is healthy and fast enough. You may >>>>> also want to review your Graylog journal settings and set a higher limit. >>>>> (Node: f121 >>>> >>>> >>>> What is this "journal"? and how to set it to "higher"? >>>> >>>> Please help! >>>> >>>> -- > You received this message because you are subscribed to the Google Groups > "Graylog Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/2288cbf2-6f37-4e77-8c32-c50ba64fe71e%40googlegroups.com > <https://groups.google.com/d/msgid/graylog2/2288cbf2-6f37-4e77-8c32-c50ba64fe71e%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog Company Poolstraße 21 20335 Hamburg Germany https://www.graylog.com <https://www.torch.sh/> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAMqbBbK3BrRgy%2B%2BnUpAGbQRsFDgsvAmMe9_8HUw%2BMBCidzc8AQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
