screenshots from my UI <https://lh3.googleusercontent.com/-tEVtJ8knQ8c/V3Euz0SdodI/AAAAAAAAfws/_kr7G20PxvE3R4WOK0AIHDKRYDGqbMwrQCLcB/s1600/journal_too_high.png>
<https://lh3.googleusercontent.com/-l6rI9eoCEb8/V3Eu2hMMM5I/AAAAAAAAfw0/iiZfY6RxadUhqGDTKr93GBqxR7wmqhN0wCLcB/s1600/node_details.png> בתאריך יום שני, 27 ביוני 2016 בשעה 15:32:52 UTC+3, מאת John: > > Hi > I checked the elasticsearch log and I don't see something special > The cluster status is green > > this is the last log file > > 2016-06-26_09:51:28.78352 [2016-06-26 12:51:28,782][INFO ][node > ] [Glenn Talbot] version[2.3.1], pid[953], > build[bd98092/2016-04-04T12:25:05Z] > 2016-06-26_09:51:28.79783 [2016-06-26 12:51:28,794][INFO ][node > ] [Glenn Talbot] initializing ... > 2016-06-26_09:51:30.17146 [2016-06-26 12:51:30,171][INFO ][plugins > ] [Glenn Talbot] modules [reindex, lang-expression, lang-groovy], > plugins [kopf], sites [kopf] > 2016-06-26_09:51:30.29289 [2016-06-26 12:51:30,292][INFO ][env > ] [Glenn Talbot] using [1] data paths, mounts [[/ > (/dev/mapper/graylog--vg-root)]], net usable_space [11gb], net total_space > [14.9gb], spins? [possibly], types [ext4] > 2016-06-26_09:51:30.29564 [2016-06-26 12:51:30,294][INFO ][env > ] [Glenn Talbot] heap size [37.6gb], compressed ordinary object > pointers [false] > 2016-06-26_09:51:30.29766 [2016-06-26 12:51:30,294][WARN ][env > ] [Glenn Talbot] max file descriptors [64000] for elasticsearch > process likely too low, consider increasing to at least [65536] > 2016-06-26_09:51:34.69050 [2016-06-26 12:51:34,690][INFO ][node > ] [Glenn Talbot] initialized > 2016-06-26_09:51:34.69107 [2016-06-26 12:51:34,690][INFO ][node > ] [Glenn Talbot] starting ... > 2016-06-26_09:51:35.32863 [2016-06-26 12:51:35,328][INFO ][transport > ] [Glenn Talbot] publish_address {172.25.232.45:9300}, > bound_addresses {172.25.232.45:9300} > 2016-06-26_09:51:35.33658 [2016-06-26 12:51:35,336][INFO ][discovery > ] [Glenn Talbot] graylog-production/th7wM-a9ThaAY_umCV3v2w > 2016-06-26_09:51:45.37933 [2016-06-26 12:51:45,379][INFO ][cluster.service > ] [Glenn Talbot] new_master {Glenn > Talbot}{th7wM-a9ThaAY_umCV3v2w}{172.25.232.45}{172.25.232.45:9300}, added > {{graylog-a0b12869-11ed-4d89-ae58-dcc7380bc3b8}{KA4cjlTpQTm9Y1Rv5wlVmw}{172.25.232.41}{172.25.232.41:9350}{client=true, > > data=false, > master=false},{graylog-2a340000-d1ba-4f21-a9df-f45901d845b7}{BiWe2Zy2Syaojr9ek0AlJQ}{172.25.232.35}{172.25.232.35:9350}{client=true, > > data=false, master=false},}, reason: zen-disco-join(elected_as_master, [0] > joins received) > 2016-06-26_09:51:45.40239 [2016-06-26 12:51:45,402][INFO ][http > ] [Glenn Talbot] publish_address {172.25.232.45:9200}, > bound_addresses {172.25.232.45:9200} > 2016-06-26_09:51:45.40350 [2016-06-26 12:51:45,403][INFO ][node > ] [Glenn Talbot] started > 2016-06-26_09:51:45.53808 [2016-06-26 12:51:45,537][INFO ][gateway > ] [Glenn Talbot] recovered [1] indices into cluster_state > 2016-06-26_09:51:45.87525 [2016-06-26 12:51:45,875][INFO > ][cluster.routing.allocation] [Glenn Talbot] Cluster health status changed > from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]). > 2016-06-26_09:57:01.91281 [2016-06-26 12:57:01,912][INFO ][cluster.service > ] [Glenn Talbot] added > {{graylog-c1be9fdd-8c8a-41b1-8a2f-dacbddbc0cc5}{-7icx5UPSrWbs9jqXVE2Mg}{172.25.232.36}{172.25.232.36:9350}{client=true, > > data=false, master=false},}, reason: zen-disco-join(join from > node[{graylog-c1be9fdd-8c8a-41b1-8a2f-dacbddbc0cc5}{-7icx5UPSrWbs9jqXVE2Mg}{172.25.232.36}{172.25.232.36:9350}{client=true, > > data=false, master=false}]) > 2016-06-26_10:17:02.43148 [2016-06-26 13:17:02,428][INFO > ][cluster.metadata ] [Glenn Talbot] [graylog_0] update_mapping > [message] > 2016-06-26_15:35:13.25159 [2016-06-26 18:35:13,250][INFO ][node > ] [Glenn Talbot] stopping ... > 2016-06-26_15:35:13.32027 [2016-06-26 18:35:13,319][INFO ][node > ] [Glenn Talbot] stopped > 2016-06-26_15:35:13.32153 [2016-06-26 18:35:13,320][INFO ][node > ] [Glenn Talbot] closing ... > 2016-06-26_15:35:13.33032 [2016-06-26 18:35:13,329][INFO ][node > ] [Glenn Talbot] closed > 2016-06-26_15:46:49.97957 [2016-06-26 18:46:49,977][INFO ][node > ] [Tether] version[2.3.1], pid[1364], > build[bd98092/2016-04-04T12:25:05Z] > 2016-06-26_15:46:49.97959 [2016-06-26 18:46:49,978][INFO ][node > ] [Tether] initializing ... > 2016-06-26_15:46:50.52052 [2016-06-26 18:46:50,519][INFO ][plugins > ] [Tether] modules [reindex, lang-expression, lang-groovy], > plugins [kopf], sites [kopf] > 2016-06-26_15:46:50.54693 [2016-06-26 18:46:50,546][INFO ][env > ] [Tether] using [1] data paths, mounts [[/ > (/dev/mapper/graylog--vg-root)]], net usable_space [11gb], net total_space > [14.9gb], spins? [possibly], types [ext4] > 2016-06-26_15:46:50.54734 [2016-06-26 18:46:50,546][INFO ][env > ] [Tether] heap size [37.6gb], compressed ordinary object pointers > [false] > 2016-06-26_15:46:50.54871 [2016-06-26 18:46:50,547][WARN ][env > ] [Tether] max file descriptors [64000] for elasticsearch process > likely too low, consider increasing to at least [65536] > 2016-06-26_15:46:53.00370 [2016-06-26 18:46:53,003][INFO ][node > ] [Tether] initialized > 2016-06-26_15:46:53.00560 [2016-06-26 18:46:53,003][INFO ][node > ] [Tether] starting ... > 2016-06-26_15:46:54.29760 [2016-06-26 18:46:54,297][INFO ][transport > ] [Tether] publish_address {172.25.232.45:9300}, bound_addresses > {172.25.232.45:9300} > 2016-06-26_15:46:54.30807 [2016-06-26 18:46:54,307][INFO ][discovery > ] [Tether] graylog-production/kMz-P-dcQZCObsEMIXZtxQ > 2016-06-26_15:47:04.35293 [2016-06-26 18:47:04,352][INFO ][cluster.service > ] [Tether] new_master > {Tether}{kMz-P-dcQZCObsEMIXZtxQ}{172.25.232.45}{172.25.232.45:9300}, > added > {{graylog-a0b12869-11ed-4d89-ae58-dcc7380bc3b8}{KA4cjlTpQTm9Y1Rv5wlVmw}{172.25.232.41}{172.25.232.41:9350}{client=true, > > data=false, > master=false},{graylog-c1be9fdd-8c8a-41b1-8a2f-dacbddbc0cc5}{-7icx5UPSrWbs9jqXVE2Mg}{172.25.232.36}{172.25.232.36:9350}{client=true, > > data=false, > master=false},{graylog-2a340000-d1ba-4f21-a9df-f45901d845b7}{BiWe2Zy2Syaojr9ek0AlJQ}{172.25.232.35}{172.25.232.35:9350}{client=true, > > data=false, master=false},}, reason: zen-disco-join(elected_as_master, [0] > joins received) > 2016-06-26_15:47:04.36292 [2016-06-26 > 18:47:04,362][DEBUG][action.admin.cluster.state] [Tether] no known master > node, scheduling a retry > 2016-06-26_15:47:04.36567 [2016-06-26 > 18:47:04,364][DEBUG][action.admin.cluster.health] [Tether] no known master > node, scheduling a retry > 2016-06-26_15:47:04.38578 [2016-06-26 18:47:04,385][INFO ][http > ] [Tether] publish_address {172.25.232.45:9200}, > bound_addresses {172.25.232.45:9200} > 2016-06-26_15:47:04.38617 [2016-06-26 18:47:04,385][INFO ][node > ] [Tether] started > 2016-06-26_15:47:04.42204 [2016-06-26 18:47:04,421][INFO ][gateway > ] [Tether] recovered [1] indices into cluster_state > 2016-06-26_15:47:04.76029 [2016-06-26 18:47:04,759][INFO > ][cluster.routing.allocation] [Tether] Cluster health status changed from > [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]). > > > > בתאריך יום שני, 27 ביוני 2016 בשעה 14:53:20 UTC+3, מאת Marius Sturm: >> >> Hi, >> this all boils down to an unstable Elasticsearch instance. When Graylog >> is not able to forward log messages to ES it buffers them on disk and tries >> to send them later. This is called journal. >> So when your ES service is not running properly the journal fills up with >> messages. Please take a look into the ES logs to figure out why it has >> problems with message ingestion. You can find them in >> /var/log/graylog/elasticsearch/current >> >> Cheers, >> Marius >> >> >> On 27 June 2016 at 13:39, John <[email protected]> wrote: >> >>> 1 and 4 >>> and the graylog server node is not sending data to elasticsearch >>> I deleted the journal but it doesn't help >>> the problems began few days after I upgraded from 1.3 to 2.0.2 >>> >>> בתאריך יום שני, 27 ביוני 2016 בשעה 14:30:28 UTC+3, מאת Joe K: >>> >>>> Which problem out of 4? >>>> >>>> >>>> On Monday, June 27, 2016 at 2:00:14 PM UTC+3, John wrote: >>>>> >>>>> Hi Joe >>>>> I have exactly the same problem few days after I upgraded from 1.3 to >>>>> 2.0.2 >>>>> Did you managed to fix this issue? >>>>> >>>>> בתאריך יום חמישי, 26 במאי 2016 בשעה 14:02:19 UTC+3, מאת Joe K: >>>>>> >>>>>> >>>>>> - We run it on t2.medium. (4GB RAM, 2 cores) >>>>>> - About 1 incoming message per second. >>>>>> - tried 2.0.0 and now running 2.0.1 >>>>>> >>>>>> Anyone use Image in real world application? Graylog 2.0 image fails >>>>>> after few days. Is this Image problem or Graylog in general? >>>>>> >>>>>> It runs fine for about a week. After that there's errors and search >>>>>> stop working. Search requests timeout. >>>>>> There's many errors and they are very cryptic, google search does not >>>>>> give any solutions how to manage them: >>>>>> >>>>>> >>>>>> *1. After about a week we have error "Uncommited messages deleted >>>>>> from journal"* >>>>>> >>>>>>> Uncommited messages deleted from journal (triggered 9 days ago) >>>>>>> Some messages were deleted from the Graylog journal before they >>>>>>> could be written to Elasticsearch. Please verify that your >>>>>>> Elasticsearch >>>>>>> cluster is healthy and fast enough. You may also want to review your >>>>>>> Graylog journal settings and set a higher limit. (Node: f12... >>>>>> >>>>>> >>>>>> What to do about this? What is "journal"? Google search produce no >>>>>> answers. >>>>>> >>>>>> *2. After about 4 days of clean install it always trigger "Cluster >>>>>> unhealthy"* >>>>>> >>>>>>> "Elasticsearch cluster unhealthy (RED)" >>>>>>> "The Elasticsearch cluster state is RED which means shards are >>>>>>> unassigned. This usually indicates a crashed and corrupt cluster and >>>>>>> needs >>>>>>> to be investigated. Graylog will write into the local disk journal. >>>>>>> Read >>>>>>> how to fix this in the Elasticsearch setup documentation." >>>>>> >>>>>> >>>>>> When you go to that documentation link it says "The red status >>>>>> indicates that some or all of the primary shards are not available. In >>>>>> this >>>>>> state, no searches can be performed until all primary shards are >>>>>> restored." >>>>>> That's it. what are you supposed to do? >>>>>> After long search finally found one solution: this was cured once >>>>>> with *curl -XPUT 'localhost:9200/_settings' -d '{ "index" : { >>>>>> "number_of_replicas" : 0}}'* >>>>>> Next time it happened, we tried the solution again, but response was >>>>>> *{"acknowledged":false}* >>>>>> So what now??? >>>>>> >>>>>> *3. Every time we perform graylog-ctl restart four more unassigled >>>>>> shards appear:* >>>>>> Elasticsearch cluster is yellow. Shards: 20 active, 0 initializing, >>>>>> 0 relocating, 8 unassigned >>>>>> graylog-ctl restart >>>>>> Elasticsearch cluster is yellow. Shards: 20 active, 0 initializing, >>>>>> 0 relocating, 12 unassigned >>>>>> Etc. >>>>>> >>>>>> >>>>>> >>>>>> *4. Journal utilization is too high without any hint on how to set it >>>>>> to higher.* >>>>>> >>>>>>> Journal utilization is too high (triggered 11 days ago) >>>>>>> Journal utilization is too high and may go over the limit soon. >>>>>>> Please verify that your Elasticsearch cluster is healthy and fast >>>>>>> enough. >>>>>>> You may also want to review your Graylog journal settings and set a >>>>>>> higher >>>>>>> limit. (Node: f121 >>>>>> >>>>>> >>>>>> What is this "journal"? and how to set it to "higher"? >>>>>> >>>>>> Please help! >>>>>> >>>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Graylog Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/graylog2/2288cbf2-6f37-4e77-8c32-c50ba64fe71e%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/graylog2/2288cbf2-6f37-4e77-8c32-c50ba64fe71e%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Developer >> >> Tel.: +49 (0)40 609 452 077 >> Fax.: +49 (0)40 609 452 078 >> >> TORCH GmbH - A Graylog Company >> Poolstraße 21 >> 20335 Hamburg >> Germany >> >> https://www.graylog.com <https://www.torch.sh/> >> >> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 >> Geschäftsführer: Lennart Koopmann (CEO) >> > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/3d642265-0ebf-4eed-abef-eca2e2120d35%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
