Just as another data point, manually cycling the deflector appears to have fixed the issue.
On Wednesday, November 2, 2016 at 1:21:08 PM UTC-4, daniel...@hagan-consulting.com wrote: > > No, I am using a straight graylog/ES stack, with logstash pre-processing > syslog and submitting to graylog via GELF connection. > > > > On Wednesday, November 2, 2016 at 1:15:59 PM UTC-4, 123Dev wrote: >> >> Just for curiosity, do you use Kibana or Marvel? >> >> As I previously have mentioned it, our problems might be related to the >> usage of Kibana / Marvel, as that's a change we recently introduced. >> >> Stopping Kibana didn't make a difference, neither removing the Marvel >> Agent from ES >> So I dug into ES Indexes / Shards >> I got rid of any kibana and marvel created indexes. >> restarted my services and ran "Recalculate Index Ranges" >> It appears to be working now, and 5m window has passed, I'm hesitantly >> optimistic. >> hardly any solution if you want to use Kibana / Marvel, but if Graylog >> goes back to normal, we'll worry about that next. >> >> >> >> On Wednesday, November 2, 2016 at 12:19:59 PM UTC-4, >> daniel...@hagan-consulting.com wrote: >>> >>> I'm seeing the same symptoms you are, at least today. If I leave it at >>> last 5 minutes, it starts failing to display results 5 minutes after the >>> last time I recalculate index ranges. So perhaps my overnight issue is >>> more complex but the same root cause? Hard to say. There must be >>> something that is triggering this syndrome though, as I didn't have any >>> problems yesterday with the "last 5 minute" query. >>> >>> On Wednesday, November 2, 2016 at 12:08:34 PM UTC-4, 123Dev wrote: >>>> >>>> Thanks for the follow up >>>> Although our symptoms are the same, probably the root causes are >>>> different. >>>> >>>> In our case, our ES never goes red (not even yellow), and the deflector >>>> alias which is pointing to today's is correct. >>>> The issue at hand is whatever else happens during "Recalculate Index >>>> Ranges" seems to make it work. >>>> but only for the window since last "Recalculate Index Ranges" which >>>> typically would be once a day. >>>> >>>> If I run now, it would work. >>>> but in 5 minutes, it stops working for "show last minute messages" >>>> because 5 minutes has already passed. >>>> But I show for 15 minutes (which hasn't yet passed) I'll see all the >>>> entries just seconds ago. >>>> As time passes on, we have to select a longer range just to get the >>>> recent logs. >>>> >>>> Searching directly on ES, works, so whatever Graylog is doing to get >>>> the search results, appears to be having problems. >>>> All of these are still in the most recent index, so it's not a >>>> deflector alias issues, at least not in our case. >>>> >>>> >>>> >>>> >>>> On Wednesday, November 2, 2016 at 11:03:52 AM UTC-4, >>>> daniel...@hagan-consulting.com wrote: >>>>> >>>>> This seems like a potential bug to me. Due to an unrelated and >>>>> undiagnosed issue, my elasticsearch cluster isn't allocating shards >>>>> successfully every night. So this problem is recurring for me each day >>>>> in >>>>> the following scenario: >>>>> >>>>> 1. Index rotation set to time based, P1D. >>>>> 2. New index is created, but shard allocation fails for some shards, >>>>> so elasticsearch status goes red. >>>>> 3. Graylog does not move the deflector alias and continues logging in >>>>> yesterday's index. >>>>> 4. Relative searches do not show data UNLESS you change the relative >>>>> search time window to a large enough value to include data from >>>>> yesterday. >>>>> Then all data shows up. >>>>> 5. Manually recalculating index ranges resolves the issue (at least >>>>> temporarily). >>>>> >>>>> Not super familiar w/ the graylog code base, but this feels like >>>>> there's a bug in the index selection algorithm and/or index range >>>>> maintenance routines. The query analyzer is excluding yesterday's index >>>>> when in fact that is the one that has the events we're looking for. >>>>> Doing >>>>> anything to trick the query analyzer into including that index makes the >>>>> query start succeeding. >>>>> >>>>> On Tuesday, November 1, 2016 at 2:47:41 PM UTC-4, 123Dev wrote: >>>>>> >>>>>> How often "Recalculate Index Ranges" are automatically performed? >>>>>> What controls that? >>>>>> >>>>>> It appears that when I "recalculate the index Ranges", messages are >>>>>> detected. >>>>>> but 5 minutes later, messages in the last 5m window are gone, >>>>>> obviously 15m window still show data. >>>>>> but as more time passes on, we have to widen the search to a larger >>>>>> window to get the data. >>>>>> >>>>>> Can I (should I?) find a way to trigger "Recalculate Index Ranges" >>>>>> every minute? >>>>>> This command doesn't seem to be doing the job >>>>>> >>>>>> curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild >>>>>> >>>>>> >>>>>> I can tell because I don't see logs in graylog, nor do I see the ui >>>>>> updated with the current timestamp. >>>>>> >>>>>> Our rotation strategy which hasn't changed for the longest time, >>>>>> which is set to: >>>>>> Index rotation strategy: Index Time >>>>>> Rotation period: P1D (1d, a day) >>>>>> Index retention strategy: Delete >>>>>> Max number of indices: 28 >>>>>> >>>>>> So all of the current data should be present in the latest index for >>>>>> 24 hours, why isn't Graylog able to find it, it's not like it is >>>>>> changing. >>>>>> >>>>>> >>>>>> >>>>>> On Tuesday, November 1, 2016 at 1:07:06 PM UTC-4, 123Dev wrote: >>>>>>> >>>>>>> I've been pulling my hair trying to figure this issue out. >>>>>>> I've done countless number of "Recalculate Index Ranges" >>>>>>> In fact maybe my problem is with "Recalculate Index Ranges" because >>>>>>> just before my last try, I was getting messages, and now I don't. >>>>>>> >>>>>>> This is also playing havoc on many of the alerts that we have setup, >>>>>>> because they're all being triggered as the system detecting inactivity. >>>>>>> >>>>>>> I'm beginning to think that this is a bug in Graylog, >>>>>>> I copied the elasticsearch query >>>>>>> ``` >>>>>>> { >>>>>>> "from": 0, >>>>>>> "size": 150, >>>>>>> "query": { >>>>>>> "bool": { >>>>>>> "must": { >>>>>>> "match_all": {} >>>>>>> }, >>>>>>> "filter": { >>>>>>> "bool": { >>>>>>> "must": { >>>>>>> "range": { >>>>>>> "timestamp": { >>>>>>> "from": "2016-11-01 16:15:32.456", >>>>>>> "to": "2016-11-01 16:20:32.456", >>>>>>> "include_lower": true, >>>>>>> "include_upper": true >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> }, >>>>>>> "sort": [ >>>>>>> { >>>>>>> "timestamp": { >>>>>>> "order": "desc" >>>>>>> } >>>>>>> } >>>>>>> ], >>>>>>> "highlight": { >>>>>>> "require_field_match": false, >>>>>>> "fields": { >>>>>>> "*": { >>>>>>> "fragment_size": 0, >>>>>>> "number_of_fragments": 0 >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> ``` >>>>>>> >>>>>>> and executed directly on the Elasticsearch cluster, and it returned >>>>>>> the results, so why isn't Graylog returning any results? >>>>>>> >>>>>>> As I was posting this, Graylog had reported that: >>>>>>> Range re-calculated 3 hours ago in 75ms. 176 segments, 1 open >>>>>>> search contexts, 0 deleted messages >>>>>>> >>>>>>> Redid the "Recalculate Index Ranges" >>>>>>> Range re-calculated a minute ago in 110ms. 182 segments, 0 open >>>>>>> search contexts, 0 deleted messages >>>>>>> >>>>>>> And I started getting results again. >>>>>>> Though couple of minutes later, NO more messages again. >>>>>>> >>>>>>> The problem is even worse, Statistics on non-numeric fields are >>>>>>> non-existant. >>>>>>> No Total, no cardinality, nada ... >>>>>>> >>>>>>> The only think I can think of that has recently changed is we added >>>>>>> "Elasticsearch client node" for Kibana. >>>>>>> Not sure if that has anything to do with it. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tuesday, November 1, 2016 at 12:25:58 PM UTC-4, >>>>>>> daniel...@hagan-consulting.com wrote: >>>>>>>> >>>>>>>> Surprisingly, I ran into this issue this morning as well. After >>>>>>>> some poking around I found that my current index time ranges hadn't >>>>>>>> been >>>>>>>> updated in 15 hours, which I assume was breaking the logic used to >>>>>>>> select >>>>>>>> indices for searching. In System -> Indices, it said the index range >>>>>>>> hadn't been updated in 15 hours. I went to System -> Indices and >>>>>>>> selected >>>>>>>> Maintenance -> Recalculate index ranges, and that fixed it. >>>>>>>> >>>>>>>> On Monday, October 31, 2016 at 4:02:44 PM UTC-4, 123Dev wrote: >>>>>>>>> >>>>>>>>> Was this ever resolved. >>>>>>>>> i never had this issue, and been running Graylog for a long time. >>>>>>>>> >>>>>>>>> Just today with the latest Graylog (2.1.1) >>>>>>>>> Search * for Last, 5m 15m, 30m, 1hr don't return and results >>>>>>>>> Search * in the last 2 hours returns messages just few seconds ago, >>>>>>>>> This is across the board, in a clustered environment, with >>>>>>>>> multiple sources. >>>>>>>>> Impossible for all those sources to have the wrong dates. >>>>>>>>> I checked all Graylog nodes ES Nodes and they all have the correct >>>>>>>>> dates / timezones. >>>>>>>>> >>>>>>>>> What gives? >>>>>>>>> Really puzzling. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> On Wednesday, April 15, 2015 at 4:22:18 PM UTC-4, Zi Dvbelju wrote: >>>>>>>>>> >>>>>>>>>> Hey Edmundo, >>>>>>>>>> >>>>>>>>>> Graylog user timezone and messages timezone are correct. Has been >>>>>>>>>> setup that way for over a year, and I have confirmed they are still >>>>>>>>>> setup >>>>>>>>>> correctly. >>>>>>>>>> >>>>>>>>>> I upgraded to the latest version hoping that would fix it, but >>>>>>>>>> still running into the same issue. >>>>>>>>>> >>>>>>>>>> Any other suggestions? Error logs don't show any issues. >>>>>>>>>> >>>>>>>>>> Only relative searches return data; nothing from absolute. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Z >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tuesday, January 20, 2015 at 2:40:53 PM UTC-5, Edmundo Alvarez >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hello Zach, >>>>>>>>>>> >>>>>>>>>>> I would start taking a look at the time configuration, specially >>>>>>>>>>> timezones. Could you verify the time settings in the machines >>>>>>>>>>> sending logs, >>>>>>>>>>> and that both your Graylog user's timezone and the messages' >>>>>>>>>>> timezones are >>>>>>>>>>> correct? >>>>>>>>>>> >>>>>>>>>>> I hope that helps. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Edmundo >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Developer >>>>>>>>>>> >>>>>>>>>>> Tel.: +49 (0)40 609 452 077 >>>>>>>>>>> Mobile: +49 (0)171 27 22 181 >>>>>>>>>>> Mobile (US): +1 (713) 321 8126 >>>>>>>>>>> Fax.: +49 (0)40 609 452 078 >>>>>>>>>>> >>>>>>>>>>> TORCH GmbH >>>>>>>>>>> Steckelhörn 11 >>>>>>>>>>> 20457 Hamburg >>>>>>>>>>> Germany >>>>>>>>>>> https://www.torch.sh/ >>>>>>>>>>> >>>>>>>>>>> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB >>>>>>>>>>> 125175 >>>>>>>>>>> Geschäftsführer: Lennart Koopmann (CEO) >>>>>>>>>>> >>>>>>>>>>> > On 20 Jan 2015, at 19:34, Zi Dvbelju <zidv...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> > >>>>>>>>>>> > Hey Team, >>>>>>>>>>> > >>>>>>>>>>> > Running into a strange issue with my Graylog2 setup. >>>>>>>>>>> > >>>>>>>>>>> > Search for anything with "last hour" returns no results. I can >>>>>>>>>>> duplicate the same exact search with "last day" and see messages >>>>>>>>>>> from >>>>>>>>>>> within the last few seconds. >>>>>>>>>>> > >>>>>>>>>>> > Only searches with the "last hour" tag are returning nothing. >>>>>>>>>>> > >>>>>>>>>>> > v92.3 >>>>>>>>>>> > >>>>>>>>>>> > Thanks, >>>>>>>>>>> > Zach >>>>>>>>>>> > >>>>>>>>>>> > -- >>>>>>>>>>> > You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "graylog2" group. >>>>>>>>>>> > To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to graylog2+u...@googlegroups.com. >>>>>>>>>>> > For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>>> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/003e5484-f674-4ab9-86a4-3bb9591c2e86%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
