I'm seeing the same symptoms you are, at least today. If I leave it at last 5 minutes, it starts failing to display results 5 minutes after the last time I recalculate index ranges. So perhaps my overnight issue is more complex but the same root cause? Hard to say. There must be something that is triggering this syndrome though, as I didn't have any problems yesterday with the "last 5 minute" query.
On Wednesday, November 2, 2016 at 12:08:34 PM UTC-4, 123Dev wrote: > > Thanks for the follow up > Although our symptoms are the same, probably the root causes are different. > > In our case, our ES never goes red (not even yellow), and the deflector > alias which is pointing to today's is correct. > The issue at hand is whatever else happens during "Recalculate Index > Ranges" seems to make it work. > but only for the window since last "Recalculate Index Ranges" which > typically would be once a day. > > If I run now, it would work. > but in 5 minutes, it stops working for "show last minute messages" because > 5 minutes has already passed. > But I show for 15 minutes (which hasn't yet passed) I'll see all the > entries just seconds ago. > As time passes on, we have to select a longer range just to get the recent > logs. > > Searching directly on ES, works, so whatever Graylog is doing to get the > search results, appears to be having problems. > All of these are still in the most recent index, so it's not a deflector > alias issues, at least not in our case. > > > > > On Wednesday, November 2, 2016 at 11:03:52 AM UTC-4, > [email protected] wrote: >> >> This seems like a potential bug to me. Due to an unrelated and >> undiagnosed issue, my elasticsearch cluster isn't allocating shards >> successfully every night. So this problem is recurring for me each day in >> the following scenario: >> >> 1. Index rotation set to time based, P1D. >> 2. New index is created, but shard allocation fails for some shards, so >> elasticsearch status goes red. >> 3. Graylog does not move the deflector alias and continues logging in >> yesterday's index. >> 4. Relative searches do not show data UNLESS you change the relative >> search time window to a large enough value to include data from yesterday. >> Then all data shows up. >> 5. Manually recalculating index ranges resolves the issue (at least >> temporarily). >> >> Not super familiar w/ the graylog code base, but this feels like there's >> a bug in the index selection algorithm and/or index range maintenance >> routines. The query analyzer is excluding yesterday's index when in fact >> that is the one that has the events we're looking for. Doing anything to >> trick the query analyzer into including that index makes the query start >> succeeding. >> >> On Tuesday, November 1, 2016 at 2:47:41 PM UTC-4, 123Dev wrote: >>> >>> How often "Recalculate Index Ranges" are automatically performed? >>> What controls that? >>> >>> It appears that when I "recalculate the index Ranges", messages are >>> detected. >>> but 5 minutes later, messages in the last 5m window are gone, obviously >>> 15m window still show data. >>> but as more time passes on, we have to widen the search to a larger >>> window to get the data. >>> >>> Can I (should I?) find a way to trigger "Recalculate Index Ranges" every >>> minute? >>> This command doesn't seem to be doing the job >>> >>> curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild >>> >>> >>> I can tell because I don't see logs in graylog, nor do I see the ui >>> updated with the current timestamp. >>> >>> Our rotation strategy which hasn't changed for the longest time, which >>> is set to: >>> Index rotation strategy: Index Time >>> Rotation period: P1D (1d, a day) >>> Index retention strategy: Delete >>> Max number of indices: 28 >>> >>> So all of the current data should be present in the latest index for 24 >>> hours, why isn't Graylog able to find it, it's not like it is changing. >>> >>> >>> >>> On Tuesday, November 1, 2016 at 1:07:06 PM UTC-4, 123Dev wrote: >>>> >>>> I've been pulling my hair trying to figure this issue out. >>>> I've done countless number of "Recalculate Index Ranges" >>>> In fact maybe my problem is with "Recalculate Index Ranges" because >>>> just before my last try, I was getting messages, and now I don't. >>>> >>>> This is also playing havoc on many of the alerts that we have setup, >>>> because they're all being triggered as the system detecting inactivity. >>>> >>>> I'm beginning to think that this is a bug in Graylog, >>>> I copied the elasticsearch query >>>> ``` >>>> { >>>> "from": 0, >>>> "size": 150, >>>> "query": { >>>> "bool": { >>>> "must": { >>>> "match_all": {} >>>> }, >>>> "filter": { >>>> "bool": { >>>> "must": { >>>> "range": { >>>> "timestamp": { >>>> "from": "2016-11-01 16:15:32.456", >>>> "to": "2016-11-01 16:20:32.456", >>>> "include_lower": true, >>>> "include_upper": true >>>> } >>>> } >>>> } >>>> } >>>> } >>>> } >>>> }, >>>> "sort": [ >>>> { >>>> "timestamp": { >>>> "order": "desc" >>>> } >>>> } >>>> ], >>>> "highlight": { >>>> "require_field_match": false, >>>> "fields": { >>>> "*": { >>>> "fragment_size": 0, >>>> "number_of_fragments": 0 >>>> } >>>> } >>>> } >>>> } >>>> ``` >>>> >>>> and executed directly on the Elasticsearch cluster, and it returned the >>>> results, so why isn't Graylog returning any results? >>>> >>>> As I was posting this, Graylog had reported that: >>>> Range re-calculated 3 hours ago in 75ms. 176 segments, 1 open search >>>> contexts, 0 deleted messages >>>> >>>> Redid the "Recalculate Index Ranges" >>>> Range re-calculated a minute ago in 110ms. 182 segments, 0 open search >>>> contexts, 0 deleted messages >>>> >>>> And I started getting results again. >>>> Though couple of minutes later, NO more messages again. >>>> >>>> The problem is even worse, Statistics on non-numeric fields are >>>> non-existant. >>>> No Total, no cardinality, nada ... >>>> >>>> The only think I can think of that has recently changed is we added >>>> "Elasticsearch client node" for Kibana. >>>> Not sure if that has anything to do with it. >>>> >>>> >>>> >>>> >>>> On Tuesday, November 1, 2016 at 12:25:58 PM UTC-4, >>>> [email protected] wrote: >>>>> >>>>> Surprisingly, I ran into this issue this morning as well. After some >>>>> poking around I found that my current index time ranges hadn't been >>>>> updated >>>>> in 15 hours, which I assume was breaking the logic used to select indices >>>>> for searching. In System -> Indices, it said the index range hadn't been >>>>> updated in 15 hours. I went to System -> Indices and selected >>>>> Maintenance >>>>> -> Recalculate index ranges, and that fixed it. >>>>> >>>>> On Monday, October 31, 2016 at 4:02:44 PM UTC-4, 123Dev wrote: >>>>>> >>>>>> Was this ever resolved. >>>>>> i never had this issue, and been running Graylog for a long time. >>>>>> >>>>>> Just today with the latest Graylog (2.1.1) >>>>>> Search * for Last, 5m 15m, 30m, 1hr don't return and results >>>>>> Search * in the last 2 hours returns messages just few seconds ago, >>>>>> This is across the board, in a clustered environment, with multiple >>>>>> sources. >>>>>> Impossible for all those sources to have the wrong dates. >>>>>> I checked all Graylog nodes ES Nodes and they all have the correct >>>>>> dates / timezones. >>>>>> >>>>>> What gives? >>>>>> Really puzzling. >>>>>> >>>>>> Thanks >>>>>> >>>>>> On Wednesday, April 15, 2015 at 4:22:18 PM UTC-4, Zi Dvbelju wrote: >>>>>>> >>>>>>> Hey Edmundo, >>>>>>> >>>>>>> Graylog user timezone and messages timezone are correct. Has been >>>>>>> setup that way for over a year, and I have confirmed they are still >>>>>>> setup >>>>>>> correctly. >>>>>>> >>>>>>> I upgraded to the latest version hoping that would fix it, but still >>>>>>> running into the same issue. >>>>>>> >>>>>>> Any other suggestions? Error logs don't show any issues. >>>>>>> >>>>>>> Only relative searches return data; nothing from absolute. >>>>>>> >>>>>>> Thanks, >>>>>>> Z >>>>>>> >>>>>>> >>>>>>> On Tuesday, January 20, 2015 at 2:40:53 PM UTC-5, Edmundo Alvarez >>>>>>> wrote: >>>>>>>> >>>>>>>> Hello Zach, >>>>>>>> >>>>>>>> I would start taking a look at the time configuration, specially >>>>>>>> timezones. Could you verify the time settings in the machines sending >>>>>>>> logs, >>>>>>>> and that both your Graylog user's timezone and the messages' timezones >>>>>>>> are >>>>>>>> correct? >>>>>>>> >>>>>>>> I hope that helps. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Edmundo >>>>>>>> >>>>>>>> -- >>>>>>>> Developer >>>>>>>> >>>>>>>> Tel.: +49 (0)40 609 452 077 >>>>>>>> Mobile: +49 (0)171 27 22 181 >>>>>>>> Mobile (US): +1 (713) 321 8126 >>>>>>>> Fax.: +49 (0)40 609 452 078 >>>>>>>> >>>>>>>> TORCH GmbH >>>>>>>> Steckelhörn 11 >>>>>>>> 20457 Hamburg >>>>>>>> Germany >>>>>>>> https://www.torch.sh/ >>>>>>>> >>>>>>>> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 >>>>>>>> Geschäftsführer: Lennart Koopmann (CEO) >>>>>>>> >>>>>>>> > On 20 Jan 2015, at 19:34, Zi Dvbelju <[email protected]> wrote: >>>>>>>> > >>>>>>>> > Hey Team, >>>>>>>> > >>>>>>>> > Running into a strange issue with my Graylog2 setup. >>>>>>>> > >>>>>>>> > Search for anything with "last hour" returns no results. I can >>>>>>>> duplicate the same exact search with "last day" and see messages from >>>>>>>> within the last few seconds. >>>>>>>> > >>>>>>>> > Only searches with the "last hour" tag are returning nothing. >>>>>>>> > >>>>>>>> > v92.3 >>>>>>>> > >>>>>>>> > Thanks, >>>>>>>> > Zach >>>>>>>> > >>>>>>>> > -- >>>>>>>> > You received this message because you are subscribed to the >>>>>>>> Google Groups "graylog2" group. >>>>>>>> > To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> > For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>>> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/3c752d87-633d-4c55-9516-1755aaa26c81%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
