Just as another data point, manually cycling the deflector appears to have fixed the issue.
On Wednesday, November 2, 2016 at 1:21:08 PM UTC-4, [email protected] wrote: > > No, I am using a straight graylog/ES stack, with logstash pre-processing > syslog and submitting to graylog via GELF connection. > > > > On Wednesday, November 2, 2016 at 1:15:59 PM UTC-4, 123Dev wrote: >> >> Just for curiosity, do you use Kibana or Marvel? >> >> As I previously have mentioned it, our problems might be related to the >> usage of Kibana / Marvel, as that's a change we recently introduced. >> >> Stopping Kibana didn't make a difference, neither removing the Marvel >> Agent from ES >> So I dug into ES Indexes / Shards >> I got rid of any kibana and marvel created indexes. >> restarted my services and ran "Recalculate Index Ranges" >> It appears to be working now, and 5m window has passed, I'm hesitantly >> optimistic. >> hardly any solution if you want to use Kibana / Marvel, but if Graylog >> goes back to normal, we'll worry about that next. >> >> >> >> On Wednesday, November 2, 2016 at 12:19:59 PM UTC-4, >> [email protected] wrote: >>> >>> I'm seeing the same symptoms you are, at least today. If I leave it at >>> last 5 minutes, it starts failing to display results 5 minutes after the >>> last time I recalculate index ranges. So perhaps my overnight issue is >>> more complex but the same root cause? Hard to say. There must be >>> something that is triggering this syndrome though, as I didn't have any >>> problems yesterday with the "last 5 minute" query. >>> >>> On Wednesday, November 2, 2016 at 12:08:34 PM UTC-4, 123Dev wrote: >>>> >>>> Thanks for the follow up >>>> Although our symptoms are the same, probably the root causes are >>>> different. >>>> >>>> In our case, our ES never goes red (not even yellow), and the deflector >>>> alias which is pointing to today's is correct. >>>> The issue at hand is whatever else happens during "Recalculate Index >>>> Ranges" seems to make it work. >>>> but only for the window since last "Recalculate Index Ranges" which >>>> typically would be once a day. >>>> >>>> If I run now, it would work. >>>> but in 5 minutes, it stops working for "show last minute messages" >>>> because 5 minutes has already passed. >>>> But I show for 15 minutes (which hasn't yet passed) I'll see all the >>>> entries just seconds ago. >>>> As time passes on, we have to select a longer range just to get the >>>> recent logs. >>>> >>>> Searching directly on ES, works, so whatever Graylog is doing to get >>>> the search results, appears to be having problems. >>>> All of these are still in the most recent index, so it's not a >>>> deflector alias issues, at least not in our case. >>>> >>>> >>>> >>>> >>>> On Wednesday, November 2, 2016 at 11:03:52 AM UTC-4, >>>> [email protected] wrote: >>>>> >>>>> This seems like a potential bug to me. Due to an unrelated and >>>>> undiagnosed issue, my elasticsearch cluster isn't allocating shards >>>>> successfully every night. So this problem is recurring for me each day >>>>> in >>>>> the following scenario: >>>>> >>>>> 1. Index rotation set to time based, P1D. >>>>> 2. New index is created, but shard allocation fails for some shards, >>>>> so elasticsearch status goes red. >>>>> 3. Graylog does not move the deflector alias and continues logging in >>>>> yesterday's index. >>>>> 4. Relative searches do not show data UNLESS you change the relative >>>>> search time window to a large enough value to include data from >>>>> yesterday. >>>>> Then all data shows up. >>>>> 5. Manually recalculating index ranges resolves the issue (at least >>>>> temporarily). >>>>> >>>>> Not super familiar w/ the graylog code base, but this feels like >>>>> there's a bug in the index selection algorithm and/or index range >>>>> maintenance routines. The query analyzer is excluding yesterday's index >>>>> when in fact that is the one that has the events we're looking for. >>>>> Doing >>>>> anything to trick the query analyzer into including that index makes the >>>>> query start succeeding. >>>>> >>>>> On Tuesday, November 1, 2016 at 2:47:41 PM UTC-4, 123Dev wrote: >>>>>> >>>>>> How often "Recalculate Index Ranges" are automatically performed? >>>>>> What controls that? >>>>>> >>>>>> It appears that when I "recalculate the index Ranges", messages are >>>>>> detected. >>>>>> but 5 minutes later, messages in the last 5m window are gone, >>>>>> obviously 15m window still show data. >>>>>> but as more time passes on, we have to widen the search to a larger >>>>>> window to get the data. >>>>>> >>>>>> Can I (should I?) find a way to trigger "Recalculate Index Ranges" >>>>>> every minute? >>>>>> This command doesn't seem to be doing the job >>>>>> >>>>>> curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild >>>>>> >>>>>> >>>>>> I can tell because I don't see logs in graylog, nor do I see the ui >>>>>> updated with the current timestamp. >>>>>> >>>>>> Our rotation strategy which hasn't changed for the longest time, >>>>>> which is set to: >>>>>> Index rotation strategy: Index Time >>>>>> Rotation period: P1D (1d, a day) >>>>>> Index retention strategy: Delete >>>>>> Max number of indices: 28 >>>>>> >>>>>> So all of the current data should be present in the latest index for >>>>>> 24 hours, why isn't Graylog able to find it, it's not like it is >>>>>> changing. >>>>>> >>>>>> >>>>>> >>>>>> On Tuesday, November 1, 2016 at 1:07:06 PM UTC-4, 123Dev wrote: >>>>>>> >>>>>>> I've been pulling my hair trying to figure this issue out. >>>>>>> I've done countless number of "Recalculate Index Ranges" >>>>>>> In fact maybe my problem is with "Recalculate Index Ranges" because >>>>>>> just before my last try, I was getting messages, and now I don't. >>>>>>> >>>>>>> This is also playing havoc on many of the alerts that we have setup, >>>>>>> because they're all being triggered as the system detecting inactivity. >>>>>>> >>>>>>> I'm beginning to think that this is a bug in Graylog, >>>>>>> I copied the elasticsearch query >>>>>>> ``` >>>>>>> { >>>>>>> "from": 0, >>>>>>> "size": 150, >>>>>>> "query": { >>>>>>> "bool": { >>>>>>> "must": { >>>>>>> "match_all": {} >>>>>>> }, >>>>>>> "filter": { >>>>>>> "bool": { >>>>>>> "must": { >>>>>>> "range": { >>>>>>> "timestamp": { >>>>>>> "from": "2016-11-01 16:15:32.456", >>>>>>> "to": "2016-11-01 16:20:32.456", >>>>>>> "include_lower": true, >>>>>>> "include_upper": true >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> }, >>>>>>> "sort": [ >>>>>>> { >>>>>>> "timestamp": { >>>>>>> "order": "desc" >>>>>>> } >>>>>>> } >>>>>>> ], >>>>>>> "highlight": { >>>>>>> "require_field_match": false, >>>>>>> "fields": { >>>>>>> "*": { >>>>>>> "fragment_size": 0, >>>>>>> "number_of_fragments": 0 >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> ``` >>>>>>> >>>>>>> and executed directly on the Elasticsearch cluster, and it returned >>>>>>> the results, so why isn't Graylog returning any results? >>>>>>> >>>>>>> As I was posting this, Graylog had reported that: >>>>>>> Range re-calculated 3 hours ago in 75ms. 176 segments, 1 open >>>>>>> search contexts, 0 deleted messages >>>>>>> >>>>>>> Redid the "Recalculate Index Ranges" >>>>>>> Range re-calculated a minute ago in 110ms. 182 segments, 0 open >>>>>>> search contexts, 0 deleted messages >>>>>>> >>>>>>> And I started getting results again. >>>>>>> Though couple of minutes later, NO more messages again. >>>>>>> >>>>>>> The problem is even worse, Statistics on non-numeric fields are >>>>>>> non-existant. >>>>>>> No Total, no cardinality, nada ... >>>>>>> >>>>>>> The only think I can think of that has recently changed is we added >>>>>>> "Elasticsearch client node" for Kibana. >>>>>>> Not sure if that has anything to do with it. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tuesday, November 1, 2016 at 12:25:58 PM UTC-4, >>>>>>> [email protected] wrote: >>>>>>>> >>>>>>>> Surprisingly, I ran into this issue this morning as well. After >>>>>>>> some poking around I found that my current index time ranges hadn't >>>>>>>> been >>>>>>>> updated in 15 hours, which I assume was breaking the logic used to >>>>>>>> select >>>>>>>> indices for searching. In System -> Indices, it said the index range >>>>>>>> hadn't been updated in 15 hours. I went to System -> Indices and >>>>>>>> selected >>>>>>>> Maintenance -> Recalculate index ranges, and that fixed it. >>>>>>>> >>>>>>>> On Monday, October 31, 2016 at 4:02:44 PM UTC-4, 123Dev wrote: >>>>>>>>> >>>>>>>>> Was this ever resolved. >>>>>>>>> i never had this issue, and been running Graylog for a long time. >>>>>>>>> >>>>>>>>> Just today with the latest Graylog (2.1.1) >>>>>>>>> Search * for Last, 5m 15m, 30m, 1hr don't return and results >>>>>>>>> Search * in the last 2 hours returns messages just few seconds ago, >>>>>>>>> This is across the board, in a clustered environment, with >>>>>>>>> multiple sources. >>>>>>>>> Impossible for all those sources to have the wrong dates. >>>>>>>>> I checked all Graylog nodes ES Nodes and they all have the correct >>>>>>>>> dates / timezones. >>>>>>>>> >>>>>>>>> What gives? >>>>>>>>> Really puzzling. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> On Wednesday, April 15, 2015 at 4:22:18 PM UTC-4, Zi Dvbelju wrote: >>>>>>>>>> >>>>>>>>>> Hey Edmundo, >>>>>>>>>> >>>>>>>>>> Graylog user timezone and messages timezone are correct. Has been >>>>>>>>>> setup that way for over a year, and I have confirmed they are still >>>>>>>>>> setup >>>>>>>>>> correctly. >>>>>>>>>> >>>>>>>>>> I upgraded to the latest version hoping that would fix it, but >>>>>>>>>> still running into the same issue. >>>>>>>>>> >>>>>>>>>> Any other suggestions? Error logs don't show any issues. >>>>>>>>>> >>>>>>>>>> Only relative searches return data; nothing from absolute. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Z >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tuesday, January 20, 2015 at 2:40:53 PM UTC-5, Edmundo Alvarez >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hello Zach, >>>>>>>>>>> >>>>>>>>>>> I would start taking a look at the time configuration, specially >>>>>>>>>>> timezones. Could you verify the time settings in the machines >>>>>>>>>>> sending logs, >>>>>>>>>>> and that both your Graylog user's timezone and the messages' >>>>>>>>>>> timezones are >>>>>>>>>>> correct? >>>>>>>>>>> >>>>>>>>>>> I hope that helps. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Edmundo >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Developer >>>>>>>>>>> >>>>>>>>>>> Tel.: +49 (0)40 609 452 077 >>>>>>>>>>> Mobile: +49 (0)171 27 22 181 >>>>>>>>>>> Mobile (US): +1 (713) 321 8126 >>>>>>>>>>> Fax.: +49 (0)40 609 452 078 >>>>>>>>>>> >>>>>>>>>>> TORCH GmbH >>>>>>>>>>> Steckelhörn 11 >>>>>>>>>>> 20457 Hamburg >>>>>>>>>>> Germany >>>>>>>>>>> https://www.torch.sh/ >>>>>>>>>>> >>>>>>>>>>> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB >>>>>>>>>>> 125175 >>>>>>>>>>> Geschäftsführer: Lennart Koopmann (CEO) >>>>>>>>>>> >>>>>>>>>>> > On 20 Jan 2015, at 19:34, Zi Dvbelju <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> > >>>>>>>>>>> > Hey Team, >>>>>>>>>>> > >>>>>>>>>>> > Running into a strange issue with my Graylog2 setup. >>>>>>>>>>> > >>>>>>>>>>> > Search for anything with "last hour" returns no results. I can >>>>>>>>>>> duplicate the same exact search with "last day" and see messages >>>>>>>>>>> from >>>>>>>>>>> within the last few seconds. >>>>>>>>>>> > >>>>>>>>>>> > Only searches with the "last hour" tag are returning nothing. >>>>>>>>>>> > >>>>>>>>>>> > v92.3 >>>>>>>>>>> > >>>>>>>>>>> > Thanks, >>>>>>>>>>> > Zach >>>>>>>>>>> > >>>>>>>>>>> > -- >>>>>>>>>>> > You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "graylog2" group. >>>>>>>>>>> > To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> > For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>>> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/003e5484-f674-4ab9-86a4-3bb9591c2e86%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
