No, I am using a straight graylog/ES stack, with logstash pre-processing syslog and submitting to graylog via GELF connection.
On Wednesday, November 2, 2016 at 1:15:59 PM UTC-4, 123Dev wrote: > > Just for curiosity, do you use Kibana or Marvel? > > As I previously have mentioned it, our problems might be related to the > usage of Kibana / Marvel, as that's a change we recently introduced. > > Stopping Kibana didn't make a difference, neither removing the Marvel > Agent from ES > So I dug into ES Indexes / Shards > I got rid of any kibana and marvel created indexes. > restarted my services and ran "Recalculate Index Ranges" > It appears to be working now, and 5m window has passed, I'm hesitantly > optimistic. > hardly any solution if you want to use Kibana / Marvel, but if Graylog > goes back to normal, we'll worry about that next. > > > > On Wednesday, November 2, 2016 at 12:19:59 PM UTC-4, > [email protected] wrote: >> >> I'm seeing the same symptoms you are, at least today. If I leave it at >> last 5 minutes, it starts failing to display results 5 minutes after the >> last time I recalculate index ranges. So perhaps my overnight issue is >> more complex but the same root cause? Hard to say. There must be >> something that is triggering this syndrome though, as I didn't have any >> problems yesterday with the "last 5 minute" query. >> >> On Wednesday, November 2, 2016 at 12:08:34 PM UTC-4, 123Dev wrote: >>> >>> Thanks for the follow up >>> Although our symptoms are the same, probably the root causes are >>> different. >>> >>> In our case, our ES never goes red (not even yellow), and the deflector >>> alias which is pointing to today's is correct. >>> The issue at hand is whatever else happens during "Recalculate Index >>> Ranges" seems to make it work. >>> but only for the window since last "Recalculate Index Ranges" which >>> typically would be once a day. >>> >>> If I run now, it would work. >>> but in 5 minutes, it stops working for "show last minute messages" >>> because 5 minutes has already passed. >>> But I show for 15 minutes (which hasn't yet passed) I'll see all the >>> entries just seconds ago. >>> As time passes on, we have to select a longer range just to get the >>> recent logs. >>> >>> Searching directly on ES, works, so whatever Graylog is doing to get the >>> search results, appears to be having problems. >>> All of these are still in the most recent index, so it's not a deflector >>> alias issues, at least not in our case. >>> >>> >>> >>> >>> On Wednesday, November 2, 2016 at 11:03:52 AM UTC-4, >>> [email protected] wrote: >>>> >>>> This seems like a potential bug to me. Due to an unrelated and >>>> undiagnosed issue, my elasticsearch cluster isn't allocating shards >>>> successfully every night. So this problem is recurring for me each day in >>>> the following scenario: >>>> >>>> 1. Index rotation set to time based, P1D. >>>> 2. New index is created, but shard allocation fails for some shards, so >>>> elasticsearch status goes red. >>>> 3. Graylog does not move the deflector alias and continues logging in >>>> yesterday's index. >>>> 4. Relative searches do not show data UNLESS you change the relative >>>> search time window to a large enough value to include data from yesterday. >>>> Then all data shows up. >>>> 5. Manually recalculating index ranges resolves the issue (at least >>>> temporarily). >>>> >>>> Not super familiar w/ the graylog code base, but this feels like >>>> there's a bug in the index selection algorithm and/or index range >>>> maintenance routines. The query analyzer is excluding yesterday's index >>>> when in fact that is the one that has the events we're looking for. Doing >>>> anything to trick the query analyzer into including that index makes the >>>> query start succeeding. >>>> >>>> On Tuesday, November 1, 2016 at 2:47:41 PM UTC-4, 123Dev wrote: >>>>> >>>>> How often "Recalculate Index Ranges" are automatically performed? >>>>> What controls that? >>>>> >>>>> It appears that when I "recalculate the index Ranges", messages are >>>>> detected. >>>>> but 5 minutes later, messages in the last 5m window are gone, >>>>> obviously 15m window still show data. >>>>> but as more time passes on, we have to widen the search to a larger >>>>> window to get the data. >>>>> >>>>> Can I (should I?) find a way to trigger "Recalculate Index Ranges" >>>>> every minute? >>>>> This command doesn't seem to be doing the job >>>>> >>>>> curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild >>>>> >>>>> >>>>> I can tell because I don't see logs in graylog, nor do I see the ui >>>>> updated with the current timestamp. >>>>> >>>>> Our rotation strategy which hasn't changed for the longest time, which >>>>> is set to: >>>>> Index rotation strategy: Index Time >>>>> Rotation period: P1D (1d, a day) >>>>> Index retention strategy: Delete >>>>> Max number of indices: 28 >>>>> >>>>> So all of the current data should be present in the latest index for >>>>> 24 hours, why isn't Graylog able to find it, it's not like it is changing. >>>>> >>>>> >>>>> >>>>> On Tuesday, November 1, 2016 at 1:07:06 PM UTC-4, 123Dev wrote: >>>>>> >>>>>> I've been pulling my hair trying to figure this issue out. >>>>>> I've done countless number of "Recalculate Index Ranges" >>>>>> In fact maybe my problem is with "Recalculate Index Ranges" because >>>>>> just before my last try, I was getting messages, and now I don't. >>>>>> >>>>>> This is also playing havoc on many of the alerts that we have setup, >>>>>> because they're all being triggered as the system detecting inactivity. >>>>>> >>>>>> I'm beginning to think that this is a bug in Graylog, >>>>>> I copied the elasticsearch query >>>>>> ``` >>>>>> { >>>>>> "from": 0, >>>>>> "size": 150, >>>>>> "query": { >>>>>> "bool": { >>>>>> "must": { >>>>>> "match_all": {} >>>>>> }, >>>>>> "filter": { >>>>>> "bool": { >>>>>> "must": { >>>>>> "range": { >>>>>> "timestamp": { >>>>>> "from": "2016-11-01 16:15:32.456", >>>>>> "to": "2016-11-01 16:20:32.456", >>>>>> "include_lower": true, >>>>>> "include_upper": true >>>>>> } >>>>>> } >>>>>> } >>>>>> } >>>>>> } >>>>>> } >>>>>> }, >>>>>> "sort": [ >>>>>> { >>>>>> "timestamp": { >>>>>> "order": "desc" >>>>>> } >>>>>> } >>>>>> ], >>>>>> "highlight": { >>>>>> "require_field_match": false, >>>>>> "fields": { >>>>>> "*": { >>>>>> "fragment_size": 0, >>>>>> "number_of_fragments": 0 >>>>>> } >>>>>> } >>>>>> } >>>>>> } >>>>>> ``` >>>>>> >>>>>> and executed directly on the Elasticsearch cluster, and it returned >>>>>> the results, so why isn't Graylog returning any results? >>>>>> >>>>>> As I was posting this, Graylog had reported that: >>>>>> Range re-calculated 3 hours ago in 75ms. 176 segments, 1 open search >>>>>> contexts, 0 deleted messages >>>>>> >>>>>> Redid the "Recalculate Index Ranges" >>>>>> Range re-calculated a minute ago in 110ms. 182 segments, 0 open >>>>>> search contexts, 0 deleted messages >>>>>> >>>>>> And I started getting results again. >>>>>> Though couple of minutes later, NO more messages again. >>>>>> >>>>>> The problem is even worse, Statistics on non-numeric fields are >>>>>> non-existant. >>>>>> No Total, no cardinality, nada ... >>>>>> >>>>>> The only think I can think of that has recently changed is we added >>>>>> "Elasticsearch client node" for Kibana. >>>>>> Not sure if that has anything to do with it. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tuesday, November 1, 2016 at 12:25:58 PM UTC-4, >>>>>> [email protected] wrote: >>>>>>> >>>>>>> Surprisingly, I ran into this issue this morning as well. After >>>>>>> some poking around I found that my current index time ranges hadn't >>>>>>> been >>>>>>> updated in 15 hours, which I assume was breaking the logic used to >>>>>>> select >>>>>>> indices for searching. In System -> Indices, it said the index range >>>>>>> hadn't been updated in 15 hours. I went to System -> Indices and >>>>>>> selected >>>>>>> Maintenance -> Recalculate index ranges, and that fixed it. >>>>>>> >>>>>>> On Monday, October 31, 2016 at 4:02:44 PM UTC-4, 123Dev wrote: >>>>>>>> >>>>>>>> Was this ever resolved. >>>>>>>> i never had this issue, and been running Graylog for a long time. >>>>>>>> >>>>>>>> Just today with the latest Graylog (2.1.1) >>>>>>>> Search * for Last, 5m 15m, 30m, 1hr don't return and results >>>>>>>> Search * in the last 2 hours returns messages just few seconds ago, >>>>>>>> This is across the board, in a clustered environment, with multiple >>>>>>>> sources. >>>>>>>> Impossible for all those sources to have the wrong dates. >>>>>>>> I checked all Graylog nodes ES Nodes and they all have the correct >>>>>>>> dates / timezones. >>>>>>>> >>>>>>>> What gives? >>>>>>>> Really puzzling. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> On Wednesday, April 15, 2015 at 4:22:18 PM UTC-4, Zi Dvbelju wrote: >>>>>>>>> >>>>>>>>> Hey Edmundo, >>>>>>>>> >>>>>>>>> Graylog user timezone and messages timezone are correct. Has been >>>>>>>>> setup that way for over a year, and I have confirmed they are still >>>>>>>>> setup >>>>>>>>> correctly. >>>>>>>>> >>>>>>>>> I upgraded to the latest version hoping that would fix it, but >>>>>>>>> still running into the same issue. >>>>>>>>> >>>>>>>>> Any other suggestions? Error logs don't show any issues. >>>>>>>>> >>>>>>>>> Only relative searches return data; nothing from absolute. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Z >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tuesday, January 20, 2015 at 2:40:53 PM UTC-5, Edmundo Alvarez >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hello Zach, >>>>>>>>>> >>>>>>>>>> I would start taking a look at the time configuration, specially >>>>>>>>>> timezones. Could you verify the time settings in the machines >>>>>>>>>> sending logs, >>>>>>>>>> and that both your Graylog user's timezone and the messages' >>>>>>>>>> timezones are >>>>>>>>>> correct? >>>>>>>>>> >>>>>>>>>> I hope that helps. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Edmundo >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Developer >>>>>>>>>> >>>>>>>>>> Tel.: +49 (0)40 609 452 077 >>>>>>>>>> Mobile: +49 (0)171 27 22 181 >>>>>>>>>> Mobile (US): +1 (713) 321 8126 >>>>>>>>>> Fax.: +49 (0)40 609 452 078 >>>>>>>>>> >>>>>>>>>> TORCH GmbH >>>>>>>>>> Steckelhörn 11 >>>>>>>>>> 20457 Hamburg >>>>>>>>>> Germany >>>>>>>>>> https://www.torch.sh/ >>>>>>>>>> >>>>>>>>>> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB >>>>>>>>>> 125175 >>>>>>>>>> Geschäftsführer: Lennart Koopmann (CEO) >>>>>>>>>> >>>>>>>>>> > On 20 Jan 2015, at 19:34, Zi Dvbelju <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> > >>>>>>>>>> > Hey Team, >>>>>>>>>> > >>>>>>>>>> > Running into a strange issue with my Graylog2 setup. >>>>>>>>>> > >>>>>>>>>> > Search for anything with "last hour" returns no results. I can >>>>>>>>>> duplicate the same exact search with "last day" and see messages >>>>>>>>>> from >>>>>>>>>> within the last few seconds. >>>>>>>>>> > >>>>>>>>>> > Only searches with the "last hour" tag are returning nothing. >>>>>>>>>> > >>>>>>>>>> > v92.3 >>>>>>>>>> > >>>>>>>>>> > Thanks, >>>>>>>>>> > Zach >>>>>>>>>> > >>>>>>>>>> > -- >>>>>>>>>> > You received this message because you are subscribed to the >>>>>>>>>> Google Groups "graylog2" group. >>>>>>>>>> > To unsubscribe from this group and stop receiving emails from >>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>> > For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>>> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/4f9c9564-7e17-4973-a7fe-251116e16641%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
