Re: [graylog2] Searching "Last Hour" returns no results
Just as another data point, manually cycling the deflector appears to have fixed the issue. On Wednesday, November 2, 2016 at 1:21:08 PM UTC-4, daniel...@hagan-consulting.com wrote: > > No, I am using a straight graylog/ES stack, with logstash pre-processing > syslog and submitting to graylog via GELF connection. > > > > On Wednesday, November 2, 2016 at 1:15:59 PM UTC-4, 123Dev wrote: >> >> Just for curiosity, do you use Kibana or Marvel? >> >> As I previously have mentioned it, our problems might be related to the >> usage of Kibana / Marvel, as that's a change we recently introduced. >> >> Stopping Kibana didn't make a difference, neither removing the Marvel >> Agent from ES >> So I dug into ES Indexes / Shards >> I got rid of any kibana and marvel created indexes. >> restarted my services and ran "Recalculate Index Ranges" >> It appears to be working now, and 5m window has passed, I'm hesitantly >> optimistic. >> hardly any solution if you want to use Kibana / Marvel, but if Graylog >> goes back to normal, we'll worry about that next. >> >> >> >> On Wednesday, November 2, 2016 at 12:19:59 PM UTC-4, >> daniel...@hagan-consulting.com wrote: >>> >>> I'm seeing the same symptoms you are, at least today. If I leave it at >>> last 5 minutes, it starts failing to display results 5 minutes after the >>> last time I recalculate index ranges. So perhaps my overnight issue is >>> more complex but the same root cause? Hard to say. There must be >>> something that is triggering this syndrome though, as I didn't have any >>> problems yesterday with the "last 5 minute" query. >>> >>> On Wednesday, November 2, 2016 at 12:08:34 PM UTC-4, 123Dev wrote: Thanks for the follow up Although our symptoms are the same, probably the root causes are different. In our case, our ES never goes red (not even yellow), and the deflector alias which is pointing to today's is correct. The issue at hand is whatever else happens during "Recalculate Index Ranges" seems to make it work. but only for the window since last "Recalculate Index Ranges" which typically would be once a day. If I run now, it would work. but in 5 minutes, it stops working for "show last minute messages" because 5 minutes has already passed. But I show for 15 minutes (which hasn't yet passed) I'll see all the entries just seconds ago. As time passes on, we have to select a longer range just to get the recent logs. Searching directly on ES, works, so whatever Graylog is doing to get the search results, appears to be having problems. All of these are still in the most recent index, so it's not a deflector alias issues, at least not in our case. On Wednesday, November 2, 2016 at 11:03:52 AM UTC-4, daniel...@hagan-consulting.com wrote: > > This seems like a potential bug to me. Due to an unrelated and > undiagnosed issue, my elasticsearch cluster isn't allocating shards > successfully every night. So this problem is recurring for me each day > in > the following scenario: > > 1. Index rotation set to time based, P1D. > 2. New index is created, but shard allocation fails for some shards, > so elasticsearch status goes red. > 3. Graylog does not move the deflector alias and continues logging in > yesterday's index. > 4. Relative searches do not show data UNLESS you change the relative > search time window to a large enough value to include data from > yesterday. > Then all data shows up. > 5. Manually recalculating index ranges resolves the issue (at least > temporarily). > > Not super familiar w/ the graylog code base, but this feels like > there's a bug in the index selection algorithm and/or index range > maintenance routines. The query analyzer is excluding yesterday's index > when in fact that is the one that has the events we're looking for. > Doing > anything to trick the query analyzer into including that index makes the > query start succeeding. > > On Tuesday, November 1, 2016 at 2:47:41 PM UTC-4, 123Dev wrote: >> >> How often "Recalculate Index Ranges" are automatically performed? >> What controls that? >> >> It appears that when I "recalculate the index Ranges", messages are >> detected. >> but 5 minutes later, messages in the last 5m window are gone, >> obviously 15m window still show data. >> but as more time passes on, we have to widen the search to a larger >> window to get the data. >> >> Can I (should I?) find a way to trigger "Recalculate Index Ranges" >> every minute? >> This command doesn't seem to be doing the job >> >> curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild >> >> >> I can tell because I don't see logs in graylog, nor do I see the ui >>>
Re: [graylog2] Searching "Last Hour" returns no results
No, I am using a straight graylog/ES stack, with logstash pre-processing syslog and submitting to graylog via GELF connection. On Wednesday, November 2, 2016 at 1:15:59 PM UTC-4, 123Dev wrote: > > Just for curiosity, do you use Kibana or Marvel? > > As I previously have mentioned it, our problems might be related to the > usage of Kibana / Marvel, as that's a change we recently introduced. > > Stopping Kibana didn't make a difference, neither removing the Marvel > Agent from ES > So I dug into ES Indexes / Shards > I got rid of any kibana and marvel created indexes. > restarted my services and ran "Recalculate Index Ranges" > It appears to be working now, and 5m window has passed, I'm hesitantly > optimistic. > hardly any solution if you want to use Kibana / Marvel, but if Graylog > goes back to normal, we'll worry about that next. > > > > On Wednesday, November 2, 2016 at 12:19:59 PM UTC-4, > daniel...@hagan-consulting.com wrote: >> >> I'm seeing the same symptoms you are, at least today. If I leave it at >> last 5 minutes, it starts failing to display results 5 minutes after the >> last time I recalculate index ranges. So perhaps my overnight issue is >> more complex but the same root cause? Hard to say. There must be >> something that is triggering this syndrome though, as I didn't have any >> problems yesterday with the "last 5 minute" query. >> >> On Wednesday, November 2, 2016 at 12:08:34 PM UTC-4, 123Dev wrote: >>> >>> Thanks for the follow up >>> Although our symptoms are the same, probably the root causes are >>> different. >>> >>> In our case, our ES never goes red (not even yellow), and the deflector >>> alias which is pointing to today's is correct. >>> The issue at hand is whatever else happens during "Recalculate Index >>> Ranges" seems to make it work. >>> but only for the window since last "Recalculate Index Ranges" which >>> typically would be once a day. >>> >>> If I run now, it would work. >>> but in 5 minutes, it stops working for "show last minute messages" >>> because 5 minutes has already passed. >>> But I show for 15 minutes (which hasn't yet passed) I'll see all the >>> entries just seconds ago. >>> As time passes on, we have to select a longer range just to get the >>> recent logs. >>> >>> Searching directly on ES, works, so whatever Graylog is doing to get the >>> search results, appears to be having problems. >>> All of these are still in the most recent index, so it's not a deflector >>> alias issues, at least not in our case. >>> >>> >>> >>> >>> On Wednesday, November 2, 2016 at 11:03:52 AM UTC-4, >>> daniel...@hagan-consulting.com wrote: This seems like a potential bug to me. Due to an unrelated and undiagnosed issue, my elasticsearch cluster isn't allocating shards successfully every night. So this problem is recurring for me each day in the following scenario: 1. Index rotation set to time based, P1D. 2. New index is created, but shard allocation fails for some shards, so elasticsearch status goes red. 3. Graylog does not move the deflector alias and continues logging in yesterday's index. 4. Relative searches do not show data UNLESS you change the relative search time window to a large enough value to include data from yesterday. Then all data shows up. 5. Manually recalculating index ranges resolves the issue (at least temporarily). Not super familiar w/ the graylog code base, but this feels like there's a bug in the index selection algorithm and/or index range maintenance routines. The query analyzer is excluding yesterday's index when in fact that is the one that has the events we're looking for. Doing anything to trick the query analyzer into including that index makes the query start succeeding. On Tuesday, November 1, 2016 at 2:47:41 PM UTC-4, 123Dev wrote: > > How often "Recalculate Index Ranges" are automatically performed? > What controls that? > > It appears that when I "recalculate the index Ranges", messages are > detected. > but 5 minutes later, messages in the last 5m window are gone, > obviously 15m window still show data. > but as more time passes on, we have to widen the search to a larger > window to get the data. > > Can I (should I?) find a way to trigger "Recalculate Index Ranges" > every minute? > This command doesn't seem to be doing the job > > curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild > > > I can tell because I don't see logs in graylog, nor do I see the ui > updated with the current timestamp. > > Our rotation strategy which hasn't changed for the longest time, which > is set to: > Index rotation strategy: Index Time > Rotation period: P1D (1d, a day) > Index retention strategy: Delete > Max number of indices:28 > > So
Re: [graylog2] Searching "Last Hour" returns no results
Just for curiosity, do you use Kibana or Marvel? As I previously have mentioned it, our problems might be related to the usage of Kibana / Marvel, as that's a change we recently introduced. Stopping Kibana didn't make a difference, neither removing the Marvel Agent from ES So I dug into ES Indexes / Shards I got rid of any kibana and marvel created indexes. restarted my services and ran "Recalculate Index Ranges" It appears to be working now, and 5m window has passed, I'm hesitantly optimistic. hardly any solution if you want to use Kibana / Marvel, but if Graylog goes back to normal, we'll worry about that next. On Wednesday, November 2, 2016 at 12:19:59 PM UTC-4, daniel...@hagan-consulting.com wrote: > > I'm seeing the same symptoms you are, at least today. If I leave it at > last 5 minutes, it starts failing to display results 5 minutes after the > last time I recalculate index ranges. So perhaps my overnight issue is > more complex but the same root cause? Hard to say. There must be > something that is triggering this syndrome though, as I didn't have any > problems yesterday with the "last 5 minute" query. > > On Wednesday, November 2, 2016 at 12:08:34 PM UTC-4, 123Dev wrote: >> >> Thanks for the follow up >> Although our symptoms are the same, probably the root causes are >> different. >> >> In our case, our ES never goes red (not even yellow), and the deflector >> alias which is pointing to today's is correct. >> The issue at hand is whatever else happens during "Recalculate Index >> Ranges" seems to make it work. >> but only for the window since last "Recalculate Index Ranges" which >> typically would be once a day. >> >> If I run now, it would work. >> but in 5 minutes, it stops working for "show last minute messages" >> because 5 minutes has already passed. >> But I show for 15 minutes (which hasn't yet passed) I'll see all the >> entries just seconds ago. >> As time passes on, we have to select a longer range just to get the >> recent logs. >> >> Searching directly on ES, works, so whatever Graylog is doing to get the >> search results, appears to be having problems. >> All of these are still in the most recent index, so it's not a deflector >> alias issues, at least not in our case. >> >> >> >> >> On Wednesday, November 2, 2016 at 11:03:52 AM UTC-4, >> daniel...@hagan-consulting.com wrote: >>> >>> This seems like a potential bug to me. Due to an unrelated and >>> undiagnosed issue, my elasticsearch cluster isn't allocating shards >>> successfully every night. So this problem is recurring for me each day in >>> the following scenario: >>> >>> 1. Index rotation set to time based, P1D. >>> 2. New index is created, but shard allocation fails for some shards, so >>> elasticsearch status goes red. >>> 3. Graylog does not move the deflector alias and continues logging in >>> yesterday's index. >>> 4. Relative searches do not show data UNLESS you change the relative >>> search time window to a large enough value to include data from yesterday. >>> Then all data shows up. >>> 5. Manually recalculating index ranges resolves the issue (at least >>> temporarily). >>> >>> Not super familiar w/ the graylog code base, but this feels like there's >>> a bug in the index selection algorithm and/or index range maintenance >>> routines. The query analyzer is excluding yesterday's index when in fact >>> that is the one that has the events we're looking for. Doing anything to >>> trick the query analyzer into including that index makes the query start >>> succeeding. >>> >>> On Tuesday, November 1, 2016 at 2:47:41 PM UTC-4, 123Dev wrote: How often "Recalculate Index Ranges" are automatically performed? What controls that? It appears that when I "recalculate the index Ranges", messages are detected. but 5 minutes later, messages in the last 5m window are gone, obviously 15m window still show data. but as more time passes on, we have to widen the search to a larger window to get the data. Can I (should I?) find a way to trigger "Recalculate Index Ranges" every minute? This command doesn't seem to be doing the job curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild I can tell because I don't see logs in graylog, nor do I see the ui updated with the current timestamp. Our rotation strategy which hasn't changed for the longest time, which is set to: Index rotation strategy: Index Time Rotation period: P1D (1d, a day) Index retention strategy: Delete Max number of indices:28 So all of the current data should be present in the latest index for 24 hours, why isn't Graylog able to find it, it's not like it is changing. On Tuesday, November 1, 2016 at 1:07:06 PM UTC-4, 123Dev wrote: > > I've been pulling my hair trying to figure this issue out. > I've done count
Re: [graylog2] Searching "Last Hour" returns no results
I'm seeing the same symptoms you are, at least today. If I leave it at
last 5 minutes, it starts failing to display results 5 minutes after the
last time I recalculate index ranges. So perhaps my overnight issue is
more complex but the same root cause? Hard to say. There must be
something that is triggering this syndrome though, as I didn't have any
problems yesterday with the "last 5 minute" query.
On Wednesday, November 2, 2016 at 12:08:34 PM UTC-4, 123Dev wrote:
>
> Thanks for the follow up
> Although our symptoms are the same, probably the root causes are different.
>
> In our case, our ES never goes red (not even yellow), and the deflector
> alias which is pointing to today's is correct.
> The issue at hand is whatever else happens during "Recalculate Index
> Ranges" seems to make it work.
> but only for the window since last "Recalculate Index Ranges" which
> typically would be once a day.
>
> If I run now, it would work.
> but in 5 minutes, it stops working for "show last minute messages" because
> 5 minutes has already passed.
> But I show for 15 minutes (which hasn't yet passed) I'll see all the
> entries just seconds ago.
> As time passes on, we have to select a longer range just to get the recent
> logs.
>
> Searching directly on ES, works, so whatever Graylog is doing to get the
> search results, appears to be having problems.
> All of these are still in the most recent index, so it's not a deflector
> alias issues, at least not in our case.
>
>
>
>
> On Wednesday, November 2, 2016 at 11:03:52 AM UTC-4,
> daniel...@hagan-consulting.com wrote:
>>
>> This seems like a potential bug to me. Due to an unrelated and
>> undiagnosed issue, my elasticsearch cluster isn't allocating shards
>> successfully every night. So this problem is recurring for me each day in
>> the following scenario:
>>
>> 1. Index rotation set to time based, P1D.
>> 2. New index is created, but shard allocation fails for some shards, so
>> elasticsearch status goes red.
>> 3. Graylog does not move the deflector alias and continues logging in
>> yesterday's index.
>> 4. Relative searches do not show data UNLESS you change the relative
>> search time window to a large enough value to include data from yesterday.
>> Then all data shows up.
>> 5. Manually recalculating index ranges resolves the issue (at least
>> temporarily).
>>
>> Not super familiar w/ the graylog code base, but this feels like there's
>> a bug in the index selection algorithm and/or index range maintenance
>> routines. The query analyzer is excluding yesterday's index when in fact
>> that is the one that has the events we're looking for. Doing anything to
>> trick the query analyzer into including that index makes the query start
>> succeeding.
>>
>> On Tuesday, November 1, 2016 at 2:47:41 PM UTC-4, 123Dev wrote:
>>>
>>> How often "Recalculate Index Ranges" are automatically performed?
>>> What controls that?
>>>
>>> It appears that when I "recalculate the index Ranges", messages are
>>> detected.
>>> but 5 minutes later, messages in the last 5m window are gone, obviously
>>> 15m window still show data.
>>> but as more time passes on, we have to widen the search to a larger
>>> window to get the data.
>>>
>>> Can I (should I?) find a way to trigger "Recalculate Index Ranges" every
>>> minute?
>>> This command doesn't seem to be doing the job
>>>
>>> curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild
>>>
>>>
>>> I can tell because I don't see logs in graylog, nor do I see the ui
>>> updated with the current timestamp.
>>>
>>> Our rotation strategy which hasn't changed for the longest time, which
>>> is set to:
>>> Index rotation strategy: Index Time
>>> Rotation period: P1D (1d, a day)
>>> Index retention strategy: Delete
>>> Max number of indices:28
>>>
>>> So all of the current data should be present in the latest index for 24
>>> hours, why isn't Graylog able to find it, it's not like it is changing.
>>>
>>>
>>>
>>> On Tuesday, November 1, 2016 at 1:07:06 PM UTC-4, 123Dev wrote:
I've been pulling my hair trying to figure this issue out.
I've done countless number of "Recalculate Index Ranges"
In fact maybe my problem is with "Recalculate Index Ranges" because
just before my last try, I was getting messages, and now I don't.
This is also playing havoc on many of the alerts that we have setup,
because they're all being triggered as the system detecting inactivity.
I'm beginning to think that this is a bug in Graylog,
I copied the elasticsearch query
```
{
"from": 0,
"size": 150,
"query": {
"bool": {
"must": {
"match_all": {}
},
"filter": {
"bool": {
"must": {
"range": {
"timestamp": {
"from": "2016-11-01 16:15:32.456",
"to": "2016-11-01 16:20:
Re: [graylog2] Searching "Last Hour" returns no results
Thanks for the follow up
Although our symptoms are the same, probably the root causes are different.
In our case, our ES never goes red (not even yellow), and the deflector
alias which is pointing to today's is correct.
The issue at hand is whatever else happens during "Recalculate Index
Ranges" seems to make it work.
but only for the window since last "Recalculate Index Ranges" which
typically would be once a day.
If I run now, it would work.
but in 5 minutes, it stops working for "show last minute messages" because
5 minutes has already passed.
But I show for 15 minutes (which hasn't yet passed) I'll see all the
entries just seconds ago.
As time passes on, we have to select a longer range just to get the recent
logs.
Searching directly on ES, works, so whatever Graylog is doing to get the
search results, appears to be having problems.
All of these are still in the most recent index, so it's not a deflector
alias issues, at least not in our case.
On Wednesday, November 2, 2016 at 11:03:52 AM UTC-4,
daniel...@hagan-consulting.com wrote:
>
> This seems like a potential bug to me. Due to an unrelated and
> undiagnosed issue, my elasticsearch cluster isn't allocating shards
> successfully every night. So this problem is recurring for me each day in
> the following scenario:
>
> 1. Index rotation set to time based, P1D.
> 2. New index is created, but shard allocation fails for some shards, so
> elasticsearch status goes red.
> 3. Graylog does not move the deflector alias and continues logging in
> yesterday's index.
> 4. Relative searches do not show data UNLESS you change the relative
> search time window to a large enough value to include data from yesterday.
> Then all data shows up.
> 5. Manually recalculating index ranges resolves the issue (at least
> temporarily).
>
> Not super familiar w/ the graylog code base, but this feels like there's a
> bug in the index selection algorithm and/or index range maintenance
> routines. The query analyzer is excluding yesterday's index when in fact
> that is the one that has the events we're looking for. Doing anything to
> trick the query analyzer into including that index makes the query start
> succeeding.
>
> On Tuesday, November 1, 2016 at 2:47:41 PM UTC-4, 123Dev wrote:
>>
>> How often "Recalculate Index Ranges" are automatically performed?
>> What controls that?
>>
>> It appears that when I "recalculate the index Ranges", messages are
>> detected.
>> but 5 minutes later, messages in the last 5m window are gone, obviously
>> 15m window still show data.
>> but as more time passes on, we have to widen the search to a larger
>> window to get the data.
>>
>> Can I (should I?) find a way to trigger "Recalculate Index Ranges" every
>> minute?
>> This command doesn't seem to be doing the job
>>
>> curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild
>>
>>
>> I can tell because I don't see logs in graylog, nor do I see the ui
>> updated with the current timestamp.
>>
>> Our rotation strategy which hasn't changed for the longest time, which is
>> set to:
>> Index rotation strategy: Index Time
>> Rotation period: P1D (1d, a day)
>> Index retention strategy: Delete
>> Max number of indices:28
>>
>> So all of the current data should be present in the latest index for 24
>> hours, why isn't Graylog able to find it, it's not like it is changing.
>>
>>
>>
>> On Tuesday, November 1, 2016 at 1:07:06 PM UTC-4, 123Dev wrote:
>>>
>>> I've been pulling my hair trying to figure this issue out.
>>> I've done countless number of "Recalculate Index Ranges"
>>> In fact maybe my problem is with "Recalculate Index Ranges" because just
>>> before my last try, I was getting messages, and now I don't.
>>>
>>> This is also playing havoc on many of the alerts that we have setup,
>>> because they're all being triggered as the system detecting inactivity.
>>>
>>> I'm beginning to think that this is a bug in Graylog,
>>> I copied the elasticsearch query
>>> ```
>>> {
>>> "from": 0,
>>> "size": 150,
>>> "query": {
>>> "bool": {
>>> "must": {
>>> "match_all": {}
>>> },
>>> "filter": {
>>> "bool": {
>>> "must": {
>>> "range": {
>>> "timestamp": {
>>> "from": "2016-11-01 16:15:32.456",
>>> "to": "2016-11-01 16:20:32.456",
>>> "include_lower": true,
>>> "include_upper": true
>>> }
>>> }
>>> }
>>> }
>>> }
>>> }
>>> },
>>> "sort": [
>>> {
>>> "timestamp": {
>>> "order": "desc"
>>> }
>>> }
>>> ],
>>> "highlight": {
>>> "require_field_match": false,
>>> "fields": {
>>> "*": {
>>> "fragment_size": 0,
>>> "number_of_fragments": 0
>>> }
>>> }
>>> }
>>> }
>>> ```
>>>
>>> and executed directly on the Elasticsearch cluster, and it returned the
>>> results, so why isn'
Re: [graylog2] Searching "Last Hour" returns no results
This seems like a potential bug to me. Due to an unrelated and undiagnosed
issue, my elasticsearch cluster isn't allocating shards successfully every
night. So this problem is recurring for me each day in the following
scenario:
1. Index rotation set to time based, P1D.
2. New index is created, but shard allocation fails for some shards, so
elasticsearch status goes red.
3. Graylog does not move the deflector alias and continues logging in
yesterday's index.
4. Relative searches do not show data UNLESS you change the relative search
time window to a large enough value to include data from yesterday. Then
all data shows up.
5. Manually recalculating index ranges resolves the issue (at least
temporarily).
Not super familiar w/ the graylog code base, but this feels like there's a
bug in the index selection algorithm and/or index range maintenance
routines. The query analyzer is excluding yesterday's index when in fact
that is the one that has the events we're looking for. Doing anything to
trick the query analyzer into including that index makes the query start
succeeding.
On Tuesday, November 1, 2016 at 2:47:41 PM UTC-4, 123Dev wrote:
>
> How often "Recalculate Index Ranges" are automatically performed?
> What controls that?
>
> It appears that when I "recalculate the index Ranges", messages are
> detected.
> but 5 minutes later, messages in the last 5m window are gone, obviously
> 15m window still show data.
> but as more time passes on, we have to widen the search to a larger window
> to get the data.
>
> Can I (should I?) find a way to trigger "Recalculate Index Ranges" every
> minute?
> This command doesn't seem to be doing the job
>
> curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild
>
>
> I can tell because I don't see logs in graylog, nor do I see the ui
> updated with the current timestamp.
>
> Our rotation strategy which hasn't changed for the longest time, which is
> set to:
> Index rotation strategy: Index Time
> Rotation period: P1D (1d, a day)
> Index retention strategy: Delete
> Max number of indices:28
>
> So all of the current data should be present in the latest index for 24
> hours, why isn't Graylog able to find it, it's not like it is changing.
>
>
>
> On Tuesday, November 1, 2016 at 1:07:06 PM UTC-4, 123Dev wrote:
>>
>> I've been pulling my hair trying to figure this issue out.
>> I've done countless number of "Recalculate Index Ranges"
>> In fact maybe my problem is with "Recalculate Index Ranges" because just
>> before my last try, I was getting messages, and now I don't.
>>
>> This is also playing havoc on many of the alerts that we have setup,
>> because they're all being triggered as the system detecting inactivity.
>>
>> I'm beginning to think that this is a bug in Graylog,
>> I copied the elasticsearch query
>> ```
>> {
>> "from": 0,
>> "size": 150,
>> "query": {
>> "bool": {
>> "must": {
>> "match_all": {}
>> },
>> "filter": {
>> "bool": {
>> "must": {
>> "range": {
>> "timestamp": {
>> "from": "2016-11-01 16:15:32.456",
>> "to": "2016-11-01 16:20:32.456",
>> "include_lower": true,
>> "include_upper": true
>> }
>> }
>> }
>> }
>> }
>> }
>> },
>> "sort": [
>> {
>> "timestamp": {
>> "order": "desc"
>> }
>> }
>> ],
>> "highlight": {
>> "require_field_match": false,
>> "fields": {
>> "*": {
>> "fragment_size": 0,
>> "number_of_fragments": 0
>> }
>> }
>> }
>> }
>> ```
>>
>> and executed directly on the Elasticsearch cluster, and it returned the
>> results, so why isn't Graylog returning any results?
>>
>> As I was posting this, Graylog had reported that:
>> Range re-calculated 3 hours ago in 75ms. 176 segments, 1 open search
>> contexts, 0 deleted messages
>>
>> Redid the "Recalculate Index Ranges"
>> Range re-calculated a minute ago in 110ms. 182 segments, 0 open search
>> contexts, 0 deleted messages
>>
>> And I started getting results again.
>> Though couple of minutes later, NO more messages again.
>>
>> The problem is even worse, Statistics on non-numeric fields are
>> non-existant.
>> No Total, no cardinality, nada ...
>>
>> The only think I can think of that has recently changed is we added
>> "Elasticsearch client node" for Kibana.
>> Not sure if that has anything to do with it.
>>
>>
>>
>>
>> On Tuesday, November 1, 2016 at 12:25:58 PM UTC-4,
>> daniel...@hagan-consulting.com wrote:
>>>
>>> Surprisingly, I ran into this issue this morning as well. After some
>>> poking around I found that my current index time ranges hadn't been updated
>>> in 15 hours, which I assume was breaking the logic used to select indices
>>> for searching. In System -> Indices, it said the index range hadn't been
>>> updated in 15 hours. I went to System
Re: [graylog2] Searching "Last Hour" returns no results
How often "Recalculate Index Ranges" are automatically performed?
What controls that?
It appears that when I "recalculate the index Ranges", messages are
detected.
but 5 minutes later, messages in the last 5m window are gone, obviously 15m
window still show data.
but as more time passes on, we have to widen the search to a larger window
to get the data.
Can I (should I?) find a way to trigger "Recalculate Index Ranges" every
minute?
This command doesn't seem to be doing the job
curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild
I can tell because I don't see logs in graylog, nor do I see the ui updated
with the current timestamp.
Our rotation strategy which hasn't changed for the longest time, which is
set to:
Index rotation strategy: Index Time
Rotation period: P1D (1d, a day)
Index retention strategy: Delete
Max number of indices:28
So all of the current data should be present in the latest index for 24
hours, why isn't Graylog able to find it, it's not like it is changing.
On Tuesday, November 1, 2016 at 1:07:06 PM UTC-4, 123Dev wrote:
>
> I've been pulling my hair trying to figure this issue out.
> I've done countless number of "Recalculate Index Ranges"
> In fact maybe my problem is with "Recalculate Index Ranges" because just
> before my last try, I was getting messages, and now I don't.
>
> This is also playing havoc on many of the alerts that we have setup,
> because they're all being triggered as the system detecting inactivity.
>
> I'm beginning to think that this is a bug in Graylog,
> I copied the elasticsearch query
> ```
> {
> "from": 0,
> "size": 150,
> "query": {
> "bool": {
> "must": {
> "match_all": {}
> },
> "filter": {
> "bool": {
> "must": {
> "range": {
> "timestamp": {
> "from": "2016-11-01 16:15:32.456",
> "to": "2016-11-01 16:20:32.456",
> "include_lower": true,
> "include_upper": true
> }
> }
> }
> }
> }
> }
> },
> "sort": [
> {
> "timestamp": {
> "order": "desc"
> }
> }
> ],
> "highlight": {
> "require_field_match": false,
> "fields": {
> "*": {
> "fragment_size": 0,
> "number_of_fragments": 0
> }
> }
> }
> }
> ```
>
> and executed directly on the Elasticsearch cluster, and it returned the
> results, so why isn't Graylog returning any results?
>
> As I was posting this, Graylog had reported that:
> Range re-calculated 3 hours ago in 75ms. 176 segments, 1 open search
> contexts, 0 deleted messages
>
> Redid the "Recalculate Index Ranges"
> Range re-calculated a minute ago in 110ms. 182 segments, 0 open search
> contexts, 0 deleted messages
>
> And I started getting results again.
> Though couple of minutes later, NO more messages again.
>
> The problem is even worse, Statistics on non-numeric fields are
> non-existant.
> No Total, no cardinality, nada ...
>
> The only think I can think of that has recently changed is we added
> "Elasticsearch client node" for Kibana.
> Not sure if that has anything to do with it.
>
>
>
>
> On Tuesday, November 1, 2016 at 12:25:58 PM UTC-4,
> daniel...@hagan-consulting.com wrote:
>>
>> Surprisingly, I ran into this issue this morning as well. After some
>> poking around I found that my current index time ranges hadn't been updated
>> in 15 hours, which I assume was breaking the logic used to select indices
>> for searching. In System -> Indices, it said the index range hadn't been
>> updated in 15 hours. I went to System -> Indices and selected Maintenance
>> -> Recalculate index ranges, and that fixed it.
>>
>> On Monday, October 31, 2016 at 4:02:44 PM UTC-4, 123Dev wrote:
>>>
>>> Was this ever resolved.
>>> i never had this issue, and been running Graylog for a long time.
>>>
>>> Just today with the latest Graylog (2.1.1)
>>> Search * for Last, 5m 15m, 30m, 1hr don't return and results
>>> Search * in the last 2 hours returns messages just few seconds ago,
>>> This is across the board, in a clustered environment, with multiple
>>> sources.
>>> Impossible for all those sources to have the wrong dates.
>>> I checked all Graylog nodes ES Nodes and they all have the correct dates
>>> / timezones.
>>>
>>> What gives?
>>> Really puzzling.
>>>
>>> Thanks
>>>
>>> On Wednesday, April 15, 2015 at 4:22:18 PM UTC-4, Zi Dvbelju wrote:
Hey Edmundo,
Graylog user timezone and messages timezone are correct. Has been setup
that way for over a year, and I have confirmed they are still setup
correctly.
I upgraded to the latest version hoping that would fix it, but still
running into the same issue.
Any other suggestions? Error logs don't show any issues.
Only relative searches return data; nothing from absolute.
Thanks,
Z
On Tue
Re: [graylog2] Searching "Last Hour" returns no results
I've been pulling my hair trying to figure this issue out.
I've done countless number of "Recalculate Index Ranges"
In fact maybe my problem is with "Recalculate Index Ranges" because just
before my last try, I was getting messages, and now I don't.
This is also playing havoc on many of the alerts that we have setup,
because they're all being triggered as the system detecting inactivity.
I'm beginning to think that this is a bug in Graylog,
I copied the elasticsearch query
```
{
"from": 0,
"size": 150,
"query": {
"bool": {
"must": {
"match_all": {}
},
"filter": {
"bool": {
"must": {
"range": {
"timestamp": {
"from": "2016-11-01 16:15:32.456",
"to": "2016-11-01 16:20:32.456",
"include_lower": true,
"include_upper": true
}
}
}
}
}
}
},
"sort": [
{
"timestamp": {
"order": "desc"
}
}
],
"highlight": {
"require_field_match": false,
"fields": {
"*": {
"fragment_size": 0,
"number_of_fragments": 0
}
}
}
}
```
and executed directly on the Elasticsearch cluster, and it returned the
results, so why isn't Graylog returning any results?
As I was posting this, Graylog had reported that:
Range re-calculated 3 hours ago in 75ms. 176 segments, 1 open search
contexts, 0 deleted messages
Redid the "Recalculate Index Ranges"
Range re-calculated a minute ago in 110ms. 182 segments, 0 open search
contexts, 0 deleted messages
And I started getting results again.
Though couple of minutes later, NO more messages again.
The problem is even worse, Statistics on non-numeric fields are
non-existant.
No Total, no cardinality, nada ...
The only think I can think of that has recently changed is we added
"Elasticsearch client node" for Kibana.
Not sure if that has anything to do with it.
On Tuesday, November 1, 2016 at 12:25:58 PM UTC-4,
daniel...@hagan-consulting.com wrote:
>
> Surprisingly, I ran into this issue this morning as well. After some
> poking around I found that my current index time ranges hadn't been updated
> in 15 hours, which I assume was breaking the logic used to select indices
> for searching. In System -> Indices, it said the index range hadn't been
> updated in 15 hours. I went to System -> Indices and selected Maintenance
> -> Recalculate index ranges, and that fixed it.
>
> On Monday, October 31, 2016 at 4:02:44 PM UTC-4, 123Dev wrote:
>>
>> Was this ever resolved.
>> i never had this issue, and been running Graylog for a long time.
>>
>> Just today with the latest Graylog (2.1.1)
>> Search * for Last, 5m 15m, 30m, 1hr don't return and results
>> Search * in the last 2 hours returns messages just few seconds ago,
>> This is across the board, in a clustered environment, with multiple
>> sources.
>> Impossible for all those sources to have the wrong dates.
>> I checked all Graylog nodes ES Nodes and they all have the correct dates
>> / timezones.
>>
>> What gives?
>> Really puzzling.
>>
>> Thanks
>>
>> On Wednesday, April 15, 2015 at 4:22:18 PM UTC-4, Zi Dvbelju wrote:
>>>
>>> Hey Edmundo,
>>>
>>> Graylog user timezone and messages timezone are correct. Has been setup
>>> that way for over a year, and I have confirmed they are still setup
>>> correctly.
>>>
>>> I upgraded to the latest version hoping that would fix it, but still
>>> running into the same issue.
>>>
>>> Any other suggestions? Error logs don't show any issues.
>>>
>>> Only relative searches return data; nothing from absolute.
>>>
>>> Thanks,
>>> Z
>>>
>>>
>>> On Tuesday, January 20, 2015 at 2:40:53 PM UTC-5, Edmundo Alvarez wrote:
Hello Zach,
I would start taking a look at the time configuration, specially
timezones. Could you verify the time settings in the machines sending
logs,
and that both your Graylog user's timezone and the messages' timezones are
correct?
I hope that helps.
Regards,
Edmundo
--
Developer
Tel.: +49 (0)40 609 452 077
Mobile: +49 (0)171 27 22 181
Mobile (US): +1 (713) 321 8126
Fax.: +49 (0)40 609 452 078
TORCH GmbH
Steckelhörn 11
20457 Hamburg
Germany
https://www.torch.sh/
Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)
> On 20 Jan 2015, at 19:34, Zi Dvbelju wrote:
>
> Hey Team,
>
> Running into a strange issue with my Graylog2 setup.
>
> Search for anything with "last hour" returns no results. I can
duplicate the same exact search with "last day" and see messages from
within the last few seconds.
>
> Only searches with the "last hour" tag are returning nothing.
>
> v92.3
>
> Thanks,
> Zach
Re: [graylog2] Searching "Last Hour" returns no results
Surprisingly, I ran into this issue this morning as well. After some poking around I found that my current index time ranges hadn't been updated in 15 hours, which I assume was breaking the logic used to select indices for searching. In System -> Indices, it said the index range hadn't been updated in 15 hours. I went to System -> Indices and selected Maintenance -> Recalculate index ranges, and that fixed it. On Monday, October 31, 2016 at 4:02:44 PM UTC-4, 123Dev wrote: > > Was this ever resolved. > i never had this issue, and been running Graylog for a long time. > > Just today with the latest Graylog (2.1.1) > Search * for Last, 5m 15m, 30m, 1hr don't return and results > Search * in the last 2 hours returns messages just few seconds ago, > This is across the board, in a clustered environment, with multiple > sources. > Impossible for all those sources to have the wrong dates. > I checked all Graylog nodes ES Nodes and they all have the correct dates / > timezones. > > What gives? > Really puzzling. > > Thanks > > On Wednesday, April 15, 2015 at 4:22:18 PM UTC-4, Zi Dvbelju wrote: >> >> Hey Edmundo, >> >> Graylog user timezone and messages timezone are correct. Has been setup >> that way for over a year, and I have confirmed they are still setup >> correctly. >> >> I upgraded to the latest version hoping that would fix it, but still >> running into the same issue. >> >> Any other suggestions? Error logs don't show any issues. >> >> Only relative searches return data; nothing from absolute. >> >> Thanks, >> Z >> >> >> On Tuesday, January 20, 2015 at 2:40:53 PM UTC-5, Edmundo Alvarez wrote: >>> >>> Hello Zach, >>> >>> I would start taking a look at the time configuration, specially >>> timezones. Could you verify the time settings in the machines sending logs, >>> and that both your Graylog user's timezone and the messages' timezones are >>> correct? >>> >>> I hope that helps. >>> >>> Regards, >>> Edmundo >>> >>> -- >>> Developer >>> >>> Tel.: +49 (0)40 609 452 077 >>> Mobile: +49 (0)171 27 22 181 >>> Mobile (US): +1 (713) 321 8126 >>> Fax.: +49 (0)40 609 452 078 >>> >>> TORCH GmbH >>> Steckelhörn 11 >>> 20457 Hamburg >>> Germany >>> https://www.torch.sh/ >>> >>> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 >>> Geschäftsführer: Lennart Koopmann (CEO) >>> >>> > On 20 Jan 2015, at 19:34, Zi Dvbelju wrote: >>> > >>> > Hey Team, >>> > >>> > Running into a strange issue with my Graylog2 setup. >>> > >>> > Search for anything with "last hour" returns no results. I can >>> duplicate the same exact search with "last day" and see messages from >>> within the last few seconds. >>> > >>> > Only searches with the "last hour" tag are returning nothing. >>> > >>> > v92.3 >>> > >>> > Thanks, >>> > Zach >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "graylog2" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to graylog2+u...@googlegroups.com. >>> > For more options, visit https://groups.google.com/d/optout. >>> >>> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/badcad55-a5cc-4d00-880d-70ade12aff92%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Searching "Last Hour" returns no results
Was this ever resolved. i never had this issue, and been running Graylog for a long time. Just today with the latest Graylog (2.1.1) Search * for Last, 5m 15m, 30m, 1hr don't return and results Search * in the last 2 hours returns messages just few seconds ago, This is across the board, in a clustered environment, with multiple sources. Impossible for all those sources to have the wrong dates. I checked all Graylog nodes ES Nodes and they all have the correct dates / timezones. What gives? Really puzzling. Thanks On Wednesday, April 15, 2015 at 4:22:18 PM UTC-4, Zi Dvbelju wrote: > > Hey Edmundo, > > Graylog user timezone and messages timezone are correct. Has been setup > that way for over a year, and I have confirmed they are still setup > correctly. > > I upgraded to the latest version hoping that would fix it, but still > running into the same issue. > > Any other suggestions? Error logs don't show any issues. > > Only relative searches return data; nothing from absolute. > > Thanks, > Z > > > On Tuesday, January 20, 2015 at 2:40:53 PM UTC-5, Edmundo Alvarez wrote: >> >> Hello Zach, >> >> I would start taking a look at the time configuration, specially >> timezones. Could you verify the time settings in the machines sending logs, >> and that both your Graylog user's timezone and the messages' timezones are >> correct? >> >> I hope that helps. >> >> Regards, >> Edmundo >> >> -- >> Developer >> >> Tel.: +49 (0)40 609 452 077 >> Mobile: +49 (0)171 27 22 181 >> Mobile (US): +1 (713) 321 8126 >> Fax.: +49 (0)40 609 452 078 >> >> TORCH GmbH >> Steckelhörn 11 >> 20457 Hamburg >> Germany >> https://www.torch.sh/ >> >> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 >> Geschäftsführer: Lennart Koopmann (CEO) >> >> > On 20 Jan 2015, at 19:34, Zi Dvbelju wrote: >> > >> > Hey Team, >> > >> > Running into a strange issue with my Graylog2 setup. >> > >> > Search for anything with "last hour" returns no results. I can >> duplicate the same exact search with "last day" and see messages from >> within the last few seconds. >> > >> > Only searches with the "last hour" tag are returning nothing. >> > >> > v92.3 >> > >> > Thanks, >> > Zach >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "graylog2" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to graylog2+u...@googlegroups.com. >> > For more options, visit https://groups.google.com/d/optout. >> >> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/ae9103cf-8eb0-4678-b5c2-06f978652c0d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Searching "Last Hour" returns no results
Hey Edmundo, Graylog user timezone and messages timezone are correct. Has been setup that way for over a year, and I have confirmed they are still setup correctly. I upgraded to the latest version hoping that would fix it, but still running into the same issue. Any other suggestions? Error logs don't show any issues. Only relative searches return data; nothing from absolute. Thanks, Z On Tuesday, January 20, 2015 at 2:40:53 PM UTC-5, Edmundo Alvarez wrote: > > Hello Zach, > > I would start taking a look at the time configuration, specially > timezones. Could you verify the time settings in the machines sending logs, > and that both your Graylog user's timezone and the messages' timezones are > correct? > > I hope that helps. > > Regards, > Edmundo > > -- > Developer > > Tel.: +49 (0)40 609 452 077 > Mobile: +49 (0)171 27 22 181 > Mobile (US): +1 (713) 321 8126 > Fax.: +49 (0)40 609 452 078 > > TORCH GmbH > Steckelhörn 11 > 20457 Hamburg > Germany > https://www.torch.sh/ > > Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 > Geschäftsführer: Lennart Koopmann (CEO) > > > On 20 Jan 2015, at 19:34, Zi Dvbelju > > wrote: > > > > Hey Team, > > > > Running into a strange issue with my Graylog2 setup. > > > > Search for anything with "last hour" returns no results. I can duplicate > the same exact search with "last day" and see messages from within the last > few seconds. > > > > Only searches with the "last hour" tag are returning nothing. > > > > v92.3 > > > > Thanks, > > Zach > > > > -- > > You received this message because you are subscribed to the Google > Groups "graylog2" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to graylog2+u...@googlegroups.com . > > For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Searching "Last Hour" returns no results
Hello Zach, I would start taking a look at the time configuration, specially timezones. Could you verify the time settings in the machines sending logs, and that both your Graylog user's timezone and the messages' timezones are correct? I hope that helps. Regards, Edmundo -- Developer Tel.: +49 (0)40 609 452 077 Mobile: +49 (0)171 27 22 181 Mobile (US): +1 (713) 321 8126 Fax.: +49 (0)40 609 452 078 TORCH GmbH Steckelhörn 11 20457 Hamburg Germany https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) > On 20 Jan 2015, at 19:34, Zi Dvbelju wrote: > > Hey Team, > > Running into a strange issue with my Graylog2 setup. > > Search for anything with "last hour" returns no results. I can duplicate the > same exact search with "last day" and see messages from within the last few > seconds. > > Only searches with the "last hour" tag are returning nothing. > > v92.3 > > Thanks, > Zach > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Searching "Last Hour" returns no results
Hey Team, Running into a strange issue with my Graylog2 setup. Search for anything with "last hour" returns no results. I can duplicate the same exact search with "last day" and see messages from within the last few seconds. Only searches with the "last hour" tag are returning nothing. v92.3 Thanks, Zach -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
