[graylog2] Graylog stop sending messages to elasticsearch

[Stefan Zahnd]

Hi

I hope someone can give me a hint. After search for two weeks now I cannot 
find the solution for my problem.

Graylog stops sending messages to elasticsearch (throuput In: xxx / Out: 
0). If I restart graylog messages are beeing sent to elasticsearch but not 
with the same performance as it did a few weeks ago, where the situation 
was normal. It begins fast and the drops until it reaches 0 for outgoing 
messages. Both elasticsearch and graylog logs don't give useful information 
in debug mode. 

Shortly before the problem occured I did some changes to the graylog 
configuration and changed the shard configuration (primary and replica). I 
think that was a bad idea and the reason for the problems right now. 

I think that a complete flush of all the data and restart with a new 
elasticsearch instance could solve the problem. But I don't know how to 
perform this correctly. my questions are

1) did anybody have the same or similar problems and can give me some hints 
where else to search?
2) how do I flush all the data correctly from the graylog database and 
start over with a fresh instance of elasticsearch? What I found was the 
following 
article: https://groups.google.com/forum/#!topic/graylog2/Dfw6uKtUF5k. But 
the solution mentioned in mongo db doesn't work as I don't have those 
options. Think the version is too old (mine is 2.6.11).

Thank you in advance for any help on this.

Kind regards

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/b5acd505-a4ed-4c99-a659-942946ee5a6b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Graylog Web Interface Internal Server Error

[Michel Laporte]

Hi There,

On the Graylog Web Interface, after a day or so, everytime i click on 
"Sources" to try and view up to date log information, i get the following 
error:


Could not load sources data
Loading of sources data failed with status: Internal Server Error. Try 
reloading the page.

However, on the right hand side if i click on anything other than Last 
hour. . It works. So if i click on Last day , Week , All it shows me the 
logs, however on the last hour i always get this error on internal server 
error. Restarting Graylog & ES on the master node always sort this issue. 
Does anyone know what the issue could be please?

Thanks

-- 
-
essencedigital.com 
Google+  • Facebook 
 • Twitter 
 • YouTube 


-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/0b9c73d1-1505-40c5-a344-3e87c070baca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] grok pattern not working

[Zsolt Osztrovszky]

Hello Guys!
I'd like to setup an extractor with Grok pattern.
This is my sample message and pattern:
10.10.1.1 - - [13/Oct/2015:17:19:54 +0200] "GET //ed98/561/this.m3u8 
HTTP/1.1" 200 388 "http://10.1.1.1/hls.php?o==2kV=BASE64; 
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11) AppleWebKit/601.1.56 (KHTML, 
like Gecko) Version/9.0 Safari/601.1.56" 3878 6090 ed98b

pattern:
%{IP:remote_addr}

If I push try, it says: Attention We were not able to run the grok 
extraction. Please check your parameters.

What am I doing wrong?
Thanks.
Cheers,
Zsolt

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/f461ed8f-4a98-4006-89b8-21b1a7e185ca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: grok pattern not working

[Jochen Schalanda]

Hi Zsolt,

did you add the required Grok patterns to your Graylog system?


Cheers,
Jochen

On Tuesday, 20 October 2015 12:56:17 UTC+2, Zsolt Osztrovszky wrote:
>
> Hello Guys!
> I'd like to setup an extractor with Grok pattern.
> This is my sample message and pattern:
> 10.10.1.1 - - [13/Oct/2015:17:19:54 +0200] "GET //ed98/561/this.m3u8 
> HTTP/1.1" 200 388 "http://10.1.1.1/hls.php?o==2kV=BASE64; 
> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11) AppleWebKit/601.1.56 (KHTML, 
> like Gecko) Version/9.0 Safari/601.1.56" 3878 6090 ed98b
>
> pattern:
> %{IP:remote_addr}
>
> If I push try, it says: Attention We were not able to run the grok 
> extraction. Please check your parameters.
>
> What am I doing wrong?
> Thanks.
> Cheers,
> Zsolt
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/ab9539b3-7c11-4665-a6f4-f6c092a2b690%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog Timestamp

[William Davis]

Bumping this thread.

On Friday, October 9, 2015 at 8:46:17 AM UTC-4, William Davis wrote:
>
> Is the timestamp displayed on the search page the timestamp when Graylog 
> received the message or some timestamp contained within the message (like 
> when it was generated)?
>
> I have created a Method Boundry Aspect (see PostSharp AOP) to log method 
> entry / exit / exception easily for debug tracing. The problem I'm having 
> is the time stamp only does to the thousandth place, and I really need to 
> add CPU ticks in so that my messages sort in the correct order.
>
> What would be the best way to achieve this?
>
> Thanks!
>
> Will.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/206e2f99-3c77-49f9-bc28-58336cc01999%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: setup ElasticSearch and Graylog

[Jochen Schalanda]

Hi Zsolt,

depending on the operating system you've installed Graylog on you can 
either use the init script (Debian Wheezy, `service graylog-server restart`), 
the Upstart service (Ubuntu, `restart graylog-server`), or the systemd 
service (Debian Jessie, `systemctl restart graylog-server`) to restart 
Graylog.

Cheers,
Jochen

On Monday, 19 October 2015 12:24:58 UTC+2, Zsolt Osztrovszky wrote:
>
> Thanks.
>
> How can I restart it if I can’t find it in the service --status-all?
>
>  
>
> Cheers, 
>
> Zsolt
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/7650aee5-f390-4a85-8d14-fda5cfc22194%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Graylog Web Interface Internal Server Error

[Edmundo Alvarez]

Hi Michel,

Could you please look into your ES and Graylog logs and share any errors that 
you see while loading the data?

Regards,

Edmundo

> On 20 Oct 2015, at 10:49, Michel Laporte  
> wrote:
> 
> Hi There,
> 
> On the Graylog Web Interface, after a day or so, everytime i click on 
> "Sources" to try and view up to date log information, i get the following 
> error:
> 
> 
> Could not load sources data
> Loading of sources data failed with status: Internal Server Error. Try 
> reloading the page.
> 
> However, on the right hand side if i click on anything other than Last hour. 
> . It works. So if i click on Last day , Week , All it shows me the logs, 
> however on the last hour i always get this error on internal server error. 
> Restarting Graylog & ES on the master node always sort this issue. Does 
> anyone know what the issue could be please?
> 
> Thanks
> 
> -
> essencedigital.com
> Google+ • Facebook • Twitter • YouTube
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/graylog2/0b9c73d1-1505-40c5-a344-3e87c070baca%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/C09CFE95-FBBF-4F50-9D2B-438CCE1AE364%40graylog.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Searching by timestamp range

[Jesse Skrivseth]

Hello everyone, 

Is there a way to do a search for all records with a timestamp that is 
outside normal business hours? I can't seem to do ranges on timestamps, 
ignoring the date. 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/986757bb-d61e-492c-8317-b18d4adb9d73%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Graylog stop sending messages to elasticsearch after adding extractor

[Stefan Zahnd]

Hi

I have the problem that right after adding a grok extractor to an input the 
system immediately stopps sending messages to elasticsearch (out = 0). Does 
somebody had the same experience?

My pattern looks like 
"client\s%{IPV4:src_ip}#%{BASE10NUM:src_port}\s\(([a-zA-Z0-9.\-_]*)+\):\squery:\s%{NOTSPACE:dns_name}\sIN\s%{WORD:dns_type}\s%{NOTSPACE:dns_flags}\s\(%{IPV4:dns_server}\)"
 
and is for bind messages.

I have the latest 1.2.1 version installed. Graylog-server, -web and 
elasticsearch are on different machines.

Thank you and kind regards

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/88f8a45e-8e14-4ec8-945a-4a47f4cad399%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Searching by timestamp range

[Jesse Skrivseth]

I ended up writing a MessageFilter plugin that intercepts the messages and 
explodes the "timestamp" field into parts:

private void explodeDateField(String field, Message message) {
if (message.hasField(field)) {

Object fieldValue = message.getField(field);
if (fieldValue != null) {
try {

//try to interpret the field as a date
DateTime date = new DateTime(fieldValue);
//assuming we have a valid date here, add the parts back on the 
message
message.addField(field + "_dow", date.dayOfWeek().get());
message.addField(field + "_day_week", 
date.dayOfWeek().getAsText());

message.addField(field + "_day", date.dayOfMonth().get());
message.addField(field + "_day_year", date.dayOfYear().get());
message.addField(field + "_month", date.monthOfYear().get());
message.addField(field + "_year", date.year().get());

message.addField(field + "_hour", date.hourOfDay().get());
message.addField(field + "_minute", date.minuteOfHour().get());

}
} catch (IllegalArgumentException e) {
log.debug("IllegalArgumentException thrown - Could not parse 
timestamp", fieldValue);
} catch (Exception e) {
log.error("Exception thrown '", e.getMessage());
}
}
}


So now I can do queries like:

timestamp_day_week:(Sunday Saturday) OR (timestamp_day_week:(Monday Tuesday 
Wednesday Thursday Friday) AND (timestamp_hour:[17 TO 23] OR 
timestamp_hour:[0 TO 9]))

Which should find all events occurring outside of M-F 9am-5pm

Not terribly pretty, but it works. 

On Tuesday, October 20, 2015 at 10:39:49 AM UTC-6, Jesse Skrivseth wrote:
>
> Hello everyone, 
>
> Is there a way to do a search for all records with a timestamp that is 
> outside normal business hours? I can't seem to do ranges on timestamps, 
> ignoring the date. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a30b0858-a204-41d0-a916-455819528248%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Searching by timestamp range

[Jesse Skrivseth]

I ended up writing a MessageFilter plugin that intercepts the messages and 
explodes the "timestamp" field into parts:

private void explodeDateField(String field, Message message) {
if (message.hasField(field)) {

Object fieldValue = message.getField(field);
if (fieldValue != null) {
try {
//try to interpret the field as a date
DateTime date = new DateTime(fieldValue);
//assuming we have a valid date here, add the parts back on the 
message
message.addField(field + "_dow", date.dayOfWeek().get());
message.addField(field + "_day_week", 
date.dayOfWeek().getAsText());

message.addField(field + "_day", date.dayOfMonth().get());
message.addField(field + "_day_year", date.dayOfYear().get());
message.addField(field + "_month", date.monthOfYear().get());
message.addField(field + "_year", date.year().get());

message.addField(field + "_hour", date.hourOfDay().get());
message.addField(field + "_minute", date.minuteOfHour().get());
   } catch (IllegalArgumentException e) {
   log.debug("IllegalArgumentException thrown - Could not parse 
timestamp", fieldValue);
   } catch (Exception e) {
   log.error("Exception thrown '", e.getMessage());
   }

}
}
}


So now I can do queries like:

timestamp_day_week:(Sunday Saturday) OR (timestamp_day_week:(Monday Tuesday 
Wednesday Thursday Friday) AND (timestamp_hour:[17 TO 23] OR 
timestamp_hour:[0 TO 9]))

Which should find all events occurring outside of M-F 9am-5pm

Not terribly pretty, but it works. 

On Tuesday, October 20, 2015 at 10:39:49 AM UTC-6, Jesse Skrivseth wrote:
>
> Hello everyone, 
>
> Is there a way to do a search for all records with a timestamp that is 
> outside normal business hours? I can't seem to do ranges on timestamps, 
> ignoring the date. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a4108268-f8b1-40d5-978e-307193506b5a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.