Our new Graylog instance has been running for awhile without so much as a
hiccup. Recently, I added new log sources from a Security Onion sensor
containing BRO and Suricata logs. It doesn't appear these new inputs have
caused any noticeable load on the system, at least not until I run them
through a pipeline processor. I am using grok and standard regex functions
in the pipeline rules to parse out bro_conn, bro_dns, etc.
Today, I noticed the output stopped during what I would consider peak load
around 11:30a<ish> CST and to the best of my recollection no changes had
been made directly proceeding the stoppage.
As I have been adding pipeline rules to parse out messages, it seems
something happens where logs stop writing to the elasticsearch nodes. I
don't see anything in the server.log that looks like a smoking gun If I
restart the graylog-server.service the logs will not begin to clear from
the output buffer. However, If I stop the graylog-server.service and then
start it, logs begin to flow again. I do not have to restart any other
service after the manual stop/start of graylog.
The only log I see that seems like it would be related is below. However, I
am not sure if it is relevant.
2016-12-07T12:27:53.601-06:00 WARN [DeadEventLoggingListener] Received
unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from
event bus <AsyncEventBus{graylog-eventbus}>
System info:
top - 15:07:03 up 18 days, 22:46, 1 user, load average: 0.68, 1.06, 1.12
Tasks: 420 total, 1 running, 419 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.3 us, 1.6 sy, 0.0 ni, 93.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
KiB Mem : 49282804 total, 34329916 free, 11754760 used, 3198128 buff/cache
KiB Swap: 1048572 total, 1048572 free, 0 used. 37276428 avail Mem
Graylog 2.1.1+01d50e5 starting up
JRE: 1.8.0_111 on Linux 3.10.0-327.36.3.el7.x86_64
OS: CentOS Linux 7 (Core) amd64
JVM arguments: -Xms8g -Xmx8g -XX:NewRatio=1 -XX:+ResizeTLAB
-XX:+UseConcMarkSweepGC -XX:+CMSConc currentMTEnabled
-XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC
-XX:-OmitStackTraceInFastThrow
-Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml
-Djava.library.path=/opt/graylog-server/lib/sigar
-Dgraylog2.installation_source=unknown
transparent_hugepage=false
I noticed when checking messages against the pipeline simulator, I see
inconsistent results in the execution times. I see this from the same
messages and I also see it with different messages at different times so it
isn't something I can reproduce at will.
Sample message:
<13>1 2016-12-08T02:36:57+00:00 BROsensor bro_notice - - -
1481164616.440593|CYxMwXBbL0TvQQKm9|10.10.10.10|51841|208.10.10.10|443|-|-|-|tcp|SSL::Invalid_Server_Cert|SSL
certificate validation failed with (self signed certificate in certificate
chain)|[email protected],CN=ADN,OU=ADN,O=External,L=Toledo,ST=Ohio,C=US|10.10.10.10|208.10.10.10|443|-|BROsensor-eth1-1|Notice::ACTION_LOG|3600.000000|F|-|-|-|-|-
Sample rule using grok:
rule "Extract bro_notice log fields"
when
has_field("message") AND
contains(value: to_string($message.application_name), search: "bro_notice"
, ignore_case: true)
then
let m = grok(
"^(?<ts>%{NUMBER}|-).?(?<uid>%{WORD}|-).?(?<id_orig_h>%{IP}|-).?(?<id_orig_p>%{INT}|-).?(?<id_resp_h>%{IP}|-).?(?<id_resp_p>%{INT}|-).?(?<fuid>%{WORD}|-).?(?<file_mime_type>%{WORD}/%{WORD}|-).?(?<file_desc>%{WORD}|-).?(?<proto>%{WORD}|-).?(?<note>%{WORD}::%{WORD}|-).?(?<msg>%{BRO_URL}).?(?<sub>%{BRO_URL}).?(?<src>%{IP}|-).?(?<dst>%{IP}|-).?(?<p>%{INT}|-).?(?<n>%{INT}|-).?(?<peer_desc>%{HOSTNAME}|-).?(?<actions>%{WORD}::%{WORD}|-).?(?<suppress_for>%{INT}\\.%{INT}|-).?(?<dropped>[TF]).?(?<remote_location_country_code>%{WORD}|-).?(?<remote_location_region>%{WORD}|-).?(?<remote_location_city>%{WORD}|-).?(?<remote_location_latitude>%{INT}\\.%{INT}|-).?(?<remote_location_longitude>%{INT}\\.%{INT}|-).?$"
, to_string($message.message), true);
set_fields(m);
end
Simulation results test 1:
These are the results of processing the loaded message. Processing took 54,
366 µs.
1 μs
Starting message processing
51 μs
Message c93a5361-bcee-11e6-9154-78e7d17bef2e running [Pipeline 'Bro IDS
Logs' (58464b367d5d445614cc4f27)] for streams [584195567d5d44076c4160f7]
84 μs
Enter Stage 0
90 μs
Evaluate Rule 'Extract bro_conn log fields' (584190b57d5d44076c415c14)
in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
105 μs
Evaluation not satisfied Rule 'Extract bro_conn log fields' (
584190b57d5d44076c415c14) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
107 μs
Evaluate Rule 'Extract bro_dns log fields' (5843002a7d5d44076c42de64) in
Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
116 μs
Evaluation not satisfied Rule 'Extract bro_dns log fields' (
5843002a7d5d44076c42de64) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
119 μs
Evaluate Rule 'Extract bro_http log fields' (584372dc7d5d443ea323761e)
in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
126 μs
Evaluation not satisfied Rule 'Extract bro_http log fields' (
584372dc7d5d443ea323761e) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
129 μs
Evaluate Rule 'Extract bro_tunnels log fields' (58476fb77d5d445614cd828a
) in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
136 μs
Evaluation not satisfied Rule 'Extract bro_tunnels log fields' (
58476fb77d5d445614cd828a) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
138 μs
Evaluate Rule 'Extract bro_notice log fields' (584795807d5d445614cdaa4e)
in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
147 μs
Evaluation satisfied Rule 'Extract bro_notice log fields' (
584795807d5d445614cdaa4e) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
150 μs
Evaluate Rule 'Extract bro_weird log fields' (5847a07f7d5d445614cdb5e2)
in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
157 μs
Evaluation not satisfied Rule 'Extract bro_weird log fields' (
5847a07f7d5d445614cdb5e2) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
161 μs
Evaluate Rule 'Extract bro_software log fields' (
5847b6017d5d445614cdcc85) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
168 μs
Evaluation not satisfied Rule 'Extract bro_software log fields' (
5847b6017d5d445614cdcc85) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
179 μs
Execute Rule 'Extract bro_notice log fields' (584795807d5d445614cdaa4e)
in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
54,335 μs
Completed Stage 0 for Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27
), continuing to next stage
54,345 μs
Exit Stage 0
54,366 μs
Finished message processing
Simulation results test 2:
These are the results of processing the loaded message. Processing took 381
µs.
3 μs
Starting message processing
36 μs
Message ead67e30-bcef-11e6-9154-78e7d17bef2e running [Pipeline 'Bro IDS
Logs' (58464b367d5d445614cc4f27)] for streams [584195567d5d44076c4160f7]
62 μs
Enter Stage 0
67 μs
Evaluate Rule 'Extract bro_conn log fields' (584190b57d5d44076c415c14)
in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
80 μs
Evaluation not satisfied Rule 'Extract bro_conn log fields' (
584190b57d5d44076c415c14) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
82 μs
Evaluate Rule 'Extract bro_dns log fields' (5843002a7d5d44076c42de64) in
Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
88 μs
Evaluation not satisfied Rule 'Extract bro_dns log fields' (
5843002a7d5d44076c42de64) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
91 μs
Evaluate Rule 'Extract bro_http log fields' (584372dc7d5d443ea323761e)
in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
97 μs
Evaluation not satisfied Rule 'Extract bro_http log fields' (
584372dc7d5d443ea323761e) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
100 μs
Evaluate Rule 'Extract bro_tunnels log fields' (58476fb77d5d445614cd828a
) in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
106 μs
Evaluation not satisfied Rule 'Extract bro_tunnels log fields' (
58476fb77d5d445614cd828a) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
109 μs
Evaluate Rule 'Extract bro_notice log fields' (584795807d5d445614cdaa4e)
in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
116 μs
Evaluation satisfied Rule 'Extract bro_notice log fields' (
584795807d5d445614cdaa4e) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
120 μs
Evaluate Rule 'Extract bro_weird log fields' (5847a07f7d5d445614cdb5e2)
in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
134 μs
Evaluation not satisfied Rule 'Extract bro_weird log fields' (
5847a07f7d5d445614cdb5e2) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
136 μs
Evaluate Rule 'Extract bro_software log fields' (
5847b6017d5d445614cdcc85) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
142 μs
Evaluation not satisfied Rule 'Extract bro_software log fields' (
5847b6017d5d445614cdcc85) in Pipeline 'Bro IDS Logs' (
58464b367d5d445614cc4f27)
148 μs
Execute Rule 'Extract bro_notice log fields' (584795807d5d445614cdaa4e)
in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
369 μs
Completed Stage 0 for Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27
), continuing to next stage
372 μs
Exit Stage 0
381 μs
Finished message processing
I am open to suggestions for where to look next.
Regards,
Brandon
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/7ebd7e00-8152-4efe-9dd8-54554091b083%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
