Additional grok patterns:
BRO_ALLBUTPIPE ([^\|]+)
BRO_URL (%{BRO_ALLBUTPIPE})
BRO_MISC (%{BRO_ALLBUTPIPE})
On Wednesday, December 7, 2016 at 9:52:31 PM UTC-6, BKeep wrote:
>
> Our new Graylog instance has been running for awhile without so much as a
> hiccup. Recently, I added new log sources from a Security Onion sensor
> containing BRO and Suricata logs. It doesn't appear these new inputs have
> caused any noticeable load on the system, at least not until I run them
> through a pipeline processor. I am using grok and standard regex functions
> in the pipeline rules to parse out bro_conn, bro_dns, etc.
>
> Today, I noticed the output stopped during what I would consider peak load
> around 11:30a<ish> CST and to the best of my recollection no changes had
> been made directly proceeding the stoppage.
>
> As I have been adding pipeline rules to parse out messages, it seems
> something happens where logs stop writing to the elasticsearch nodes. I
> don't see anything in the server.log that looks like a smoking gun If I
> restart the graylog-server.service the logs will not begin to clear from
> the output buffer. However, If I stop the graylog-server.service and then
> start it, logs begin to flow again. I do not have to restart any other
> service after the manual stop/start of graylog.
>
> The only log I see that seems like it would be related is below. However,
> I am not sure if it is relevant.
> 2016-12-07T12:27:53.601-06:00 WARN [DeadEventLoggingListener] Received
> unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from
> event bus <AsyncEventBus{graylog-eventbus}>
>
> System info:
> top - 15:07:03 up 18 days, 22:46, 1 user, load average: 0.68, 1.06, 1.12
> Tasks: 420 total, 1 running, 419 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 5.3 us, 1.6 sy, 0.0 ni, 93.0 id, 0.0 wa, 0.0 hi, 0.0 si,
> 0.0 st
> KiB Mem : 49282804 total, 34329916 free, 11754760 used, 3198128 buff/cache
> KiB Swap: 1048572 total, 1048572 free, 0 used. 37276428 avail Mem
>
> Graylog 2.1.1+01d50e5 starting up
> JRE: 1.8.0_111 on Linux 3.10.0-327.36.3.el7.x86_64
> OS: CentOS Linux 7 (Core) amd64
>
> JVM arguments: -Xms8g -Xmx8g -XX:NewRatio=1 -XX:+ResizeTLAB
> -XX:+UseConcMarkSweepGC -XX:+CMSConc currentMTEnabled
> -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC
> -XX:-OmitStackTraceInFastThrow
> -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml
> -Djava.library.path=/opt/graylog-server/lib/sigar
> -Dgraylog2.installation_source=unknown
>
> transparent_hugepage=false
>
>
> I noticed when checking messages against the pipeline simulator, I see
> inconsistent results in the execution times. I see this from the same
> messages and I also see it with different messages at different times so it
> isn't something I can reproduce at will.
>
> Sample message:
> <13>1 2016-12-08T02:36:57+00:00 BROsensor bro_notice - - -
> 1481164616.440593|CYxMwXBbL0TvQQKm9|10.10.10.10|51841|208.10.10.10|443|-|-|-|tcp|SSL::Invalid_Server_Cert|SSL
>
> certificate validation failed with (self signed certificate in certificate
> chain)|[email protected]
> ,CN=ADN,OU=ADN,O=External,L=Toledo,ST=Ohio,C=US|10.10.10.10|208.10.10.10|443|-|BROsensor-eth1-1|Notice::ACTION_LOG|3600.000000|F|-|-|-|-|-
>
> Sample rule using grok:
> rule "Extract bro_notice log fields"
> when
> has_field("message") AND
> contains(value: to_string($message.application_name), search:
> "bro_notice", ignore_case: true)
> then
> let m = grok(
> "^(?<ts>%{NUMBER}|-).?(?<uid>%{WORD}|-).?(?<id_orig_h>%{IP}|-).?(?<id_orig_p>%{INT}|-).?(?<id_resp_h>%{IP}|-).?(?<id_resp_p>%{INT}|-).?(?<fuid>%{WORD}|-).?(?<file_mime_type>%{WORD}/%{WORD}|-).?(?<file_desc>%{WORD}|-).?(?<proto>%{WORD}|-).?(?<note>%{WORD}::%{WORD}|-).?(?<msg>%{BRO_URL}).?(?<sub>%{BRO_URL}).?(?<src>%{IP}|-).?(?<dst>%{IP}|-).?(?<p>%{INT}|-).?(?<n>%{INT}|-).?(?<peer_desc>%{HOSTNAME}|-).?(?<actions>%{WORD}::%{WORD}|-).?(?<suppress_for>%{INT}\\.%{INT}|-).?(?<dropped>[TF]).?(?<remote_location_country_code>%{WORD}|-).?(?<remote_location_region>%{WORD}|-).?(?<remote_location_city>%{WORD}|-).?(?<remote_location_latitude>%{INT}\\.%{INT}|-).?(?<remote_location_longitude>%{INT}\\.%{INT}|-).?$"
> , to_string($message.message), true);
>
> set_fields(m);
> end
>
> Simulation results test 1:
> These are the results of processing the loaded message. Processing took 54
> ,366 µs.
>
> 1 μs
> Starting message processing
> 51 μs
> Message c93a5361-bcee-11e6-9154-78e7d17bef2e running [Pipeline 'Bro
> IDS Logs' (58464b367d5d445614cc4f27)] for streams [
> 584195567d5d44076c4160f7]
> 84 μs
> Enter Stage 0
> 90 μs
> Evaluate Rule 'Extract bro_conn log fields' (584190b57d5d44076c415c14)
> in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
> 105 μs
> Evaluation not satisfied Rule 'Extract bro_conn log fields' (
> 584190b57d5d44076c415c14) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 107 μs
> Evaluate Rule 'Extract bro_dns log fields' (5843002a7d5d44076c42de64)
> in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
> 116 μs
> Evaluation not satisfied Rule 'Extract bro_dns log fields' (
> 5843002a7d5d44076c42de64) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 119 μs
> Evaluate Rule 'Extract bro_http log fields' (584372dc7d5d443ea323761e)
> in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
> 126 μs
> Evaluation not satisfied Rule 'Extract bro_http log fields' (
> 584372dc7d5d443ea323761e) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 129 μs
> Evaluate Rule 'Extract bro_tunnels log fields' (
> 58476fb77d5d445614cd828a) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 136 μs
> Evaluation not satisfied Rule 'Extract bro_tunnels log fields' (
> 58476fb77d5d445614cd828a) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 138 μs
> Evaluate Rule 'Extract bro_notice log fields' (
> 584795807d5d445614cdaa4e) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 147 μs
> Evaluation satisfied Rule 'Extract bro_notice log fields' (
> 584795807d5d445614cdaa4e) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 150 μs
> Evaluate Rule 'Extract bro_weird log fields' (5847a07f7d5d445614cdb5e2
> ) in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
> 157 μs
> Evaluation not satisfied Rule 'Extract bro_weird log fields' (
> 5847a07f7d5d445614cdb5e2) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 161 μs
> Evaluate Rule 'Extract bro_software log fields' (
> 5847b6017d5d445614cdcc85) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 168 μs
> Evaluation not satisfied Rule 'Extract bro_software log fields' (
> 5847b6017d5d445614cdcc85) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 179 μs
> Execute Rule 'Extract bro_notice log fields' (584795807d5d445614cdaa4e
> ) in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
> 54,335 μs
> Completed Stage 0 for Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27), continuing to next stage
> 54,345 μs
> Exit Stage 0
> 54,366 μs
> Finished message processing
>
> Simulation results test 2:
> These are the results of processing the loaded message. Processing took
> 381 µs.
>
> 3 μs
> Starting message processing
> 36 μs
> Message ead67e30-bcef-11e6-9154-78e7d17bef2e running [Pipeline 'Bro
> IDS Logs' (58464b367d5d445614cc4f27)] for streams [
> 584195567d5d44076c4160f7]
> 62 μs
> Enter Stage 0
> 67 μs
> Evaluate Rule 'Extract bro_conn log fields' (584190b57d5d44076c415c14)
> in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
> 80 μs
> Evaluation not satisfied Rule 'Extract bro_conn log fields' (
> 584190b57d5d44076c415c14) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 82 μs
> Evaluate Rule 'Extract bro_dns log fields' (5843002a7d5d44076c42de64)
> in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
> 88 μs
> Evaluation not satisfied Rule 'Extract bro_dns log fields' (
> 5843002a7d5d44076c42de64) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 91 μs
> Evaluate Rule 'Extract bro_http log fields' (584372dc7d5d443ea323761e)
> in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
> 97 μs
> Evaluation not satisfied Rule 'Extract bro_http log fields' (
> 584372dc7d5d443ea323761e) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 100 μs
> Evaluate Rule 'Extract bro_tunnels log fields' (
> 58476fb77d5d445614cd828a) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 106 μs
> Evaluation not satisfied Rule 'Extract bro_tunnels log fields' (
> 58476fb77d5d445614cd828a) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 109 μs
> Evaluate Rule 'Extract bro_notice log fields' (
> 584795807d5d445614cdaa4e) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 116 μs
> Evaluation satisfied Rule 'Extract bro_notice log fields' (
> 584795807d5d445614cdaa4e) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 120 μs
> Evaluate Rule 'Extract bro_weird log fields' (5847a07f7d5d445614cdb5e2
> ) in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
> 134 μs
> Evaluation not satisfied Rule 'Extract bro_weird log fields' (
> 5847a07f7d5d445614cdb5e2) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 136 μs
> Evaluate Rule 'Extract bro_software log fields' (
> 5847b6017d5d445614cdcc85) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 142 μs
> Evaluation not satisfied Rule 'Extract bro_software log fields' (
> 5847b6017d5d445614cdcc85) in Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27)
> 148 μs
> Execute Rule 'Extract bro_notice log fields' (584795807d5d445614cdaa4e
> ) in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
> 369 μs
> Completed Stage 0 for Pipeline 'Bro IDS Logs' (
> 58464b367d5d445614cc4f27), continuing to next stage
> 372 μs
> Exit Stage 0
> 381 μs
> Finished message processing
>
>
> I am open to suggestions for where to look next.
> Regards,
> Brandon
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/7048791a-22e2-446f-895a-d1b02324f0f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
