hmm,
https://www.graylog.org/blog/77-announcing-graylog-2-2-0-beta-2
- Stabilized Pipeline Processor! We’ve improved performance with faster
processing speed
I will let you know ;)
On Sunday, December 11, 2016 at 11:01:34 PM UTC-6, BKeep wrote:
>
> No one has any ideas?
>
> On Friday, December 9, 2016 at 12:17:17 PM UTC-6, BKeep wrote:
>>
>> It happened again today. However, for whatever reason, stopping and then
>> starting the graylog-server.service did not allow operations to continue
>> when graylog came back up. I had to disconnect the pipeline processor.
>>
>>
>>
>> Once I did that, logs started flowing to elasticsearch again.
>>
>>
>>
>> Without running anything through the pipeline processor this is the load
>> on the system.
>> top - 12:12:22 up 20 days, 19:51, 1 user, load average: 1.35, 2.60, 2.17
>> Tasks: 420 total, 1 running, 419 sleeping, 0 stopped, 0 zombie
>> %Cpu(s): 2.1 us, 0.8 sy, 0.0 ni, 97.1 id, 0.0 wa, 0.0 hi, 0.0 si,
>> 0.0 st
>> KiB Mem : 49282804 total, 36668764 free, 8659352 used, 3954688
>> buff/cache
>> KiB Swap: 1048572 total, 1048572 free, 0 used. 40362516 avail Mem
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>> 18329 graylog 20 0 22.554g 6.058g 70608 S 161.5 12.9 28:45.28 java
>> 1905 elastic+ 20 0 29.972g 1.941g 15520 S 17.3 4.1 220:31.90 java
>> 2030 mongod 20 0 444924 57660 7512 S 1.3 0.1 263:08.27 mongod
>>
>> I'm open to suggestions.
>>
>> Regards,
>> Brandon
>>
>> On Wednesday, December 7, 2016 at 9:52:31 PM UTC-6, BKeep wrote:
>>>
>>> Our new Graylog instance has been running for awhile without so much as
>>> a hiccup. Recently, I added new log sources from a Security Onion sensor
>>> containing BRO and Suricata logs. It doesn't appear these new inputs have
>>> caused any noticeable load on the system, at least not until I run them
>>> through a pipeline processor. I am using grok and standard regex functions
>>> in the pipeline rules to parse out bro_conn, bro_dns, etc.
>>>
>>> Today, I noticed the output stopped during what I would consider peak
>>> load around 11:30a<ish> CST and to the best of my recollection no changes
>>> had been made directly proceeding the stoppage.
>>>
>>> As I have been adding pipeline rules to parse out messages, it seems
>>> something happens where logs stop writing to the elasticsearch nodes. I
>>> don't see anything in the server.log that looks like a smoking gun If I
>>> restart the graylog-server.service the logs will not begin to clear from
>>> the output buffer. However, If I stop the graylog-server.service and then
>>> start it, logs begin to flow again. I do not have to restart any other
>>> service after the manual stop/start of graylog.
>>>
>>> The only log I see that seems like it would be related is below.
>>> However, I am not sure if it is relevant.
>>> 2016-12-07T12:27:53.601-06:00 WARN [DeadEventLoggingListener] Received
>>> unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from
>>> event bus <AsyncEventBus{graylog-eventbus}>
>>>
>>> System info:
>>> top - 15:07:03 up 18 days, 22:46, 1 user, load average: 0.68, 1.06,
>>> 1.12
>>> Tasks: 420 total, 1 running, 419 sleeping, 0 stopped, 0 zombie
>>> %Cpu(s): 5.3 us, 1.6 sy, 0.0 ni, 93.0 id, 0.0 wa, 0.0 hi, 0.0 si,
>>> 0.0 st
>>> KiB Mem : 49282804 total, 34329916 free, 11754760 used, 3198128
>>> buff/cache
>>> KiB Swap: 1048572 total, 1048572 free, 0 used. 37276428 avail
>>> Mem
>>>
>>> Graylog 2.1.1+01d50e5 starting up
>>> JRE: 1.8.0_111 on Linux 3.10.0-327.36.3.el7.x86_64
>>> OS: CentOS Linux 7 (Core) amd64
>>>
>>> JVM arguments: -Xms8g -Xmx8g -XX:NewRatio=1 -XX:+ResizeTLAB
>>> -XX:+UseConcMarkSweepGC -XX:+CMSConc currentMTEnabled
>>> -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC
>>> -XX:-OmitStackTraceInFastThrow
>>> -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml
>>> -Djava.library.path=/opt/graylog-server/lib/sigar
>>> -Dgraylog2.installation_source=unknown
>>>
>>> transparent_hugepage=false
>>>
>>>
>>> I noticed when checking messages against the pipeline simulator, I see
>>> inconsistent results in the execution times. I see this from the same
>>> messages and I also see it with different messages at different times so it
>>> isn't something I can reproduce at will.
>>>
>>> Sample message:
>>> <13>1 2016-12-08T02:36:57+00:00 BROsensor bro_notice - - -
>>> 1481164616.440593|CYxMwXBbL0TvQQKm9|10.10.10.10|51841|208.10.10.10|443|-|-|-|tcp|SSL::Invalid_Server_Cert|SSL
>>>
>>> certificate validation failed with (self signed certificate in certificate
>>> chain)|[email protected]
>>> ,CN=ADN,OU=ADN,O=External,L=Toledo,ST=Ohio,C=US|10.10.10.10|208.10.10.10|443|-|BROsensor-eth1-1|Notice::ACTION_LOG|3600.000000|F|-|-|-|-|-
>>>
>>> Sample rule using grok:
>>> rule "Extract bro_notice log fields"
>>> when
>>> has_field("message") AND
>>> contains(value: to_string($message.application_name), search:
>>> "bro_notice", ignore_case: true)
>>> then
>>> let m = grok(
>>> "^(?<ts>%{NUMBER}|-).?(?<uid>%{WORD}|-).?(?<id_orig_h>%{IP}|-).?(?<id_orig_p>%{INT}|-).?(?<id_resp_h>%{IP}|-).?(?<id_resp_p>%{INT}|-).?(?<fuid>%{WORD}|-).?(?<file_mime_type>%{WORD}/%{WORD}|-).?(?<file_desc>%{WORD}|-).?(?<proto>%{WORD}|-).?(?<note>%{WORD}::%{WORD}|-).?(?<msg>%{BRO_URL}).?(?<sub>%{BRO_URL}).?(?<src>%{IP}|-).?(?<dst>%{IP}|-).?(?<p>%{INT}|-).?(?<n>%{INT}|-).?(?<peer_desc>%{HOSTNAME}|-).?(?<actions>%{WORD}::%{WORD}|-).?(?<suppress_for>%{INT}\\.%{INT}|-).?(?<dropped>[TF]).?(?<remote_location_country_code>%{WORD}|-).?(?<remote_location_region>%{WORD}|-).?(?<remote_location_city>%{WORD}|-).?(?<remote_location_latitude>%{INT}\\.%{INT}|-).?(?<remote_location_longitude>%{INT}\\.%{INT}|-).?$"
>>> , to_string($message.message), true);
>>>
>>> set_fields(m);
>>> end
>>>
>>> Simulation results test 1:
>>> These are the results of processing the loaded message. Processing took
>>> 54,366 µs.
>>>
>>> 1 μs
>>> Starting message processing
>>> 51 μs
>>> Message c93a5361-bcee-11e6-9154-78e7d17bef2e running [Pipeline 'Bro
>>> IDS Logs' (58464b367d5d445614cc4f27)] for streams [
>>> 584195567d5d44076c4160f7]
>>> 84 μs
>>> Enter Stage 0
>>> 90 μs
>>> Evaluate Rule 'Extract bro_conn log fields' (
>>> 584190b57d5d44076c415c14) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 105 μs
>>> Evaluation not satisfied Rule 'Extract bro_conn log fields' (
>>> 584190b57d5d44076c415c14) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 107 μs
>>> Evaluate Rule 'Extract bro_dns log fields' (5843002a7d5d44076c42de64
>>> ) in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
>>> 116 μs
>>> Evaluation not satisfied Rule 'Extract bro_dns log fields' (
>>> 5843002a7d5d44076c42de64) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 119 μs
>>> Evaluate Rule 'Extract bro_http log fields' (
>>> 584372dc7d5d443ea323761e) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 126 μs
>>> Evaluation not satisfied Rule 'Extract bro_http log fields' (
>>> 584372dc7d5d443ea323761e) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 129 μs
>>> Evaluate Rule 'Extract bro_tunnels log fields' (
>>> 58476fb77d5d445614cd828a) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 136 μs
>>> Evaluation not satisfied Rule 'Extract bro_tunnels log fields' (
>>> 58476fb77d5d445614cd828a) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 138 μs
>>> Evaluate Rule 'Extract bro_notice log fields' (
>>> 584795807d5d445614cdaa4e) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 147 μs
>>> Evaluation satisfied Rule 'Extract bro_notice log fields' (
>>> 584795807d5d445614cdaa4e) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 150 μs
>>> Evaluate Rule 'Extract bro_weird log fields' (
>>> 5847a07f7d5d445614cdb5e2) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 157 μs
>>> Evaluation not satisfied Rule 'Extract bro_weird log fields' (
>>> 5847a07f7d5d445614cdb5e2) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 161 μs
>>> Evaluate Rule 'Extract bro_software log fields' (
>>> 5847b6017d5d445614cdcc85) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 168 μs
>>> Evaluation not satisfied Rule 'Extract bro_software log fields' (
>>> 5847b6017d5d445614cdcc85) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 179 μs
>>> Execute Rule 'Extract bro_notice log fields' (
>>> 584795807d5d445614cdaa4e) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 54,335 μs
>>> Completed Stage 0 for Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27), continuing to next stage
>>> 54,345 μs
>>> Exit Stage 0
>>> 54,366 μs
>>> Finished message processing
>>>
>>> Simulation results test 2:
>>> These are the results of processing the loaded message. Processing took
>>> 381 µs.
>>>
>>> 3 μs
>>> Starting message processing
>>> 36 μs
>>> Message ead67e30-bcef-11e6-9154-78e7d17bef2e running [Pipeline 'Bro
>>> IDS Logs' (58464b367d5d445614cc4f27)] for streams [
>>> 584195567d5d44076c4160f7]
>>> 62 μs
>>> Enter Stage 0
>>> 67 μs
>>> Evaluate Rule 'Extract bro_conn log fields' (
>>> 584190b57d5d44076c415c14) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 80 μs
>>> Evaluation not satisfied Rule 'Extract bro_conn log fields' (
>>> 584190b57d5d44076c415c14) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 82 μs
>>> Evaluate Rule 'Extract bro_dns log fields' (5843002a7d5d44076c42de64
>>> ) in Pipeline 'Bro IDS Logs' (58464b367d5d445614cc4f27)
>>> 88 μs
>>> Evaluation not satisfied Rule 'Extract bro_dns log fields' (
>>> 5843002a7d5d44076c42de64) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 91 μs
>>> Evaluate Rule 'Extract bro_http log fields' (
>>> 584372dc7d5d443ea323761e) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 97 μs
>>> Evaluation not satisfied Rule 'Extract bro_http log fields' (
>>> 584372dc7d5d443ea323761e) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 100 μs
>>> Evaluate Rule 'Extract bro_tunnels log fields' (
>>> 58476fb77d5d445614cd828a) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 106 μs
>>> Evaluation not satisfied Rule 'Extract bro_tunnels log fields' (
>>> 58476fb77d5d445614cd828a) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 109 μs
>>> Evaluate Rule 'Extract bro_notice log fields' (
>>> 584795807d5d445614cdaa4e) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 116 μs
>>> Evaluation satisfied Rule 'Extract bro_notice log fields' (
>>> 584795807d5d445614cdaa4e) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 120 μs
>>> Evaluate Rule 'Extract bro_weird log fields' (
>>> 5847a07f7d5d445614cdb5e2) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 134 μs
>>> Evaluation not satisfied Rule 'Extract bro_weird log fields' (
>>> 5847a07f7d5d445614cdb5e2) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 136 μs
>>> Evaluate Rule 'Extract bro_software log fields' (
>>> 5847b6017d5d445614cdcc85) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 142 μs
>>> Evaluation not satisfied Rule 'Extract bro_software log fields' (
>>> 5847b6017d5d445614cdcc85) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 148 μs
>>> Execute Rule 'Extract bro_notice log fields' (
>>> 584795807d5d445614cdaa4e) in Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27)
>>> 369 μs
>>> Completed Stage 0 for Pipeline 'Bro IDS Logs' (
>>> 58464b367d5d445614cc4f27), continuing to next stage
>>> 372 μs
>>> Exit Stage 0
>>> 381 μs
>>> Finished message processing
>>>
>>>
>>> I am open to suggestions for where to look next.
>>> Regards,
>>> Brandon
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/e36d5c1f-d862-42b0-b0b4-e9b0468dc32b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
