Our new Graylog instance has been running for awhile without so much as a
hiccup. Recently, I added new log sources from a Security Onion sensor
containing BRO and Suricata logs. It doesn't appear these new inputs have
caused any noticeable load on the system, at least not until I run them
through a pipeline processor. I am using grok and standard regex functions
in the pipeline rules to parse out bro_conn, bro_dns, etc.
Today, I noticed the output stopped during what I would consider peak load
around 11:30a<ish> CST and to the best of my recollection no changes had
been made directly proceeding the stoppage.
As I have been adding pipeline rules to parse out messages, it seems
something happens where logs stop writing to the elasticsearch nodes. I
don't see anything in the server.log that looks like a smoking gun If I
restart the graylog-server.service the logs will not begin to clear from
the output buffer. However, If I stop the graylog-server.service and then
start it, logs begin to flow again. I do not have to restart any other
service after the manual stop/start of graylog.
The only log I see that seems like it would be related is below. However, I
am not sure if it is relevant.
2016-12-07T12:27:53.601-06:00 WARN [DeadEventLoggingListener] Received
unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from
event bus <AsyncEventBus{graylog-eventbus}>
top - 15:07:03 up 18 days, 22:46, 1 user, load average: 0.68, 1.06, 1.12
Tasks: 420 total, 1 running, 419 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.3 us, 1.6 sy, 0.0 ni, 93.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
KiB Mem : 49282804 total, 34329916 free, 11754760 used, 3198128 buff/cache
KiB Swap: 1048572 total, 1048572 free, 0 used. 37276428 avail Mem
Graylog 2.1.1+01d50e5 starting up
JRE: 1.8.0_111 on Linux 3.10.0-327.36.3.el7.x86_64
OS: CentOS Linux 7 (Core) amd64
JVM arguments: -Xms8g -Xmx8g -XX:NewRatio=1 -XX:+ResizeTLAB
-XX:+UseConcMarkSweepGC -XX:+CMSConc currentMTEnabled
-XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC
-XX:-OmitStackTraceInFastThrow
-Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml
-Djava.library.path=/opt/graylog-server/lib/sigar
-Dgraylog2.installation_source=unknown
transparent_hugepage=false
If someone could point me to a place where I can get better insight into
which pipeline rule(s) may be causing the problem, I would appreciate it. I
also am open to other suggestions too.
Regards,
Brandon
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/c48bebd2-adbb-4681-97c8-c70eb978af10%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
