[ossec-list] Re: Simultaneous Events at 25 EPS, but Missing Alerts
Hi Jon, OSSEC connects through UDP protocol, that doesn't guarantee that messages arrive in the same order they were delivered. In order to prevent replay attacks, OSSEC verifies the counter from every message. I think there is a network issue, perhaps congestion, so messages arrive disordered. In your case, disabling remoted.verity_msg_id is a good solution. Best regards, Victor. On Tuesday, October 11, 2016 at 8:57:26 PM UTC+2, Jon Goodgion wrote: > > Hey there! I think it's actually due to the *remoted.verify_msg_id* > option in internal_options.conf > > Once I turned this off, messages were coming in out of order, but all > messages were getting received! > > On Tuesday, October 4, 2016 at 5:15:25 AM UTC-4, Pedro S wrote: >> >> Hi Jon, >> >> This is an interesting test, I think we can get a lot of useful >> information from here. >> >> On my experience probably the bottleneck is on remoted socket/buffer or >> logcollector speed performance to read each log line. >> >> For Remoted, try to enable debug mode at the agent, internal_options.conf >> file, remoted.debug=2, and agent.debug=2. You will find at ossec.log each >> line read by logcollector and sent to remoted, this way we can figure out >> if the problem is related to gathering/sending or lately >> receiving/proccesing. >> >> On my tests I can see how everything is being pushed, but just a few >> events on archives are displayed, I think OSSEC also have some protection >> for multiple identical messages. >> >> 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: >>> 'test55' >>> 2016/10/04 11:07:39 ossec-logcollector: DEBUG: Reading syslog message: >>> 'test55' >>> 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to >>> server. >>> 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: >>> 'test55' >>> 2016/10/04 11:07:39 ossec-logcollector: DEBUG: Reading syslog message: >>> 'test55' >>> 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to >>> server. >>> 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: >>> 'test55' >>> 2016/10/04 11:07:39 ossec-logcollector: DEBUG: Reading syslog message: >>> 'test55' >>> 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to >>> server. >>> 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: >>> 'test55' >>> 2016/10/04 11:07:39 ossec-logcollector: DEBUG: Reading syslog message: >>> 'test55' >>> 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to >>> server. >>> 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: >>> 'test55' >>> 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to >>> server. >>> 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: >>> '--MARK--: ' >>> 2016/10/04 11:07:43 ossec-agent: DEBUG: Attempting to send message to >>> server. >> >> >> >> >> >> On Monday, October 3, 2016 at 8:27:04 PM UTC+2, Jon Goodgion wrote: >>> >>> I've been curious about the performance of OSSEC in a server/agent >>> architecture, so I have been emulating simultaneous events on a single >>> agent by appending log entries to the agent's syslog. >>> >>> Using a shell script for loop on the agent, I append 25 consecutive logs >>> that match the format of a telnet failed password log. I figured 25 EPS >>> should be easily captured by OSSEC. >>> >>> However, on the server, (after enabling logall to archives), it doesn't >>> seem like it is processing all the logs. >>> /var/ossec/logs/archives/archives.log shows: >>> >>> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >>> 14:15:55 queen telnetd[*1*]: refused connect from 81.215.42.24 >>> >>> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >>> 14:15:55 queen telnetd[*13*]: refused connect from 81.215.42.158 >>> >>> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >>> 14:15:55 queen telnetd[14]: refused connect from 81.215.42.69 >>> >>> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >>> 14:15:55 queen telnetd[15]: refused connect from 81.215.42.32 >>> >>> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >>> 14:15:55 queen telnetd[16]: refused connect from 81.215.42.41 >>> >>> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >>> 14:15:55 queen telnetd[17]: refused connect from 81.215.42.74 >>> >>> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >>> 14:15:55 queen telnetd[18]: refused connect from 81.215.42.32 >>> >>> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >>> 14:15:55 queen telnetd[19]: refused connect from 81.215.42.222 >>> >>> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >>> 14:15:55 queen telnetd[20]: refused connect from 81.215.42.25 >>> >>> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >>> 14:15:55 queen telnetd[21]: refused connect
[ossec-list] Re: Simultaneous Events at 25 EPS, but Missing Alerts
Hey there! I think it's actually due to the *remoted.verify_msg_id* option in internal_options.conf Once I turned this off, messages were coming in out of order, but all messages were getting received! On Tuesday, October 4, 2016 at 5:15:25 AM UTC-4, Pedro S wrote: > > Hi Jon, > > This is an interesting test, I think we can get a lot of useful > information from here. > > On my experience probably the bottleneck is on remoted socket/buffer or > logcollector speed performance to read each log line. > > For Remoted, try to enable debug mode at the agent, internal_options.conf > file, remoted.debug=2, and agent.debug=2. You will find at ossec.log each > line read by logcollector and sent to remoted, this way we can figure out > if the problem is related to gathering/sending or lately > receiving/proccesing. > > On my tests I can see how everything is being pushed, but just a few > events on archives are displayed, I think OSSEC also have some protection > for multiple identical messages. > > 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: 'test55' >> 2016/10/04 11:07:39 ossec-logcollector: DEBUG: Reading syslog message: >> 'test55' >> 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to >> server. >> 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: >> 'test55' >> 2016/10/04 11:07:39 ossec-logcollector: DEBUG: Reading syslog message: >> 'test55' >> 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to >> server. >> 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: >> 'test55' >> 2016/10/04 11:07:39 ossec-logcollector: DEBUG: Reading syslog message: >> 'test55' >> 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to >> server. >> 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: >> 'test55' >> 2016/10/04 11:07:39 ossec-logcollector: DEBUG: Reading syslog message: >> 'test55' >> 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to >> server. >> 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: >> 'test55' >> 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to >> server. >> 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: >> '--MARK--: ' >> 2016/10/04 11:07:43 ossec-agent: DEBUG: Attempting to send message to >> server. > > > > > > On Monday, October 3, 2016 at 8:27:04 PM UTC+2, Jon Goodgion wrote: >> >> I've been curious about the performance of OSSEC in a server/agent >> architecture, so I have been emulating simultaneous events on a single >> agent by appending log entries to the agent's syslog. >> >> Using a shell script for loop on the agent, I append 25 consecutive logs >> that match the format of a telnet failed password log. I figured 25 EPS >> should be easily captured by OSSEC. >> >> However, on the server, (after enabling logall to archives), it doesn't >> seem like it is processing all the logs. >> /var/ossec/logs/archives/archives.log shows: >> >> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >> 14:15:55 queen telnetd[*1*]: refused connect from 81.215.42.24 >> >> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >> 14:15:55 queen telnetd[*13*]: refused connect from 81.215.42.158 >> >> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >> 14:15:55 queen telnetd[14]: refused connect from 81.215.42.69 >> >> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >> 14:15:55 queen telnetd[15]: refused connect from 81.215.42.32 >> >> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >> 14:15:55 queen telnetd[16]: refused connect from 81.215.42.41 >> >> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >> 14:15:55 queen telnetd[17]: refused connect from 81.215.42.74 >> >> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >> 14:15:55 queen telnetd[18]: refused connect from 81.215.42.32 >> >> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >> 14:15:55 queen telnetd[19]: refused connect from 81.215.42.222 >> >> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >> 14:15:55 queen telnetd[20]: refused connect from 81.215.42.25 >> >> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >> 14:15:55 queen telnetd[21]: refused connect from 81.215.42.141 >> >> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >> 14:15:55 queen telnetd[22]: refused connect from 81.215.42.248 >> >> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >> 14:15:55 queen telnetd[23]: refused connect from 81.215.42.45 >> >> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >> 14:15:55 queen telnetd[24]: refused connect from 81.215.42.166 >> 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 >> 14:15:55 queen telnetd[25]: refused connect from
[ossec-list] Re: Simultaneous Events at 25 EPS, but Missing Alerts
Hi Jon, This is an interesting test, I think we can get a lot of useful information from here. On my experience probably the bottleneck is on remoted socket/buffer or logcollector speed performance to read each log line. For Remoted, try to enable debug mode at the agent, internal_options.conf file, remoted.debug=2, and agent.debug=2. You will find at ossec.log each line read by logcollector and sent to remoted, this way we can figure out if the problem is related to gathering/sending or lately receiving/proccesing. On my tests I can see how everything is being pushed, but just a few events on archives are displayed, I think OSSEC also have some protection for multiple identical messages. 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: 'test55' > 2016/10/04 11:07:39 ossec-logcollector: DEBUG: Reading syslog message: > 'test55' > 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to > server. > 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: 'test55' > 2016/10/04 11:07:39 ossec-logcollector: DEBUG: Reading syslog message: > 'test55' > 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to > server. > 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: 'test55' > 2016/10/04 11:07:39 ossec-logcollector: DEBUG: Reading syslog message: > 'test55' > 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to > server. > 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: 'test55' > 2016/10/04 11:07:39 ossec-logcollector: DEBUG: Reading syslog message: > 'test55' > 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to > server. > 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: 'test55' > 2016/10/04 11:07:39 ossec-agent: DEBUG: Attempting to send message to > server. > 2016/10/04 11:07:39 ossec-agent: DEBUG: Sending message to server: > '--MARK--: ' > 2016/10/04 11:07:43 ossec-agent: DEBUG: Attempting to send message to > server. On Monday, October 3, 2016 at 8:27:04 PM UTC+2, Jon Goodgion wrote: > > I've been curious about the performance of OSSEC in a server/agent > architecture, so I have been emulating simultaneous events on a single > agent by appending log entries to the agent's syslog. > > Using a shell script for loop on the agent, I append 25 consecutive logs > that match the format of a telnet failed password log. I figured 25 EPS > should be easily captured by OSSEC. > > However, on the server, (after enabling logall to archives), it doesn't > seem like it is processing all the logs. > /var/ossec/logs/archives/archives.log shows: > > 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 > 14:15:55 queen telnetd[*1*]: refused connect from 81.215.42.24 > > 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 > 14:15:55 queen telnetd[*13*]: refused connect from 81.215.42.158 > > 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 > 14:15:55 queen telnetd[14]: refused connect from 81.215.42.69 > > 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 > 14:15:55 queen telnetd[15]: refused connect from 81.215.42.32 > > 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 > 14:15:55 queen telnetd[16]: refused connect from 81.215.42.41 > > 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 > 14:15:55 queen telnetd[17]: refused connect from 81.215.42.74 > > 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 > 14:15:55 queen telnetd[18]: refused connect from 81.215.42.32 > > 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 > 14:15:55 queen telnetd[19]: refused connect from 81.215.42.222 > > 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 > 14:15:55 queen telnetd[20]: refused connect from 81.215.42.25 > > 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 > 14:15:55 queen telnetd[21]: refused connect from 81.215.42.141 > > 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 > 14:15:55 queen telnetd[22]: refused connect from 81.215.42.248 > > 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 > 14:15:55 queen telnetd[23]: refused connect from 81.215.42.45 > > 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 > 14:15:55 queen telnetd[24]: refused connect from 81.215.42.166 > 2016 Oct 03 13:53:31 (agent101) 192.168.0.101->/var/log/syslog Oct 1 > 14:15:55 queen telnetd[25]: refused connect from 81.215.42.178 > > I put the for loop sequence identifier in the telnetd brackets [ ], and as > you can see, this particular test didn't catch anything between the 2nd and > 12th log. > > Does this have to do with UDP loss? Am I missing something, or possibly > need to reconfigure OSSEC a certain way? Any help would be greatly > appreciated! > -- --- You received this message because you are subscribed to
