Hi Graeme. I agree, it would be great to print on the log that the agent became disconnected. The SEC_ERROR definition is shared between manager and agents, but it's possible to extend some other messages. In fact, the line at sendmsg.c that tests if the agent is disconnected (more than 20 minutes since the last keep-alive) is the only one that doesn't log an error.
I did some modifications at the Wazuh repository, maybe it's useful to you: https://github.com/wazuh/ossec-wazuh/commit/efbd5c17cc3ea5109ea978208b11da6c98fa3083 See below an example of the new log format for the error: 2016/07/29 11:43:57 ossec-remoted(1245): ERROR: Sending message to disconnected agent '001'. 2016/07/29 11:43:57 ossec-remoted(1217): ERROR: Error creating encrypted message. 2016/07/29 11:43:57 ossec-remoted(1246): ERROR: Unable to send file 'merged.mg' to agent '001' (centos). I hope this leads you to find the problem. Kind regards. On Friday, July 29, 2016 at 8:19:56 AM UTC-7, Graeme Stewart wrote: > > Hi Victor, > > Huge thanks for the detail, this would explain exactly why we're seeing > this; our OSSEC managers are likely overloaded. > > It would be very helpful to include the agentid in the logfile to > understand / track where this is occurring and the number of unique agents > that are impacted, perhaps something like: > > From: src/error_messages/error_messages.h > #define SEC_ERROR "%s(1217): ERROR: Error creating encrypted message > for: '%s')." > > Then inside: src/remoted/sendmsg.c > msg_size = CreateSecMSG(&keys, msg, crypt_msg, agentid); > if (msg_size == 0) { > merror(SEC_ERROR, ARGV0, agentid); > return (-1); > } > > The clustered nature of this issue leads me to suspect it's repeating this > error in the logfiles multiple times for a connection attempt across only > one or two agents. > > Again, many thanks for the detailed response. > > Graeme > > On Thursday, July 28, 2016 at 5:33:29 PM UTC-7, Victor Fernandez wrote: >> >> Hi Graeme. >> >> According to the log, I think the problem occurs when the manager tries >> to send the merged.mg to an agent that has not sent the keep-alive in >> the last 20 minutes. This may happen if a lot of agents get connected, or >> send the keep-alive at the same time. >> >> So, if many agents send a keep-alive, the manager takes more than 20 >> minutes to send the merged.mg to an agent, and that agent hasn't sent >> the keep-alive again, this problem occurs. >> >> I did some math: the manager sleeps one second every time it sends 27 KB. >> With a 150 KB merged.mg, OSSEC takes 20 minutes to send the complete >> file to about 216 agents. >> >> The 20-minutes check appears on src/remoted/sendmsg.c: >> >> /* If we don't have the agent id, ignore it */ >> if (keys.keyentries[agentid]->rcvd < (time(0) - (2 * NOTIFY_TIME))) { >> return (-1); >> } >> >> NOTIFY_TIME is 600 (10 minutes) by default. Nevertheless OSSEC labels an >> agent as disconnected when it hasn't send the keep-alive in the last 30:30 >> minutes, as we can see at src/shared/read-agents.c: >> >> if (file_status.st_mtime > (time(0) - (3 * NOTIFY_TIME + 30))) { >> return (GA_STATUS_ACTIVE); >> } >> >> Because of this, I think that this may be an issue. >> >> I think that a good approach would be to check that there aren't alerts >> about disconnected agents that connected recently. >> >> Kind regards. >> >> >> On Thursday, July 28, 2016 at 9:43:32 AM UTC-7, Graeme Stewart wrote: >>> >>> Seeing a lot of errors in the logfiles like this: >>> >>> 2016/07/28 16:41:48 ossec-remoted: ERROR: Unable to send file 'merged.mg' >>> to agent. >>> 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating encrypted >>> message. >>> 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file 'merged.mg' >>> to agent. >>> 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating encrypted >>> message. >>> 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file 'merged.mg' >>> to agent. >>> 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating encrypted >>> message. >>> 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file 'merged.mg' >>> to agent. >>> 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating encrypted >>> message. >>> 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file 'merged.mg' >>> to agent. >>> 2016/07/28 16:41:54 ossec-remoted(1217): ERROR: Error creating encrypted >>> message. >>> 2016/07/28 16:41:54 ossec-remoted: ERROR: Unable to send file 'merged.mg' >>> to agent. >>> 2016/07/28 16:41:56 ossec-remoted(1217): ERROR: Error creating encrypted >>> message. >>> >>> Any guidance on troubleshooting? Search hasn't turned up much other than >>> delete merged.mg and restart (which we've tried to no success)... >>> >> -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
