Awesome! Many thanks, this is exactly what I was looking for. On Friday, July 29, 2016 at 12:16:35 PM UTC-7, Victor Fernandez wrote: > > Hi Graeme. > > I agree, it would be great to print on the log that the agent became > disconnected. The SEC_ERROR definition is shared between manager and > agents, but it's possible to extend some other messages. In fact, the line > at sendmsg.c that tests if the agent is disconnected (more than 20 > minutes since the last keep-alive) is the only one that doesn't log an > error. > > I did some modifications at the Wazuh repository, maybe it's useful to you: > > > https://github.com/wazuh/ossec-wazuh/commit/efbd5c17cc3ea5109ea978208b11da6c98fa3083 > > See below an example of the new log format for the error: > > 2016/07/29 11:43:57 ossec-remoted(1245): ERROR: Sending message to > disconnected agent '001'. > 2016/07/29 11:43:57 ossec-remoted(1217): ERROR: Error creating encrypted > message. > 2016/07/29 11:43:57 ossec-remoted(1246): ERROR: Unable to send file ' > merged.mg' to agent '001' (centos). > > I hope this leads you to find the problem. > > Kind regards. > > > On Friday, July 29, 2016 at 8:19:56 AM UTC-7, Graeme Stewart wrote: >> >> Hi Victor, >> >> Huge thanks for the detail, this would explain exactly why we're seeing >> this; our OSSEC managers are likely overloaded. >> >> It would be very helpful to include the agentid in the logfile to >> understand / track where this is occurring and the number of unique agents >> that are impacted, perhaps something like: >> >> From: src/error_messages/error_messages.h >> #define SEC_ERROR "%s(1217): ERROR: Error creating encrypted >> message for: '%s')." >> >> Then inside: src/remoted/sendmsg.c >> msg_size = CreateSecMSG(&keys, msg, crypt_msg, agentid); >> if (msg_size == 0) { >> merror(SEC_ERROR, ARGV0, agentid); >> return (-1); >> } >> >> The clustered nature of this issue leads me to suspect it's repeating >> this error in the logfiles multiple times for a connection attempt across >> only one or two agents. >> >> Again, many thanks for the detailed response. >> >> Graeme >> >> On Thursday, July 28, 2016 at 5:33:29 PM UTC-7, Victor Fernandez wrote: >>> >>> Hi Graeme. >>> >>> According to the log, I think the problem occurs when the manager tries >>> to send the merged.mg to an agent that has not sent the keep-alive in >>> the last 20 minutes. This may happen if a lot of agents get connected, or >>> send the keep-alive at the same time. >>> >>> So, if many agents send a keep-alive, the manager takes more than 20 >>> minutes to send the merged.mg to an agent, and that agent hasn't sent >>> the keep-alive again, this problem occurs. >>> >>> I did some math: the manager sleeps one second every time it sends 27 >>> KB. With a 150 KB merged.mg, OSSEC takes 20 minutes to send the >>> complete file to about 216 agents. >>> >>> The 20-minutes check appears on src/remoted/sendmsg.c: >>> >>> /* If we don't have the agent id, ignore it */ >>> if (keys.keyentries[agentid]->rcvd < (time(0) - (2 * NOTIFY_TIME))) { >>> return (-1); >>> } >>> >>> NOTIFY_TIME is 600 (10 minutes) by default. Nevertheless OSSEC labels an >>> agent as disconnected when it hasn't send the keep-alive in the last 30:30 >>> minutes, as we can see at src/shared/read-agents.c: >>> >>> if (file_status.st_mtime > (time(0) - (3 * NOTIFY_TIME + 30))) { >>> return (GA_STATUS_ACTIVE); >>> } >>> >>> Because of this, I think that this may be an issue. >>> >>> I think that a good approach would be to check that there aren't alerts >>> about disconnected agents that connected recently. >>> >>> Kind regards. >>> >>> >>> On Thursday, July 28, 2016 at 9:43:32 AM UTC-7, Graeme Stewart wrote: >>>> >>>> Seeing a lot of errors in the logfiles like this: >>>> >>>> 2016/07/28 16:41:48 ossec-remoted: ERROR: Unable to send file ' >>>> merged.mg' to agent. >>>> 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating >>>> encrypted message. >>>> 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file ' >>>> merged.mg' to agent. >>>> 2016/07/28 16:41:50 ossec-remoted(1217): ERROR: Error creating >>>> encrypted message. >>>> 2016/07/28 16:41:50 ossec-remoted: ERROR: Unable to send file ' >>>> merged.mg' to agent. >>>> 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating >>>> encrypted message. >>>> 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file ' >>>> merged.mg' to agent. >>>> 2016/07/28 16:41:52 ossec-remoted(1217): ERROR: Error creating >>>> encrypted message. >>>> 2016/07/28 16:41:52 ossec-remoted: ERROR: Unable to send file ' >>>> merged.mg' to agent. >>>> 2016/07/28 16:41:54 ossec-remoted(1217): ERROR: Error creating >>>> encrypted message. >>>> 2016/07/28 16:41:54 ossec-remoted: ERROR: Unable to send file ' >>>> merged.mg' to agent. >>>> 2016/07/28 16:41:56 ossec-remoted(1217): ERROR: Error creating >>>> encrypted message. >>>> >>>> Any guidance on troubleshooting? Search hasn't turned up much other >>>> than delete merged.mg and restart (which we've tried to no success)... >>>> >>>
-- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
