This is not a traffic issue, the packets (if sent) DO get where they are expected to get. Reasons why i'm sure of this: - Logs from the broken agent DO get on the server and i can see them in the WUI, just the agent in WUI and tools is reported as Inactive. - On a good and on the broken agent machines i DO see ESTABLISHED connections from the machine to the OSSEC server on port 1514. - All agents have only one network inteface except for the broken agent that has an external, internal and VPN tunnel interface. - On the broken agent packets from the agent towards the server DO reach the server (i can see events in the WUI) - I could not "see" either on the broken agent or on the server any reply packets that the server should send to an agent.
Also, from my testing, unfortunatelly i'm very dissapointed with OSSEC debug support. I started two agents' agentd (a good one and the broken one) in debug mode, as well as the server's remoted in debug mode, and i see no logs whatsoever that might enlighten what is going on. Both the good and the broken agent report lines like this every 8 minutes: Good agent: 2011/04/08 11:34:24 ossec-agentd: DEBUG: Sending agent notification. 2011/04/08 11:42:33 ossec-agentd: DEBUG: Sending agent notification. [snip until agent restart] 2011/04/08 11:51:24 ossec-agentd: INFO: Assigning counter for agent xxx-snl17: '4:8881'. 2011/04/08 11:51:24 ossec-agentd: INFO: Assigning sender counter: 297:5009 Broken agent: 2011/04/08 11:33:04 ossec-agentd: DEBUG: Sending agent notification. 2011/04/08 11:41:36 ossec-agentd: DEBUG: Sending agent notification. [snip until agent restart] 2011/04/08 11:52:15 ossec-agentd: INFO: Assigning counter for agent xxx_snl1: '2:4459'. 2011/04/08 11:52:15 ossec-agentd: INFO: Assigning sender counter: 179:8253 so i can't tell anything from this. The server logs only the following when started in debug: 2011/04/08 11:51:17 ossec-remoted: DEBUG: Starting ... 2011/04/08 11:51:17 ossec-remoted: INFO: Started (pid: 15679). 2011/04/08 11:51:17 ossec-remoted: DEBUG: Forking remoted: '0'. 2011/04/08 11:51:17 ossec-remoted: INFO: Started (pid: 15681). 2011/04/08 11:51:17 ossec-remoted: DEBUG: Running manager_init 2011/04/08 11:51:17 ossec-remoted: INFO: (unix_domain) Maximum send buffer set to: '108544'. 2011/04/08 11:51:17 ossec-remoted(4111): INFO: Maximum number of agents allowed: '255'. 2011/04/08 11:51:17 ossec-remoted(1410): INFO: Reading authentication keys file. 2011/04/08 11:51:17 ossec-remoted: DEBUG: OS_StartCounter. 2011/04/08 11:51:17 ossec-remoted: OS_StartCounter: keysize: 11 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent xxx-snl17: '297:4858'. 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent xxx_snl8: '227:2834'. 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent xxx_snl9: '287:7960'. 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent xxx_snl12: '194:9729'. 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent xxx_snl3: '655:7394'. 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent xxx_snl13: '28:8244'. 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent xxx_snl15: '60:6841'. 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent xxx_snl16: '70:170'. 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent xxx_snl5: '35:2146'. 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent xxx_snl11: '3334:9345'. 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent xxx_snl1: '98:5516'. 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning sender counter: 4:9901 2011/04/08 11:51:17 ossec-remoted: DEBUG: OS_StartCounter completed. However, as you can see from the log snippets above, first i restarted the server in debug mode (at 11:51:17 - remoted.debug = 2), then restarted the agents in debug mode (at 11:51:24 and 11:52:15 - agentd.debug = 2). The server does not log anything about that agents "reconnecting" (WTF?, it is in DEBUG mode, i would expect detailed debug logging). But: - the good agent reports counter 297:5009 while the server a few seconds ago reported 297:4858 -> LOOKS GOOD - the broken agent reports counter 179:8253 while the server a few seconds ago reported 98:5516 -> BROKEN So at least i have confirmation something inside OSSEC logic should detect something is broken, but not even in debug mode nothing gets logged. I will try to make OSSEC server listen again on both the LAN and the VPN tunnel interface again, maybe some magic (from not listening on 0.0.0.0 but on explicit IPs) will fix the problem. As a final note, i have run a tcpdump on the server, checking for packets that might originate from the server (on all IPs possible) towards the broken agent and.. no traffic whatsoever. So i keep my belief that for some reason OSSEC remoted does NOT send the reply packets it's supposed to send. Any other ideas? On Thu, Apr 7, 2011 at 12:32 AM, joshua.gruber <[email protected]>wrote: > Have you tried testing traffic on a test port to make certain traffic > is getting there and coming from the direction you expect? For > instance: > > on the good server: > tcpdump -n 'port 55512' > > on the bad agent machine: > echo "hello?" | netcat goodserverip 55512 > > See if the traffic gets to the good server, and if so whether it comes > from the IP you expected it to. > > On Apr 4, 2:07 pm, Valentin Avram <[email protected]> wrote: >
