Problem solved after relaxing some firewall rules on the ossec server. It seems it was a problem with the ip remoted uses to send the reply packets to the agent. For now i added all possibilities of ip combinations, so i will try to find the needed once and post back here. I still believe ossec uses a wrong ip somewhere.
I also found out that local_ip configuration entry overrides any previous local_ip entry, so you cant listen on 2 ips from X ips on a server. On Apr 8, 2011 12:27 PM, "Valentin Avram" <[email protected]> wrote: > This is not a traffic issue, the packets (if sent) DO get where they are > expected to get. > Reasons why i'm sure of this: > - Logs from the broken agent DO get on the server and i can see them in the > WUI, just the agent in WUI and tools is reported as Inactive. > - On a good and on the broken agent machines i DO see ESTABLISHED > connections from the machine to the OSSEC server on port 1514. > - All agents have only one network inteface except for the broken agent that > has an external, internal and VPN tunnel interface. > - On the broken agent packets from the agent towards the server DO reach the > server (i can see events in the WUI) > - I could not "see" either on the broken agent or on the server any reply > packets that the server should send to an agent. > > Also, from my testing, unfortunatelly i'm very dissapointed with OSSEC debug > support. I started two agents' agentd (a good one and the broken one) in > debug mode, as well as the server's remoted in debug mode, and i see no logs > whatsoever that might enlighten what is going on. > > Both the good and the broken agent report lines like this every 8 minutes: > > Good agent: > 2011/04/08 11:34:24 ossec-agentd: DEBUG: Sending agent notification. > 2011/04/08 11:42:33 ossec-agentd: DEBUG: Sending agent notification. > [snip until agent restart] > 2011/04/08 11:51:24 ossec-agentd: INFO: Assigning counter for agent > xxx-snl17: '4:8881'. > 2011/04/08 11:51:24 ossec-agentd: INFO: Assigning sender counter: 297:5009 > > Broken agent: > 2011/04/08 11:33:04 ossec-agentd: DEBUG: Sending agent notification. > 2011/04/08 11:41:36 ossec-agentd: DEBUG: Sending agent notification. > [snip until agent restart] > 2011/04/08 11:52:15 ossec-agentd: INFO: Assigning counter for agent > xxx_snl1: '2:4459'. > 2011/04/08 11:52:15 ossec-agentd: INFO: Assigning sender counter: 179:8253 > > so i can't tell anything from this. > > The server logs only the following when started in debug: > 2011/04/08 11:51:17 ossec-remoted: DEBUG: Starting ... > 2011/04/08 11:51:17 ossec-remoted: INFO: Started (pid: 15679). > 2011/04/08 11:51:17 ossec-remoted: DEBUG: Forking remoted: '0'. > 2011/04/08 11:51:17 ossec-remoted: INFO: Started (pid: 15681). > 2011/04/08 11:51:17 ossec-remoted: DEBUG: Running manager_init > 2011/04/08 11:51:17 ossec-remoted: INFO: (unix_domain) Maximum send buffer > set to: '108544'. > 2011/04/08 11:51:17 ossec-remoted(4111): INFO: Maximum number of agents > allowed: '255'. > 2011/04/08 11:51:17 ossec-remoted(1410): INFO: Reading authentication keys > file. > 2011/04/08 11:51:17 ossec-remoted: DEBUG: OS_StartCounter. > 2011/04/08 11:51:17 ossec-remoted: OS_StartCounter: keysize: 11 > 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent > xxx-snl17: '297:4858'. > 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent > xxx_snl8: '227:2834'. > 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent > xxx_snl9: '287:7960'. > 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent > xxx_snl12: '194:9729'. > 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent > xxx_snl3: '655:7394'. > 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent > xxx_snl13: '28:8244'. > 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent > xxx_snl15: '60:6841'. > 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent > xxx_snl16: '70:170'. > 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent > xxx_snl5: '35:2146'. > 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent > xxx_snl11: '3334:9345'. > 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning counter for agent > xxx_snl1: '98:5516'. > 2011/04/08 11:51:17 ossec-remoted: INFO: Assigning sender counter: 4:9901 > 2011/04/08 11:51:17 ossec-remoted: DEBUG: OS_StartCounter completed. > > However, as you can see from the log snippets above, first i restarted the > server in debug mode (at 11:51:17 - remoted.debug = 2), then restarted the > agents in debug mode (at 11:51:24 and 11:52:15 - agentd.debug = 2). > > The server does not log anything about that agents "reconnecting" (WTF?, it > is in DEBUG mode, i would expect detailed debug logging). > But: > - the good agent reports counter 297:5009 while the server a few seconds ago > reported 297:4858 -> LOOKS GOOD > - the broken agent reports counter 179:8253 while the server a few seconds > ago reported 98:5516 -> BROKEN > > So at least i have confirmation something inside OSSEC logic should detect > something is broken, but not even in debug mode nothing gets logged. > > I will try to make OSSEC server listen again on both the LAN and the VPN > tunnel interface again, maybe some magic (from not listening on 0.0.0.0 but > on explicit IPs) will fix the problem. > > As a final note, i have run a tcpdump on the server, checking for packets > that might originate from the server (on all IPs possible) towards the > broken agent and.. no traffic whatsoever. So i keep my belief that for some > reason OSSEC remoted does NOT send the reply packets it's supposed to send. > > Any other ideas? > > On Thu, Apr 7, 2011 at 12:32 AM, joshua.gruber <[email protected] >wrote: > >> Have you tried testing traffic on a test port to make certain traffic >> is getting there and coming from the direction you expect? For >> instance: >> >> on the good server: >> tcpdump -n 'port 55512' >> >> on the bad agent machine: >> echo "hello?" | netcat goodserverip 55512 >> >> See if the traffic gets to the good server, and if so whether it comes >> from the IP you expected it to. >> >> On Apr 4, 2:07 pm, Valentin Avram <[email protected]> wrote: >>
