Dave - first, thank you very much for such a detailed explanation and submitting this back to OSSEC for the community!
I just attempted an upgrade from 2.8.3. to 2.9.3 and had, what sounds like, similar issues. Our CentOS 6.5 server didn't have ipv6 support installed or enabled. I recompiled with a workaround noted in this github issue - https://github.com/ossec/ossec-hids/pull/1259 - and was able to get syslog clients working, but my agents would still not connect. My agent logs contained the same errors... trying to connnect. Connected, and then waiting for reply. I ended up restoring my old server. Have you looked at the pull request I linked above? Are the issues related? Thanks again! On Wednesday, May 2, 2018 at 6:25:18 PM UTC-7, Dave Stoddard wrote: > > I just submitted a pull request to correct a connectivity issue I was > having with OSSEC servers running on FreeBSD 11.1. I mentioned this on the > email list a few weeks ago, but it took some time to do the work to > identify and correct the problem. I encountered a significant issue in > OSSEC 2.9.3 and the current OSSEC beta (GitHub repository) release that > affects the ability of OSSEC clients to be able to connect to FreeBSD > servers. The issue did not exist for OSSEC 2.8.3 possibly due to the fact > that it was IPv4 only. This connectivity issue probably affects all current > BSD derivatives and possibly some of the more obscure releases of Linux as > well. I have identified and resolved the issue in OSSEC by rewriting some > of the networking code, however, before I describe the solution, I wanted > to provide some background information on the problem first. > > *Problem Symptoms* > > The server is running FreeBSD 11.1 with all of the current updates. This > is a standard OSSEC server installation with all standard options enabled > and no bells and whistles (no MySQL and no MQ). After correcting for some > BSD-specific issues for header files and libraries, OSSEC installs normally > on the server with the install.sh script and the compilation completes > without any fatal errors (GCC does generate a few warnings on > analysisd/decoders/syscheck.c and analysisd/decoders/syscheck-test.c, but > these are not show stoppers). The server has an assigned IP address of > 192.168.1.80 (IPv4 private IP space) and all of the clients it associates > with are on the local 192.168.1.0/24 network. > > The OSSEC 2.9.3 client software is installed on each Windows workstation > and valid client keys are obtained for each client system. Everything > appears fine, except that there are no alerts logged on the server for any > of the configured clients. In short, the problem manifests itself as an > inability for the client systems to connect to the OSSEC server. In the > ossec.log file on each client there are messages like this that repeat over > and over: > > 2018/03/08 12:11:06 ossec-agentd: INFO: Trying to connect to server > 192.168.1.80, port 1514. > 2018/03/08 12:11:06 INFO: Connected to 192.168.1.80 at address 192.168. > 1.80:1514, port 1514 > 2018/03/08 12:11:27 ossec-agentd(4101): WARN: Waiting for server reply ( > not started). Tried: '192.168.1.80'. > > > 2018/03/08 12:12:41 ossec-agentd: INFO: Trying to connect to server > 192.168.1.80, port 1514. > 2018/03/08 12:12:41 INFO: Connected to 192.168.1.80 at address 192.168. > 1.80:1514, port 1514 > 2018/03/08 12:13:02 ossec-agentd(4101): WARN: Waiting for server reply ( > not started). Tried: '192.168.1.80'. > > > Because the clients use UDP, and UDP is a connectionless protocol, there > is no protocol handshake to ensure a connection has actually been > established like there would be with TCP. As long as the sendto() function > returns a positive integer (the number of bytes sent), the call is deemed > successful. This is true even when packets are not received on the > destination host. To compensate for this, the server is supposed to return > a control message to indicate the packet was received. Because the client > never receives this confirmation, a warning is generated in the client log > "Waiting for server reply (not started)". > > On the server, remoted starts fine. It displays the maximum number of > agents allowed (2048) and logs the fact that it successfully read the > authentication keys file. However, the server logs hint that there is an > issue when remoted starts because each client in the client.keys file > results in a message stating "Assigning counter for agent Workstation01: > '0:0'", or "Assigning sender counter: 0:0". These messages are generated > when a client is defined in the etc/client.keys file, but no initial > connection has ever occurred. This is normal for a first time start of > OSSEC, but if this happens every time you restart OSSEC for every client > defined in the system, your clients are not communicating. > > To identify whether remoted was listening on UDP port 1514, I used the > "netstat -an" command and the FreeBSD "sockstat" command (sockstat is like > "lsof" for sockets). The only address bound to port 1514 was an IPv6 > address, not the IPv4 address. The server uses only a single IPv4 address > for network communication and no IPv6 addresses. However, it turns out that > FreeBSD (and many other systems) create a link-local IPv6 address based on > the MAC address of each interface when IPv6 is not specifically configured > for the interface. This is specified in RFC 2462. > > The standard solution for this would be to specify an IP address for the > "secure" connection in the remote configuration using the <local_ip> > option. However, when that is used, OSSEC ignores it and the system > continues to bind to the IPv6 link-local address. While I could have > experimented with disabling IPv6 in the operating system, this could have > an adverse impact on other applications running on the server. > > Normally, the system should simply query all available addresses on the > system using getaddrinfo(), create a socket for each address, and bind the > address to the socket. If the protocol is TCP, then you also need to use > listen() to establish a queue. Then select() can be used to handle the > dispatch of the individual sockets as messages arrive. I have a lot of > experience with writing network interfaces in C, so I decided I would take > a look at what was going on inside OSSEC that was creating this issue for > me. > > *Initial Analysis* > > The OSSEC remoted process is responsible for handling the connections > between the clients and the server. All of this code, except for the actual > networking code, is found in the src/remoted directory. The initial > connections for the UDP secure server and the syslog server are initiated > in the HandleRemote() function inside remoted.c through calls to > OS_Bindportudp() or OS_Bindporttcp() which are found in os_net/os_net.c. > What is immediately obvious is that only one socket is returned as an > integer value, instead of a group of sockets (one for each interface and > protocol family). > > The actual processing of incoming messages is divided between secure.c for > secure UDP connections, syslog.c for syslog UDP connections, and > syslogtcp.c for syslog TCP connections. Most of the outgoing connections > are handled by send_msg.c (with some exceptions). Because there is only one > socket used per communication strategy (secure UDP, syslog UDP, and syslog > TCP), all of the routines use a while(1) loop with blocking reads to > process incoming messages. To handle multiple sockets without dragging the > CPU into the mud, these would have to be modified to use select(). > > The next thing I looked at was the code that was doing the network calls, > which is found in the file os_net/os_net.c. Both the OS_Bindportudp() and > OS_Bindporttcp() calls referred to a function called OS_Bindport() to > handle all of the network initialization code including querying the > available addresses, creating the socket, binding the address, invoking > listen() for the queue (TCP only), and returning the active socket to the > caller. > > The code in os_net/os_net.c clearly showed my dilemma; the call to > getaddrinfo() was setup to use AF_INET6 for the address family and > AI_V4MAPPED, AI_PASSIVE, and AI_ADDRCONFIG for flags. Under this > specification, IPv6 is preferred over IPv4, and if no IPv6 addresses exist > and IPv4 addresses do exist, the IPv4 address is to be returned as a mapped > IPv4 address using IPv6. A better solution is to use PF_UNSPEC for the > address family with AI_PASSIVE and AI_ADDRCONFIG to pick up all addressed > on configured interfaces. > > The original author of this module recognized this as a problem early on, > but s/he assumed that if they had AI_V4MAPPED defined in netdb.h, then the > system would handle it properly. However, if IPv4 is preferred and there > are any IPv6 addresses configured on the system, AI_V4MAPPED will not yield > any IPv4 addresses. There is a good article by IBM on this that you can > read here: > > > https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.hale001/ipv6d0131000748.htm > > This is further compounded because OSSEC is returning only a single socket > bound to a single address making the process of supporting both IPv4 and > IPv6 at the same time is impossible. The only correct solution is to modify > OSSEC to handle multiple IP addresses under both IPv4 and IPv6 > simultaneously. > > *The Solution* > > The overall solution to this problem was to return all of the addresses > that meet the selection criteria and use the select() function to return > sockets that are ready for reading. While this change affected a number of > modules in the OSSEC system, the change was relatively straight forward to > implement. A list of modules affected is provided at the end of this > document. > > Under the current version of OSSEC, a function named OS_Bindport() is used > to locate the first address that is returned from getaddrinfo(). Then a > socket for the appropriate connection type is created (SOCK_DGRAM for UDP > or SOCK_STREAM for TCP), and the address is bound to the socket. If the > connection is a TCP connection, a queue is established for the bound socket > using listen(). Then the integer value socket is returned. If the socket > value is -1, an error is assumed and various error actions are taken by the > caller. > > Under the modified version, OS_Bindport() was modified in a number of > ways. First, instead of returning an integer value representing a bound > socket to the caller, the routine now returns a pointer to an OSNetInfo > structure. The OSNetInfo structure is defined in os_net/os_net.h as follows: > > typedef struct _OSNetInfo { > fd_set fdset; /* set of sockets used by select > ()*/ > int fdmax; /* fd max socket for select() */ > int fds[FD_SETSIZE]; /* array of bound sockets for > send() */ > int fdcnt; /* number of sockets in array */ > int status; /* return status (-1 is error) */ > int retval; /* return value (additional info) > */ > } OSNetInfo; > > > In order to multiplex multiple sockets efficiently, we need to use the > select() function. The select() function operates on a struct called fd_set > that is built on the successful acquisition of a socket with a bound > address. Each successful acquisition is added to the set using the FD_SET > macro. We also track the highest value socket number returned in fdmax. > When all processing is completed, fdmax is incremented by 1 so it can be > used directly with the select() function in other modules. We also track > the individual bound sockets in an integer table called fds, and a count of > the number of sockets in the table using fdcnt. > > Because the original implementation relied upon a return value of -1 to > indicate an error, we include an element in the struct called status that > performs the same function. If status is -1, then an error has occurred. > Upon a successful return, the status element will be 0. If status is -1, > the return value element retval will contain the error number associated > with the error status. The OSNetInfo struct made the process of modifying > the OSSEC networking code relatively simple because it preserved all of the > functionality that was inherent with the original code. > > In addition to the OSNetInfo structure as a return value, the second major > change to os_net/os_net.c was the definition of hints to the getaddrinfo() > function. The getaddrinfo() function was created by the POSIX standards > committee to provide support both IPv4 and IPv6 simultaneously. The > getaddrinfo() function modifies a struct that returns a pointer to a linked > list of addresses that match the selection criteria passed to getaddrinfo() > through the hints parameter struct. > > Under the original OSSEC code, if the operating system platform had > AI_V4MAPPED defined in <netdb.h> (the vast majority of modern OS's support > this), then the hints structure passed to getaddrinfo() used AF_INET6 for > the protocol family. This essentially guaranteed that only IPv6 addresses > would be returned if they were defined on the system. Even if there are no > IPv6 addresses configured on the system, the AF_INET6 protocol family with > the AI_V4MAPPED flag will return a mapped IPv4 address in IPv6 format. > > For current *BSD systems, if an IPv6 address is not defined on an > interface, the system will create a link-local IPv6 address based on the > MAC address of each interface. IPv6 link-local addresses are essentially > useless for external communication, so the net result is that *BSD systems > (and others) are not able to communicate using IPv4. > > By modifying this to use AF_UNSPEC instead of AF_INET6, eliminating the > AI_V4MAPPED flag, and utilizing all of the addresses returned by the call > to getaddrinfo(), we have the ability to communicate through all of the > interfaces defined on the system using both IPv4 nd IPv6. Nonetheless, > there are a number of Linux systems that are working fine for OSSEC with > the AF_INET6 family and the AI_V4MAPPED flag. As such, an "#if defined" > definition was placed in the code to support the original behavior on Linux > systems, with the PF_UNSPEC behavior defined for non-Linux systems. In the > event you do want to use the PF_UNSPEC code with a Linux system (I suspect > this will become the norm as more versions of Linux adopt RFC 2462 and the > link-local address), all you need to do is to uncomment the line in the > Makefile that defines NOV4MAP and the system will use AF_UNSPEC instead of > AF_INET6. > > Once the call is made to getaddrinfo(), we utilize a loop to process each > of the addresses returned in the linked list of addresses. Each socket that > is bound successfully is added to the fd_set for subsequent select() > processing, and also added to an array of integers containing bound > sockets. The only other substantive change that was made to this module is > that we set a socket option for the SO_REUSEADDR flag. In the event OSSEC > is stopped and restarted again, the use of the SO_REUSEADDR socket option > allows OSSEC to restart immediately without having to wait for a lingering > socket to close. It also improves performance for short duration > transactions. > > The other changes in the os_net/os_net.c module include changing the > return parameter from integer to a pointer to an OSNetInfo struct for > OS_Bindportudp() and OS_Bindporttcp(), and a number of helper functions > were added to the bottom of the module to provide additional information as > the addresses are retrieved, sockets created, and addresses are bound to > sockets. > > Once the changes were made to os_net/os_net.c, the remoted struct in > config/remote-config.h was modified to add a pointer called netinfo to the > remoted struct using a typedef of OSNetInfo (directly below where the sock > element is defined), and remoted/remoted.c was modified to accept the new > OSNetInfo pointer and store it in the netinfo element in the logr > configuration struct. > > Next, I modified the routines that use the sockets to use the logr.netinfo > element instead of the logr.sock element that was originally used for > reads. Blocking reads were replaced with select() to multiplex the sockets > that became available for reading. This change affected secure.c, syslog.c, > and syslogtcp.c in the remoted subdirectory, and main-server.c in the > os_auth subdirectory. > > The remoted/sendmsg.c module was modified to use multiple sockets for > sending. In the event a message was received on one of the sockets, that > socket was used for sending the packet out. In the event no packet had been > received, the routine was modified to loop through the list of available > sockets to ensure the packet is sent. > > The Makefile was also modified to added a define for NOV4MAP, which is > commented out by default. When this is defined, Linux systems can build > remoted with AF_UNSPEC instead of AF_INET6 and the AI_V4MAPPED attribute > that was described earlier. As more operating systems adopt the link-local > specification to establish an IPv6 address based on the MAC address, this > could help Linux users with being able to support IPv4 on their systems. > > There are two other changes I made that are not related to the IP address > issue but relate to successfully compiling and installing OSSEC on FreeBSD. > Because I had to modify the Makefile for the NOV4MAP define, it made sense > to include these changes as part of this pull request. Both of these > changes are to make files that are provided with the distribution. > > The first change is to the Makefile in the top src directory. BSD systems > use the /usr/local directory for managing files and programs that are not > part of the standard operating system distribution. MySQL, Postgres, MQ, > readline, and other applications are extensions that get installed under > /usr/local. As such, it is necessary to add /usr/local/include directory > for C header files and /usr/local/lib for object libraries. This already > existed in Makefile for OpenBSD, but not for FreeBSD or NetBSD. > > The second Makefile change I made was to the LUA Makefile for BSD > distributions. Like the Makefile in the src directory described above, the > LUA build needs to have the same definitions for header files and > libraries. The file I modified under the src directory is > external/lua/src/Makefile. The modification added parameters for > "-I/usr/local/include" and "-L/usr/local/lib". With these changes, FreeBSD > and other BSD systems will compile without problems using gmake and GCC. > > *Header Files Modified* > > os_net/os_net.h - Defined a new typedef struct called OSNetInfo that is > used to return a list of bound sockets back to the caller, and changed the > return values defined for OS_Bindporttcp() and OS_Bindportudp() to use a > pointer to the OSNetInfo struct instead of the integer value they used to > return. The pointer to the OSNetInfo struct is stored in the remoted struct > that is defined as "logr" in the code. The pointer to the OSNetInfo pointer > is named netinfo. > > config/remote-config.h - Added a #include for "os_net/os_net.h" to ensure > the appropriate network function headers were defined for select() and > added a pointer to an OSNetInfo struct named netinfo to the remoted struct. > The netinfo pointer serves the same purpose as the old sock element, except > that it handles multiple sockets instead of just a single socket. The > remoted struct, which is usually referred to as "logr" in the remoted code, > is used to pass a number of configuration elements between the remoted > modules. > > *Source Modules Modified* > > os_net/os_net.c - Modified the module to return a struct called OSNetInfo > that contains the bound sockets for each of the addresses in a format ready > for use by select(), and an array of sockets that can be referenced for > individual socket usage. The OSNetInfo struct is described earlier in this > document. Also added code to use PF_UNSPEC for all interfaces except those > that are for Linux and also have AI_V4MAPPED defined in <netdb.h>. Also > added a number of helper functions to display addresses and other status > information in the ossec.log file using verbose(). > > remoted/remoted.c - Modified the module to use the new OSNetInfo struct > for return values from calls to OS_Bindporttcp() and OS_Bindportudp(). > > remoted/secure.c - Modified secure.c to utilize OSNetInfo data with > multiple bound sockets via logr.netinfo instead of the original single > bound socket with logr.sock. Also changed the message receipt loop to use > select() to multiplex the sockets, instead of blocking on a read for a > single socket. > > remoted/syslog.c - Modified syslog.c to support UDP messages the same way > we modified secure.c, including using the OSNetInfo struct for multiple > sockets and the select() call to support multiplexing. > > remoted/syslogtcp.c - Modified syslogtcp.c to support TCP messages the > same way we modified secure.c and syslog.c, including using the OSNetInfo > struct for multiple sockets and the select() call to support multiplexing. > > remoted/sendmsg.c - Modified sendmsg.c to use the bound socket array in > the netinfo structure. If we have previously identified a valid socket, we > use that socket to send on. Otherwise, we cycle through the array of valid > bound sockets until we get a successful completion on the sendto() call. > This approach appears to work well. > > os_auth/main-server.c - This module was an outlier, but because it called > OS_Bindporttcp() it had to be modified to support the new multi-address > approach. Modified main-server.c to use new OSNetInfo struct for data > returned by OS_Bindporttcp and implemented the select() call to multiplex > the group of sockets. > > *Makefile Modifications* > > Makefile - Found in the src directory at the top of the directory tree, I > added /usr/local/include to CFLAGS and /usr/local/lib OSSEC_LDFLAGS to > support successful compilation on FreeBSD and NetBSD. Also added the define > for NOV4MAP that is commented out. When NOV4MAP is defined, Linux > distributions will use AF_UNSPEC to bind sockets instead of AF_INET6 with > the AI_V4MAPPED flag. > > external/lua/src/Makefile - LUA compilation fails with the current > Makefile because the /usr/local/include directory is not specified for > headers, and the /usr/local/lib is not specified for object libraries. I > added parameters to the definitions for bsd and freebsd in the Makefile to > make sure those directories were included. This allows the LUA compile > process to complete without errors. > > *Modification Summary and Pull Request* > > I had originally performed this modification for the OSSEC 2.9.3 release > and tested it thoroughly on FreeBSD 11.1 for several weeks. I then ported > these changes into the current code branch of OSSEC that I retrieved from > GitHub on May 1, 2018. This required my changes to be re-ported into newer > versions of os_auth/main-server.c, os_net/os_net.c, and os_net/os_net.h. I > also had to re-port changes into the two Makefiles I identified earlier. I > only have a day of use on this, but it seemd to be running ithout issues. > > If anyone is interested in the changes that work for the 2.9.3 release, > you can download the modified OSSEC 2.9.3 release from > > https://networkalarmcorp.com/dnld/ossec-hids-2.9.3.1.tgz > > I should have the 2.9.3 code up on my server by Thursday noon EDT. I have > submitted a pull request for this modification to be incorporated into the > OSSEC code base. If anyone has any questions or suggestions related to this > modification, I am happy to discuss them. I hope this modification helps > the OSSEC community at large. > > Dave Stoddard > Network Alarm Corporation > https://networkalarmcorp.com > https://redgravity.net > Office: 301-850-0668 x101 > Email: dgs at networkalarmcorp dot com > -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
