I just submitted a pull request to correct a connectivity issue I was having with OSSEC servers running on FreeBSD 11.1. I mentioned this on the email list a few weeks ago, but it took some time to do the work to identify and correct the problem. I encountered a significant issue in OSSEC 2.9.3 and the current OSSEC beta (GitHub repository) release that affects the ability of OSSEC clients to be able to connect to FreeBSD servers. The issue did not exist for OSSEC 2.8.3 possibly due to the fact that it was IPv4 only. This connectivity issue probably affects all current BSD derivatives and possibly some of the more obscure releases of Linux as well. I have identified and resolved the issue in OSSEC by rewriting some of the networking code, however, before I describe the solution, I wanted to provide some background information on the problem first.
*Problem Symptoms* The server is running FreeBSD 11.1 with all of the current updates. This is a standard OSSEC server installation with all standard options enabled and no bells and whistles (no MySQL and no MQ). After correcting for some BSD-specific issues for header files and libraries, OSSEC installs normally on the server with the install.sh script and the compilation completes without any fatal errors (GCC does generate a few warnings on analysisd/decoders/syscheck.c and analysisd/decoders/syscheck-test.c, but these are not show stoppers). The server has an assigned IP address of 192.168.1.80 (IPv4 private IP space) and all of the clients it associates with are on the local 192.168.1.0/24 network. The OSSEC 2.9.3 client software is installed on each Windows workstation and valid client keys are obtained for each client system. Everything appears fine, except that there are no alerts logged on the server for any of the configured clients. In short, the problem manifests itself as an inability for the client systems to connect to the OSSEC server. In the ossec.log file on each client there are messages like this that repeat over and over: 2018/03/08 12:11:06 ossec-agentd: INFO: Trying to connect to server 192.168. 1.80, port 1514. 2018/03/08 12:11:06 INFO: Connected to 192.168.1.80 at address 192.168.1.80: 1514, port 1514 2018/03/08 12:11:27 ossec-agentd(4101): WARN: Waiting for server reply (not started). Tried: '192.168.1.80'. 2018/03/08 12:12:41 ossec-agentd: INFO: Trying to connect to server 192.168. 1.80, port 1514. 2018/03/08 12:12:41 INFO: Connected to 192.168.1.80 at address 192.168.1.80: 1514, port 1514 2018/03/08 12:13:02 ossec-agentd(4101): WARN: Waiting for server reply (not started). Tried: '192.168.1.80'. Because the clients use UDP, and UDP is a connectionless protocol, there is no protocol handshake to ensure a connection has actually been established like there would be with TCP. As long as the sendto() function returns a positive integer (the number of bytes sent), the call is deemed successful. This is true even when packets are not received on the destination host. To compensate for this, the server is supposed to return a control message to indicate the packet was received. Because the client never receives this confirmation, a warning is generated in the client log "Waiting for server reply (not started)". On the server, remoted starts fine. It displays the maximum number of agents allowed (2048) and logs the fact that it successfully read the authentication keys file. However, the server logs hint that there is an issue when remoted starts because each client in the client.keys file results in a message stating "Assigning counter for agent Workstation01: '0:0'", or "Assigning sender counter: 0:0". These messages are generated when a client is defined in the etc/client.keys file, but no initial connection has ever occurred. This is normal for a first time start of OSSEC, but if this happens every time you restart OSSEC for every client defined in the system, your clients are not communicating. To identify whether remoted was listening on UDP port 1514, I used the "netstat -an" command and the FreeBSD "sockstat" command (sockstat is like "lsof" for sockets). The only address bound to port 1514 was an IPv6 address, not the IPv4 address. The server uses only a single IPv4 address for network communication and no IPv6 addresses. However, it turns out that FreeBSD (and many other systems) create a link-local IPv6 address based on the MAC address of each interface when IPv6 is not specifically configured for the interface. This is specified in RFC 2462. The standard solution for this would be to specify an IP address for the "secure" connection in the remote configuration using the <local_ip> option. However, when that is used, OSSEC ignores it and the system continues to bind to the IPv6 link-local address. While I could have experimented with disabling IPv6 in the operating system, this could have an adverse impact on other applications running on the server. Normally, the system should simply query all available addresses on the system using getaddrinfo(), create a socket for each address, and bind the address to the socket. If the protocol is TCP, then you also need to use listen() to establish a queue. Then select() can be used to handle the dispatch of the individual sockets as messages arrive. I have a lot of experience with writing network interfaces in C, so I decided I would take a look at what was going on inside OSSEC that was creating this issue for me. *Initial Analysis* The OSSEC remoted process is responsible for handling the connections between the clients and the server. All of this code, except for the actual networking code, is found in the src/remoted directory. The initial connections for the UDP secure server and the syslog server are initiated in the HandleRemote() function inside remoted.c through calls to OS_Bindportudp() or OS_Bindporttcp() which are found in os_net/os_net.c. What is immediately obvious is that only one socket is returned as an integer value, instead of a group of sockets (one for each interface and protocol family). The actual processing of incoming messages is divided between secure.c for secure UDP connections, syslog.c for syslog UDP connections, and syslogtcp.c for syslog TCP connections. Most of the outgoing connections are handled by send_msg.c (with some exceptions). Because there is only one socket used per communication strategy (secure UDP, syslog UDP, and syslog TCP), all of the routines use a while(1) loop with blocking reads to process incoming messages. To handle multiple sockets without dragging the CPU into the mud, these would have to be modified to use select(). The next thing I looked at was the code that was doing the network calls, which is found in the file os_net/os_net.c. Both the OS_Bindportudp() and OS_Bindporttcp() calls referred to a function called OS_Bindport() to handle all of the network initialization code including querying the available addresses, creating the socket, binding the address, invoking listen() for the queue (TCP only), and returning the active socket to the caller. The code in os_net/os_net.c clearly showed my dilemma; the call to getaddrinfo() was setup to use AF_INET6 for the address family and AI_V4MAPPED, AI_PASSIVE, and AI_ADDRCONFIG for flags. Under this specification, IPv6 is preferred over IPv4, and if no IPv6 addresses exist and IPv4 addresses do exist, the IPv4 address is to be returned as a mapped IPv4 address using IPv6. A better solution is to use PF_UNSPEC for the address family with AI_PASSIVE and AI_ADDRCONFIG to pick up all addressed on configured interfaces. The original author of this module recognized this as a problem early on, but s/he assumed that if they had AI_V4MAPPED defined in netdb.h, then the system would handle it properly. However, if IPv4 is preferred and there are any IPv6 addresses configured on the system, AI_V4MAPPED will not yield any IPv4 addresses. There is a good article by IBM on this that you can read here: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.hale001/ipv6d0131000748.htm This is further compounded because OSSEC is returning only a single socket bound to a single address making the process of supporting both IPv4 and IPv6 at the same time is impossible. The only correct solution is to modify OSSEC to handle multiple IP addresses under both IPv4 and IPv6 simultaneously. *The Solution* The overall solution to this problem was to return all of the addresses that meet the selection criteria and use the select() function to return sockets that are ready for reading. While this change affected a number of modules in the OSSEC system, the change was relatively straight forward to implement. A list of modules affected is provided at the end of this document. Under the current version of OSSEC, a function named OS_Bindport() is used to locate the first address that is returned from getaddrinfo(). Then a socket for the appropriate connection type is created (SOCK_DGRAM for UDP or SOCK_STREAM for TCP), and the address is bound to the socket. If the connection is a TCP connection, a queue is established for the bound socket using listen(). Then the integer value socket is returned. If the socket value is -1, an error is assumed and various error actions are taken by the caller. Under the modified version, OS_Bindport() was modified in a number of ways. First, instead of returning an integer value representing a bound socket to the caller, the routine now returns a pointer to an OSNetInfo structure. The OSNetInfo structure is defined in os_net/os_net.h as follows: typedef struct _OSNetInfo { fd_set fdset; /* set of sockets used by select ()*/ int fdmax; /* fd max socket for select() */ int fds[FD_SETSIZE]; /* array of bound sockets for send() */ int fdcnt; /* number of sockets in array */ int status; /* return status (-1 is error) */ int retval; /* return value (additional info) */ } OSNetInfo; In order to multiplex multiple sockets efficiently, we need to use the select() function. The select() function operates on a struct called fd_set that is built on the successful acquisition of a socket with a bound address. Each successful acquisition is added to the set using the FD_SET macro. We also track the highest value socket number returned in fdmax. When all processing is completed, fdmax is incremented by 1 so it can be used directly with the select() function in other modules. We also track the individual bound sockets in an integer table called fds, and a count of the number of sockets in the table using fdcnt. Because the original implementation relied upon a return value of -1 to indicate an error, we include an element in the struct called status that performs the same function. If status is -1, then an error has occurred. Upon a successful return, the status element will be 0. If status is -1, the return value element retval will contain the error number associated with the error status. The OSNetInfo struct made the process of modifying the OSSEC networking code relatively simple because it preserved all of the functionality that was inherent with the original code. In addition to the OSNetInfo structure as a return value, the second major change to os_net/os_net.c was the definition of hints to the getaddrinfo() function. The getaddrinfo() function was created by the POSIX standards committee to provide support both IPv4 and IPv6 simultaneously. The getaddrinfo() function modifies a struct that returns a pointer to a linked list of addresses that match the selection criteria passed to getaddrinfo() through the hints parameter struct. Under the original OSSEC code, if the operating system platform had AI_V4MAPPED defined in <netdb.h> (the vast majority of modern OS's support this), then the hints structure passed to getaddrinfo() used AF_INET6 for the protocol family. This essentially guaranteed that only IPv6 addresses would be returned if they were defined on the system. Even if there are no IPv6 addresses configured on the system, the AF_INET6 protocol family with the AI_V4MAPPED flag will return a mapped IPv4 address in IPv6 format. For current *BSD systems, if an IPv6 address is not defined on an interface, the system will create a link-local IPv6 address based on the MAC address of each interface. IPv6 link-local addresses are essentially useless for external communication, so the net result is that *BSD systems (and others) are not able to communicate using IPv4. By modifying this to use AF_UNSPEC instead of AF_INET6, eliminating the AI_V4MAPPED flag, and utilizing all of the addresses returned by the call to getaddrinfo(), we have the ability to communicate through all of the interfaces defined on the system using both IPv4 nd IPv6. Nonetheless, there are a number of Linux systems that are working fine for OSSEC with the AF_INET6 family and the AI_V4MAPPED flag. As such, an "#if defined" definition was placed in the code to support the original behavior on Linux systems, with the PF_UNSPEC behavior defined for non-Linux systems. In the event you do want to use the PF_UNSPEC code with a Linux system (I suspect this will become the norm as more versions of Linux adopt RFC 2462 and the link-local address), all you need to do is to uncomment the line in the Makefile that defines NOV4MAP and the system will use AF_UNSPEC instead of AF_INET6. Once the call is made to getaddrinfo(), we utilize a loop to process each of the addresses returned in the linked list of addresses. Each socket that is bound successfully is added to the fd_set for subsequent select() processing, and also added to an array of integers containing bound sockets. The only other substantive change that was made to this module is that we set a socket option for the SO_REUSEADDR flag. In the event OSSEC is stopped and restarted again, the use of the SO_REUSEADDR socket option allows OSSEC to restart immediately without having to wait for a lingering socket to close. It also improves performance for short duration transactions. The other changes in the os_net/os_net.c module include changing the return parameter from integer to a pointer to an OSNetInfo struct for OS_Bindportudp() and OS_Bindporttcp(), and a number of helper functions were added to the bottom of the module to provide additional information as the addresses are retrieved, sockets created, and addresses are bound to sockets. Once the changes were made to os_net/os_net.c, the remoted struct in config/remote-config.h was modified to add a pointer called netinfo to the remoted struct using a typedef of OSNetInfo (directly below where the sock element is defined), and remoted/remoted.c was modified to accept the new OSNetInfo pointer and store it in the netinfo element in the logr configuration struct. Next, I modified the routines that use the sockets to use the logr.netinfo element instead of the logr.sock element that was originally used for reads. Blocking reads were replaced with select() to multiplex the sockets that became available for reading. This change affected secure.c, syslog.c, and syslogtcp.c in the remoted subdirectory, and main-server.c in the os_auth subdirectory. The remoted/sendmsg.c module was modified to use multiple sockets for sending. In the event a message was received on one of the sockets, that socket was used for sending the packet out. In the event no packet had been received, the routine was modified to loop through the list of available sockets to ensure the packet is sent. The Makefile was also modified to added a define for NOV4MAP, which is commented out by default. When this is defined, Linux systems can build remoted with AF_UNSPEC instead of AF_INET6 and the AI_V4MAPPED attribute that was described earlier. As more operating systems adopt the link-local specification to establish an IPv6 address based on the MAC address, this could help Linux users with being able to support IPv4 on their systems. There are two other changes I made that are not related to the IP address issue but relate to successfully compiling and installing OSSEC on FreeBSD. Because I had to modify the Makefile for the NOV4MAP define, it made sense to include these changes as part of this pull request. Both of these changes are to make files that are provided with the distribution. The first change is to the Makefile in the top src directory. BSD systems use the /usr/local directory for managing files and programs that are not part of the standard operating system distribution. MySQL, Postgres, MQ, readline, and other applications are extensions that get installed under /usr/local. As such, it is necessary to add /usr/local/include directory for C header files and /usr/local/lib for object libraries. This already existed in Makefile for OpenBSD, but not for FreeBSD or NetBSD. The second Makefile change I made was to the LUA Makefile for BSD distributions. Like the Makefile in the src directory described above, the LUA build needs to have the same definitions for header files and libraries. The file I modified under the src directory is external/lua/src/Makefile. The modification added parameters for "-I/usr/local/include" and "-L/usr/local/lib". With these changes, FreeBSD and other BSD systems will compile without problems using gmake and GCC. *Header Files Modified* os_net/os_net.h - Defined a new typedef struct called OSNetInfo that is used to return a list of bound sockets back to the caller, and changed the return values defined for OS_Bindporttcp() and OS_Bindportudp() to use a pointer to the OSNetInfo struct instead of the integer value they used to return. The pointer to the OSNetInfo struct is stored in the remoted struct that is defined as "logr" in the code. The pointer to the OSNetInfo pointer is named netinfo. config/remote-config.h - Added a #include for "os_net/os_net.h" to ensure the appropriate network function headers were defined for select() and added a pointer to an OSNetInfo struct named netinfo to the remoted struct. The netinfo pointer serves the same purpose as the old sock element, except that it handles multiple sockets instead of just a single socket. The remoted struct, which is usually referred to as "logr" in the remoted code, is used to pass a number of configuration elements between the remoted modules. *Source Modules Modified* os_net/os_net.c - Modified the module to return a struct called OSNetInfo that contains the bound sockets for each of the addresses in a format ready for use by select(), and an array of sockets that can be referenced for individual socket usage. The OSNetInfo struct is described earlier in this document. Also added code to use PF_UNSPEC for all interfaces except those that are for Linux and also have AI_V4MAPPED defined in <netdb.h>. Also added a number of helper functions to display addresses and other status information in the ossec.log file using verbose(). remoted/remoted.c - Modified the module to use the new OSNetInfo struct for return values from calls to OS_Bindporttcp() and OS_Bindportudp(). remoted/secure.c - Modified secure.c to utilize OSNetInfo data with multiple bound sockets via logr.netinfo instead of the original single bound socket with logr.sock. Also changed the message receipt loop to use select() to multiplex the sockets, instead of blocking on a read for a single socket. remoted/syslog.c - Modified syslog.c to support UDP messages the same way we modified secure.c, including using the OSNetInfo struct for multiple sockets and the select() call to support multiplexing. remoted/syslogtcp.c - Modified syslogtcp.c to support TCP messages the same way we modified secure.c and syslog.c, including using the OSNetInfo struct for multiple sockets and the select() call to support multiplexing. remoted/sendmsg.c - Modified sendmsg.c to use the bound socket array in the netinfo structure. If we have previously identified a valid socket, we use that socket to send on. Otherwise, we cycle through the array of valid bound sockets until we get a successful completion on the sendto() call. This approach appears to work well. os_auth/main-server.c - This module was an outlier, but because it called OS_Bindporttcp() it had to be modified to support the new multi-address approach. Modified main-server.c to use new OSNetInfo struct for data returned by OS_Bindporttcp and implemented the select() call to multiplex the group of sockets. *Makefile Modifications* Makefile - Found in the src directory at the top of the directory tree, I added /usr/local/include to CFLAGS and /usr/local/lib OSSEC_LDFLAGS to support successful compilation on FreeBSD and NetBSD. Also added the define for NOV4MAP that is commented out. When NOV4MAP is defined, Linux distributions will use AF_UNSPEC to bind sockets instead of AF_INET6 with the AI_V4MAPPED flag. external/lua/src/Makefile - LUA compilation fails with the current Makefile because the /usr/local/include directory is not specified for headers, and the /usr/local/lib is not specified for object libraries. I added parameters to the definitions for bsd and freebsd in the Makefile to make sure those directories were included. This allows the LUA compile process to complete without errors. *Modification Summary and Pull Request* I had originally performed this modification for the OSSEC 2.9.3 release and tested it thoroughly on FreeBSD 11.1 for several weeks. I then ported these changes into the current code branch of OSSEC that I retrieved from GitHub on May 1, 2018. This required my changes to be re-ported into newer versions of os_auth/main-server.c, os_net/os_net.c, and os_net/os_net.h. I also had to re-port changes into the two Makefiles I identified earlier. I only have a day of use on this, but it seemd to be running ithout issues. If anyone is interested in the changes that work for the 2.9.3 release, you can download the modified OSSEC 2.9.3 release from https://networkalarmcorp.com/dnld/ossec-hids-2.9.3.1.tgz I should have the 2.9.3 code up on my server by Thursday noon EDT. I have submitted a pull request for this modification to be incorporated into the OSSEC code base. If anyone has any questions or suggestions related to this modification, I am happy to discuss them. I hope this modification helps the OSSEC community at large. Dave Stoddard Network Alarm Corporation https://networkalarmcorp.com https://redgravity.net Office: 301-850-0668 x101 Email: dgs at networkalarmcorp dot com -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
