Hi Fred, We'll, problem solved...
I was able to duplicate the problem with the IP configured on em3 using # nc -lvu 10.102.88.81 5060. No traffic.
As stated before, the IP works on interface em1(10g), not em3(1g). Centos 7, Dell R630. Firewalld correct.
I started digging into the Dell Broadcom BCM57800 card. Numerous reports of issues with the ports not working. Which comes as quite a surprise, considering we have 8 x R630's running the cards since January!
I've never seen a network interface not pass the packet to the stack - kind of the whole point. Ordering replacement Dell/Intel cards now.
Thanks to all for your help!!! Anxious to put Kamailio in production to handle STIR/SHAKEN (and maybe some cool routing).
Thanks, Dan Fred Posner wrote on 10/13/21 9:39 AM:
I think at this point, it would be best to rule out your system/environment before debugging code. Have you see if another application layer program can see the SIP traffic (such as sipp)? If a netstat shows kamailio listening on the IP/port, and kamailio didn't indicate any errors binding to the IP provided, we must further rule out something on the system. Fred Posner | palner.com Matrix: @fred:matrix.lod.com o: +1 (212) 937-7844 On 10/13/21 7:58 AM, [email protected] wrote:Hi Sergey, Only testing with the SIP INVITE from one of our production SIP trunk providers. The 'length's are listed below for the tcpdump. Same packet size/contents for all tests. Only change was moving IP from interface em3 to em1. Doesn't look like packet size is the issue. Looking at some of the socket code. Could be related to socket structs, maybe an array enumeration issue. We have 4 interfaces: em1 ((10g up), em2(10g down), em3 (1g up), em4 (1g up). Just switched IP from 3rd interface (2nd up) to 1st interface (1st up) - it worked. If somebody would like to provide a debug message/location, I could recompile to test the array bucket theory. Quick way to check. Note, the config below is the entire test config. No changes. Only purpose was to receive SIP INVITE to test socket. Thanks, Dan Sergey Safarov wrote on 10/13/21 4:08 AM:you also need to check UDP packets are fragmented or not. you must receive all fragments before Kamailio is able to parse the message. On Wed, Oct 13, 2021 at 1:28 AM David Villasmil <[email protected] <mailto:[email protected]>> wrote: use the listen=udp:YOUR-LOCAL-IP:5060 then do a netstat to see if it's listening on 5060 Regards, David Villasmil email: [email protected] <mailto:[email protected]> phone: +34669448337 On Tue, Oct 12, 2021 at 11:17 PM <[email protected] <mailto:[email protected]>> wrote: Okay, think I discovered something.... I'm attempting to run the kamailio on interface em3 - no joy. I moved the 10.102.88.81 ip to the em1 interface. kamailio is showing the UDP packets as received (via my horribly simple config below). This test also answers the firewall question, UDP port is allowed. So, that prompts the question... how do you use kamailio with UDP on a specific interface ? I'm thinking it might not do it currently ? Also, forgive me if I messed up the prior reply to the list. Thanks, Dan [email protected] <mailto:[email protected]> wrote on 10/12/21 5:28 PM:Hello, The tcpdump (below) shows packet reaching em3 interface. The firewalld command (below) shows UDP port 5060 open. netcat doesn't do specific interfaces. Can't do ALL interfaces, since em4 is a pcap of hundreds of thousands of live UDP 5060. I'll have to write a standalone to prove that the firewall allows UDP 5060 through the em3 interface, post firewall... (betting it does). Should have test prog tomorrow. Thanks, Dan Daniel-Constantin Mierla wrote on 10/12/21 4:27 PM:Hello, tcpdump and other network sniffers (e.g., ngrep, sngrep) capture the sip traffic before firewall, so even if you see packets with them, the application can still not get them due to firewall. To test, you can replace kamailio with netcat listening on UDP port 5060 and see if it receives the sip traffic. Cheers, Daniel On 12.10.21 19:08, [email protected] <mailto:[email protected]> wrote:After 4 (long) days of trying to receive any UDP SIP, I'm hoping someone can point out what I'm doing wrong. I've looked at the kamailio code. I'm beginning to think it's the 'polling' code, but it looks fine. It could be Centos or a Dell network driver, but not seeing any errors anywhere. - stock build/compile... Version: kamailio 5.5.2 (x86_64/linux) 0d53d9 Compile flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES MAX_RECV_BUFFER_SIZE=262144 MAX_URI_SIZE=1024 BUF_SIZE=65535 DEFAULT PKG_SIZE=8MB DEFAULT SHM_SIZE=64MB ADAPTIVE_WAIT_LOOPS=1024 TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select Source code revision ID: 0d53d9 Compiled with: gcc 4.8.5 Compiled architecture: x86_64 Compiled on: 08:44:19 Oct 11 2021 Running Centos 7 all updated. Dell R630. # sestatus SELinux status: disabled # firewall-cmd --list-all ports: 5666/tcp 873/tcp 873/udp 80/tcp 443/tcp 5060/tcp 5060/udp 5061/tcp 5061/udp - Just trying to send SIP INVITE from trunk provider to R630 kamailio... (SIP trunk) UDP 172.200.200.202:5060 <http://172.200.200.202:5060> -> (firewall) UDP 172.250.250.162:5060 <http://172.250.250.162:5060> -> 1:1 NAT -> (R630) UDP 10.102.88.81:5060 <http://10.102.88.81:5060> - I can 'see' the packet hit the em3 network... # tcpdump -i em3 -e -vv -n port 5060 11:53:33.260892 99:99:ef:3e:69:c1 > 99:99:1c:48:4e:e3, ethertype IPv4 (0x0800), length 1366: (tos 0x0, ttl 54, id 20430, offset 0, flags [none], proto UDP (17), length 1352) 172.200.200.202.sip > 10.102.88.81.sip: [udp sum ok] SIP, length: 1324 INVITE sip:[email protected]:5060 <mailto:sip:[email protected]:5060> SIP/2.0 ... At this point, I just need to see a SIP INVITE received. Then I can go back to working on a real config. I've tried with TCP on/off, listening on all networks, alias on/off, strace and everything else I could think of.Truly appreciate any guidance. Tried every sample config possible, but I just need to test receiving the SIP UDP packet, so this 'should' be minimum config... #!KAMAILIO debug=9 log_stderror=no log_facility=LOG_LOCAL0 log_prefix="{$mt $hdr(CSeq) $ci} " children=2 disable_tcp=yes udp4_raw=on auto_aliases=no alias="sbc01.mysite.com <http://sbc01.mysite.com>" listen=udp:10.102.88.81:5060 loadmodule "debugger.so" loadmodule "jsonrpcs.so" loadmodule "kex.so" loadmodule "corex.so" loadmodule "sl.so" loadmodule "rr.so" loadmodule "pv.so" loadmodule "textops.so" loadmodule "siputils.so" loadmodule "xlog.so" loadmodule "sanity.so" loadmodule "ctl.so" loadmodule "cfg_rpc.so" modparam("debugger", "cfgtrace", 1) request_route { xlog("SIP received xxxxxxxxxxxxxxxxxxx\n"); } # kamcmd ps 11578 main process - attendant 11579 udp receiver child=0 sock=10.102.88.81:5060 <http://10.102.88.81:5060> 11580 udp receiver child=1 sock=10.102.88.81:5060 <http://10.102.88.81:5060> 11581 slow timer 11583 timer 11584 secondary timer 11585 JSONRPCS FIFO 11586 JSONRPCS DATAGRAM 11587 ctl handler - I get plenty of logging, but the logging just waits here... Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10060]: INFO: ctl [io_listener.c:210]: io_listen_loop(): io_listen_loop: using epoll_lt as the io watch method (auto detected) Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10060]: DEBUG: <core> [core/mem/q_malloc.c:374]: qm_malloc(): qm_malloc(0x7fd2a8bae010, 4800) called from core: core/io_wait.c: init_io_wait(469) Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10059]: DEBUG: <core> [core/mem/q_malloc.c:419]: qm_malloc(): qm_malloc(0x7fd2a8bae010, 65456) returns address 0x7fd2a8c652d8 frag. 0x7fd2a8c652a0 (size=65456) on 1 -th hit Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10060]: DEBUG: <core> [core/mem/q_malloc.c:419]: qm_malloc(): qm_malloc(0x7fd2a8bae010, 4800) returns address 0x7fd2a8c652d8 frag. 0x7fd2a8c652a0 (size=4800) on 1 -th hit Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10060]: DEBUG: <core> [core/mem/q_malloc.c:374]: qm_malloc(): qm_malloc(0x7fd2a8bae010, 2400) called from core: core/io_wait.c: init_io_wait(516) Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10060]: DEBUG: <core> [core/mem/q_malloc.c:419]: qm_malloc(): qm_malloc(0x7fd2a8bae010, 2400) returns address 0x7fd2a8c66600 frag. 0x7fd2a8c665c8 (size=2400) on 1 -th hit Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10060]: DEBUG: ctl [io_listener.c:246]: io_listen_loop(): io_listen_loop: adding socket 8, type 2, transport 3 (/var/run/kamailio//kamailio_ctl) Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10060]: DEBUG: ctl [../../core/io_wait.h:375]: io_watch_add(): DBG: io_watch_add(0x7fd2a70a5360, 8, 2, 0x2439960), fd_no=0 - The only other thing that I see is transport 3. Isn't that a UNIXS_SOCK? __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * [email protected] <mailto:[email protected]> Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio Advanced Training - Online Nov 08-11, 2021 (Europe Timezone) - Nov 22-25, 2021 (America Timezone) * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * [email protected] <mailto:[email protected]> Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users> __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * [email protected] <mailto:[email protected]> Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * [email protected] Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * [email protected] Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * [email protected] Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
