> Why have you chosen to hide information that may be useful in debugging your > problem?
I’m truly sorry for the inconvenience but I do have some concerns of security and privacy. I confirm it is not a broadcast address because it is the public IP of the server and this issue has a probability of 1% to happen. The address cannot just be a broadcast address at 1% of the time while not at the rest of 99%. I also double checked it by SSHing to the address I copied from the kdump, if it makes sense. > So, since the manpage mentions blocking pf, I suggest the hypothesis "it > returns EACCES because pf is blocking your packets". I can think of several > ways to test that; what testing have you performed to confirm or rule out > that possibility? "doas pfctl -d; run test; doas pfctl -e”? This issue is really hard to reproduce because the application works at most of the time, but I think you are right. I’ll be watching the pf log in next weeks. > Alternatively: what's different about *that* call? Does every sento() call > on that socket fail? What is special about that socket? If other sendto() > calls succeed, what is different about that call? Earlier setsockopt() calls? The call failed once after a few days of running. I have no idea of how to present this but it seems the second and third arguments of sendto call are always changing? 3058 myapp CALL sendto(5,0x1689f5f6100,0x5d,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10) 3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy } 3058 myapp GIO fd 5 wrote 93 bytes "\M^Ai9\M^N\M^U\M-@9 \M-7\M-B\M-m\M^C.\M-ut\M^E~)\M^H'w\M-A\M-p|\M-+DK\M-V\M-8v\M-~c\240\M-=y\M-E\M-*\M-+.\M-I\M-m\M^[\M-s\M-&\\\M^GwP\M-I\r\M^[\M-C\^]B\M-%\M-<\M-q\ \M-rk\M-_\M-g\M-9Y\^B\M-+Y\M-:K\M^X\^E/*\M-4_\M-x\at\M-{\M-$\M-[\^NU\\\M^HPk\M-i\\\M-M\M-H8t " 3058 myapp RET sendto 93/0x5d 3058 myapp CALL kevent(3,0,0,0x168a7f0c000,128,0) 3058 myapp STRU struct kevent { ident=4, filter=EVFILT_READ, flags=0x61<EV_ADD|EV_CLEAR|EV_RECEIPT>, fflags=0<>, data=73, udata=0x2 } 3058 myapp RET kevent 1 3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0) 3058 myapp GIO fd 4 read 73 bytes "\0\0\0\^BE\0\0E\M^_?\0\0@\^Q\M-9\M-:\M-@\M-(_\^D\^A\^A\^A\^A\M-x\M-U\0005\0001\M^U\M-Q\M-B\M-U\^A\0\0\^A\0\0\0\0\0\^A\^Bbt\^Ehenbt\^Ccom\0\0\^A\0\^A\0\0)\^D\M-P\0\ \0\M^@\0\0\0" 3058 myapp RET read 73/0x49 3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0) 3058 myapp GIO fd 4 read 79 bytes "\0\0\0\^BE\0\0K\M-i\M^E\0\0@\^Qpo\M-@\M-(_\^D\^A\0\0\^A\M-Sy\0005\0007\M-8 \M-dB\^A\0\0\^A\0\0\0\0\0\^A\atracker\^Fcity9x\^Ccom\0\0\^A\0\^A\0\0)\^D\M-P\0\0\M^@\0\0\ \0" 3058 myapp RET read 79/0x4f 3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0) 3058 myapp RET read -1 errno 35 Resource temporarily unavailable 3058 myapp CALL read(7,0x7f7fffff07e0,0xc) 3058 myapp GIO fd 7 read 12 bytes "4z.\M-"\M-)-\M-o\^E\M-v\^Q\M-5u" 3058 myapp RET read 12/0xc 3058 myapp CALL sendto(5,0x1690776aa00,0x61,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10) 3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy } 3058 myapp GIO fd 5 wrote 97 bytes "\M-)\M-,\M-r\M-b\^R\M-Jb\M-@\M-b\M-{\M^G"\M-Z\v\M-v\M-L\^^\M-BI\M-X\M-S\M-L]\M-&\^R%\M-?\M-\\M-kN\M^G:\M-NVI\M-@s\0#\M-V\M-}\M-h\M-T\M^C\^R\M-W\M-"\^?\M^Q[:u\M-=\ \M-Q%\M-c\M-+\M-xZ\^B\M-$\M-N\M-1\^Zy\M^D 9#\^Q\M--\M^Jb\M^T( \M^@\M^L;~\M-<\M-}0\M-G\M-`4z.\M-"\M-)-\M-o\^E\M-v\^Q\M-5u" 3058 myapp RET sendto 97/0x61 3058 myapp CALL read(7,0x7f7fffff07e0,0xc) 3058 myapp GIO fd 7 read 12 bytes "\M-j\M->\M-9\^V\^U\^^\M-c/h',\M^D" 3058 myapp RET read 12/0xc 3058 myapp CALL sendto(5,0x1690776a600,0x67,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10) 3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy } 3058 myapp GIO fd 5 wrote 103 bytes "\^W\^\\M-B\M^E\M-~\M-c\M-3\M-f\M-X\M^Y\M-Z\^V\M-G\M-p\M^L\M^G\M-b\M-T\M^I\M-SWu\M-]\M^Jq\M^Q\M-Z\^Z\M^[\bW>`/fW\b\M^K\M-*\^]:E\M-T\M-[\M^X\M-5M\M^J\M-j\M-v\^^Z@:\ \M-QfQ\a-\M-\)\M-O$[\M-x]\b\M-Wh5\M^F\M^B|I> \M^CY\M-%\M-6'\M^?g9F?y\^T\M-}\^E\M-g\M-j\M->\M-9\^V\^U\^^\M-c/h',\M^D" 3058 myapp RET sendto 103/0x67 3058 myapp CALL kevent(3,0,0,0x168a7f0c000,128,0) 3058 myapp STRU struct kevent { ident=4, filter=EVFILT_READ, flags=0x61<EV_ADD|EV_CLEAR|EV_RECEIPT>, fflags=0<>, data=75, udata=0x2 } 3058 myapp RET kevent 1 3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0) 3058 myapp GIO fd 4 read 75 bytes "\0\0\0\^BE\0\0GS\M^L\0\0@\^Q\^Fm\M-@\M-(_\^D\^A\0\0\^A\M-n\M-)\0005\0003m\M-_N\M-#\^A\0\0\^A\0\0\0\0\0\^A\^Cbtx\aanifilm\^Btv\0\0\^A\0\^A\0\0)\^D\M-P\0\0\M^@\0\0\0" 3058 myapp RET read 75/0x4b 3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0) 3058 myapp RET read -1 errno 35 Resource temporarily unavailable 3058 myapp CALL read(7,0x7f7fffff07e0,0xc) 3058 myapp GIO fd 7 read 12 bytes "\M^T\M-|\M-e\M-Fk-\M^P\M-u}\M-j\M^U=" 3058 myapp RET read 12/0xc 3058 myapp CALL sendto(5,0x1690776a500,0x63,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10) 3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy } 3058 myapp GIO fd 5 wrote 99 bytes "\M-3\M-l7X{PM[\M^[\M^S\M-Rv:\M^C2\^_=-?\M^[\M-p\M-Kq;/0\M^_\M-`\M-Yx\M-B_\^CyX\M-B\M^?\M^N\M^H\M-[ykO\M-d\M-qg\M-]\M^L\M-BulvAF\M-"\M-d%)\M-Y\M-mK*`\M^L\M^B\M-I\ \M-5\^B\M-/<\M-?\0005\^Y3\M^?q\M-n\M^Ui\M-^\M^?\M^X!\M-w\^RA\M^T\M-|\M-e\M-Fk-\M^P\M-u}\M-j\M^U=" 3058 myapp RET sendto 99/0x63 3058 myapp CALL kevent(3,0,0,0x168a7f0c000,128,0) 3058 myapp STRU struct kevent { ident=4, filter=EVFILT_READ, flags=0x61<EV_ADD|EV_CLEAR|EV_RECEIPT>, fflags=0<>, data=69, udata=0x2 } 3058 myapp RET kevent 1 3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0) 3058 myapp GIO fd 4 read 69 bytes "\0\0\0\^BE\0\0AHN\0\0@\^Q\^Q\M-1\M-@\M-(_\^D\^A\0\0\^A\M-u\M-D\0005\0-\M-w\r{\M-{\^A\0\0\^A\0\0\0\0\0\^A\^Dpow7\^Ccom\0\0\^A\0\^A\0\0)\^D\M-P\0\0\M^@\0\0\0" 3058 myapp RET read 69/0x45 3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0) 3058 myapp GIO fd 4 read 79 bytes "\0\0\0\^BE\0\0K\M-{\M-3\0\0@\^Q^A\M-@\M-(_\^D\^A\0\0\^A\M-U\M^F\0005\0007\^V\M-J\M^C\M^L\^A\0\0\^A\0\0\0\0\0\^A\atracker\^Fcity9x\^Ccom\0\0\^A\0\^A\0\0)\^D\M-P\0\0\ \M^@\0\0\0" 3058 myapp RET read 79/0x4f 3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0) 3058 myapp RET read -1 errno 35 Resource temporarily unavailable 3058 myapp CALL read(7,0x7f7fffff07e0,0xc) 3058 myapp GIO fd 7 read 12 bytes "\M^Xr\M-r\M-[%\M-n\\\M^L\M-e&\M^_\M-i" 3058 myapp RET read 12/0xc 3058 myapp CALL sendto(5,0x1689f5f6500,0x5d,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10) 3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy } 3058 myapp RET sendto -1 errno 13 Permission denied 3058 myapp CALL close(5) 3058 myapp RET close 0 3058 myapp CALL munmap(0x1688d3c8000,0x1a1000) 3058 myapp RET munmap 0 3058 myapp CALL munmap(0x169192ad000,0x1a1000) 3058 myapp RET munmap 0 3058 myapp CALL munmap(0x1691cb22000,0x1a1000) 3058 myapp RET munmap 0 3058 myapp CALL munmap(0x168220f8000,0x1a1000) 3058 myapp RET munmap 0 3058 myapp CALL close(4) 3058 myapp RET close 0 3058 myapp CALL close(3) 3058 myapp RET close 0 Siegfried siegfried.le...@gmail.com > On Jun 22, 2021, at 3:20 PM, Philip Guenther <guent...@gmail.com> wrote: > > On Mon, Jun 21, 2021 at 9:07 PM Siegfried Levin <siegfried.le...@gmail.com > <mailto:siegfried.le...@gmail.com>> wrote: > Thanks a lot for the hint. Unfortunately I’m still not able to see why sendto > failed with 13 Permission denied. The AF_INET address masked is the correct > one of my server, not a broadcast address. A sendto before this one to the > same address just worked. > > 3058 myapp CALL > sendto(5,0x1689f5f6500,0x5d,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10) > 3058 myapp STRU struct sockaddr { AF_INET, xxx.xxx.xxx.xxx:yyyy } > > Why have you chosen to hide information that may be useful in debugging your > problem? > > "Hi, I'm asking for help but I have to hide addresses because...this > application is insecure if anyone else has its IP+port? Because I've never > heard of shodan and don't believe that people are constantly scanning the > Internet? And while I don't know why it's failing I'm 1000% sure that > there's no information to be gained from seeing the IP, so if it later turns > out my understanding of 'broadcast address' is incorrect, the time I've > wasted for myself and others will be...a total loss?" > > > 3058 myapp RET sendto -1 errno 13 Permission denied > 3058 myapp CALL close(5) > 3058 myapp RET close 0 > The dump file is like 600MB. I can provide more trace log if it is necessary > for locating the root cause. > > Use the scientific method: > * make a testable hypothesis > * devise a test for that > * perform the test > * determine whether the hypothesis has been ruled out or confirmed > > So, since the manpage mentions blocking pf, I suggest the hypothesis "it > returns EACCES because pf is blocking your packets". I can think of several > ways to test that; what testing have you performed to confirm or rule out > that possibility? "doas pfctl -d; run test; doas pfctl -e"? > > > Alternatively: what's different about *that* call? Does every sento() call > on that socket fail? What is special about that socket? If other sendto() > calls succeed, what is different about that call? Earlier setsockopt() calls? > > > You say "I can confirm the packet was not sent to a broadcast address": *how* > have you confirmed that your understanding of 'broadcast address' matches the > kernel's understanding? It ain't just 255.255.255.255.... > > > Philip Guenther > > > > > Siegfried > siegfried.le...@gmail.com <mailto:siegfried.le...@gmail.com> > > > > > > On Jun 15, 2021, at 8:50 PM, Theo de Raadt <dera...@openbsd.org > > <mailto:dera...@openbsd.org>> wrote: > > > > use ktrace > > > > Siegfried Levin <siegfried.le...@gmail.com > > <mailto:siegfried.le...@gmail.com>> wrote: > > > >> Hi, > >> > >> I have a application run by a normal user communicating with the server > >> with UDP. It crashes very occasionally, like once per week, due to EACCES > >> when sending a UDP packet. According to the manpage > >> (https://man.openbsd.org/OpenBSD-6.9/sendmsg.2#EACCES > >> <https://man.openbsd.org/OpenBSD-6.9/sendmsg.2#EACCES>), the reason might > >> be either being blocked by PF or sending to a broadcast address. I can > >> confirm the packet was not sent to a broadcast address. However, I cannot > >> figure out what rule could block the connection occasionally either. The > >> application can be brought back online without changing any configuration. > >> Does anyone know what might fix this? I can also rewrite the code to make > >> it ignore the error and keep trying but that might not be a proper > >> solution. Running it as root might not be a good idea, too. > >> > >> It happens since OpenBSD 6.8. Now I’m running it on 6.9. The application > >> is written in Rust. > >> > >> Siegfried > >> siegfried.le...@gmail.com <mailto:siegfried.le...@gmail.com> > >> > >> > >> > >> >