Re: Possible sasyncd memory leak ?
On Sat, Mar 09, 2019 at 12:10:34PM +0100, Michał Koc wrote: > W dniu 09.03.2019 o 08:15, Otto Moerbeek pisze: > > On Fri, Mar 08, 2019 at 12:03:25PM +0100, Michał Koc wrote: > > > > > Hi all, > > > > > > We have a triple redundant vpn gateway setup with sasyncd running and tons > > > of tunnels, about 1000 flows. > > > > > > Looking at the graph of memory usage, you can clearly see that something > > > is > > > sucking up the memory. > > > > > > The graph can be viewed here: https://pasteboard.co/I4sjzQ8.jpg > > > > > > Looking at the ps, sasyncd shows huge memory consumption: > > > > > > USER PID %CPU %MEM VSZ RSS TT STAT > > > STARTED TIME COMMAND > > > _isakmpd 33560 0.0 17.0 699264 708508 ?? S > > > 26Feb19 6:58.81 /usr/sbin/sasyncd > > > > > > It only happens on the master node. Slaves do not show such a behavior. > > > > > > There is nothing about sasyncd in the logs. > > > > > > After sasyncd restart memory consumption is minimal, but tends to grow. > > > > > > Is it normal ? or am I missing something ? > > > > > > Best regards > > > M.K. > > > > > This is not normal. You could try to run with -vv to see if some error > > path is taken that triggers a leak. > > > > -Otto > > > Should I look for something specific ? > > The log grows pretty fast and it looks like it could contain some security > data which I wouldn't like to post online. > > The statistics of the log(about 2 hours) looks like this: > carp_init: 1 > config: 7 > monitor_get_pfkey_snap: 4 > monitor_loop: 1 > net: 1 > net_connect: 3 > net_ctl: 4 > net_disconnect_peer: 3 > net_handle_messages: 2 > net_queue: 91780 > net_read: 10 > net_send_messages: 39192 > pfkey_send_flush: 4 > pfkey_snapshot: 6832 > timer_add: 19 > timer_run: 18 > > Best regards > M.K. > Just the counts does not reveal anything. I did a quick audit of the memory allocation logic of sasyncd and did not spot an error. If you do not want to post the logs, you'll neeed to analyze them yourself. This requires matching the log lines to the code and tracking where stuff gets allocated and deallocated. Some digging could reveal the error. I used to run sasyncd, but I do no longer. Settig up a test env is quite some work I do not have time for. -Otto
How to make X listen tcp again?
The default changed, X does not receive Tcp connections. In FreeBSD I solved the problem with a file .xserverrc in my home directory with exec /usr/X11R6/bin/Xorg -listen tcp But this does not work with OpenBSD 6.4 (X does not even execute .xinitrc, I start X with xinit). Any hint? Thanks Rodrigo
Re: How to make X listen tcp again?
Well, now the way below reads .xinitrc, perhaps a lapsus when typing before. With "ssh -X" on the X server I get widgets from the remote server, but not without "ssh -X". Typing in the remote computer I get the following error: # env DISPLAY=192.168.178.47:0.0 xterm xterm: Xt error: Can't open display: 192.168.178.47:0.0 Is there another new security measure I do not know? BTW: I did "xhost +" in the server. Rodrigo On Sat, 9 Mar 2019, Roderick wrote: The default changed, X does not receive Tcp connections. In FreeBSD I solved the problem with a file .xserverrc in my home directory with exec /usr/X11R6/bin/Xorg -listen tcp But this does not work with OpenBSD 6.4 (X does not even execute .xinitrc, I start X with xinit).
Re: Possible sasyncd memory leak ?
W dniu 09.03.2019 o 08:15, Otto Moerbeek pisze: On Fri, Mar 08, 2019 at 12:03:25PM +0100, Michał Koc wrote: Hi all, We have a triple redundant vpn gateway setup with sasyncd running and tons of tunnels, about 1000 flows. Looking at the graph of memory usage, you can clearly see that something is sucking up the memory. The graph can be viewed here: https://pasteboard.co/I4sjzQ8.jpg Looking at the ps, sasyncd shows huge memory consumption: USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND _isakmpd 33560 0.0 17.0 699264 708508 ?? S 26Feb19 6:58.81 /usr/sbin/sasyncd It only happens on the master node. Slaves do not show such a behavior. There is nothing about sasyncd in the logs. After sasyncd restart memory consumption is minimal, but tends to grow. Is it normal ? or am I missing something ? Best regards M.K. This is not normal. You could try to run with -vv to see if some error path is taken that triggers a leak. -Otto Should I look for something specific ? The log grows pretty fast and it looks like it could contain some security data which I wouldn't like to post online. The statistics of the log(about 2 hours) looks like this: carp_init: 1 config: 7 monitor_get_pfkey_snap: 4 monitor_loop: 1 net: 1 net_connect: 3 net_ctl: 4 net_disconnect_peer: 3 net_handle_messages: 2 net_queue: 91780 net_read: 10 net_send_messages: 39192 pfkey_send_flush: 4 pfkey_snapshot: 6832 timer_add: 19 timer_run: 18 Best regards M.K.
Re: How to make X listen tcp again?
On 3/9/19 2:03 PM, Roderick wrote: > > Well, now the way below reads .xinitrc, perhaps a lapsus when typing > before. > > With "ssh -X" on the X server I get widgets from the remote server, > but not without "ssh -X". > > Typing in the remote computer I get the following error: > > # env DISPLAY=192.168.178.47:0.0 xterm > xterm: Xt error: Can't open display: 192.168.178.47:0.0 > > Is there another new security measure I do not know? I think you also need the "-Y" option in ssh. Robert > > BTW: I did "xhost +" in the server. > > Rodrigo > > On Sat, 9 Mar 2019, Roderick wrote: > >> >> The default changed, X does not receive Tcp connections. In FreeBSD >> I solved the problem with a file .xserverrc in my home directory >> with >> >> exec /usr/X11R6/bin/Xorg -listen tcp >> >> But this does not work with OpenBSD 6.4 (X does not even >> execute .xinitrc, I start X with xinit). >
problem with site-to-site VPN between local machine and remote LAN (OpenIKED)
Hello,
I have a local_machine and testing remote_gateway/NAT with one
remoteLAN_machine behind it. All the boxes are running OpenBSD. I can log in
(ssh) to remoteLAN_machine through port forwarded on remote_gateway/NAT.
I'm trying to setup Site-To-Site VPN between local_machine and the remote LAN.
When I set it up (iked) the local_machine can ping (only ping)
remoteLAN_machine through VPN tunnel.
I CANNOT log in (ssh) to remoteLAN_machine from local_machine, both through VPN
and from outside (on gateway's public IP and forwarded port).
I need to have both ways access to behind_NAT services/boxes. I don't know what
I'm doing wrong.
Could you shed some light on my problem/configs please?
Thank you!
local_machine# cat /etc/iked.conf | grep "^[^#;]"
remote_gw_FW70 = "240.240.10.70"
remote_lan_FW70= "10.0.100.0/24"
ikev2 quick active esp from egress to $remote_lan_FW70 \
peer $remote_gw_FW70 \
psk "aaa"
local_machine# cat /etc/pf.conf | grep "^[^#;]"
set skip on lo
block all
table const {240.240.10.96, 240.240.10.70 }
pass out quick on egress proto esp from (egress:0) to
keep state
pass out quick on egress proto udp from (egress:0) to port {500,
4500} keep state
pass in quick on egress proto esp from to (egress:0)
keep state
pass in quick on egress proto udp from to (egress:0) port {500,
4500} keep state
pass out quick on trust received-on enc0 keep state
pass out
block return in on ! lo0 proto tcp to port 6000:6010
block return out log proto {tcp udp} user _pbuild
local_machine# ipsecctl -sa
FLOWS:
flow esp in from 10.0.100.0/24 to 240.240.10.69 peer 240.240.10.70 srcid
FQDN/desk.pk dstid FQDN/fw63 type use
flow esp out from 240.240.10.69 to 10.0.100.0/24 peer 240.240.10.70 srcid
FQDN/desk.pk dstid FQDN/fw63 type require
flow esp out from ::/0 to ::/0 type deny
SAD:
esp tunnel from 240.240.10.69 to 240.240.10.70 spi 0x3428e2ee auth
hmac-sha2-256 enc aes-256
esp tunnel from 240.240.10.70 to 240.240.10.69 spi 0x4b96dca8 auth
hmac-sha2-256 enc aes-256
remote_gateway/NAT# cat /etc/iked.conf | grep "^[^#;]"
local_lan_FW70 = "10.0.100.0/24"
remote_desk_RDK= "240.240.10.69"
ikev2 quick active esp \
from $local_lan_FW70 to $remote_desk_RDK peer $remote_desk_RDK \
psk "aaa"
remote_gateway/NAT# cat /etc/pf.conf | grep "^[^#;]"
sql_soe = "10.0.100.123"
ssh_port= "1071"
icmp_types = "{ echoreq, unreach }"
ssh_soe_int = "1071"
ssh_soe_ext = "22123"
set block-policy drop
set optimization normal
set ruleset-optimization basic
set skip on lo
set fingerprints "/dev/null"
antispoof quick for lo0
block all
match out log on egress from vr3:network nat-to egress:0
match in all scrub (no-df random-id)
match out all scrub (no-df random-id)
table const {240.240.10.96, 240.240.10.69 }
pass out quick on egress proto esp from (egress:0) to
keep state
pass out quick on egress proto udp from (egress:0) to port {500,
4500} keep state
pass in quick on egress proto esp from to (egress:0)
keep state
pass in quick on egress proto udp from to (egress:0) port {500,
4500} keep state
pass out quick on trust received-on enc0 keep state
pass out log proto tcp keep state
pass log proto udp keep state
pass in log quick inet proto tcp from any to egress port $ssh_port flags S/SA
keep state
pass in log quick on egress inet proto tcp from any to egress port $ssh_soe_ext
rdr-to $sql_soe port $ssh_soe_int keep state
pass inet proto icmp all icmp-type $icmp_types keep state
pass log inet proto { tcp, udp, esp } from vr3:network to any keep state
block in log on ! lo0 proto tcp to port 6000:6010
remote_gateway/NAT# ipsecctl -sa
FLOWS:
flow esp in from 240.240.10.69 to 10.0.100.0/24 peer 240.240.10.69 srcid
FQDN/fw63 dstid FQDN/desk.pk type use
flow esp out from 10.0.100.0/24 to 240.240.10.69 peer 240.240.10.69 srcid
FQDN/fw63 dstid FQDN/desk.pk type require
flow esp out from ::/0 to ::/0 type deny
SAD:
esp tunnel from 240.240.10.70 to 240.240.10.69 spi 0x09952f16 auth
hmac-sha2-256 enc aes-256
esp tunnel from 240.240.10.70 to 240.240.10.69 spi 0x216a3871 auth
hmac-sha2-256 enc aes-256
esp tunnel from 240.240.10.69 to 240.240.10.70 spi 0x3428e2ee auth
hmac-sha2-256 enc aes-256
esp tunnel from 240.240.10.70 to 240.240.10.69 spi 0x4b96dca8 auth
hmac-sha2-256 enc aes-256
esp tunnel from 240.240.10.69 to 240.240.10.70 spi 0x62c0615a auth
hmac-sha2-256 enc aes-256
esp tunnel from 240.240.10.69 to 240.240.10.70 spi 0x97cc9e5f auth
hmac-sha2-256 enc aes-256
remoteLAN_machine# cat /etc/pf.conf | grep "^[^#;]"
set skip on {lo, enc}
match in all scrub (no-df random-id)
match out all scrub (no-df random-id)
pass all
--
radek
Re: How to make X listen tcp again?
On Sat, 9 Mar 2019, Robert Paschedag wrote: I think you also need the "-Y" option in ssh. Thanks, but perhaps I did not express myself correct. I do not want to use ssh. No encription of X11. Just to do as I always did: call an x client specifying a remote display (OpenBSD) either with envirenment variable DISPLAY or with the option -display. Rodrigo
Re: usbd_free_xfer: xfer=0xffffff041e9651e0 not free
I am on OpenBSD 6.4 amd64 also getting this error message. It seems to be caused by my USB serial device. If I plug it into a Windows machine then Windows reports that the device is malfunctioning. If I plug into OpenBSD 6.4 then I get the usbd_free_xfer not free error and my USB ports stop working and then I have to reboot. If a dev is interested in my malfunctioning USB serial cable in order to reproduce and fix the issue then I will be happy to mail it to you. I will send my dmesg on request. Leo wrote on 2018-10-13 16:52:36: > Hello, > i just upgraded to the latest snapshot and i noticed that all my > external USB drives are not working anymore. (i tryed 3 different > external drives) As soon as i plug them into an USB port I get the > following message in my dmesg: > > > usbd_free_xfer: xfer=0xff041e9651e0 not free -- -alfred
Re: How to make X listen tcp again?
Roderick writes:
>
> The default changed, X does not receive Tcp connections. In FreeBSD
> I solved the problem with a file .xserverrc in my home directory
> with
>
> exec /usr/X11R6/bin/Xorg -listen tcp
>
> But this does not work with OpenBSD 6.4 (X does not even
> execute .xinitrc, I start X with xinit).
>
> Any hint?
Yes: you would be better off using xenodm and editing
/etc/X11/xenodm/Xservers, with a script and a doas config around 'rcctl
start xenodm' if you really must start it manually, but if you don't
want to do that for whatever poorly-considered reasons then xinit does
in fact use the xserver and xinit rc scripts provided they're not
overridden on the command line or in the environment:
/*
* if no server arguments given, check for a startup file and copy
* that into the argument list
*/
if (!server_given) {
char *cp;
Bool required = False;
xserverrcbuf[0] = '\0';
if ((cp = getenv("XSERVERRC")) != NULL) {
snprintf(xserverrcbuf, sizeof(xserverrcbuf), "%s", cp);
required = True;
} else if ((cp = getenv("HOME")) != NULL) {
snprintf(xserverrcbuf, sizeof(xserverrcbuf),
"%s/%s", cp, XSERVERRC);
}
etc.
Also you're right: X does not execute .xinitrc. xinit does that.
Matthew
Re: How to make X listen tcp again?
Roderick writes: > On Sat, 9 Mar 2019, cho...@jtan.com wrote: > > > > Any hint? > > > > Yes: [...] > > Did you try it? No but I think you missed the part where I said "and editing /etc/X11/xenodm/Xservers" which is the where the server binary's options (and server binary) are specified. For what it's worth I run a rather unusual X configuration so I've had to dig into X' startup sequence quite extensively. I can't come up with any reason why the ability to listen on plain TCP has been completely restricted, though if it has I can understand it because having X listen on TCP is nuts[*], but if you want to customise the X server which xenodm starts, eg. by allowing it to listen on TCP, that will be the place to do it. That said, I think the ssh -X solution is a lot more sensible. Matthew [*] This doesn't stop me but in my case the X server runs on linux so not applicable to your case.
Re: How to make X listen tcp again?
On 2019-03-09, Roderick wrote: > The default changed, X does not receive Tcp connections. In addition, the default /etc/pf.conf blocks connections to the X11 server: # By default, do not permit remote connections to X11 block return in on ! lo0 proto tcp to port 6000:6010 -- Christian "naddy" Weisgerber na...@mips.inka.de
Re: How to make X listen tcp again?
On Sat, 9 Mar 2019, cho...@jtan.com wrote: > Any hint? Yes: [...] Did you try it? BTW, I started xenodm, logged in, did "xhost +", tried to run an X client on another compiter with this display, and got the same error. Rodrigo
Re: Possible sasyncd memory leak ?
W dniu 09.03.2019 o 12:43, Otto Moerbeek pisze: On Sat, Mar 09, 2019 at 12:10:34PM +0100, Michał Koc wrote: W dniu 09.03.2019 o 08:15, Otto Moerbeek pisze: On Fri, Mar 08, 2019 at 12:03:25PM +0100, Michał Koc wrote: Hi all, We have a triple redundant vpn gateway setup with sasyncd running and tons of tunnels, about 1000 flows. Looking at the graph of memory usage, you can clearly see that something is sucking up the memory. The graph can be viewed here: https://pasteboard.co/I4sjzQ8.jpg Looking at the ps, sasyncd shows huge memory consumption: USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND _isakmpd 33560 0.0 17.0 699264 708508 ?? S 26Feb19 6:58.81 /usr/sbin/sasyncd It only happens on the master node. Slaves do not show such a behavior. There is nothing about sasyncd in the logs. After sasyncd restart memory consumption is minimal, but tends to grow. Is it normal ? or am I missing something ? Best regards M.K. This is not normal. You could try to run with -vv to see if some error path is taken that triggers a leak. -Otto Should I look for something specific ? The log grows pretty fast and it looks like it could contain some security data which I wouldn't like to post online. The statistics of the log(about 2 hours) looks like this: carp_init: 1 config: 7 monitor_get_pfkey_snap: 4 monitor_loop: 1 net: 1 net_connect: 3 net_ctl: 4 net_disconnect_peer: 3 net_handle_messages: 2 net_queue: 91780 net_read: 10 net_send_messages: 39192 pfkey_send_flush: 4 pfkey_snapshot: 6832 timer_add: 19 timer_run: 18 Best regards M.K. Just the counts does not reveal anything. I did a quick audit of the memory allocation logic of sasyncd and did not spot an error. If you do not want to post the logs, you'll neeed to analyze them yourself. This requires matching the log lines to the code and tracking where stuff gets allocated and deallocated. Some digging could reveal the error. I used to run sasyncd, but I do no longer. Settig up a test env is quite some work I do not have time for. -Otto First of all, thank You for your time. I know it's one of the most valuable resource. We have done some analysis and we have found the problem. The problem is that the very master machine exists as a peer in it's config. The purpose of this is to make all of the config files to be as similar as possible for all of the members of the cluster. Removing it from peers fixes the problem. So there are three main roads we can follow: 1. Fix sasyncd to recognize self and handle it properly (a result) 2. It should be mentioned in manual, not to set self as a peer (an excuse) 3. We can change our internal config handling (no one else benefits) What would You recommend as a next step ? we will do much to follow road 1, but we might need help, eg. code review and some guidance to meet OpenBSD needs Furthermore if we follow road 1, is there any chance to get the reviewed, correct, accepted fix into the tree ? Best regards M.K.
Re: Possible sasyncd memory leak ?
On Sat, Mar 09, 2019 at 11:19:11PM +0100, Michał Koc wrote: > W dniu 09.03.2019 o 12:43, Otto Moerbeek pisze: > > On Sat, Mar 09, 2019 at 12:10:34PM +0100, Michał Koc wrote: > > > > > W dniu 09.03.2019 o 08:15, Otto Moerbeek pisze: > > > > On Fri, Mar 08, 2019 at 12:03:25PM +0100, Michał Koc wrote: > > > > > > > > > Hi all, > > > > > > > > > > We have a triple redundant vpn gateway setup with sasyncd running and > > > > > tons > > > > > of tunnels, about 1000 flows. > > > > > > > > > > Looking at the graph of memory usage, you can clearly see that > > > > > something is > > > > > sucking up the memory. > > > > > > > > > > The graph can be viewed here: https://pasteboard.co/I4sjzQ8.jpg > > > > > > > > > > Looking at the ps, sasyncd shows huge memory consumption: > > > > > > > > > > USER PID %CPU %MEM VSZ RSS TT STAT > > > > > STARTED TIME COMMAND > > > > > _isakmpd 33560 0.0 17.0 699264 708508 ?? S > > > > > 26Feb19 6:58.81 /usr/sbin/sasyncd > > > > > > > > > > It only happens on the master node. Slaves do not show such a > > > > > behavior. > > > > > > > > > > There is nothing about sasyncd in the logs. > > > > > > > > > > After sasyncd restart memory consumption is minimal, but tends to > > > > > grow. > > > > > > > > > > Is it normal ? or am I missing something ? > > > > > > > > > > Best regards > > > > > M.K. > > > > > > > > > This is not normal. You could try to run with -vv to see if some error > > > > path is taken that triggers a leak. > > > > > > > > -Otto > > > > > > > Should I look for something specific ? > > > > > > The log grows pretty fast and it looks like it could contain some security > > > data which I wouldn't like to post online. > > > > > > The statistics of the log(about 2 hours) looks like this: > > > carp_init: 1 > > > config: 7 > > > monitor_get_pfkey_snap: 4 > > > monitor_loop: 1 > > > net: 1 > > > net_connect: 3 > > > net_ctl: 4 > > > net_disconnect_peer: 3 > > > net_handle_messages: 2 > > > net_queue: 91780 > > > net_read: 10 > > > net_send_messages: 39192 > > > pfkey_send_flush: 4 > > > pfkey_snapshot: 6832 > > > timer_add: 19 > > > timer_run: 18 > > > > > > Best regards > > > M.K. > > > > > Just the counts does not reveal anything. I did a quick audit of the > > memory allocation logic of sasyncd and did not spot an error. If you > > do not want to post the logs, you'll neeed to analyze them yourself. > > This requires matching the log lines to the code and tracking where > > stuff gets allocated and deallocated. Some digging could reveal the error. > > > > I used to run sasyncd, but I do no longer. Settig up a test env is > > quite some work I do not have time for. > > > > -Otto > > > First of all, thank You for your time. I know it's one of the most valuable > resource. > > We have done some analysis and we have found the problem. > > The problem is that the very master machine exists as a peer in it's config. > The purpose of this is to make all of the config files to be as similar as > possible for all of the members of the cluster. > > Removing it from peers fixes the problem. > > So there are three main roads we can follow: > 1. Fix sasyncd to recognize self and handle it properly (a result) > 2. It should be mentioned in manual, not to set self as a peer (an excuse) > 3. We can change our internal config handling (no one else benefits) > > What would You recommend as a next step ? we will do much to follow road 1, > but we might need help, eg. code review and some guidance to meet OpenBSD > needs > > Furthermore if we follow road 1, is there any chance to get the reviewed, > correct, accepted fix into the tree ? > > Best regards > M.K. > Good you found the problem. As for number 1, in general it is hard to determine if an IP is your own (or redirects to yourself), especially since network interfaces can be added and changed dynamically. But a starup check against getifaddrs(3) would catch the most obvious I'm my own peer cases. A dynamic form of loop detection would be nice, it would require adding a hostid or equivalent. That is not normally set these days and also would change the wire format of the messages. But even if you start work on code, a manual page change warning against this would be a good thing. If you post diffs to tech@ with a proper explanation they will be picked up and reviewed, either by me or somee other developer. BTW, I noticed there are order dependencies in sasyncd.cnf that are not mentioned in the man page. Also the example in src/etc/examples/sasyncd.conf does not work if you just uncomment the lines because of those order dependencies. So there's more room for improvement in the man page and example config. -Otto
