Re: Racoon and setkey problems

[Andrey V. Elsukov]

On 26.02.2018 15:53, Misak Khachatryan wrote:
> Hi Andrey,
> 
> thanks for the patch!  Is it safe to use it on 10.3?

It should be applicable to 10.3, but I don't know how it is safe :)
When there are no errors, it should work like before. When some error
occurs like you have, it will be ignore and this can lead to some
unknown results. Maybe racoon will do something strange.

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature

Re: Racoon and setkey problems

[Misak Khachatryan]

Hi Andrey,

thanks for the patch!  Is it safe to use it on 10.3?

Best regards,
Misak Khachatryan


On Mon, Feb 26, 2018 at 4:39 PM, Andrey V. Elsukov  wrote:
> On 22.02.2018 22:12, Misak Khachatryan wrote:
  kernel`key_sendup0+0xee
  kernel`key_sendup_mbuf+0x1e6
  kernel`key_parse+0x87f

>>>
>>> Then probably this output will be changed.
>
> I think the problem is that there are several PF_KEY sockets present,
> but some socket has overfilled its buffers. key_sendup_mbuf() function
> tries to send data to all sockets and fails on this mentioned socket.
>
> If you can, please, try the attached patch. It changes the behavior to
> always try to send data to all sockets and ignore some possible errors
> on intermediate sockets. I think with this patch you will be able to
> clear SAs with `setkey -F` command.
>
> You need to rebuild and reinstall the kernel. The patch is for stable/10.
>
> --
> WBR, Andrey V. Elsukov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Andrey V. Elsukov]

On 22.02.2018 22:12, Misak Khachatryan wrote:
>>>  kernel`key_sendup0+0xee
>>>  kernel`key_sendup_mbuf+0x1e6
>>>  kernel`key_parse+0x87f
>>>
>>
>> Then probably this output will be changed.

I think the problem is that there are several PF_KEY sockets present,
but some socket has overfilled its buffers. key_sendup_mbuf() function
tries to send data to all sockets and fails on this mentioned socket.

If you can, please, try the attached patch. It changes the behavior to
always try to send data to all sockets and ignore some possible errors
on intermediate sockets. I think with this patch you will be able to
clear SAs with `setkey -F` command.

You need to rebuild and reinstall the kernel. The patch is for stable/10.

-- 
WBR, Andrey V. Elsukov
Index: stable/10/sys/netipsec/keysock.c
===
--- stable/10/sys/netipsec/keysock.c	(revision 329557)
+++ stable/10/sys/netipsec/keysock.c	(working copy)
@@ -333,16 +333,14 @@ key_sendup_mbuf(struct socket *so, struct mbuf *m,
 			continue;
 
 		if ((n = m_copy(m, 0, (int)M_COPYALL)) == NULL) {
-			m_freem(m);
 			PFKEYSTAT_INC(in_nomem);
-			mtx_unlock(_mtx);
-			return ENOBUFS;
+			/* Try with next socket */
+			continue;
 		}
 
 		if ((error = key_sendup0(rp, n, 0)) != 0) {
-			m_freem(m);
-			mtx_unlock(_mtx);
-			return error;
+			/* Try with next socket */
+			continue;
 		}
 
 		n = NULL;


signature.asc
Description: OpenPGP digital signature

Re: Racoon and setkey problems

[Misak Khachatryan]

Here is changed output:

# sysctl net.raw
net.raw.recvspace: 8192
net.raw.sendspace: 8192
# sysctl net.raw.recvspace=65535
net.raw.recvspace: 8192 -> 65535
# sysctl net.raw.sendspace=65535
net.raw.sendspace: 8192 -> 65535
#
#
#
# setkey -x
setkey: send: No buffer space available
# dtrace -s /tmp/key.d
dtrace: script '/tmp/key.d' matched 14 probes
CPU IDFUNCTION:NAME
  3  25400  soreserve:entry 32768 65536
  kernel`sonewconn+0x1b1
  kernel`syncache_expand+0x6e1
  kernel`tcp_input+0xdc4

  5  25400  soreserve:entry 65535 65535
  kernel`raw_attach+0x2a
  kernel`key_attach+0x57
  kernel`socreate+0x1af

  5   7957key_attach:return 0
  5   6405 sbappendaddr_locked_internal:return 1
  kernel`sbappendaddr_locked+0x90
  kernel`sbappendaddr+0x61
  kernel`key_sendup0+0xee

  5  24460   sbappendaddr_locked:return 1
  kernel`sbappendaddr+0x61
  kernel`key_sendup0+0xee
  kernel`key_sendup_mbuf+0x14b

  5  12872  sbappendaddr:return 1
  kernel`key_sendup0+0xee
  kernel`key_sendup_mbuf+0x14b
  kernel`key_parse+0x87f

  5   7969   key_sendup0:return 0
  kernel`key_sendup_mbuf+0x14b
  kernel`key_parse+0x87f
  kernel`sosend_generic+0x476

  5  24460   sbappendaddr_locked:return 0
  kernel`sbappendaddr+0x61
  kernel`key_sendup0+0xee
  kernel`key_sendup_mbuf+0x1e6

  5  12872  sbappendaddr:return 0
  kernel`key_sendup0+0xee
  kernel`key_sendup_mbuf+0x1e6
  kernel`key_parse+0x87f

  5   7969   key_sendup0:return 55
  kernel`key_sendup_mbuf+0x1e6
  kernel`key_parse+0x87f
  kernel`sosend_generic+0x476

  5  24402   key_sendup_mbuf:return 55
  kernel`key_parse+0x87f
  kernel`sosend_generic+0x476
  kernel`kern_sendit+0x245

  5  11197 key_parse:return 55
^C

#

Best regards,
Misak Khachatryan


On Thu, Feb 22, 2018 at 8:34 PM, Andrey V. Elsukov  wrote:
> On 22.02.2018 18:28, Misak Khachatryan wrote:
>> # dtrace -s key.d
>> dtrace: script 'key.d' matched 14 probes
>> CPU IDFUNCTION:NAME
>
> So, what I can say:
>
>>  4  25400  soreserve:entry 8192 8192
>>  kernel`raw_attach+0x2a
>>  kernel`key_attach+0x57
>>  kernel`socreate+0x1af
>
> First of try to increase both values of net.raw:
>
> # sysctl net.raw
> net.raw.recvspace: 65535
> net.raw.sendspace: 65535
>
>>
>>  4  24460   sbappendaddr_locked:return 0
>>  kernel`sbappendaddr+0x61
>>  kernel`key_sendup0+0xee
>>  kernel`key_sendup_mbuf+0x1e6
>>
>>  4  12872  sbappendaddr:return 0
>>  kernel`key_sendup0+0xee
>>  kernel`key_sendup_mbuf+0x1e6
>>  kernel`key_parse+0x87f
>>
>
> Then probably this output will be changed.
>
> --
> WBR, Andrey V. Elsukov
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Andrey V. Elsukov]

On 22.02.2018 18:28, Misak Khachatryan wrote:
> # dtrace -s key.d
> dtrace: script 'key.d' matched 14 probes
> CPU IDFUNCTION:NAME

So, what I can say:

>  4  25400  soreserve:entry 8192 8192
>  kernel`raw_attach+0x2a
>  kernel`key_attach+0x57
>  kernel`socreate+0x1af

First of try to increase both values of net.raw:

# sysctl net.raw
net.raw.recvspace: 65535
net.raw.sendspace: 65535

> 
>  4  24460   sbappendaddr_locked:return 0
>  kernel`sbappendaddr+0x61
>  kernel`key_sendup0+0xee
>  kernel`key_sendup_mbuf+0x1e6
> 
>  4  12872  sbappendaddr:return 0
>  kernel`key_sendup0+0xee
>  kernel`key_sendup_mbuf+0x1e6
>  kernel`key_parse+0x87f
> 

Then probably this output will be changed.

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature

Re: Racoon and setkey problems

[Misak Khachatryan]

# dtrace -s key.d
dtrace: script 'key.d' matched 14 probes
CPU IDFUNCTION:NAME
 2  25400  soreserve:entry 2048 4096
 kernel`uipc_attach+0x76
 kernel`socreate+0x1af
 kernel`sys_socket+0xf7

 3  25400  soreserve:entry 32768 65536
 kernel`sonewconn+0x1b1
 kernel`syncache_expand+0x6e1
 kernel`tcp_input+0xdc4

 3  25400  soreserve:entry 32768 65536
 kernel`sonewconn+0x1b1
 kernel`syncache_expand+0x6e1
 kernel`tcp_input+0xdc4

 4  25400  soreserve:entry 8192 8192
 kernel`raw_attach+0x2a
 kernel`key_attach+0x57
 kernel`socreate+0x1af

 4   7957key_attach:return 0
 4   6405 sbappendaddr_locked_internal:return 1
 kernel`sbappendaddr_locked+0x90
 kernel`sbappendaddr+0x61
 kernel`key_sendup0+0xee

 4  24460   sbappendaddr_locked:return 1
 kernel`sbappendaddr+0x61
 kernel`key_sendup0+0xee
 kernel`key_sendup_mbuf+0x14b

 4  12872  sbappendaddr:return 1
 kernel`key_sendup0+0xee
 kernel`key_sendup_mbuf+0x14b
 kernel`key_parse+0x87f

 4   7969   key_sendup0:return 0
 kernel`key_sendup_mbuf+0x14b
 kernel`key_parse+0x87f
 kernel`sosend_generic+0x476

 4  24460   sbappendaddr_locked:return 0
 kernel`sbappendaddr+0x61
 kernel`key_sendup0+0xee
 kernel`key_sendup_mbuf+0x1e6

 4  12872  sbappendaddr:return 0
 kernel`key_sendup0+0xee
 kernel`key_sendup_mbuf+0x1e6
 kernel`key_parse+0x87f

 4   7969   key_sendup0:return 55
 kernel`key_sendup_mbuf+0x1e6
 kernel`key_parse+0x87f
 kernel`sosend_generic+0x476

 4  24402   key_sendup_mbuf:return 55
 kernel`key_parse+0x87f
 kernel`sosend_generic+0x476
 kernel`kern_sendit+0x245

 4  11197 key_parse:return 55
^C

#

Best regards,
Misak Khachatryan


On Thu, Feb 22, 2018 at 5:54 PM, Andrey V. Elsukov  wrote:
> On 22.02.2018 16:27, Misak Khachatryan wrote:
>> Here is the result:
>>
>> # dtrace -s key.d
>> dtrace: script 'key.d' matched 8 probes
>> CPU IDFUNCTION:NAME
>>  3  25400  soreserve:entry 32768 65536
>
> I hope the last update, to understand what is going on.
>
> --
> WBR, Andrey V. Elsukov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Andrey V. Elsukov]

On 22.02.2018 16:27, Misak Khachatryan wrote:
> Here is the result:
> 
> # dtrace -s key.d
> dtrace: script 'key.d' matched 8 probes
> CPU IDFUNCTION:NAME
>  3  25400  soreserve:entry 32768 65536

I hope the last update, to understand what is going on.

-- 
WBR, Andrey V. Elsukov


signature.asc
Description: OpenPGP digital signature

Re: Racoon and setkey problems

[Misak Khachatryan]

Here is the result:

# dtrace -s key.d
dtrace: script 'key.d' matched 8 probes
CPU IDFUNCTION:NAME
 3  25400  soreserve:entry 32768 65536
 7  25400  soreserve:entry 8192 8192
 7   7957key_attach:return 0
 7  12872  sbappendaddr:return 1
 kernel`key_sendup0+0xee
 kernel`key_sendup_mbuf+0x14b
 kernel`key_parse+0x87f

 7   7969   key_sendup0:return 0
 kernel`key_sendup_mbuf+0x14b
 kernel`key_parse+0x87f
 kernel`sosend_generic+0x476

 7  12872  sbappendaddr:return 0
 kernel`key_sendup0+0xee
 kernel`key_sendup_mbuf+0x1e6
 kernel`key_parse+0x87f

 7   7969   key_sendup0:return 55
 kernel`key_sendup_mbuf+0x1e6
 kernel`key_parse+0x87f
 kernel`sosend_generic+0x476

 7  24402   key_sendup_mbuf:return 55
 kernel`key_parse+0x87f
 kernel`sosend_generic+0x476
 kernel`kern_sendit+0x245

 7  11197 key_parse:return 55


^C

Best regards,
Misak Khachatryan


On Thu, Feb 22, 2018 at 5:11 PM, Andrey V. Elsukov  wrote:
> On 22.02.2018 15:13, Misak Khachatryan wrote:
>> I did this way:
>>
>> # dtrace -s key.d
>> dtrace: script 'key.d' matched 6 probes
>> CPU IDFUNCTION:NAME
>>  7   7957key_attach:return 0
>>  7   7969   key_sendup0:return 0
>>  7   7969   key_sendup0:return 55
>>  7  24402   key_sendup_mbuf:return 55
>>  7  11197 key_parse:return 55
>> ^C
>
> Ok, I updated the script, please, show new result
>
> --
> WBR, Andrey V. Elsukov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Andrey V. Elsukov]

On 22.02.2018 15:13, Misak Khachatryan wrote:
> I did this way:
> 
> # dtrace -s key.d
> dtrace: script 'key.d' matched 6 probes
> CPU IDFUNCTION:NAME
>  7   7957key_attach:return 0
>  7   7969   key_sendup0:return 0
>  7   7969   key_sendup0:return 55
>  7  24402   key_sendup_mbuf:return 55
>  7  11197 key_parse:return 55
> ^C

Ok, I updated the script, please, show new result

-- 
WBR, Andrey V. Elsukov


signature.asc
Description: OpenPGP digital signature

Re: Racoon and setkey problems

[Misak Khachatryan]

I did this way:

# dtrace -s key.d
dtrace: script 'key.d' matched 6 probes
CPU IDFUNCTION:NAME
 7   7957key_attach:return 0
 7   7969   key_sendup0:return 0
 7   7969   key_sendup0:return 55
 7  24402   key_sendup_mbuf:return 55
 7  11197 key_parse:return 55
^C


Best regards,
Misak Khachatryan


On Thu, Feb 22, 2018 at 4:09 PM, Misak Khachatryan  wrote:
> I'm getting this:
>
> # ./key.d
> : No such file or directory
> # which dtrace
> /usr/sbin/dtrace
>
> Best regards,
> Misak Khachatryan
>
>
> On Thu, Feb 22, 2018 at 3:42 PM, Andrey V. Elsukov  wrote:
>> On 22.02.2018 12:08, Misak Khachatryan wrote:
>>> That didn help.
>>>
>>> Best regards,
>>> Misak Khachatryan
>>
>> Can you stop racoon and use the following commands and then show the output?
>>
>> # kldload dtraceall
>> # chmod +x ./key.d
>> # ./key.d
>>
>> and from another console run `setkey -x`, show what key.d will print out.
>>
>> --
>> WBR, Andrey V. Elsukov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Misak Khachatryan]

I'm getting this:

# ./key.d
: No such file or directory
# which dtrace
/usr/sbin/dtrace

Best regards,
Misak Khachatryan


On Thu, Feb 22, 2018 at 3:42 PM, Andrey V. Elsukov  wrote:
> On 22.02.2018 12:08, Misak Khachatryan wrote:
>> That didn help.
>>
>> Best regards,
>> Misak Khachatryan
>
> Can you stop racoon and use the following commands and then show the output?
>
> # kldload dtraceall
> # chmod +x ./key.d
> # ./key.d
>
> and from another console run `setkey -x`, show what key.d will print out.
>
> --
> WBR, Andrey V. Elsukov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Andrey V. Elsukov]

On 22.02.2018 12:08, Misak Khachatryan wrote:
> That didn help.
> 
> Best regards,
> Misak Khachatryan

Can you stop racoon and use the following commands and then show the output?

# kldload dtraceall
# chmod +x ./key.d
# ./key.d

and from another console run `setkey -x`, show what key.d will print out.

-- 
WBR, Andrey V. Elsukov


signature.asc
Description: OpenPGP digital signature

Re: Racoon and setkey problems

[Misak Khachatryan]

That didn help.

Best regards,
Misak Khachatryan


On Thu, Feb 22, 2018 at 11:50 AM, Eugene Grosbein  wrote:
> On 22.02.2018 14:10, Misak Khachatryan wrote:
>> Hello there,
>>
>> just a quick feedback. I've added rules to my ipfw to block all isakmp
>> ports on interfaces not involved in ipsec and rebooted 3 of 4
>> machines. Situation returned to normal on them, but rebooting fourth
>> host is very painful. It seems i have some kind of massive ipsec
>> probes from botnet which fills all my SAD and SPD entries or PFKEY
>> sockets.
>>
>> All i need is to flush all SAD and SDP entries, but setkey can't do
>> that. Is there any other way?
>
> Try to increase sysctl kern.ipc.maxsockbuf upto some huge value like 80MB
> and re-try with setkey.
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Eugene Grosbein]

On 22.02.2018 14:10, Misak Khachatryan wrote:
> Hello there,
> 
> just a quick feedback. I've added rules to my ipfw to block all isakmp
> ports on interfaces not involved in ipsec and rebooted 3 of 4
> machines. Situation returned to normal on them, but rebooting fourth
> host is very painful. It seems i have some kind of massive ipsec
> probes from botnet which fills all my SAD and SPD entries or PFKEY
> sockets.
> 
> All i need is to flush all SAD and SDP entries, but setkey can't do
> that. Is there any other way?

Try to increase sysctl kern.ipc.maxsockbuf upto some huge value like 80MB
and re-try with setkey.

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Misak Khachatryan]

Hello there,

just a quick feedback. I've added rules to my ipfw to block all isakmp
ports on interfaces not involved in ipsec and rebooted 3 of 4
machines. Situation returned to normal on them, but rebooting fourth
host is very painful. It seems i have some kind of massive ipsec
probes from botnet which fills all my SAD and SPD entries or PFKEY
sockets.

All i need is to flush all SAD and SDP entries, but setkey can't do
that. Is there any other way?


Best regards,
Misak Khachatryan


On Tue, Feb 20, 2018 at 4:47 PM, Andrey V. Elsukov  wrote:
> On 20.02.2018 08:55, Eugene Grosbein wrote:
>>> yes, all output is from same machine. I'll recheck all configs again,
>>> or, if it's OK, I can post them here. The most confusing thing is that
>>> everything worked as a charm several years. And nothing changed in
>>> configurations until logs stars to fill up with these messages and i
>>> tried to play with some settings to troubleshoot.
>>
>> You may be suffering from some kind of massive IPSEC-scanning bots activity
>> that try to expoit IPSEC-related bugs and trigger some memory leak.
>>
>> You should really try 11.1.
>
> 11.1-RELEASE had several bugs in new IPsec code, that were fixed in
> stable/11 branch. So, if you want to try, I recommend to use stable/11.
> Also there is very little chance that some problem will be fixed in 10.x
> branch.
>
> --
> WBR, Andrey V. Elsukov
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Andrey V. Elsukov]

On 20.02.2018 08:55, Eugene Grosbein wrote:
>> yes, all output is from same machine. I'll recheck all configs again,
>> or, if it's OK, I can post them here. The most confusing thing is that
>> everything worked as a charm several years. And nothing changed in
>> configurations until logs stars to fill up with these messages and i
>> tried to play with some settings to troubleshoot.
> 
> You may be suffering from some kind of massive IPSEC-scanning bots activity
> that try to expoit IPSEC-related bugs and trigger some memory leak.
> 
> You should really try 11.1.

11.1-RELEASE had several bugs in new IPsec code, that were fixed in
stable/11 branch. So, if you want to try, I recommend to use stable/11.
Also there is very little chance that some problem will be fixed in 10.x
branch.

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature

Re: Racoon and setkey problems

[Misak Khachatryan]

One of the machines didn't connected to the Internet, have only private ip
address on it's interfaces, so i have doubts about that. But thanks, I'll
check for that too. I'm exporting traffic from two machines to netflow
collector, should be easy.

On Feb 20, 2018 9:55 AM, "Eugene Grosbein"  wrote:

On 20.02.2018 00:44, Misak Khachatryan wrote:
> Hi Andrey,
>
> yes, all output is from same machine. I'll recheck all configs again,
> or, if it's OK, I can post them here. The most confusing thing is that
> everything worked as a charm several years. And nothing changed in
> configurations until logs stars to fill up with these messages and i
> tried to play with some settings to troubleshoot.

You may be suffering from some kind of massive IPSEC-scanning bots activity
that try to expoit IPSEC-related bugs and trigger some memory leak.

You should really try 11.1.
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Eugene Grosbein]

On 20.02.2018 00:44, Misak Khachatryan wrote:
> Hi Andrey,
> 
> yes, all output is from same machine. I'll recheck all configs again,
> or, if it's OK, I can post them here. The most confusing thing is that
> everything worked as a charm several years. And nothing changed in
> configurations until logs stars to fill up with these messages and i
> tried to play with some settings to troubleshoot.

You may be suffering from some kind of massive IPSEC-scanning bots activity
that try to expoit IPSEC-related bugs and trigger some memory leak.

You should really try 11.1.



___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Misak Khachatryan]

Hi Andrey,

yes, all output is from same machine. I'll recheck all configs again,
or, if it's OK, I can post them here. The most confusing thing is that
everything worked as a charm several years. And nothing changed in
configurations until logs stars to fill up with these messages and i
tried to play with some settings to troubleshoot.

Best regards,
Misak Khachatryan


On Mon, Feb 19, 2018 at 2:56 PM, Andrey V. Elsukov  wrote:
> On 19.02.2018 12:28, Misak Khachatryan wrote:
>> Hi,
>>
>> # vmstat -m | egrep "sec|sah|pol"
>>  inpcbpolicy   122 4K   -  4955796  32
>> secasvar 48558 12140K   -  1572045  256
>>   sahead 3 1K   -   15  256
>>  ipsecpolicy   25664K   -  9911740  256
>> ipsecrequest12 2K   -   48  128
>>   ipsec-misc 389632 12176K   - 12575976  16,32,64
>>ipsec-saq 3 1K   -   15  128
>>ipsec-reg 3 1K   -   12  32
>>histogram by message type:
>>getspi: 1533688
>>update: 1533640
>>add: 25
>>delete: 1
>>acquire: 1569975
>>register: 16
>>expire: 2968244
>>flush: 10
>>dump: 111982
>>x_promisc: 48
>>x_spdadd: 48
>>x_spddump: 60
>>x_spdflush: 7
>
> This looks very strange. Are these from the same machine?
> You said the system has only 3 tunnels. From this output I can say, that
> you have too many SAs. Huge numbers for getspi, update, and acquire
> messages means that you have security policy that produces many SAs.
> Probably something wrong with your configs.
>
> --
> WBR, Andrey V. Elsukov
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Andrey V. Elsukov]

On 19.02.2018 12:28, Misak Khachatryan wrote:
> Hi,
> 
> # vmstat -m | egrep "sec|sah|pol"
>  inpcbpolicy   122 4K   -  4955796  32
> secasvar 48558 12140K   -  1572045  256
>   sahead 3 1K   -   15  256
>  ipsecpolicy   25664K   -  9911740  256
> ipsecrequest12 2K   -   48  128
>   ipsec-misc 389632 12176K   - 12575976  16,32,64
>ipsec-saq 3 1K   -   15  128
>ipsec-reg 3 1K   -   12  32
>histogram by message type:
>getspi: 1533688
>update: 1533640
>add: 25
>delete: 1
>acquire: 1569975
>register: 16
>expire: 2968244
>flush: 10
>dump: 111982
>x_promisc: 48
>x_spdadd: 48
>x_spddump: 60
>x_spdflush: 7

This looks very strange. Are these from the same machine?
You said the system has only 3 tunnels. From this output I can say, that
you have too many SAs. Huge numbers for getspi, update, and acquire
messages means that you have security policy that produces many SAs.
Probably something wrong with your configs.

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature

Re: Racoon and setkey problems

[Misak Khachatryan]

HThis machine was rebooted few days ago and immediately it starts
behave like this,

FreeBSD xx.net 10.4-RELEASE-p1 FreeBSD 10.4-RELEASE-p1 #0: Mon Oct
30 21:13:49 +04 2017 x...@xx.net:/usr/obj/usr/src/sys/RTR
amd64

It's 64 bit system with 2 MB of memory:

# vmstat
procs  memory  pagedisks faults cpu
r b w avmfre   flt  re  pi  pofr  sr md0 ad0   in   sy   cs us sy id
1 0 0   2145M   716M   384   0   0   0   617 229   0   0 3678 2043 8230  0  1 99

Flushing rules doesn't help, there is 3 IPSEC tunnels in racoon.conf
overall, IPv4 and IPv6, so 12 rules in setkey.conf




Best regards,
Misak Khachatryan


On Mon, Feb 19, 2018 at 1:40 PM, Eugene Grosbein  wrote:
> 19.02.2018 16:28, Misak Khachatryan wrote:
>
>> # vmstat -m | egrep "sec|sah|pol"
>>  inpcbpolicy   122 4K   -  4955796  32
>> secasvar 48558 12140K   -  1572045  256
>>   sahead 3 1K   -   15  256
>>  ipsecpolicy   25664K   -  9911740  256
>> ipsecrequest12 2K   -   48  128
>>   ipsec-misc 389632 12176K   - 12575976  16,32,64
>
> Looking at huge "MemUse" values for secasvar and ipsec-misc,
> I suspect some kind of memory leak.
>
> FreeBSD 11.1 has new IPSEC implementation and you may consider trying new 
> version.
>
> Meantime, you can try to flush all IPSEC-related data from the system:
>
> service racoon stop
> setkey -F; setkey -FP
> service racoon start
>
> If that does not help, reboot and start monitoring these numbers for secasvar 
> and ipsec-misc.
>
> How many IPSEC tunnells/associations do you have simultaneously?
> And again, are those systems 32 bit or 64 bit?
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Eugene Grosbein]

19.02.2018 16:28, Misak Khachatryan wrote:

> # vmstat -m | egrep "sec|sah|pol"
>  inpcbpolicy   122 4K   -  4955796  32
> secasvar 48558 12140K   -  1572045  256
>   sahead 3 1K   -   15  256
>  ipsecpolicy   25664K   -  9911740  256
> ipsecrequest12 2K   -   48  128
>   ipsec-misc 389632 12176K   - 12575976  16,32,64

Looking at huge "MemUse" values for secasvar and ipsec-misc,
I suspect some kind of memory leak.

FreeBSD 11.1 has new IPSEC implementation and you may consider trying new 
version.

Meantime, you can try to flush all IPSEC-related data from the system:

service racoon stop
setkey -F; setkey -FP
service racoon start

If that does not help, reboot and start monitoring these numbers for secasvar 
and ipsec-misc.

How many IPSEC tunnells/associations do you have simultaneously?
And again, are those systems 32 bit or 64 bit?

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Misak Khachatryan]

Hi,

# vmstat -m | egrep "sec|sah|pol"
 inpcbpolicy   122 4K   -  4955796  32
secasvar 48558 12140K   -  1572045  256
  sahead 3 1K   -   15  256
 ipsecpolicy   25664K   -  9911740  256
ipsecrequest12 2K   -   48  128
  ipsec-misc 389632 12176K   - 12575976  16,32,64
   ipsec-saq 3 1K   -   15  128
   ipsec-reg 3 1K   -   12  32

# netstat -m
3178/8717/11895 mbufs in use (current/cache/total)
3075/4025/7100/524288 mbuf clusters in use (current/cache/total/max)
3075/4009 mbuf+clusters out of packet secondary zone in use (current/cache)
0/248/248/126182 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/37387 9k jumbo clusters in use (current/cache/total/max)
0/0/0/21030 16k jumbo clusters in use (current/cache/total/max)
6944K/11221K/18165K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

# vmstat -z | egrep 'ITEM|mbuf'
ITEM   SIZE  LIMIT USED FREE  REQ FAIL SLEEP
mbuf_packet:256, 1615140,3084,4000,835233903,   0,   0
mbuf:   256, 1615140, 104,4707,1373358845,   0,   0
mbuf_cluster:  2048, 524288,7084,  16,7084,   0,   0
mbuf_jumbo_page:   4096, 126182,   0, 248,  309522,   0,   0
mbuf_jumbo_9k: 9216,  37387,   0,   0,   0,   0,   0
mbuf_jumbo_16k:   16384,  21030,   0,   0,   0,   0,   0
mbuf_ext_refcnt:  4,  0,   0,   0,   0,   0,   0

# sysctl kern.ipc.nmbclusters
kern.ipc.nmbclusters: 524288


Nothing new in messages with net.inet.ipsec.debug=1


Best regards,
Misak Khachatryan


On Mon, Feb 19, 2018 at 1:25 PM, Eugene Grosbein  wrote:
> 19.02.2018 13:27, Misak Khachatryan wrote:
>
>>1644111 messages with memory allocation failure
>>
>> 3 of machines running   10.4-RELEASE-p1, one 10.3.
>> Two of the machine almost the same, only ip addresses and few lines of
>> configs differ. One is OK, other one have problem.
>>
>> Running almost any setkey command leads to:
>>
>>  # setkey -x
>> setkey: send: No buffer space available
>>
>> All packet versions are completely the same, binaries exactly same size.
>>
>> Any help will be appreciated.
>
> Perhaps, that is mbuf cluster exhaustion. Please show output of commands:
>
> netstat -m
> vmstat -z | egrep 'ITEM|mbuf'
> sysctl kern.ipc.nmbclusters
>
> How much RAM do they have? Do they run 32 bit or 64 bit system?
>
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Eugene Grosbein]

19.02.2018 13:27, Misak Khachatryan wrote:

>1644111 messages with memory allocation failure
> 
> 3 of machines running   10.4-RELEASE-p1, one 10.3.
> Two of the machine almost the same, only ip addresses and few lines of
> configs differ. One is OK, other one have problem.
> 
> Running almost any setkey command leads to:
> 
>  # setkey -x
> setkey: send: No buffer space available
> 
> All packet versions are completely the same, binaries exactly same size.
> 
> Any help will be appreciated.

Perhaps, that is mbuf cluster exhaustion. Please show output of commands:

netstat -m
vmstat -z | egrep 'ITEM|mbuf'
sysctl kern.ipc.nmbclusters

How much RAM do they have? Do they run 32 bit or 64 bit system?


___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Misak Khachatryan]

Thanks, will try right now!

Best regards,
Misak Khachatryan


On Mon, Feb 19, 2018 at 12:23 PM, Andrey V. Elsukov  wrote:
> On 19.02.2018 09:27, Misak Khachatryan wrote:
>>1644111 messages with memory allocation failure
>>
>> 3 of machines running   10.4-RELEASE-p1, one 10.3.
>> Two of the machine almost the same, only ip addresses and few lines of
>> configs differ. One is OK, other one have problem.
>
> You can inspect the output of following commands to find where is the
> problem:
>
> % vmstat -m | egrep "sec|sah|pol"
> % netstat -m
>
> Also net.inet.ipsec.debug=1 will enable some debugging output that can
> shed light to what happens.
>
> --
> WBR, Andrey V. Elsukov
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Racoon and setkey problems

[Andrey V. Elsukov]

On 19.02.2018 09:27, Misak Khachatryan wrote:
>1644111 messages with memory allocation failure
> 
> 3 of machines running   10.4-RELEASE-p1, one 10.3.
> Two of the machine almost the same, only ip addresses and few lines of
> configs differ. One is OK, other one have problem.

You can inspect the output of following commands to find where is the
problem:

% vmstat -m | egrep "sec|sah|pol"
% netstat -m

Also net.inet.ipsec.debug=1 will enable some debugging output that can
shed light to what happens.

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature