Hi Tobias,
Thanks for the quick response.
Can you explain your below comment a little more
“You could try if the settings in the
charon.plugins.kernel-netlink.spdh_thresh section in strongswan.conf [1]
help, which configure the kernel to hash parts of the selectors of the
policies (how well that works depends on the negotiated traffic
selectors and the configured thresholds). Note that the size of the
policy hash table can't be configured, it's limited to 8 buckets (so if
you manage to hash policies you get eight lists instead of one, or nine
if you consider the overflow list for policies that aren't hashed due to
the thresholds).”
Are you talking about the below parameters
kernel-netlink {
spdh_thresh {
ipv4 {
lbits = 16
rbits = 32
}
ipv6 {
lbits = 128
rbits = 128
}
}
}
Is the Limit 8 for the policy table set in kernel?
What are the best values for the Hash to get to at least 100K tunnels at
200tunnels/sec.
And other kernel buffer parameters needs to be set from below list?
charon.plugins.kernel-netlink.buflen
min(PAGE_SIZE, 8192)
Buffer size for received Netlink messages.
charon.plugins.kernel-netlink.force_receive_buffer_size
no
If the maximum Netlink socket receive buffer in bytes set by
receive_buffer_size exceeds the system-wide maximum from
/proc/sys/net/core/rmem_max, this option can be used to override the limit.
Enabling this option requires special priviliges (CAP_NET_ADMIN).
charon.plugins.kernel-netlink.fwmark
Firewall mark to set on the routing rule that directs traffic to our own
routing table. The format is [!]mark[/mask], where the optional exclamation
mark inverts the meaning (i.e. the rule only applies to packets that don't
match the mark). A possible use case are host-to-host tunnels with
kernel-libipsec<https://wiki.strongswan.org/projects/strongswan/wiki/Kernel-libipsec#Host-to-Host-Tunnels>.
When set to !<mark> a more efficient lookup for source and next-hop addresses
may also be used since 5.3.3<https://wiki.strongswan.org/versions/58>.
charon.plugins.kernel-netlink.mss
0
MSS to set on installed routes, 0 to disable.
charon.plugins.kernel-netlink.mtu
0
MTU to set on installed routes, 0 to disable.
charon.plugins.kernel-netlink.process_rules
no
Whether to process changes in routing rules to trigger roam events. This is
currently only useful if the kernel based route lookup is used (i.e. if route
installation is disabled or an inverted fwmark match is configured).
charon.plugins.kernel-netlink.receive_buffer_size
0
Maximum Netlink socket receive buffer in bytes. This value controls how many
bytes of Netlink messages can be received on a Netlink socket. The default
value is set by /proc/sys/net/core/rmem_default. The specified value cannot
exceed the system-wide maximum from /proc/sys/net/core/rmem_max, unless
force_receive_buffer_size is enabled.
charon.plugins.kernel-netlink.roam_events
yes
Whether to trigger roam events when interfaces, addresses or routes change.
charon.plugins.kernel-netlink.set_proto_port_transport_sa
no
Whether to set protocol and ports in the selector installed on transport mode
IPsec SAs in the kernel. While doing so enforces policies for inbound traffic,
it also prevents the use of a single IPsec SA by more than one traffic selector.
charon.plugins.kernel-netlink.spdh_thresh
Subsection to configure XFRM policy hashing thresholds for IPv4 and IPv6. The
section defines hashing thresholds to configure in the kernel during daemon
startup. Each address family takes a threshold for the local subnet of an IPsec
policy (src in out-policies, dst in in- and forward-policies) and the remote
subnet (dst in out-policies, src in in- and forward-policies).
If the subnet has more or equal net bits than the threshold, the first
threshold bits are used to calculate a hash to lookup the policy.
Policy hashing thresholds are not supported before Linux 3.18 and might
conflict with socket policies before Linux 4.8.
charon.plugins.kernel-netlink.spdh_thresh.ipv4.lbits
32
Local subnet XFRM policy hashing threshold for IPv4.
charon.plugins.kernel-netlink.spdh_thresh.ipv4.rbits
32
Remote subnet XFRM policy hashing threshold for IPv4.
charon.plugins.kernel-netlink.spdh_thresh.ipv6.lbits
128
Local subnet XFRM policy hashing threshold for IPv6.
charon.plugins.kernel-netlink.spdh_thresh.ipv6.rbits
128
Remote subnet XFRM policy hashing threshold for IPv6.
Thanks & Regards,
Vinay
From: Tobias Brunner <[email protected]>
Sent: Tuesday, December 04, 2018 5:42 AM
To: Vinay G. Pullela <[email protected]>; [email protected]
Subject: Re: [strongSwan-dev] StrongSwan 5.6.3, Netlink performance issue as
responder.
Hi Vinay,
> What I mean by this is request and response for the kernel-netlink
> messages are taking more and more time as the number of established
> tunnels count, I have tried both with the parallel option enabled and
> disabled, see the observation for both below.
I'd say that's to be expected and probably related to how the kernel
manages policies. They are stored in a linear list, sorted by priority.
So every additional CHILD_SA takes longer to install because the
policies have to be inserted in the correct location in that list.
> 1. When parallel flag
I wouldn't expect this to change anything because the kernel has to lock
the policy lists and SA hash tables globally.
You could try if the settings in the
charon.plugins.kernel-netlink.spdh_thresh section in strongswan.conf [1]
help, which configure the kernel to hash parts of the selectors of the
policies (how well that works depends on the negotiated traffic
selectors and the configured thresholds). Note that the size of the
policy hash table can't be configured, it's limited to 8 buckets (so if
you manage to hash policies you get eight lists instead of one, or nine
if you consider the overflow list for policies that aren't hashed due to
the thresholds).
> Note: Route installation is disabled.
You mean via charon.install_routes? That's good, as installing one or
more routes for every client is another bottleneck with lots of clients.
And if e.g. virtual IPs are used, a single route can easily cover all
clients if it's even necessary.
Regards,
Tobias
[1]
https://wiki.strongswan.org/projects/strongswan/wiki/StrongswanConf<https://wiki.strongswan.org/projects/strongswan/wiki/StrongswanConf>
