Re: haproxy and solarflare onload

[Christian Ruppert]

Oh, btw, I'm just reading that onload documentation.

<snip>
Filters
Filters are used to deliver packets received from the wire to the 
appropriate
application. When filters are exhausted it is not possible to create new 
accelerated
sockets. The general recommendation is that applications do not allocate 
more than
4096 filters ‐ or applications should not create more than 4096 outgoing
connections.
The limit does not apply to inbound connections to a listening socket.
</snip>
On 2017-12-20 13:11, Christian Ruppert wrote:
Hi Elias,

I'm currently preparing a test setup including a SFN8522 + onload.
How did you measure it? When did those errors (drops/discard?) appear,
during a test or some real traffic?
The first thing I did is updating the driver + firmware. Is both
up2date in your case?

I haven't measured / compared the SFN8522 against a X520 nor X710 yet
but do you have RSS / affinity or something related, enabled/set?
Intel has some features and Solarflare may have its own stuff.

On 2017-12-20 11:48, Elias Abacioglu wrote:
Hi,

Yes on the LD_PRELOAD.

Yes, I have one node running with Solarflare SFN8522 2p 10Gbit/s
currently without Onload enabled.
it has 17.5K http_request_rate and ~26% server interrupts on core 0
and 1 where the NIC IRQ is bound to.

And I have a similar node with Intel X710 2p 10Gbit/s.
It has 26.1K http_request_rate and ~26% server interrupts on core 0
and 1 where the NIC IRQ is bound to.

both nodes have 1 socket, Intel Xeon CPU E3-1280 v6, 32 GB RAM.

So without Onload Solarflare performs worse than the X710 since it has
the same amount of SI load with less traffic. And a side note is that
I haven't compared the ethtool settings between Intel and Solarflare,
just running with the defaults of both cards.

I currently have a support ticket open with the Solarflare team to
about the issues I mentioned in my previous mail, if they sort that
out I can perhaps setup a test server if I can manage to free up one
server.
Then we can do some synthetic benchmarks with a set of parameters of
your choosing.

Regards,

/Elias

On Wed, Dec 20, 2017 at 9:48 AM, Willy Tarreau <w...@1wt.eu> wrote:

Hi Elias,

On Tue, Dec 19, 2017 at 02:23:21PM +0100, Elias Abacioglu wrote:
Hi,

I recently bought a solarflare NIC with (ScaleOut) Onload /
OpenOnload to
test it with HAproxy.

Have anyone tried running haproxy with solarflare onload
functions?

After I started haproxy with onload, this started spamming on the
kernel
log:
Dec 12 14:11:54 dflb06 kernel: [357643.035355] [onload]
oof_socket_add_full_hw: 6:3083 ERROR: FILTER TCP 10.3.54.43:4147
[1]
10.3.20.116:80 [2] failed (-16)
Dec 12 14:11:54 dflb06 kernel: [357643.064395] [onload]
oof_socket_add_full_hw: 6:3491 ERROR: FILTER TCP 10.3.54.43:39321
[3]
10.3.20.113:80 [4] failed (-16)
Dec 12 14:11:54 dflb06 kernel: [357643.081069] [onload]
oof_socket_add_full_hw: 3:2124 ERROR: FILTER TCP 10.3.54.43:62403
[5]
10.3.20.30:445 [6] failed (-16)
Dec 12 14:11:54 dflb06 kernel: [357643.082625] [onload]
oof_socket_add_full_hw: 3:2124 ERROR: FILTER TCP 10.3.54.43:62403
[5]
10.3.20.30:445 [6] failed (-16)

And this in haproxy log:
Dec 12 14:12:07 dflb06 haproxy[21145]: Proxy ssl-relay reached
system
memory limit at 9931 sockets. Please check system tunables.
Dec 12 14:12:07 dflb06 haproxy[21146]: Proxy ssl-relay reached
system
memory limit at 9184 sockets. Please check system tunables.
Dec 12 14:12:07 dflb06 haproxy[21145]: Proxy HTTP reached system
memory
limit at 9931 sockets. Please check system tunables.
Dec 12 14:12:07 dflb06 haproxy[21145]: Proxy HTTP reached system
memory
limit at 9931 sockets. Please check system tunables.


Apparently I've hit the max hardware filter limit on the  card.
Does anyone here have experience in running haproxy with onload
features?

I've never got any report of any such test, though in the past I
thought
it would be nice to run such a test, at least to validate the
perimeter
covered by the library (you're using it as LD_PRELOAD, that's it ?).

Mind sharing insights and advice on how to get a functional setup?

I really don't know what can reasonably be expected from code trying
to
partially bypass a part of the TCP stack to be honnest. From what
I've
read a long time ago, onload might be doing its work in a not very
intrusive way but judging by your messages above I'm having some
doubts
now.

Have you tried without this software, using the card normally ? I
mean,
2 years ago I had the opportunity to test haproxy on a dual-40G
setup
and we reached 60 Gbps of forwarded traffic with all machines in the
test bench reaching their limits (and haproxy reaching 100% as
well),
so for me that proves that the TCP stack still scales extremely well
and that while such acceleration software might make sense for a
next
generation NIC running on old hardware (eg: when 400 Gbps NICs start
to appear), I'm really not convinced that it makes any sense to use
them on well supported setups like 2-4 10Gbps links which are very
common nowadays. I mean, I managed to run haproxy at 10Gbps 10 years
ago on a core2-duo! Hardware has evolved quite a bit since :-)

Regards,
Willy



Links:
------
[1] http://10.3.54.43:4147
[2] http://10.3.20.116:80
[3] http://10.3.54.43:39321
[4] http://10.3.20.113:80
[5] http://10.3.54.43:62403
[6] http://10.3.20.30:445

--
Regards,
Christian Ruppert