Re: [External] Re: QAT intermittent healthcheck errors

2019-05-13 Thread Emeric Brun
Hi Marcin, > > Thank you Marcin, It shows that haproxy is waiting for an event on all those > fds because a crypto jobs were launched on the engine > and we can't free the session until the end of this job (it would result in a > segfault). > > So the processes are stucked, unable to free

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-07 Thread Emeric Brun
On 5/7/19 3:35 PM, Marcin Deranek wrote: > Hi Emeric, > > On 5/7/19 1:53 PM, Emeric Brun wrote: >> On 5/7/19 1:24 PM, Marcin Deranek wrote: >>> Hi Emeric, >>> >>> On 5/7/19 11:44 AM, Emeric Brun wrote: Hi Marcin,>> As I use HAProxy 1.8 I had to adjust the patch (see attachment for

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-07 Thread Marcin Deranek
Hi Emeric, On 5/7/19 1:53 PM, Emeric Brun wrote: On 5/7/19 1:24 PM, Marcin Deranek wrote: Hi Emeric, On 5/7/19 11:44 AM, Emeric Brun wrote: Hi Marcin,>> As I use HAProxy 1.8 I had to adjust the patch (see attachment for end result). Unfortunately after applying the patch there is no

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-07 Thread Emeric Brun
On 5/7/19 1:24 PM, Marcin Deranek wrote: > Hi Emeric, > > On 5/7/19 11:44 AM, Emeric Brun wrote: >> Hi Marcin,>> As I use HAProxy 1.8 I had to adjust the patch (see >> attachment for end result). Unfortunately after applying the patch there is >> no change in behavior: we still leak

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-07 Thread Marcin Deranek
Hi Emeric, On 5/7/19 11:44 AM, Emeric Brun wrote: Hi Marcin,>> As I use HAProxy 1.8 I had to adjust the patch (see attachment for end result). Unfortunately after applying the patch there is no change in behavior: we still leak /dev/usdm_drv descriptors and have "stuck" HAProxy instances

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-03 Thread Emeric Brun
Hi Marcin, On 5/3/19 4:56 PM, Marcin Deranek wrote: > Hi Emeric, > > On 5/3/19 4:50 PM, Emeric Brun wrote: > >> I've a testing platform here but I don't use the usdm_drv but the >> qat_contig_mem and I don't reproduce this issue (I'm using QAT 1.5, as the >> doc says to use with my chip) . >

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-03 Thread Marcin Deranek
Hi Emeric, On 5/3/19 4:50 PM, Emeric Brun wrote: I've a testing platform here but I don't use the usdm_drv but the qat_contig_mem and I don't reproduce this issue (I'm using QAT 1.5, as the doc says to use with my chip) . I see. I use qat 1.7 and qat-engine 0.5.40. Anyway, could you

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-03 Thread Emeric Brun
Hi Marcin, Good so we progress! I've a testing platform here but I don't use the usdm_drv but the qat_contig_mem and I don't reproduce this issue (I'm using QAT 1.5, as the doc says to use with my chip) . Anyway, could you re-compile a haproxy's binary if I provide you a testing patch? The

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-03 Thread Marcin Deranek
Hi Emeric, It looks like on every reload master leaks /dev/usdm_drv device: # systemctl restart haproxy.service # ls -la /proc/$(cat haproxy.pid)/fd|fgrep dev lr-x-- 1 root root 64 May 3 15:40 0 -> /dev/null lrwx-- 1 root root 64 May 3 15:40 7 -> /dev/usdm_drv # systemctl reload

Re: [External] Re: QAT intermittent healthcheck errors

2019-05-03 Thread Emeric Brun
Hi Marcin, On 4/29/19 6:41 PM, Marcin Deranek wrote: > Hi Emeric, > > On 4/29/19 3:42 PM, Emeric Brun wrote: >> Hi Marcin, >> >>> I've also a contact at intel who told me to try this option on the qat engine: >

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-29 Thread Marcin Deranek
Hi Emeric, On 4/29/19 3:42 PM, Emeric Brun wrote: Hi Marcin, I've also a contact at intel who told me to try this option on the qat engine: --disable-qat_auto_engine_init_on_fork/--enable-qat_auto_engine_init_on_fork Disable/Enable the engine from being initialized automatically

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-29 Thread Emeric Brun
Hi Marcin, On 4/19/19 3:26 PM, Marcin Deranek wrote: > Hi Emeric, > > On 4/18/19 4:35 PM, Emeric Brun wrote: >>> An other interesting trace would be to perform a "show sess" command on a >>> stucked process through the master cli. >> >> And also the "show fd" > > Here it is: > > show proc > # 

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-19 Thread Marcin Deranek
Hi Emeric, On 4/18/19 4:35 PM, Emeric Brun wrote: An other interesting trace would be to perform a "show sess" command on a stucked process through the master cli. And also the "show fd" Here it is: show proc # 13409 master 0 1

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-12 Thread Emeric Brun
Hi Marcin, Do you have ssl enabled on the server side? If it is the case could replace health check with a simple tcp check (without ssl)? Regarding the show info/lsoff it seems there is no more sessions on client side but remaining ssl jobs (CurrSslConns) and I supsect the health checks to

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-12 Thread Marcin Deranek
Hi Emeric, On 4/10/19 2:20 PM, Emeric Brun wrote: On 4/10/19 1:02 PM, Marcin Deranek wrote: Hi Emeric, Our process limit in QAT configuration is quite high (128) and I was able to run 100+ openssl processes without a problem. According to Joel from Intel problem is in cleanup code -

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-10 Thread Emeric Brun
Hi Marcin, > You can also use the 'master CLI' using '-S' and you could check if it > remains sessions on those older processes (doc is available in management.txt) Here the doc: https://cbonte.github.io/haproxy-dconv/1.9/management.html#9.4 Emeric

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-10 Thread Emeric Brun
Hi Marcin, On 4/10/19 1:02 PM, Marcin Deranek wrote: > Hi Emeric, > > Our process limit in QAT configuration is quite high (128) and I was able to > run 100+ openssl processes without a problem. According to Joel from Intel > problem is in cleanup code - presumably when HAProxy exits and frees

Re: [External] Re: QAT intermittent healthcheck errors

2019-04-10 Thread Marcin Deranek
Hi Emeric, Our process limit in QAT configuration is quite high (128) and I was able to run 100+ openssl processes without a problem. According to Joel from Intel problem is in cleanup code - presumably when HAProxy exits and frees up QAT resources. Will try to see if I can get more debug

Re: [External] Re: QAT intermittent healthcheck errors

2019-03-13 Thread Emeric Brun
Hi Marcin, On 3/11/19 4:27 PM, Marcin Deranek wrote: > On 3/11/19 11:51 AM, Emeric Brun wrote: > >> Mode async is enabled on both sides, server and frontend side. >> >> But on server side, haproxy is using session resuming, so there is a new key >> computation (full handshake with RSA/DSA

Re: [External] Re: QAT intermittent healthcheck errors

2019-03-11 Thread Marcin Deranek
Hi Emeric, On 3/11/19 2:48 PM, Emeric Brun wrote: Once again, you could add the "no-ssl-reuse" statement if you want to check if QAT offloads the backend side, but it is clearly not an optimal option for production because it will generate an heavy load on your servers and force them to

Re: [External] Re: QAT intermittent healthcheck errors

2019-03-11 Thread Marcin Deranek
On 3/11/19 11:51 AM, Emeric Brun wrote: Mode async is enabled on both sides, server and frontend side. But on server side, haproxy is using session resuming, so there is a new key computation (full handshake with RSA/DSA computation) only every 5 minutes (openssl default value). You can

Re: [External] Re: QAT intermittent healthcheck errors

2019-03-11 Thread Marcin Deranek
Hi Emeric, On 3/8/19 4:43 PM, Emeric Brun wrote: I've just realized that if your server are TLSv1.3 ssl-default-server-ciphers won't force anything (see ssl-default-server-ciphersuites documentation) Backend servers are 'only' TLS 1.2, so it should have desired effect. Will test suggested