Re:Re: 回复：Haproxy SSl Termination performance issue

[Mike G]

Hi,  Emeric


got it.


Thanks a lot




Mike








At 2017-12-21 18:19:50, "Emeric Brun"  wrote:
>Hi Mike,
>
>Thierry is right, 4096 rsa key computation have clearly an heavy CPU cost.
>In our internal benchmark we notice:
>Using one process of haproxy on one core of i7-4790K CPU @ 4.00GHz we reach 
>170 con/s (comparatively 1350 con/s using 2048 rsa key).
>
>Usually this CPU usage isn't so high because software clients use "session ID" 
>or "tls tickets" witch avoid to recompute an rsa key for each new connection 
>from the same client.
>
>I don't know if what you observe is due to a production traffic or a benchmark 
>tool but in any case your software clients
>don't seem to reuse sessions.
>
>Anyway, Thierry has well described how to scale your arch using the max of 
>available cpu.
>
>R,
>Emeric 
> 
>
>
>On 12/19/2017 04:14 PM, hongw...@163.com wrote:
>> Hi, Thierry.
>> 
>> Thanks again.
>> 
>> One more question about you talking about, can i just think like this way: 
>> assume we got a 8core cpu, we use 7 of them for ssl termination and one is 
>> for http forward? If it is, is there any document for this soulution?
>> 
>> Thanks a lot
>> 
>> Mike
>> 
>> 
>> 
>>  原始邮件 
>> 主题：Re: Haproxy SSl Termination performance issue
>> 发件人：Thierry Fournier
>> 收件人：Mike G
>> 抄送：Haproxy
>> 
>> 
>> Ok, you’re using HAProxy as SSL offloading. HAProxy is one of the
>> right solutions for doing this. You’re performance problem is not
>> due to HAProxy, each component using OpenSSL will reach the same
>> limits.
>> 
>> Classic setup is to configure many process for the SSL offloading
>> (proxy in TCP mode), and only one for the HTTP. This setup works
>> fine because it allow many CPU for the SSL which require computing,
>> and the HTTP processing is done by only one process which can
>> perform accounting and apply limits (rate limit, connexion limit,
>> ...).
>> 
>> Thierry
>> 
>> 
>> > On 19 Dec 2017, at 12:44, Mike G wrote:
>> >
>> >
>> >
>> > Hi, Thierry.
>> >
>> > our case is like this: we put a haproxy as ssl termination. and 
>> haproxy got the https requirement. and then go throught SSL ternimation. and 
>> then forward
>> > the request to web (by HTTP), also, get the Http request and encrypt 
>> it, and return HTTPS to client.
>> >
>> >
>> > thanks
>> >
>> > Mike
>> >
>> >
>> >
>> >
>> >
>> > At 2017-12-19 19:25:09, "Thierry Fournier" wrote:
>> > >Hi,
>> > >
>> > >What kind of job ?
>> > >
>> > >Thierry
>> > >
>> > >> On 19 Dec 2017, at 12:17, hongw...@163.com wrote:
>> > >>
>> > >> Hi，Thierry
>> > >>
>> > >> got it. Thanks！
>> > >>
>> > >> By the way, may I ask the ssl termination is best solution for this 
>> kind of job?
>> > >>
>> > >>
>> > >> Many thanks
>> > >>
>> > >> Mike
>> > >>
>> > >>
>> > >>
>> > >>  原始邮件 
>> > >> 主题：Re: Haproxy SSl Termination performance issue
>> > >> 发件人：Thierry Fournier
>> > >> 收件人：Mike G
>> > >> 抄送：Haproxy
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> I gues that 130 is 130 SSL requests per seconds ?
>> > >>
>> > >> SSL is a very heavy processing. The 4096 bits certificates consume 
>> more
>> > >> CPU that 2048 (thanks captain obvious). Your capacity processing is
>> > >> capped by your CPU. You must check the CPU of your server during 
>> your
>> > >> test. If the CPU consummation is 100%, you reach the limit of your 
>> server.
>> > >>
>> > >> If you reach the limit of one CPU (nbproc), you can use more CPU 
>> and/or more
>> > >> servers.
>> > >>
>> > >> Thierry
>> > >>
>> > >>
>> > >> > On 19 Dec 2017, at 08:36, Mike G wrote:
>> > >> >
>> > >> > Hi, everyone.
>> > >> >
>> > >> > I just got a problem about the haproxy ssl termination 
>> performance issues.
>> > >> > we have a case which want to use SSL Termination. so, we did some 
>> testing before online, I know the virtual machine will not good choice, but 
>> it make feel so supriose the cur link can be more than 130 when I use 4096 
>> key.
>> > >> > here's my configuration about the haproxy:
>> > >> >
>> > >> > haproxy as SSL termination layer before web server.
>> > >> > the haproxy version is 1.8.1
>> > >> > I compile it by myself:
>> > >> > use the parameter:
>> > >> > make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1
>> > >> >
>> > >> > also, I use download openssl 1.0.2n from openssl.org, and compile 
>> by those parameters:
>> > >> > ./config -d zlib
>> > >> >
>> > >> > after install openssl and haproxy.
>> > >> > here's my configuration about the haproxy:
>> > >> > global
>> > >> > log 127.0.0.1 local0
>> > >> >
>> > >> > chroot /var/lib/haproxy
>> > >> > pidfile 

Re: 回复：Haproxy SSl Termination performance issue

[Emeric Brun]

Hi Mike,

Thierry is right, 4096 rsa key computation have clearly an heavy CPU cost.
In our internal benchmark we notice:
Using one process of haproxy on one core of i7-4790K CPU @ 4.00GHz we reach 170 
con/s (comparatively 1350 con/s using 2048 rsa key).

Usually this CPU usage isn't so high because software clients use "session ID" 
or "tls tickets" witch avoid to recompute an rsa key for each new connection 
from the same client.

I don't know if what you observe is due to a production traffic or a benchmark 
tool but in any case your software clients
don't seem to reuse sessions.

Anyway, Thierry has well described how to scale your arch using the max of 
available cpu.

R,
Emeric 
 


On 12/19/2017 04:14 PM, hongw...@163.com wrote:
> Hi, Thierry.
> 
> Thanks again.
> 
> One more question about you talking about, can i just think like this way: 
> assume we got a 8core cpu, we use 7 of them for ssl termination and one is 
> for http forward? If it is, is there any document for this soulution?
> 
> Thanks a lot
> 
> Mike
> 
> 
> 
>  原始邮件 
> 主题：Re: Haproxy SSl Termination performance issue
> 发件人：Thierry Fournier
> 收件人：Mike G
> 抄送：Haproxy
> 
> 
> Ok, you’re using HAProxy as SSL offloading. HAProxy is one of the
> right solutions for doing this. You’re performance problem is not
> due to HAProxy, each component using OpenSSL will reach the same
> limits.
> 
> Classic setup is to configure many process for the SSL offloading
> (proxy in TCP mode), and only one for the HTTP. This setup works
> fine because it allow many CPU for the SSL which require computing,
> and the HTTP processing is done by only one process which can
> perform accounting and apply limits (rate limit, connexion limit,
> ...).
> 
> Thierry
> 
> 
> > On 19 Dec 2017, at 12:44, Mike G wrote:
> >
> >
> >
> > Hi, Thierry.
> >
> > our case is like this: we put a haproxy as ssl termination. and haproxy 
> got the https requirement. and then go throught SSL ternimation. and then 
> forward
> > the request to web (by HTTP), also, get the Http request and encrypt 
> it, and return HTTPS to client.
> >
> >
> > thanks
> >
> > Mike
> >
> >
> >
> >
> >
> > At 2017-12-19 19:25:09, "Thierry Fournier" wrote:
> > >Hi,
> > >
> > >What kind of job ?
> > >
> > >Thierry
> > >
> > >> On 19 Dec 2017, at 12:17, hongw...@163.com wrote:
> > >>
> > >> Hi，Thierry
> > >>
> > >> got it. Thanks！
> > >>
> > >> By the way, may I ask the ssl termination is best solution for this 
> kind of job?
> > >>
> > >>
> > >> Many thanks
> > >>
> > >> Mike
> > >>
> > >>
> > >>
> > >>  原始邮件 
> > >> 主题：Re: Haproxy SSl Termination performance issue
> > >> 发件人：Thierry Fournier
> > >> 收件人：Mike G
> > >> 抄送：Haproxy
> > >>
> > >>
> > >> Hi,
> > >>
> > >> I gues that 130 is 130 SSL requests per seconds ?
> > >>
> > >> SSL is a very heavy processing. The 4096 bits certificates consume 
> more
> > >> CPU that 2048 (thanks captain obvious). Your capacity processing is
> > >> capped by your CPU. You must check the CPU of your server during your
> > >> test. If the CPU consummation is 100%, you reach the limit of your 
> server.
> > >>
> > >> If you reach the limit of one CPU (nbproc), you can use more CPU 
> and/or more
> > >> servers.
> > >>
> > >> Thierry
> > >>
> > >>
> > >> > On 19 Dec 2017, at 08:36, Mike G wrote:
> > >> >
> > >> > Hi, everyone.
> > >> >
> > >> > I just got a problem about the haproxy ssl termination performance 
> issues.
> > >> > we have a case which want to use SSL Termination. so, we did some 
> testing before online, I know the virtual machine will not good choice, but 
> it make feel so supriose the cur link can be more than 130 when I use 4096 
> key.
> > >> > here's my configuration about the haproxy:
> > >> >
> > >> > haproxy as SSL termination layer before web server.
> > >> > the haproxy version is 1.8.1
> > >> > I compile it by myself:
> > >> > use the parameter:
> > >> > make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1
> > >> >
> > >> > also, I use download openssl 1.0.2n from openssl.org, and compile 
> by those parameters:
> > >> > ./config -d zlib
> > >> >
> > >> > after install openssl and haproxy.
> > >> > here's my configuration about the haproxy:
> > >> > global
> > >> > log 127.0.0.1 local0
> > >> >
> > >> > chroot /var/lib/haproxy
> > >> > pidfile /var/run/haproxy.pid
> > >> > maxconn 65535
> > >> > group haproxy
> > >> > user haproxy
> > >> > daemon
> > >> > nbproc 1
> > >> >
> > >> > stats socket /var/lib/haproxy/stats
> > >> > tune.ssl.default-dh-param 2048
> > >> >
> > >> > defaults
>

回复：Haproxy SSl Termination performance issue

[hongw...@163.com]

Hi, JohanThanks a lotMikeSent from my HuaWei mate9 Phone 原始邮件 主题：Re: Haproxy SSl Termination performance issue发件人：Johan Hendriks 收件人：haproxy@formilux.org,hongw...@163.com抄送：Op di 19 dec. 2017 om 16:16 schreef hongw...@163.com Hi, Thierry.Thanks again.One more question about you talking about, can i just think like this way: assume we got a 8core cpu, we use 7 of them for ssl termination and one is for http forward?  If it is, is there any document for this soulution?Thanks a lotMikeI think the following blog can help you with that.https://medium.com/cagataygurturk/using-haproxy-in-multi-core-environments-68ee2d3ae39eRegardsJohan

回复：Haproxy SSl Termination performance issue

[hongw...@163.com]

Hi, Thierry.Thanks again.One more question about you talking about, can i just think like this way: assume we got a 8core cpu, we use 7 of them for ssl termination and one is for http forward?  If it is, is there any document for this soulution?Thanks a lotMike 原始邮件 主题：Re: Haproxy SSl Termination performance issue发件人：Thierry Fournier 收件人：Mike G 抄送：Haproxy Ok, you’re using HAProxy as SSL offloading. HAProxy is one of theright solutions for doing this. You’re performance problem is notdue to HAProxy, each component using OpenSSL will reach the samelimits.Classic setup is to configure many process for the SSL offloading(proxy in TCP mode), and only one for the HTTP. This setup worksfine because it allow many CPU for the SSL which require computing,and the HTTP processing is done by only one process which canperform accounting and apply limits (rate limit, connexion limit,...). Thierry> On 19 Dec 2017, at 12:44, Mike G  wrote:> > > > Hi, Thierry.> > our case is like this: we put a haproxy as ssl termination.  and haproxy got the https requirement. and then go throught SSL ternimation. and then forward > the request to web (by HTTP), also, get the Http request and encrypt it, and return HTTPS to client.> > > thanks> > Mike> > > > > > At 2017-12-19 19:25:09, "Thierry Fournier"  wrote:> >Hi,> >> >What kind of job ?> >> >Thierry> >> >> On 19 Dec 2017, at 12:17, hongw...@163.com wrote:> >> > >> Hi，Thierry> >> > >> got it. Thanks！> >> > >> By the way, may I ask the ssl termination is best solution for this kind of job?> >> > >> > >> Many thanks> >> > >> Mike> >> > >> > >> > >>  原始邮件 > >> 主题：Re: Haproxy SSl Termination performance issue> >> 发件人：Thierry Fournier > >> 收件人：Mike G > >> 抄送：Haproxy > >> > >> > >> Hi,> >> > >> I gues that 130 is 130 SSL requests per seconds ?> >> > >> SSL is a very heavy processing. The 4096 bits certificates consume more> >> CPU that 2048 (thanks captain obvious). Your capacity processing is> >> capped by your CPU. You must check the CPU of your server during your> >> test. If the CPU consummation is 100%, you reach the limit of your server.> >> > >> If you reach the limit of one CPU (nbproc), you can use more CPU and/or more> >> servers.> >> > >> Thierry> >> > >> > >> > On 19 Dec 2017, at 08:36, Mike G wrote:> >> > > >> > Hi, everyone. > >> > > >> > I just got a problem about the haproxy ssl termination performance issues. > >> > we have a case which want to use SSL Termination. so, we did some testing before online, I know the virtual machine will not good choice, but it make feel so supriose the cur link can be more than 130 when I use 4096 key.> >> > here's my configuration about the haproxy:> >> > > >> > haproxy as SSL termination layer before web server.  > >> > the haproxy version is 1.8.1> >> > I compile it by myself:> >> > use the parameter:> >> > make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1> >> > > >> > also, I use download openssl 1.0.2n from openssl.org, and compile by those parameters:> >> > ./config -d zlib> >> > > >> > after install openssl and haproxy.> >> > here's my configuration about the haproxy:> >> > global> >> > log 127.0.0.1 local0> >> > > >> > chroot /var/lib/haproxy> >> > pidfile /var/run/haproxy.pid> >> > maxconn 65535> >> > group haproxy> >> > user haproxy> >> > daemon> >> > nbproc 1> >> > > >> > stats socket /var/lib/haproxy/stats> >> > tune.ssl.default-dh-param 2048> >> > > >> > defaults> >> > mode http> >> > log global> >> > option redispatch> >> > option abortonclose> >> > log 127.0.0.1 local0> >> > retries 3> >> > maxconn 65535> >> > timeout connect 10s> >> > timeout client 1m> >> > timeout queue 1m> >> > timeout http-request 30s> >> > timeout server 1m> >> > timeout check 5s> >> > > >> > listen admin_stats> >> > bind 0.0.0.0:20123> >> > maxconn 10> >> > stats refresh 10s> >> > stats uri /web/status> >> > stats auth admin:1> >> > stats hide-version> >> > > >> > > >> > frontend localhost> >> > bind *:80> >> > bind *:443 ssl crt /etc/ssl/web-zhengshu.pem> >> > option httpclose> >> > mode http> >> > default_backend nodes> >> > > >> > backend nodes> >> > mode http> >> > balance roundrobin> >> > option forwardfor> >> > option httpchk GET /check.html> >> > server web01 127.0.0.1:8080 check> >> > http-request set-header X-Forwarded-Port %[dst_port]> >> > http-request add-header X-Forwarded-Proto https if { ssl_fc }> >> > > >> > > >> > note: about the option httpclose, I make it for purpose.> >> > > >> > also, I use vegeta for test.> >> > here's the testing command line:> >> > echo "GET https://10.77.77.215/check.html" | ./vegeta.vegeta -cpus=8 attack -duration=90s -rate=800 -insecure | tee reports.bin | ./vegeta.vegeta report> >> > > >> > I found the cpu is get more than 90% usage very soon. but the haproxy status picture like in attachment.> >> > > >> > the max links is less than 130 around.> >> > > 

回复：Haproxy SSl Termination performance issue

[hongw...@163.com]

Hi，ThierryMany thanksMike发自我的华为手机 原始邮件 主题：Re: Haproxy SSl Termination performance issue发件人：Thierry Fournier 收件人：Mike G 抄送：Haproxy Ok, you’re using HAProxy as SSL offloading. HAProxy is one of theright solutions for doing this. You’re performance problem is notdue to HAProxy, each component using OpenSSL will reach the samelimits.Classic setup is to configure many process for the SSL offloading(proxy in TCP mode), and only one for the HTTP. This setup worksfine because it allow many CPU for the SSL which require computing,and the HTTP processing is done by only one process which canperform accounting and apply limits (rate limit, connexion limit,...). Thierry> On 19 Dec 2017, at 12:44, Mike G  wrote:> > > > Hi, Thierry.> > our case is like this: we put a haproxy as ssl termination.  and haproxy got the https requirement. and then go throught SSL ternimation. and then forward > the request to web (by HTTP), also, get the Http request and encrypt it, and return HTTPS to client.> > > thanks> > Mike> > > > > > At 2017-12-19 19:25:09, "Thierry Fournier"  wrote:> >Hi,> >> >What kind of job ?> >> >Thierry> >> >> On 19 Dec 2017, at 12:17, hongw...@163.com wrote:> >> > >> Hi，Thierry> >> > >> got it. Thanks！> >> > >> By the way, may I ask the ssl termination is best solution for this kind of job?> >> > >> > >> Many thanks> >> > >> Mike> >> > >> > >> > >>  原始邮件 > >> 主题：Re: Haproxy SSl Termination performance issue> >> 发件人：Thierry Fournier > >> 收件人：Mike G > >> 抄送：Haproxy > >> > >> > >> Hi,> >> > >> I gues that 130 is 130 SSL requests per seconds ?> >> > >> SSL is a very heavy processing. The 4096 bits certificates consume more> >> CPU that 2048 (thanks captain obvious). Your capacity processing is> >> capped by your CPU. You must check the CPU of your server during your> >> test. If the CPU consummation is 100%, you reach the limit of your server.> >> > >> If you reach the limit of one CPU (nbproc), you can use more CPU and/or more> >> servers.> >> > >> Thierry> >> > >> > >> > On 19 Dec 2017, at 08:36, Mike G wrote:> >> > > >> > Hi, everyone. > >> > > >> > I just got a problem about the haproxy ssl termination performance issues. > >> > we have a case which want to use SSL Termination. so, we did some testing before online, I know the virtual machine will not good choice, but it make feel so supriose the cur link can be more than 130 when I use 4096 key.> >> > here's my configuration about the haproxy:> >> > > >> > haproxy as SSL termination layer before web server.  > >> > the haproxy version is 1.8.1> >> > I compile it by myself:> >> > use the parameter:> >> > make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1> >> > > >> > also, I use download openssl 1.0.2n from openssl.org, and compile by those parameters:> >> > ./config -d zlib> >> > > >> > after install openssl and haproxy.> >> > here's my configuration about the haproxy:> >> > global> >> > log 127.0.0.1 local0> >> > > >> > chroot /var/lib/haproxy> >> > pidfile /var/run/haproxy.pid> >> > maxconn 65535> >> > group haproxy> >> > user haproxy> >> > daemon> >> > nbproc 1> >> > > >> > stats socket /var/lib/haproxy/stats> >> > tune.ssl.default-dh-param 2048> >> > > >> > defaults> >> > mode http> >> > log global> >> > option redispatch> >> > option abortonclose> >> > log 127.0.0.1 local0> >> > retries 3> >> > maxconn 65535> >> > timeout connect 10s> >> > timeout client 1m> >> > timeout queue 1m> >> > timeout http-request 30s> >> > timeout server 1m> >> > timeout check 5s> >> > > >> > listen admin_stats> >> > bind 0.0.0.0:20123> >> > maxconn 10> >> > stats refresh 10s> >> > stats uri /web/status> >> > stats auth admin:1> >> > stats hide-version> >> > > >> > > >> > frontend localhost> >> > bind *:80> >> > bind *:443 ssl crt /etc/ssl/web-zhengshu.pem> >> > option httpclose> >> > mode http> >> > default_backend nodes> >> > > >> > backend nodes> >> > mode http> >> > balance roundrobin> >> > option forwardfor> >> > option httpchk GET /check.html> >> > server web01 127.0.0.1:8080 check> >> > http-request set-header X-Forwarded-Port %[dst_port]> >> > http-request add-header X-Forwarded-Proto https if { ssl_fc }> >> > > >> > > >> > note: about the option httpclose, I make it for purpose.> >> > > >> > also, I use vegeta for test.> >> > here's the testing command line:> >> > echo "GET https://10.77.77.215/check.html" | ./vegeta.vegeta -cpus=8 attack -duration=90s -rate=800 -insecure | tee reports.bin | ./vegeta.vegeta report> >> > > >> > I found the cpu is get more than 90% usage very soon. but the haproxy status picture like in attachment.> >> > > >> > the max links is less than 130 around.> >> > > >> > but when I changed the ssl certication file back to 2048, it will be increase to around 800.> >> > > >> > is there anyone can help me about how to improve the haproxy ssl termination performance?> >> > > >> > > >> > Many 

回复：Haproxy SSl Termination performance issue

[hongw...@163.com]

Hi，Thierrygot it. Thanks！By the way, may I ask the ssl termination is best solution for this kind of job?Many thanksMike 原始邮件 主题：Re: Haproxy SSl Termination performance issue发件人：Thierry Fournier 收件人：Mike G 抄送：Haproxy Hi,I gues that 130 is 130 SSL requests per seconds ?SSL is a very heavy processing. The 4096 bits certificates consume moreCPU that 2048 (thanks captain obvious). Your capacity processing iscapped by your CPU. You must check the CPU of your server during yourtest. If the CPU consummation is 100%, you reach the limit of your server.If you reach the limit of one CPU (nbproc), you can use more CPU and/or moreservers.Thierry> On 19 Dec 2017, at 08:36, Mike G  wrote:> > Hi, everyone. > > I just got a problem about the haproxy ssl termination performance issues. > we have a case which want to use SSL Termination. so, we did some testing before online, I know the virtual machine will not good choice, but it make feel so supriose the cur link can be more than 130 when I use 4096 key.> here's my configuration about the haproxy:> > haproxy as SSL termination layer before web server.  > the haproxy version is 1.8.1> I compile it by myself:> use the parameter:>  make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1> > also,  I use download openssl 1.0.2n from openssl.org, and compile by those parameters:> ./config -d zlib> > after install openssl and haproxy.> here's my configuration about the haproxy:> global> log 127.0.0.1   local0> > chroot  /var/lib/haproxy> pidfile /var/run/haproxy.pid> maxconn 65535> group haproxy> user haproxy> daemon> nbproc 1> > stats socket /var/lib/haproxy/stats> tune.ssl.default-dh-param 2048> > defaults> modehttp> log global> option  redispatch> option  abortonclose> log 127.0.0.1 local0> retries 3> maxconn 65535> timeout connect 10s> timeout client  1m> timeout queue   1m> timeout http-request30s> timeout server  1m> timeout check   5s> > listen  admin_stats> bind 0.0.0.0:20123> maxconn 10> stats refresh 10s> stats uri /web/status> stats auth admin:1> stats hide-version> > > frontend localhost> bind *:80> bind *:443 ssl crt /etc/ssl/web-zhengshu.pem> option httpclose> mode http> default_backend nodes> > backend nodes> mode http> balance roundrobin> option forwardfor> option httpchk GET /check.html> server web01 127.0.0.1:8080 check> http-request set-header X-Forwarded-Port %[dst_port]> http-request add-header X-Forwarded-Proto https if { ssl_fc }> > > note: about the option httpclose,  I make it for purpose.> > also, I use vegeta for test.> here's the testing command line:> echo "GET https://10.77.77.215/check.html" | ./vegeta.vegeta -cpus=8 attack -duration=90s -rate=800 -insecure | tee reports.bin | ./vegeta.vegeta report> > I found the cpu is get more than 90% usage very soon. but the haproxy status picture like in attachment.> > the max links is less than 130 around.> > but when I changed the ssl certication file back to 2048, it will be increase to around 800.> > is there anyone can help me about how to improve the haproxy ssl termination performance?> > > Many thanks> > > Mike> > > > > > > > > > > >  >