Re: How to configure frontend/backend for SSL OR Non SSL Backend?

[Baptiste]

So basically, here is what you want to do:
peers mypeers
 # read the doc for the info to store here

frontend ftapp
 bind :80
 bind :443
 mode tcp
 default_backend bkapp

backend bkapp
 mode tcp
 stick-table type ip size 10k peers mypeers
 stick on src
 server s1 a.b.c.d check port 80
 server s2 a.b.c.e check port 80


Baptiste


On Mon, Oct 12, 2015 at 4:40 PM, Daren Sefcik  wrote:
> humm...not sure I know how to answer that...we have servers that require SSL
> for some requests and not for others. I am not needing to do anything other
> than pass the traffic along, not doing any inspection or verifying of cert
> or anything. I tried to setup a frontend with 2 servers in the backend, one
> with 443 and the other with 80 but that didn't seem to work, like it would
> pick the wrong one to send to.
>
> On Mon, Oct 12, 2015 at 7:29 AM, Baptiste  wrote:
>>
>> Hi Daren,
>>
>> Do you want/need to decipher the traffic when using SSL?
>>
>> Baptiste
>>
>> On Mon, Oct 12, 2015 at 4:24 PM, Daren Sefcik 
>> wrote:
>> > I am probably totally overlooking something but how do I configure a
>> > frontend/backend to pass to the same server for both SSL and Non SSL
>> > requests?  We have server that require ssl for some applications but
>> > most of
>> > the time not.
>> >
>
>

Fix for s390x

[Neale Ferguson]

Hi,
 I would like to contribute the following fix to enable the Linux s390x
platform. The fix was built against today¹s git master. I¹ve attached the
patch for review. Depending on your buildbot/jenkins/Š requirements I can
set up a virtual machine for automated building/testing of the package in
this environment.

Neale



haproxy-s390x.patch
Description: haproxy-s390x.patch

Re: rebalance sessions when re-restarting server

[Baptiste]

Hi Stephen,

you have to wait for either the client or the server to close the connection.
As you said, the "sessions don't end", so your problem is by design on
your application.

Baptiste


On Mon, Oct 12, 2015 at 5:59 PM, Walsh, Stephen
 wrote:
> Hi all,
>
>
>
> We are using HaProxy in trial. We use it as a TCP Load Balancer for SSL
> connections.
>
> These sessions don’t end and are persistent.
>
>
>
> However when a node is restarted all sessions are moved the other nodes and
> never come back to the restarted one.
>
> How can we rebalance these backend nodes without restarting Ha Proxy?
>
>
>
>
>
> Regards
>
> Stephen W
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and destroy
> any copies. You may not further disclose or distribute this email or its
> attachments.

rebalance sessions when re-restarting server

[Walsh, Stephen]

Hi all,

We are using HaProxy in trial. We use it as a TCP Load Balancer for SSL 
connections.
These sessions don't end and are persistent.

However when a node is restarted all sessions are moved the other nodes and 
never come back to the restarted one.
How can we rebalance these backend nodes without restarting Ha Proxy?


Regards
Stephen W
This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

Re: Interactive stats socket broken on master

[Baptiste]

On Mon, Oct 12, 2015 at 12:06 AM, Willy Tarreau  wrote:
> On Sat, Oct 10, 2015 at 08:55:44PM -0500, Andrew Hayworth wrote:
>> Bump -
>>
>> I don't mind maintaining my own HAProxy package, but it seems bad to
>> release a major version with the interactive stats socket broken. Any
>> thoughts on the patch?
>
> Has anyone else tested it ? Since the beginning of the thread I must
> confess it's unclear to me as Jesse reported the issue, you said that
> your patch works for you then Jesse asks whether we should merge it.
> Jesse, have you tested it as well, so that we ensure you're facing
> the same issue ?
>
> Andrew BTW, your patch looks good and seems to do what you described
> in the message, I'm just asking to be sure that it addresses Jesse's
> bug as well.
>
> Last point guys, please keep in mind that not everybody reads all
> e-mails, so when you want to have a patch integrated, clearly mark
> it in the subject and don't leave it pending at the trail of a
> thread like this.
>
> Thanks,
> Willy
>


Hi all,

I confirm attached patch fix the issue: natively on my computer
(Willy, I was wrong this morning, I still had the bug) and when
HAProxy runs in docker too.

Baptiste

7inch rugged IP68 good cost Rugged tablet from china

[gavin.fan]

This email newsletter was sent to you in graphical HTML format.
If you're seeing this version, your email program prefers plain text emails.
You can read the original version online:
http://ymlp169.com/zRqprx



Hello Dear
Recommend one 7inch IP68 rugged tablet for you!
1. rugged IP68 grate, 1.5 meter droping test
2. qual core 1.3GHz, MSM8382
3. Android 4.4
4. 7inch HD/FFS 1280*800 screen
5. WIFI 802.11a b/g/n, BT4.0 3G WCDMA
6. 3.7V  8000mAh battery
more information, please click this link:
http://www.sincoole.com/English/Product/0243984012.html
any question, you are welcome.

BR

Gavin.fan
Email: gavin@sincoole.com
Mobile: +8613682649015
TEL:+8675527584269 ( tel:+8675527584269 )
skype: sincoole
qq: 2502265974
whatapp: +8613682649015
WEB: www.sincoole.com/english
www.sincoole.net
ADD: Fuzhongfu D6 xixiang town, Shenzhen city,China

_
Unsubscribe / Change Profile: http://ymlp169.com/ughweywjgsgeuhbujghbmgguyjyq
Powered by YourMailingListProvider

Re: How to configure frontend/backend for SSL OR Non SSL Backend?

[Daren Sefcik]

humm...not sure I know how to answer that...we have servers that require
SSL for some requests and not for others. I am not needing to do anything
other than pass the traffic along, not doing any inspection or verifying of
cert or anything. I tried to setup a frontend with 2 servers in the
backend, one with 443 and the other with 80 but that didn't seem to work,
like it would pick the wrong one to send to.

On Mon, Oct 12, 2015 at 7:29 AM, Baptiste  wrote:

> Hi Daren,
>
> Do you want/need to decipher the traffic when using SSL?
>
> Baptiste
>
> On Mon, Oct 12, 2015 at 4:24 PM, Daren Sefcik 
> wrote:
> > I am probably totally overlooking something but how do I configure a
> > frontend/backend to pass to the same server for both SSL and Non SSL
> > requests?  We have server that require ssl for some applications but
> most of
> > the time not.
> >
>

Re: How to configure frontend/backend for SSL OR Non SSL Backend?

[Baptiste]

Hi Daren,

Do you want/need to decipher the traffic when using SSL?

Baptiste

On Mon, Oct 12, 2015 at 4:24 PM, Daren Sefcik  wrote:
> I am probably totally overlooking something but how do I configure a
> frontend/backend to pass to the same server for both SSL and Non SSL
> requests?  We have server that require ssl for some applications but most of
> the time not.
>

Optimal Configuration examples for Squid or other cache backends?

[Daren Sefcik]

Can anyone point me to any references that explain optimal squid/web cache
specific examples of configuring haproxy? Everything I am finding so far
seems specific to web servers being the backend. Things like should I be
using stick tables or cookies or best client time out, etc

TIA

How to configure frontend/backend for SSL OR Non SSL Backend?

[Daren Sefcik]

I am probably totally overlooking something but how do I configure a
frontend/backend to pass to the same server for both SSL and Non SSL
requests?  We have server that require ssl for some applications but most
of the time not.

Re: Interactive stats socket broken on master

[Jesse Hathaway]

On Sun, Oct 11, 2015 at 5:06 PM, Willy Tarreau  wrote:
>
> Has anyone else tested it ? Since the beginning of the thread I must
> confess it's unclear to me as Jesse reported the issue, you said that
> your patch works for you then Jesse asks whether we should merge it.
> Jesse, have you tested it as well, so that we ensure you're facing
> the same issue ?

I can confirm the patch does indeed fix the issue for me, sorry for
any confusion.

Re: Fix for s390x

[Willy Tarreau]

On Mon, Oct 12, 2015 at 04:03:15PM +, Neale Ferguson wrote:
> Hi,
>  I would like to contribute the following fix to enable the Linux s390x
> platform. The fix was built against today¹s git master. I¹ve attached the
> patch for review. Depending on your buildbot/jenkins/? requirements I can
> set up a virtual machine for automated building/testing of the package in
> this environment.

Patch applied, thanks Neale!

Willy

Re: Interactive stats socket broken on master

[Willy Tarreau]

On Mon, Oct 12, 2015 at 12:33:39PM -0500, Jesse Hathaway wrote:
> On Sun, Oct 11, 2015 at 5:06 PM, Willy Tarreau  wrote:
> >
> > Has anyone else tested it ? Since the beginning of the thread I must
> > confess it's unclear to me as Jesse reported the issue, you said that
> > your patch works for you then Jesse asks whether we should merge it.
> > Jesse, have you tested it as well, so that we ensure you're facing
> > the same issue ?
> 
> I can confirm the patch does indeed fix the issue for me, sorry for
> any confusion.

Great, thanks for clearing this up, I've applied it now. Funnily, I just
faced the bug while writing the management doc and looking for an example
output from the CLI to copy-paste :-)

Willy

Re: HTTP Response Rewriting to Replace Internal IP with FQDN

[Susheel Jalali]

Dear HAProxy Developer community,

We are seeking your inputs in the following issue we are facing:

We would like to access Product1 via URL: 
https://coscend.com:14443/Product1/ , wherein 14443 is port forwarded by 
router to 443 inside the server.


Output URL from the Product1 server should be: 
https://coscend.com:14443/Product1/signin?xyz


Current set up is giving: 
http://coscend.com:/Product1/signin?xyz


(1)  instead of 14443 (forwarded port).

(2) http, instead of https


@Aleks

Thank you for your insights.  It helped make some progress, but are 
short of conquering the final frontier.  Your guidance will be appreciated.



Setup you advised:  Client -> haproxy -> Product1 app server (Tomcat), 
which is talking HTTP in its standard deployment 
(http://IP:port/appname) and has no custom rewriting or SSL or other 
proxy (Apache or nginx).


As the content is dynamic (real-time video), we are not using Varnish.

Logs and config deployed for HAProxy 1.5.14 (stable)

We are not getting any errors in the logs, as the right Web page is 
being displayed.



frontend webapps-frontend
bind  *:80 name http
bind  *:443 name https ssl crt /path/to/server.pem

log   global
optionforwardfor
optionhttplog clf

reqadd X-Forwarded-Proto:\ https if { ssl_fc }
reqadd X-Forwarded-Proto:\ http if !{ ssl_fc }

acl host_httpsreq.hdr(Host) coscend.com:14443  # 14443 is due 
to port forwarding deployment

acl path_subdomain_p1 path_beg -i /Product1

use_backend subdomain_p1-backend if host_https path_subdomain_p1


backend subdomain_p1-backend
http-request set-header Host 
reqirep ^([^\ ]*)\ /Product1/*([^\ ]*)\ (.*)$   \1\ /Product1\2\ \3

acl hdr_location res.hdr(Location) -m found
rspirep ^Location:\ (https?://)([^/]*)(:[0-9]+)(/.*)$ Location:\ 
\1coscend.com\3\4 if hdr_location


server Product1.VM0  
cookie c check



Output of haproxy -f … -db -V
Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result FAILED
Total: 3 (2 usable), will use epoll.

Thank you.

Sincerely,

--
Susheel Jalali

Coscend Communications Solutions
Web site: www(DOT)Coscend(DOT)com
--
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail 
Messages from Coscend Communications Solutions' posted at:

www(DOT)Coscend(DOT)com/Terms_and_Conditions.html


On 10/08/15 03:50, Aleksandar Lazic wrote:

Dear Susheel Jalali.

Am 07-10-2015 23:24, schrieb Susheel Jalali:

Dear Igor and Aleks,

Thank you for your insights.  Very useful to us, as we are implementing
HAProxy for the first time.  Below we have described how we have
implemented your advise and the result. Output of "haproxy -vv" is
given at end.

We have also provided the configuration file and relevant logs. We
would appreciate any insights to replace the internal IP address
occurring in server-response URL with the externally valid domain name
either by using the rewriting of Location and Host headers or the
complete URL, using %HP.

We would like to access Product1 via URL:
https://coscend.com:14443/Product1/
Output URL from the Product1 server should be:
https://coscend.com:14443/Product1/signin?xyz

what we are getting: http://Internal_IP:14443/Product1/signin?xyz

--
Responses to your insights / questions

@ Aleks:  Yes, Tomcat has a reverse proxy setting for our Product1.  
Can we

not have two reverse proxies to Product1?


Sorry I do not understand what you mean.

Have your read and understood the proxy-howto?

https://tomcat.apache.org/tomcat-8.0-doc/proxy-howto.html

I assume your setup is like this.

Client->haproxy->apache-mod_proxy_ajp->tomcat-X

Is this right?

Maybe you can omit apache-mod_proxy_ajp and talk http with tomcat.
Client->haproxy->tomcat-X HTTP Connector

In case that you want to deliver static content you should consider to 
use such a setup.


Client->haproxy->varnish->tomcat-X HTTP Connector
Client->haproxy->nginx+cache->tomcat-X HTTP Connector

Take care that you setup one of the *NIO* or *Apr* protocol handler
https://tomcat.apache.org/tomcat-8.0-doc/config/http.html



@Igor,

(1) As you rightly pointed out, we are getting http, not https

(2) As you advised, we moved these two lines from backend to frontend,
but did not find any change.

acl hdr_location res.hdr(Location) -m found
rspirep ^(Location:)\ (https?://([^/]*))/(.*)$\1\
http://\3/Product1/\4 if hdr_location



Maybe you can offer (gist,download,...) some debug output from

haproxy -f ... ... -db -V'


(3) Configuration file

global
log 127.0.0.1 local2
log-tag haproxy
chroot  /var/haproxy/lib
pidfile /var/run/haproxy.pid
userhaproxy
group   haproxy
nbproc  1
maxconn 5000
spread-checks 5
daemon
#debug
stats socket  

Re: core dump, lua service, 1.6-dev6 ss-20150930

[PiBa-NL]

Hi Willy,

Op 12-10-2015 om 7:28 schreef Willy Tarreau:

Hi Pieter,

On Mon, Oct 12, 2015 at 01:22:48AM +0200, PiBa-NL wrote:

#1  0x00417388 in buffer_slow_realign (buf=0x7d3c90) at
src/buffer.c:166
   block1 = -3306
   block2 = 0

I'm puzzled by this above, no block should have a negative size.


#2  0x00480c42 in http_wait_for_request (s=0x80247d600,
req=0x80247d610, an_bit=4)
   at src/proto_http.c:2686
   cur_idx = -6336
   sess = (struct session *) 0x80241e400
   txn = (struct http_txn *) 0x802bb2140
   msg = (struct http_msg *) 0x802bb21a0
   ctx = {line = 0x2711079 , idx = 3, val = 0, vlen = 7, tws = 0, del = 33, prev = 0}

And this above, similarly cur_idx shouldn't be negative.


Seems that buffer_slow_realign() isn't used regularly during normal
haproxy operation, and it crashes first time that specific function
gets called.
Reproduction is pretty consistent with chrome browser refreshing stats
every second.
Then starting: wrk -c 200 -t 2 -d 10 http://127.0.0.1:801/
I tried adding some Alert(); items in the code to see what parameters
are set at what step, but am not understanding the exact internals of
that code..

This negative bh=-7800 is not supposed to be there i think? It is from
one of the dprintf statements, how are those supposed generate output?..
[891069718] http_wait_for_request: stream=0x80247d600 b=0x80247d610,
exp(r,w)=0,0 bf=00c08200 bh=-7800 analysers=34

Anything else i can check or provide to help get this fixed?

Best regards,
PiBa-NL

Just a little 'bump' to this issue..

Anyone know when/how this buffer_slow_realign() is suppose to work?

Yes, it's supposed to be used only when a request or response is wrapped
in the request or response buffer. It uses memcpy(), hence the "slow"
aspect of the realign.


I suspect it either contains a bug, or is called with bogus parameters..

It's very sensitive to the consistency of the buffer being realigned. So
errors such as buf->i + buf->o > buf->size, or buf->p > buf->data + buf->size,
or buf->p < buf->data etc... can lead to crashes. But these must never happen
at all otherwise it proves that there's a bug somewhere else.

Here since block1 is -3306 and block2 = 0, I suspect that they were assigned
at line 159 from buf->i, which definitely means that the buffer was already
corrupted.


How can we/i determine which it is?

The difficulty consists in finding what can lead to a corrupted buffer :-/
In the past we had such issues when trying to forward more data than was
available in the buffer, due to option send-name-header. I wouldn't be
surprized that it can happen here on corner cases when building a message
from Lua if the various message pointers are not all perfectly correct.


Even though with a small change in the config (adding a backend) i cant
reproduce it, that doesnt mean there isn't a problem with the fuction..
As the whole function doesn't seem to get called in that circumstance..

It could be related to an uninitialized variable somewhere as well. You
can try to start haproxy with "-dM" to see if it makes the issues 100%

-dM doesnt seem to make much difference if any i this case..

reproducible or not. This poisons all buffers (fills them with a constant
byte 0x50 after malloc) so that we don't rely on an uninitialized zero byte
somewhere.

Regards,
Willy

Been running some more tests with the information that req->buf->i 
should be >= 0.


What i find is that after 1 request i already see rqh=-103 , it seems 
like the initial request size which in this case is also is 103 bytes is 
subtracted twice? It does not immediately crash, but if this is already 
a sign of 'corruption' then the cause should be a little more easy to find..


@Willy can you confirm this indicates the problem could be in progress 
of heading to a crash? Even though in the last line it restores to 0..


See its full output below replaced the DPRINTF already in code of 
stream.c with Alert..


Thanks in advance,
PiBa-NL

root@OPNsense:/usr/ports/net/haproxy-devel # haproxy -f 
/var/haproxy.config -d

[ALERT] 277/063055 (61489) : SSLv3 support requested but unavailable.
Available polling systems :
 kqueue : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result FAILED
Total: 3 (2 usable), will use kqueue.
Using kqueue() as the polling mechanism.
:http_frt.accept(0006)=0008 from [127.0.0.1:62358]
[ALERT] 277/063058 (61489) : [910446771] process_stream:1655: 
task=0x80244b9b0 s=0x80247d600, sfl=0x0080, rq=0x80247d610, 
rp=0x80247d650, exp(r,w)=0,0 rqf=00d08000 rpf=8000 rqh=0 rqt=0 rph=0 
rpt=0 cs=7 ss=0, cet=0x0 set=0x0 retr=0
[ALERT] 277/063058 (61489) : [910446771] http_wait_for_request: 
stream=0x80247d600 b=0x80247d610, exp(r,w)=0,0 bf=00d08000 bh=0 analysers=34
[ALERT] 277/063058 (61489) : [910446772] process_stream:1655: 
task=0x80244b9b0 s=0x80247d600, sfl=0x0080, rq=0x80247d610, 
rp=0x80247d650, 

Re: HTTP Response Rewriting to Replace Internal IP with FQDN

[Aleksandar Lazic]

Dear Susheel Jalali.

Am 12-10-2015 22:06, schrieb Susheel Jalali:

Dear HAProxy Developer community,

We are seeking your inputs in the following issue we are facing:

We would like to access Product1 via URL:
https://coscend.com:14443/Product1/ , wherein 14443 is port forwarded 
by

router to 443 inside the server.

Output URL from the Product1 server should be:
https://coscend.com:14443/Product1/signin?xyz

Current set up is giving:
http://coscend.com:/Product1/signin?xyz

(1)  instead of 14443 (forwarded port).

(2) http, instead of https

@Aleks

Thank you for your insights.  It helped make some progress, but are
short of conquering the final frontier.  Your guidance will be
appreciated.

Setup you advised:  Client -> haproxy -> Product1 app server (Tomcat),
which is talking HTTP in its standard deployment
(http://IP:port/appname) and has no custom rewriting or SSL or other
proxy (Apache or nginx).


Please can you also add the tomcat config and web xml.


As the content is dynamic (real-time video), we are not using Varnish.

Logs and config deployed for HAProxy 1.5.14 (stable)

We are not getting any errors in the logs, as the right Web page is
being displayed.

frontend webapps-frontend
bind  *:80 name http
bind  *:443 name https ssl crt /path/to/server.pem

log   global
optionforwardfor
optionhttplog clf

reqadd X-Forwarded-Proto:\ https if { ssl_fc }
reqadd X-Forwarded-Proto:\ http if !{ ssl_fc }

acl host_httpsreq.hdr(Host) coscend.com:14443  # 14443 is due 
to

port forwarding deployment
acl path_subdomain_p1 path_beg -i /Product1

use_backend subdomain_p1-backend if host_https path_subdomain_p1

backend subdomain_p1-backend
http-request set-header Host

reqirep ^([^\ ]*)\ /Product1/*([^\ ]*)\ (.*)$   \1\ 
/Product1\2\

\3


shouldn't be here the  in the replacment?


acl hdr_location res.hdr(Location) -m found
rspirep ^Location:\ (https?://)([^/]*)(:[0-9]+)(/.*)$ Location:\
\1coscend.com\3\4 if hdr_location


Maybe I misunderstood your request but shouldn't be in the replacement 
the port 14443?


rspirep ^Location:\ (https?://)([^/]*)(:[0-9]+)(/.*)$ Location:\ 
\1coscend.com:14443\4 if hdr_location


But still I think you should use

https://tomcat.apache.org/tomcat-8.0-doc/config/filter.html#Remote_IP_Filter
https://tomcat.apache.org/tomcat-8.0-doc/proxy-howto.html
https://tomcat.apache.org/tomcat-8.0-doc/config/http.html

proxyName
proxyPort
redirectPort


server Product1.VM0  
cookie

c check

Output of haproxy -f … -db -V
Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result FAILED
Total: 3 (2 usable), will use epoll.


Please can you post some output from the -db thanks.

BR Aleks


Thank you.

Sincerely,

 --
Susheel Jalali

Coscend Communications Solutions
Web site: www(DOT)Coscend(DOT)com
--
CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail
Messages from Coscend Communications Solutions' posted at:
www(DOT)Coscend(DOT)com/Terms_and_Conditions.html

On 10/08/15 03:50, Aleksandar Lazic wrote:


Dear Susheel Jalali.

Am 07-10-2015 23:24, schrieb Susheel Jalali:


Dear Igor and Aleks,

Thank you for your insights.  Very useful to us, as we are
implementing
HAProxy for the first time.  Below we have described how we have
implemented your advise and the result. Output of "haproxy -vv" is
given at end.

We have also provided the configuration file and relevant logs.  We
would appreciate any insights to replace the internal IP address
occurring in server-response URL with the externally valid domain
name
either by using the rewriting of Location and Host headers or the
complete URL, using %HP.

We would like to access Product1 via URL:
https://coscend.com:14443/Product1/
Output URL from the Product1 server should be:
https://coscend.com:14443/Product1/signin?xyz

what we are getting:   http://Internal_IP:14443/Product1/signin?xyz

--
Responses to your insights / questions

@ Aleks:  Yes, Tomcat has a reverse proxy setting for our Product1.
Can we
not have two reverse proxies to Product1?


Sorry I do not understand what you mean.

Have your read and understood the proxy-howto?

https://tomcat.apache.org/tomcat-8.0-doc/proxy-howto.html

I assume your setup is like this.

Client->haproxy->apache-mod_proxy_ajp->tomcat-X

Is this right?

Maybe you can omit apache-mod_proxy_ajp and talk http with tomcat.
Client->haproxy->tomcat-X HTTP Connector

In case that you want to deliver static content you should consider to
use such a setup.

Client->haproxy->varnish->tomcat-X HTTP Connector
Client->haproxy->nginx+cache->tomcat-X HTTP Connector

Take care that you setup one of the *NIO* or *Apr* protocol handler
https://tomcat.apache.org/tomcat-8.0-doc/config/http.html


@Igor,

(1) As you rightly pointed out, we are 

Re: core dump, lua service, 1.6-dev6 ss-20150930

[Willy Tarreau]

Hi Pieter,

On Mon, Oct 12, 2015 at 10:29:05PM +0200, PiBa-NL wrote:
> Been running some more tests with the information that req->buf->i 
> should be >= 0.
> 
> What i find is that after 1 request i already see rqh=-103 , it seems 
> like the initial request size which in this case is also is 103 bytes is 
> subtracted twice? It does not immediately crash, but if this is already 
> a sign of 'corruption' then the cause should be a little more easy to 
> find..

Oh yes definitely, good catch!

> @Willy can you confirm this indicates the problem could be in progress 
> of heading to a crash? Even though in the last line it restores to 0..

Absolutely. Everytime we're trying to track such a painful bug, I end
up looking for initial symptoms, like this one. In general the first
corruption is minor and undetected but once it's seeded its disease,
the problem will definitely occur.

> See its full output below replaced the DPRINTF already in code of 
> stream.c with Alert..
> 
> Thanks in advance,
> PiBa-NL
> 
> root@OPNsense:/usr/ports/net/haproxy-devel # haproxy -f 
> /var/haproxy.config -d
> [ALERT] 277/063055 (61489) : SSLv3 support requested but unavailable.
> Available polling systems :
>  kqueue : pref=300,  test result OK
>poll : pref=200,  test result OK
>  select : pref=150,  test result FAILED
> Total: 3 (2 usable), will use kqueue.
> Using kqueue() as the polling mechanism.
> :http_frt.accept(0006)=0008 from [127.0.0.1:62358]
> [ALERT] 277/063058 (61489) : [910446771] process_stream:1655: 
> task=0x80244b9b0 s=0x80247d600, sfl=0x0080, rq=0x80247d610, 
> rp=0x80247d650, exp(r,w)=0,0 rqf=00d08000 rpf=8000 rqh=0 rqt=0 rph=0 
> rpt=0 cs=7 ss=0, cet=0x0 set=0x0 retr=0
> [ALERT] 277/063058 (61489) : [910446771] http_wait_for_request: 
> stream=0x80247d600 b=0x80247d610, exp(r,w)=0,0 bf=00d08000 bh=0 
> analysers=34
> [ALERT] 277/063058 (61489) : [910446772] process_stream:1655: 
> task=0x80244b9b0 s=0x80247d600, sfl=0x0080, rq=0x80247d610, 
> rp=0x80247d650, exp(r,w)=0,0 rqf=0002 rpf=8000 rqh=103 rqt=0 
> rph=0 rpt=0 cs=7 ss=0, cet=0x0 set=0x0 retr=0
> [ALERT] 277/063058 (61489) : [910446772] http_wait_for_request: 
> stream=0x80247d600 b=0x80247d610, exp(r,w)=0,0 bf=00808002 bh=103 
> analysers=34
> :http_frt.clireq[0008:]: GET / HTTP/1.1
> :http_frt.clihdr[0008:]: Host: 127.0.0.1:801
> :http_frt.clihdr[0008:]: Accept: */*
> :http_frt.clihdr[0008:]: User-Agent: fetch libfetch/2.0
> :http_frt.clihdr[0008:]: Connection: close
> [ALERT] 277/063058 (61489) : [910446772] process_switching_rules: 
> stream=0x80247d600 b=0x80247d610, exp(r,w)=0,0 bf=00808002 bh=103 
> analysers=00
> [ALERT] 277/063058 (61489) : [910446772] sess_prepare_conn_req: 
> sess=0x80247d600 rq=0x80247d610, rp=0x80247d650, exp(r,w)=0,0 
> rqf=00808002 rpf=8000 rqh=0 rqt=103 rph=0 rpt=0 cs=7 ss=1
> [ALERT] 277/063058 (61489) : [910446772] process_stream:1655: 
> task=0x80244b9b0 s=0x80247d600, sfl=0x048a, rq=0x80247d610, 
> rp=0x80247d650, exp(r,w)=910456772,0 rqf=00808200 rpf=8023 rqh=0 
> rqt=0 rph=97 rpt=0 cs=7 ss=7, cet=0x0 set=0x0 retr=0
> :http_frt.srvrep[0008:]: HTTP/1.1 200 OK
> :http_frt.srvhdr[0008:]: content-length: 13
> :http_frt.srvhdr[0008:]: content-type: text/plain
> :http_frt.srvhdr[0008:]: Connection: close
> [ALERT] 277/063058 (61489) : [910446773] process_stream:1655: 
> task=0x80244b9b0 s=0x80247d600, sfl=0x048a, rq=0x80247d610, 
> rp=0x80247d650, exp(r,w)=910456773,0 rqf=00808200 rpf=8004a221 rqh=-103 
> rqt=0 rph=0 rpt=0 cs=7 ss=7, cet=0x0 set=0x0 retr=0

Yep so here rqh is in fact req->buf->i and as you noticed it's been
decremented a second time.

I'm seeing this which I find suspicious in hlua.c :

  5909
  5910  /* skip the requests bytes. */
  5911  bo_skip(si_oc(si), strm->txn->req.eoh + 2);

First I don't understand why "eoh+2", I suspect that it's for the CRLF
in which case it's wrong since it can be a lone LF. Second, I'm not
seeing sov being reset afterwards. Could you please just add this
after this line :

   strm->txn->req.next -= strm->txn->req.sov;
   strm->txn->req.sov = 0;

That's equivalent to what we're doing when dealing with a redirect (http.c:4258
if you're curious) since we also have to "eat" the request. There may be a few
other corner cases, the use-service mechanism is fairly new and puts its feet
in a place where things used to work just because they were trained to... But
it's a terribly powerful thing to have so we must fix it even if it needs a
few -stable cycles.

Thanks!
Willy

Re: core dump, lua service, 1.6-dev6 ss-20150930

[Willy Tarreau]

On Tue, Oct 13, 2015 at 12:32:08AM +0200, PiBa-NL wrote:
> >Yep so here rqh is in fact req->buf->i and as you noticed it's been
> >decremented a second time.
> >
> >I'm seeing this which I find suspicious in hlua.c :
> >
> >   5909
> >   5910  /* skip the requests bytes. */
> >   5911  bo_skip(si_oc(si), strm->txn->req.eoh + 2);
> >
> >First I don't understand why "eoh+2", I suspect that it's for the CRLF
> >in which case it's wrong since it can be a lone LF. Second, I'm not
> >seeing sov being reset afterwards. Could you please just add this
> >after this line :
> >
> >strm->txn->req.next -= strm->txn->req.sov;
> >strm->txn->req.sov = 0;
> This did not seem to resolve the issue.

OK thanks for testing at least!

> If you have any other idea where it might go wrong please let me know :)
> Ill try and dig a little further tomorrow evening.

Unfortunately no I don't have any other idea. At this point I think
I'll have to discuss with Thierry about this. We're in a situation
where I know pretty well how HTTP forwarding works, while he knows
very well how Lua works, but both of us have very unclear idea of
the other one's part, so that doesn't help much :-/

I'm still working on the management doc that I hoped to finish by today
and seems to take longer, so the release might be deferred to tomorrow,
maybe in the mean time we'll have the opportunity to find something odd,
but I prefer not to put my nose there myself or it will further postpone
the doc and the release which is waiting for it.

Thanks for your feedback!

Willy
 

Re: core dump, lua service, 1.6-dev6 ss-20150930

[PiBa-NL]

Hi Willy,

Op 12-10-2015 om 23:06 schreef Willy Tarreau:

Hi Pieter,

On Mon, Oct 12, 2015 at 10:29:05PM +0200, PiBa-NL wrote:

Been running some more tests with the information that req->buf->i
should be >= 0.

What i find is that after 1 request i already see rqh=-103 , it seems
like the initial request size which in this case is also is 103 bytes is
subtracted twice? It does not immediately crash, but if this is already
a sign of 'corruption' then the cause should be a little more easy to
find..

Oh yes definitely, good catch!


@Willy can you confirm this indicates the problem could be in progress
of heading to a crash? Even though in the last line it restores to 0..

Absolutely. Everytime we're trying to track such a painful bug, I end
up looking for initial symptoms, like this one. In general the first
corruption is minor and undetected but once it's seeded its disease,
the problem will definitely occur.


See its full output below replaced the DPRINTF already in code of
stream.c with Alert..

Thanks in advance,
PiBa-NL

root@OPNsense:/usr/ports/net/haproxy-devel # haproxy -f
/var/haproxy.config -d
[ALERT] 277/063055 (61489) : SSLv3 support requested but unavailable.
Available polling systems :
  kqueue : pref=300,  test result OK
poll : pref=200,  test result OK
  select : pref=150,  test result FAILED
Total: 3 (2 usable), will use kqueue.
Using kqueue() as the polling mechanism.
:http_frt.accept(0006)=0008 from [127.0.0.1:62358]
[ALERT] 277/063058 (61489) : [910446771] process_stream:1655:
task=0x80244b9b0 s=0x80247d600, sfl=0x0080, rq=0x80247d610,
rp=0x80247d650, exp(r,w)=0,0 rqf=00d08000 rpf=8000 rqh=0 rqt=0 rph=0
rpt=0 cs=7 ss=0, cet=0x0 set=0x0 retr=0
[ALERT] 277/063058 (61489) : [910446771] http_wait_for_request:
stream=0x80247d600 b=0x80247d610, exp(r,w)=0,0 bf=00d08000 bh=0
analysers=34
[ALERT] 277/063058 (61489) : [910446772] process_stream:1655:
task=0x80244b9b0 s=0x80247d600, sfl=0x0080, rq=0x80247d610,
rp=0x80247d650, exp(r,w)=0,0 rqf=0002 rpf=8000 rqh=103 rqt=0
rph=0 rpt=0 cs=7 ss=0, cet=0x0 set=0x0 retr=0
[ALERT] 277/063058 (61489) : [910446772] http_wait_for_request:
stream=0x80247d600 b=0x80247d610, exp(r,w)=0,0 bf=00808002 bh=103
analysers=34
:http_frt.clireq[0008:]: GET / HTTP/1.1
:http_frt.clihdr[0008:]: Host: 127.0.0.1:801
:http_frt.clihdr[0008:]: Accept: */*
:http_frt.clihdr[0008:]: User-Agent: fetch libfetch/2.0
:http_frt.clihdr[0008:]: Connection: close
[ALERT] 277/063058 (61489) : [910446772] process_switching_rules:
stream=0x80247d600 b=0x80247d610, exp(r,w)=0,0 bf=00808002 bh=103
analysers=00
[ALERT] 277/063058 (61489) : [910446772] sess_prepare_conn_req:
sess=0x80247d600 rq=0x80247d610, rp=0x80247d650, exp(r,w)=0,0
rqf=00808002 rpf=8000 rqh=0 rqt=103 rph=0 rpt=0 cs=7 ss=1
[ALERT] 277/063058 (61489) : [910446772] process_stream:1655:
task=0x80244b9b0 s=0x80247d600, sfl=0x048a, rq=0x80247d610,
rp=0x80247d650, exp(r,w)=910456772,0 rqf=00808200 rpf=8023 rqh=0
rqt=0 rph=97 rpt=0 cs=7 ss=7, cet=0x0 set=0x0 retr=0
:http_frt.srvrep[0008:]: HTTP/1.1 200 OK
:http_frt.srvhdr[0008:]: content-length: 13
:http_frt.srvhdr[0008:]: content-type: text/plain
:http_frt.srvhdr[0008:]: Connection: close
[ALERT] 277/063058 (61489) : [910446773] process_stream:1655:
task=0x80244b9b0 s=0x80247d600, sfl=0x048a, rq=0x80247d610,
rp=0x80247d650, exp(r,w)=910456773,0 rqf=00808200 rpf=8004a221 rqh=-103
rqt=0 rph=0 rpt=0 cs=7 ss=7, cet=0x0 set=0x0 retr=0

Yep so here rqh is in fact req->buf->i and as you noticed it's been
decremented a second time.

I'm seeing this which I find suspicious in hlua.c :

   5909
   5910  /* skip the requests bytes. */
   5911  bo_skip(si_oc(si), strm->txn->req.eoh + 2);

First I don't understand why "eoh+2", I suspect that it's for the CRLF
in which case it's wrong since it can be a lone LF. Second, I'm not
seeing sov being reset afterwards. Could you please just add this
after this line :

strm->txn->req.next -= strm->txn->req.sov;
strm->txn->req.sov = 0;

This did not seem to resolve the issue.


That's equivalent to what we're doing when dealing with a redirect (http.c:4258
if you're curious) since we also have to "eat" the request. There may be a few
other corner cases, the use-service mechanism is fairly new and puts its feet
in a place where things used to work just because they were trained to... But
it's a terribly powerful thing to have so we must fix it even if it needs a
few -stable cycles.

Thanks!
Willy


If you have any other idea where it might go wrong please let me know :)
Ill try and dig a little further tomorrow evening.

Regards,
PiBa-NL