Re: Try request again if response body is empty?

[Baptiste]

On Sun, Oct 11, 2015 at 5:29 AM, Shawn Heisey  wrote:
> On 10/10/2015 12:31 AM, Willy Tarreau wrote:
>> Is the response closed when this happens (eg: server crash) ? If so,
>> we could add some sample fetches to detect that the request or response
>> channels are closed in case that could help. This is trivial to do, but
>> it will only be reliable if the close is immediately encountered, so it
>> still depends on the timing.
>
> We don't really understand why it happens, though we have been able to
> track down an exception that we *think* is related.  It's a common
> problem seen with servlet containers: "java.lang.IllegalStateException:
> Cannot forward after response has been committed" and
> "java.lang.IllegalStateException: Cannot call sendError() after the
> response has been committed".  The underlying cause virtually every time
> these exceptions occur is programmer error.  The difficult part is
> tracking it down and getting a fix deployed.


Usually, all the app servers serves the same code, so if app on
serverA is buggy, then app on serverB must end up with the same
result.
This type of replay may make sense in a A/B testing mode, but slightly
revisited.
IE use farm A until one fail occurs, then replay to farm B where we
have a more verbose (hence much slower) version of the app or a
"fixed" version under testing, etc...

In such case, I would agree it may make sense.

Baptiste

Re: Interactive stats socket broken on master

[Willy Tarreau]

On Sat, Oct 10, 2015 at 08:55:44PM -0500, Andrew Hayworth wrote:
> Bump -
> 
> I don't mind maintaining my own HAProxy package, but it seems bad to
> release a major version with the interactive stats socket broken. Any
> thoughts on the patch?

Has anyone else tested it ? Since the beginning of the thread I must
confess it's unclear to me as Jesse reported the issue, you said that
your patch works for you then Jesse asks whether we should merge it.
Jesse, have you tested it as well, so that we ensure you're facing
the same issue ?

Andrew BTW, your patch looks good and seems to do what you described
in the message, I'm just asking to be sure that it addresses Jesse's
bug as well.

Last point guys, please keep in mind that not everybody reads all
e-mails, so when you want to have a patch integrated, clearly mark
it in the subject and don't leave it pending at the trail of a
thread like this.

Thanks,
Willy

Re: core dump, lua service, 1.6-dev6 ss-20150930

[PiBa-NL]

Hi All,

Op 7-10-2015 om 0:31 schreef PiBa-NL:

Hi Thierry,
Op 6-10-2015 om 9:47 schreef Thierry FOURNIER:

On Mon, 5 Oct 2015 21:04:08 +0200
PiBa-NL  wrote:


Hi Thierry,

Hi Pieter,



With or without "option http-server-close" does not seem to make any
difference.


Sure, it is only an answer to the Cyril keep alive problem. I encounter
again the keepalive problem :(

The HAProxy applet (services) can't directly uses the keepalive. The
service send its response with an "internal" connection: close. If you
activate the debug, you will see the header "connection: close".

You must configure HAProxy to use keepalive between the frontend and
the client.
Ok. whell without further specific configuration it is keeping 
connections alive, but as that is the default thats ok.


Adding a empty backend does seem to resolve the problem, stats also 
show

the backend handling connections and tracking its 2xx http result
session totals when configured like this.:

frontend http_frt
mode http
bind :801
http-request use-service lua.hello-world
default_backend http-lua-service
backend http-lua-service
mode http


I can't reproduce the problem with the last dev version. But, I
regognize the backtrace, I already encounter the same. I'm believe that
is fixed in the dev6 :(

Using dev7 i can still reproduce it..

I try to bench with my http injector, and I try with ab with and
without keep alive. I try also to stress the admin page, and I can't
reproduce pthe problem.

Argg, I see a major difference: to use freebsd. I don't have the
environment for testing it. I must install a VM.



Op 5-10-2015 om 16:06 schreef Thierry FOURNIER:

Hi,

I process this email later. For waiting, I propose to you to set the
"option http-server-close". Actually, the "services" doesn't support
itself the keepalive, but HAProxy does this job.

The "option http-server-close" expectes a server-close from the 
service

stream. The front of HAProxy maintains the keep-alive between the
client and the haproxy.

This method embbed a limitation: if some servers are declared in the
backend, the "option http-server-close" forbid the keepalive between
haproxy and the serveur.

Can you test with this option ?

Thierry



On Thu, 1 Oct 2015 23:00:45 +0200
Cyril Bonté  wrote:


Hi,

Le 01/10/2015 20:52, PiBa-NL a écrit :

Hi List,

With the config below while running 'siege' i get a core dump 
within a
few hundreds of requests.. Viewing the stats page from a chrome 
browser

while siege is running seems to crash it sooner..

Is below enough to find the cause? Anything else i should try?
This is embarrassing because with your configuration, I currently 
can't

reproduce a segfault but I can reproduce another issue with HTTP
keep-alive requests !

(details below)


Using the haproxy snapshot from: 1.6-dev6 ss-20150930
Or perhaps i just did compile it wrong?..
make NO_CHECKSUM=yes clean debug=1 reinstall WITH_DEBUG=yes

global
   stats socket /tmp/hap.socket level admin
   maxconn 6
   lua-load /haproxy/brute/hello.lua

defaults
   timeout client 1
   timeout connect 1
   timeout server 1

frontend HAProxyLocalStats
   bind :2300 name localstats
   mode http
   stats enable
   stats refresh 1000
   stats admin if TRUE
   stats uri /
frontend http_frt
 bind :801
 mode http
 http-request use-service lua.hello-world

Here, if I use :
$ ab -n100 -c1 -k http://127.0.0.1:801/
There will be a 1ms delay after the first request.

Or with another test case :
echo -ne "GET / HTTP/1.1\r\nHost: localhost\r\n\r\nGET /
HTTP/1.1\r\nHost: localhost\r\n\r\n"| nc localhost 801
HTTP/1.1 200 OK
content-type: text/plain
Transfer-encoding: chunked

d
Hello World !
0

=> only 1 response

Now, if I change "frontend http_frt" to "listen http_frt", I get the
awaited behaviour.

The second test case with "listen" :
echo -ne "GET / HTTP/1.1\r\nHost: localhost\r\n\r\nGET /
HTTP/1.1\r\nHost: localhost\r\n\r\n"| nc localhost 801
HTTP/1.1 200 OK
content-type: text/plain
Transfer-encoding: chunked

d
Hello World !
0

HTTP/1.1 200 OK
content-type: text/plain
Transfer-encoding: chunked

d
Hello World !
0

=> the 2 responses are returned


core.register_service("hello-world", "http", function(applet)
  local response = "Hello World !"
  applet:set_status(200)
  applet:add_header("content-type", "text/plain")
  applet:start_response()
  applet:send(response)
end )

(gdb) bt full
#0  0x000801a2da75 in memcpy () from /lib/libc.so.7
No symbol table info available.
#1  0x00417388 in buffer_slow_realign (buf=0x7d3c90) at
src/buffer.c:166
   block1 = -3306
   block2 = 0
#2  0x00480c42 in http_wait_for_request (s=0x80247d600,
req=0x80247d610, an_bit=4)
   at src/proto_http.c:2686
   cur_idx = -6336
   sess = (struct session *) 0x80241e400
   txn = (struct http_txn *) 0x802bb2140
   

Re: RE : FW: HAProxy

[Willy Tarreau]

On Mon, Oct 12, 2015 at 04:48:41AM +, Cédric Petter wrote:
> Thanks Willy and thanks Thierry Fournier too (He answers some days before and
> didn't get the time to test before)
> 
> It work like a charm now :-)

Great!

> It's weird but no blogs speaks about this. All blogs I found do redirect to
> 80 on backend. Or they use 443 but there is nothing in the config explained
> :-(

Well maybe that leaves an opportunity for you to post a blog article
somewhere. Also, please keep in mind that the documentation is supposed
to be used before blogs, although I admit it's a bit large now and I
understand why some people prefer too look for a blog post before reading
all the doc!

Regards,
Willy

[SPAM] Novidade - Curso Scan N Cut Brother e mais novidades!

[Loja Arte em DVD]

Title: Vlady Ensina - Escultura em Papel Princesas - Curso em DVD



Caso não esteja conseguindo visualizar nossa 
  mensagem, clique aqui 
  ou acesse - www.arteemdvd.com.br

  

  


  Chegou 
  a hora de tirar o máximo proveito de sua máquina de corte 
  ScanNCut. Descubra como utilizar as lâminas de 
  forma correta, como trabalhar com as esteiras, tire suas 
  dúvidas sobre as configurações de corte, aproveite 
  ao máximo o recurso do scanner e vá além com o 
  software gratuito e online ScanNCut Canvas. 
Neste curso, com aproximadamente 2 horas de 
  duração, você terá acesso a todos 
  os segredos desta brilhante máquina da Brother que permite trabalhar 
  com tecido, EVA, papel e muito mais.. 
Primeiro curso em DVD da máquina lançado 
  no Brasil. Chega de perder tempo e dinheiro com cursos não oficiais 
  ou com informações desencontradas! 

   
Através do famoso sistema multicameras da Artesanato 
  Digital, você terá acesso aos 
  principais detalhes como se estivesse tendo uma aula presencial com 
  a professora em seu ateliê ou no conforto de sua casa, com a 
  vantagem de poder assistir tudo a hora que quiser e quantas vezes desejar. 

Este curso em DVD foi gravado em alta definição 
  e conta com som digital para que você tenha a certeza de aprender 
  a trabalhar com este equipamento de verdade e de forma correta. 
Nunca mais perca seus tapetes de corte, suas agulhas 
  ou até mesmo seu material! Aprenda como fazer a máquina 
  trabalhar para você!




  
	   
  


  
  Para a maior comodidade de nossos clientes trabalhamos em parceira 
com o PAGSEGURO do portal UOL, e PAYPAL BRASIL. Através destes serviços 
você pode realizar o pagamento de suas compras com total segurança 
e com um detalhe importante: enquanto os produtos não são de fato 
entregues aos clientes, estas empresas não fazem o repasse do valor 
da compra para nossa loja. Desta forma, você conta com o suporte adicional 
para ter absoluta certeza que receberá os seus produtos! Em caso de 
dúvidas, nossa central de atendimento está a sua inteira disposição 
pelo e-mail: ven...@arteemdvd.com.br - Todas as questões 
são respondidas em até 24 horas, incluindo sábados, domingos e feriados. 
  
  

  

Para 
  não receber mais nossos informativos simplesmente responda está mensagem trocando 
  o ASSUNTO por DESCADASTRO ou, se preferir, clique 
  aqui para solicitar o imediato descadastramento de nosso sistema de newsletter. 
  Ainda, por medida de segurança você também pode criar um regra em cliente de 
  e-mails, eliminando automaticamente qualquer mensagem que seja disparada por 
  ven...@arteemdvd.com.br

RE : FW: HAProxy

[Cédric Petter]

Thanks Willy and thanks Thierry Fournier too (He answers some days before and 
didn't get the time to test before)

It work like a charm now :-)
It's weird but no blogs speaks about this. All blogs I found do redirect to 80 
on backend. Or they use 443 but there is nothing in the config explained :-(

So I really appreciate your help. 
It saves me some sleep hours :-)

Kind Regards

Cédric Petter
VP of Support & IT

BPA Solutions
Headquarters – Switzerland

Build Closer Relationships with SharePoint

p. +41 24 524 25 50
e. cedric.pet...@bpa-solutions.net


De : Willy Tarreau [w...@1wt.eu]
Date d'envoi : samedi 10 octobre 2015 08:12
À : Cédric Petter
Cc: haproxy@formilux.org
Objet : Re: FW: HAProxy

Hello Cédric,

On Tue, Oct 06, 2015 at 01:56:41PM +, Cédric Petter wrote:
> Bonjour
>
> First of all, if I need to explain in English, please tell me.

Yes the list is in english, but I understood your problem so I'll
put out a quick summary and will respond :-)

> Je suis bloqué avec HAProxy sur une VM.
> J'ai un serveur Debian 8.2 avec HAProxy 1.5.14.
> Et "derrière", j'ai 2 serveurs Windows avec IIS 8.5 & SharePoint 2013.
>
> Cela fonctionne bien en HTTP, mais pas en HTTPS.
> avec HTTPS j'ai soit des erreurs 503 & 504 en altérnance.
> Si quelqu'un à une idée ca serait cool.

In short Cedric faces an issue where he gets errors 503/504 on haproxy
when passing HTTPS requests to IIS but that's OK with HTTP.

Cedric, the problem is that you are connecting to port 443 in clear
because you didnt specify "ssl" on the server lines :

   backend www-backend-https
 server www-1 192.168.1.2:443 check
 server www-2 192.168.1.3:443 check

Just add "ssl" at the end of the line and it will work better. You'll
get a warning upon startup that you need to add "ssl-verify-none" or
to put a CA file. If haproxy and the servers are on the same local
network and you consider this network to be safe, you can easily add
that option.

Additionnally, maybe you don't even need to pass again via port 443
and you can pass everything to port 80 ? That can make a simpler
config and avoid to re-encrypt+decrypt.

Last, since you're on haproxy 1.5, if you're observing important
CPU usage when using SSL, you can enable HTTP keep-alive to the
servers by removing this line :

   option http-server-close

It will use more memory by maintaining more connections though.

Regards,
Willy

[SPAM] e-banking alert

[Westpac Bank]

It’s important you protect your Westpac. Please update your westpac account now: 
Login to view new updates.

Re: core dump, lua service, 1.6-dev6 ss-20150930

[Willy Tarreau]

Hi Pieter,

On Mon, Oct 12, 2015 at 01:22:48AM +0200, PiBa-NL wrote:
> >>#1  0x00417388 in buffer_slow_realign (buf=0x7d3c90) at
> >>src/buffer.c:166
> >>   block1 = -3306
> >>   block2 = 0

I'm puzzled by this above, no block should have a negative size.

> >>#2  0x00480c42 in http_wait_for_request (s=0x80247d600,
> >>req=0x80247d610, an_bit=4)
> >>   at src/proto_http.c:2686
> >>   cur_idx = -6336
> >>   sess = (struct session *) 0x80241e400
> >>   txn = (struct http_txn *) 0x802bb2140
> >>   msg = (struct http_msg *) 0x802bb21a0
> >>   ctx = {line = 0x2711079  >>bounds>, idx = 3, val = 0, vlen = 7, tws = 0, del = 33, prev = 0}

And this above, similarly cur_idx shouldn't be negative.

> >Seems that buffer_slow_realign() isn't used regularly during normal 
> >haproxy operation, and it crashes first time that specific function 
> >gets called.
> >Reproduction is pretty consistent with chrome browser refreshing stats 
> >every second.
> >Then starting: wrk -c 200 -t 2 -d 10 http://127.0.0.1:801/
> >I tried adding some Alert(); items in the code to see what parameters 
> >are set at what step, but am not understanding the exact internals of 
> >that code..
> >
> >This negative bh=-7800 is not supposed to be there i think? It is from 
> >one of the dprintf statements, how are those supposed generate output?..
> >[891069718] http_wait_for_request: stream=0x80247d600 b=0x80247d610, 
> >exp(r,w)=0,0 bf=00c08200 bh=-7800 analysers=34
> >
> >Anything else i can check or provide to help get this fixed?
> >
> >Best regards,
> >PiBa-NL
> Just a little 'bump' to this issue..
> 
> Anyone know when/how this buffer_slow_realign() is suppose to work?

Yes, it's supposed to be used only when a request or response is wrapped
in the request or response buffer. It uses memcpy(), hence the "slow"
aspect of the realign.

> I suspect it either contains a bug, or is called with bogus parameters..

It's very sensitive to the consistency of the buffer being realigned. So
errors such as buf->i + buf->o > buf->size, or buf->p > buf->data + buf->size,
or buf->p < buf->data etc... can lead to crashes. But these must never happen
at all otherwise it proves that there's a bug somewhere else.

Here since block1 is -3306 and block2 = 0, I suspect that they were assigned
at line 159 from buf->i, which definitely means that the buffer was already
corrupted.

> How can we/i determine which it is?

The difficulty consists in finding what can lead to a corrupted buffer :-/
In the past we had such issues when trying to forward more data than was
available in the buffer, due to option send-name-header. I wouldn't be
surprized that it can happen here on corner cases when building a message
from Lua if the various message pointers are not all perfectly correct.

> Even though with a small change in the config (adding a backend) i cant 
> reproduce it, that doesnt mean there isn't a problem with the fuction.. 
> As the whole function doesn't seem to get called in that circumstance..

It could be related to an uninitialized variable somewhere as well. You
can try to start haproxy with "-dM" to see if it makes the issues 100%
reproducible or not. This poisons all buffers (fills them with a constant
byte 0x50 after malloc) so that we don't rely on an uninitialized zero byte
somewhere.

Regards,
Willy