Hi All,

Op 7-10-2015 om 0:31 schreef PiBa-NL:
Hi Thierry,
Op 6-10-2015 om 9:47 schreef Thierry FOURNIER:
On Mon, 5 Oct 2015 21:04:08 +0200
PiBa-NL <piba.nl....@gmail.com> wrote:

Hi Thierry,
Hi Pieter,


With or without "option http-server-close" does not seem to make any
difference.

Sure, it is only an answer to the Cyril keep alive problem. I encounter
again the keepalive problem :(

The HAProxy applet (services) can't directly uses the keepalive. The
service send its response with an "internal" connection: close. If you
activate the debug, you will see the header "connection: close".

You must configure HAProxy to use keepalive between the frontend and
the client.
Ok. whell without further specific configuration it is keeping connections alive, but as that is the default thats ok.

Adding a empty backend does seem to resolve the problem, stats also show
the backend handling connections and tracking its 2xx http result
session totals when configured like this.:

frontend http_frt
    mode http
    bind :801
    http-request use-service lua.hello-world
    default_backend http-lua-service
backend http-lua-service
    mode http

I can't reproduce the problem with the last dev version. But, I
regognize the backtrace, I already encounter the same. I'm believe that
is fixed in the dev6 :(
Using dev7 i can still reproduce it..
I try to bench with my http injector, and I try with ab with and
without keep alive. I try also to stress the admin page, and I can't
reproduce pthe problem.

Argg, I see a major difference: to use freebsd. I don't have the
environment for testing it. I must install a VM.


Op 5-10-2015 om 16:06 schreef Thierry FOURNIER:
Hi,

I process this email later. For waiting, I propose to you to set the
"option http-server-close". Actually, the "services" doesn't support
itself the keepalive, but HAProxy does this job.

The "option http-server-close" expectes a server-close from the service
stream. The front of HAProxy maintains the keep-alive between the
client and the haproxy.

This method embbed a limitation: if some servers are declared in the
backend, the "option http-server-close" forbid the keepalive between
haproxy and the serveur.

Can you test with this option ?

Thierry



On Thu, 1 Oct 2015 23:00:45 +0200
Cyril Bonté <cyril.bo...@free.fr> wrote:

Hi,

Le 01/10/2015 20:52, PiBa-NL a écrit :
Hi List,

With the config below while running 'siege' i get a core dump within a few hundreds of requests.. Viewing the stats page from a chrome browser
while siege is running seems to crash it sooner..

Is below enough to find the cause? Anything else i should try?
This is embarrassing because with your configuration, I currently can't
reproduce a segfault but I can reproduce another issue with HTTP
keep-alive requests !

(details below)

Using the haproxy snapshot from: 1.6-dev6 ss-20150930
Or perhaps i just did compile it wrong?..
make NO_CHECKSUM=yes clean debug=1 reinstall WITH_DEBUG=yes

global
       stats socket /tmp/hap.socket level admin
       maxconn 60000
       lua-load /haproxy/brute/hello.lua

defaults
       timeout client 10000
       timeout connect 10000
       timeout server 10000

frontend HAProxyLocalStats
       bind :2300 name localstats
       mode http
       stats enable
       stats refresh 1000
       stats admin if TRUE
       stats uri /
frontend http_frt
     bind :801
     mode http
     http-request use-service lua.hello-world
Here, if I use :
$ ab -n100 -c1 -k http://127.0.0.1:801/
There will be a 10000ms delay after the first request.

Or with another test case :
echo -ne "GET / HTTP/1.1\r\nHost: localhost\r\n\r\nGET /
HTTP/1.1\r\nHost: localhost\r\n\r\n"| nc localhost 801
HTTP/1.1 200 OK
content-type: text/plain
Transfer-encoding: chunked

d
Hello World !
0

=> only 1 response

Now, if I change "frontend http_frt" to "listen http_frt", I get the
awaited behaviour.

The second test case with "listen" :
echo -ne "GET / HTTP/1.1\r\nHost: localhost\r\n\r\nGET /
HTTP/1.1\r\nHost: localhost\r\n\r\n"| nc localhost 801
HTTP/1.1 200 OK
content-type: text/plain
Transfer-encoding: chunked

d
Hello World !
0

HTTP/1.1 200 OK
content-type: text/plain
Transfer-encoding: chunked

d
Hello World !
0

=> the 2 responses are returned

core.register_service("hello-world", "http", function(applet)
      local response = "Hello World !"
      applet:set_status(200)
      applet:add_header("content-type", "text/plain")
      applet:start_response()
      applet:send(response)
end )

(gdb) bt full
#0  0x0000000801a2da75 in memcpy () from /lib/libc.so.7
No symbol table info available.
#1  0x0000000000417388 in buffer_slow_realign (buf=0x7d3c90) at
src/buffer.c:166
           block1 = -3306
           block2 = 0
#2  0x0000000000480c42 in http_wait_for_request (s=0x80247d600,
req=0x80247d610, an_bit=4)
       at src/proto_http.c:2686
           cur_idx = -6336
           sess = (struct session *) 0x80241e400
           txn = (struct http_txn *) 0x802bb2140
           msg = (struct http_msg *) 0x802bb21a0
ctx = {line = 0x2711079 <Address 0x2711079 out of bounds>, idx
= 3, val = 0, vlen = 7, tws = 0,
     del = 33, prev = 0}
#3  0x00000000004d55b1 in process_stream (t=0x80244b390) at
src/stream.c:1759
           max_loops = 199
           ana_list = 52
           ana_back = 52
           flags = 4227584
           srv = (struct server *) 0x0
           s = (struct stream *) 0x80247d600
           sess = (struct session *) 0x80241e400
           rqf_last = 8397312
           rpf_last = 2248179715
           rq_prod_last = 7
           rq_cons_last = 9
           rp_cons_last = 7
           rp_prod_last = 0
           req_ana_back = 8192
           req = (struct channel *) 0x80247d610
           res = (struct channel *) 0x80247d650
           si_f = (struct stream_interface *) 0x80247d7f8
           si_b = (struct stream_interface *) 0x80247d818
#4 0x000000000041fe78 in process_runnable_tasks () at src/task.c:238
           t = (struct task *) 0x80244b390
           max_processed = 0
#5  0x000000000040cc4e in run_poll_loop () at src/haproxy.c:1539
           next = 549107027
#6  0x000000000040daee in main (argc=4, argv=0x7fffffffeaf0) at
src/haproxy.c:1892
           err = 0
           retry = 200
           limit = {rlim_cur = 120032, rlim_max = 120032}
           errmsg =
"\000êÿÿÿ\177\000\000\030ëÿÿÿ\177\000\000ðêÿÿÿ\177\000\000\004\000\000\000\000\000\000\000Ðêÿÿÿ\177\000\000]A}\000\b\000\000\000pêÿÿÿ\177\000\000\000\000\000\000\000\000\000\000èêÿÿÿ\177\000\000\030ëÿÿÿ\177\000\000ðêÿÿÿ\177\000\000\004\000\000\000\000\000\000\000\220êÿÿ"

           pidfd = -1

# haproxy -vv
[ALERT] 273/021153 (10691) : SSLv3 support requested but unavailable.
HA-Proxy version 1.6-dev6-10770fa 2015/09/29
Copyright 2000-2015 Willy Tarreau <wi...@haproxy.org>

Build options :
     TARGET  = freebsd
     CPU     = generic
     CC      = cc
     CFLAGS  = -pipe -g -fstack-protector -fno-strict-aliasing
-DFREEBSD_PORTS
     OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1
USE_STATIC_PCRE=1 USE_PCRE_JIT=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2d 9 Jul 2015
Running on OpenSSL version : OpenSSL 1.0.2d 9 Jul 2015
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.37 2015-04-28
PCRE library supports JIT : yes
Built with Lua version : Lua 5.3.0
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY

Available polling systems :
        kqueue : pref=300,  test result OK
          poll : pref=200,  test result OK
        select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.




--
Cyril Bonté


Seems that buffer_slow_realign() isn't used regularly during normal haproxy operation, and it crashes first time that specific function gets called. Reproduction is pretty consistent with chrome browser refreshing stats every second.
Then starting: wrk -c 200 -t 2 -d 10 http://127.0.0.1:801/
I tried adding some Alert(); items in the code to see what parameters are set at what step, but am not understanding the exact internals of that code..

This negative bh=-7800 is not supposed to be there i think? It is from one of the dprintf statements, how are those supposed generate output?.. [891069718] http_wait_for_request: stream=0x80247d600 b=0x80247d610, exp(r,w)=0,0 bf=00c08200 bh=-7800 analysers=34

Anything else i can check or provide to help get this fixed?

Best regards,
PiBa-NL
Just a little 'bump' to this issue..

Anyone know when/how this buffer_slow_realign() is suppose to work?
I suspect it either contains a bug, or is called with bogus parameters..

How can we/i determine which it is?

Even though with a small change in the config (adding a backend) i cant reproduce it, that doesnt mean there isn't a problem with the fuction.. As the whole function doesn't seem to get called in that circumstance..

@Willy, hoping you perhaps have a good idea how to pursue this.?

Best regards,
PiBa-NL

Reply via email to