Re: HAProxy 1.7.9 FreeBSD 100% CPU usage

[Willy Tarreau]

Hi Pieter,

On Thu, Nov 09, 2017 at 02:28:46AM +0100, PiBa-NL wrote:
> Actually haproxy has been running for a few weeks with 100% and i didnt
> notice.. it does keep working it seems..
> 
> Anyhow thought i would try and capture the next event if it would happen
> again. It did after a few hours..
> 
> After the truss output below the last line keeps repeating fast lots and
> lots of times.
> 
> kevent(0,0x0,0,{ },7,{ 1.0 })         = 0 (0x0)
> kevent(0,0x0,0,{ },7,{ 1.0 })         = 0 (0x0)
> kevent(0,0x0,0,{ },7,{ 1.0 })         = 0 (0x0)
> kevent(0,0x0,0,{ 1,EVFILT_READ,EV_EOF,0x0,0x0,0x0 },7,{ 0.99400 }) = 1
> (0x1)
> recvfrom(1,0x8024ed972,16290,0,NULL,0x0)     = 0 (0x0)
> kevent(0,{ 1,EVFILT_READ,EV_DELETE,0x0,0x0,0x0 },1,0x0,0,0x0) = 0 (0x0)
> kevent(0,0x0,0,{ },7,{ 0.0 })         = 0 (0x0)
> kevent(0,0x0,0,{ },7,{ 0.0 })         = 0 (0x0)
> kevent(0,0x0,0,{ },7,{ 0.0 })         = 0 (0x0)
> kevent(0,0x0,0,{ },7,{ 0.0 })         = 0 (0x0)
> kevent(0,0x0,0,{ },7,{ 0.0 })         = 0 (0x0)

We had something similar on Linux in relation with TCP splicing and the
fd cache, for which a fix was emitted. But yesterday Christopher explained
me that the fix has an impact on the way applets are scheduled in 1.8, so
actually it could mean that the initial bug might possibly cover a larger
scope than splicing only, and that recv+send might also be affected. If
you're interested in testing, the commit in 1.7 is
c040c1f ("BUG/MAJOR: stream-int: don't re-arm recv if send fails") and
is present in the latest snapshot (we really need to emit 1.7.10 BTW).

I'd be curious to know if it fixes it or not. At least it will tell us
if that's related to this fd cache thing or to something completely
different such as Lua.

I also need to check with Thierry if we could find a way to add some
stats about the time spent in Lua to "show info" to help debugging such
cases where Lua is involved.

By the way, thanks for your dump, we'll check the sessions' statuses.
There are not that many, and maybe it will give us a useful indication!

Cheers,
Willy

Re: Error in `haproxy': munmap_chunk(): invalid pointer:

[Willy Tarreau]

Hi Tim,

On Thu, Nov 09, 2017 at 01:00:22AM +0100, Tim Düsterhus wrote:
> Hi
> 
> I get the following crash when running:
> 
> [timwolla@/t/h/haproxy-1.8-rc2]./haproxy -V
> HA-Proxy version 1.8-rc2-a8d8d6e 2017/11/03
> Copyright 2000-2017 Willy Tarreau 
> 
> with the configuration at the bottom of this email as follows:
> 
> root@node42:/tmp/haproxy/haproxy-1.8-rc2# ./haproxy -W -f
> /tmp/haproxy/haproxy-1.8-rc2/haproxy.cfg
> [WARNING] 312/004845 (31835) : Can't open server state file
> '/etc/haproxy/state/global': No such file or directory
> [WARNING] 312/004845 (31835) : Can't open server state file
> '/etc/haproxy/state/global': No such file or directory
> 
> and then killing the master process using a SIGHUP:
> 
> > *** Error in `./haproxy': munmap_chunk(): invalid pointer: 
> > 0x00515028 ***
> > === Backtrace: =
> > /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fda72bbc7e5]
> > /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1a8)[0x7fda72bc9698]
> > ./haproxy[0x4a28d0]
> > ./haproxy[0x4a2c4e]
> > ./haproxy[0x4a2f57]
> > ./haproxy[0x40b16f]
> > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fda72b65830]
> > ./haproxy[0x40c149]
> > === Memory map: 
> > 0040-00532000 r-xp  08:03 2765518
> > /tmp/haproxy/haproxy-1.8-rc2/haproxy
> > 00731000-00732000 r--p 00131000 08:03 2765518
> > /tmp/haproxy/haproxy-1.8-rc2/haproxy
> > 00732000-00749000 rw-p 00132000 08:03 2765518
> > /tmp/haproxy/haproxy-1.8-rc2/haproxy
> > 00749000-0074e000 rw-p  00:00 0 
> > 01d7f000-01de3000 rw-p  00:00 0  
> > [heap]
> > 7fda7272b000-7fda72741000 r-xp  08:03 528903 
> > /lib/x86_64-linux-gnu/libgcc_s.so.1
> > 7fda72741000-7fda7294 ---p 00016000 08:03 528903 
> > /lib/x86_64-linux-gnu/libgcc_s.so.1
> > 7fda7294-7fda72941000 rw-p 00015000 08:03 528903 
> > /lib/x86_64-linux-gnu/libgcc_s.so.1
> > 7fda72941000-7fda72944000 r-xp  08:03 533578 
> > /lib/x86_64-linux-gnu/libdl-2.23.so
> > 7fda72944000-7fda72b43000 ---p 3000 08:03 533578 
> > /lib/x86_64-linux-gnu/libdl-2.23.so
> > 7fda72b43000-7fda72b44000 r--p 2000 08:03 533578 
> > /lib/x86_64-linux-gnu/libdl-2.23.so
> > 7fda72b44000-7fda72b45000 rw-p 3000 08:03 533578 
> > /lib/x86_64-linux-gnu/libdl-2.23.so
> > 7fda72b45000-7fda72d05000 r-xp  08:03 533590 
> > /lib/x86_64-linux-gnu/libc-2.23.so
> > 7fda72d05000-7fda72f05000 ---p 001c 08:03 533590 
> > /lib/x86_64-linux-gnu/libc-2.23.so
> > 7fda72f05000-7fda72f09000 r--p 001c 08:03 533590 
> > /lib/x86_64-linux-gnu/libc-2.23.so
> > 7fda72f09000-7fda72f0b000 rw-p 001c4000 08:03 533590 
> > /lib/x86_64-linux-gnu/libc-2.23.so
> > 7fda72f0b000-7fda72f0f000 rw-p  00:00 0 
> > 7fda72f0f000-7fda72f7d000 r-xp  08:03 541796 
> > /lib/x86_64-linux-gnu/libpcre.so.3.13.2
> > 7fda72f7d000-7fda7317d000 ---p 0006e000 08:03 541796 
> > /lib/x86_64-linux-gnu/libpcre.so.3.13.2
> > 7fda7317d000-7fda7317e000 r--p 0006e000 08:03 541796 
> > /lib/x86_64-linux-gnu/libpcre.so.3.13.2
> > 7fda7317e000-7fda7317f000 rw-p 0006f000 08:03 541796 
> > /lib/x86_64-linux-gnu/libpcre.so.3.13.2
> > 7fda7317f000-7fda73399000 r-xp  08:03 529753 
> > /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
> > 7fda73399000-7fda73598000 ---p 0021a000 08:03 529753 
> > /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
> > 7fda73598000-7fda735b4000 r--p 00219000 08:03 529753 
> > /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
> > 7fda735b4000-7fda735c rw-p 00235000 08:03 529753 
> > /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
> > 7fda735c-7fda735c3000 rw-p  00:00 0 
> > 7fda735c3000-7fda73621000 r-xp  08:03 524697 
> > /lib/x86_64-linux-gnu/libssl.so.1.0.0
> > 7fda73621000-7fda73821000 ---p 0005e000 08:03 524697 
> > /lib/x86_64-linux-gnu/libssl.so.1.0.0
> > 7fda73821000-7fda73825000 r--p 0005e000 08:03 524697 
> > /lib/x86_64-linux-gnu/libssl.so.1.0.0
> > 7fda73825000-7fda7382c000 rw-p 00062000 08:03 524697 
> > /lib/x86_64-linux-gnu/libssl.so.1.0.0
> > 7fda7382c000-7fda73844000 r-xp  08:03 530442 
> > /lib/x86_64-linux-gnu/libpthread-2.23.so
> > 7fda73844000-7fda73a43000 ---p 00018000 08:03 530442 
> > /lib/x86_64-linux-gnu/libpthread-2.23.so
> > 7fda73a43000-7fda73a44000 r--p 00017000 08:03 530442 
> > /lib/x86_64-linux-gnu/libpthread-2.23.so
> > 7fda73a44000-7fda73a45000 rw-p 00018000 08:03 

HAProxy 1.7.9 FreeBSD 100% CPU usage

[PiBa-NL]

Hi List,

I've experienced a issue where its using 100% cpu usage with haproxy 
1.7.9 on FreeBSD 11.1p3 / pfSense 2.4.2dev.


There is very little traffic actually hitting this haproxy instance. But 
it happened for the second time in a few days now.
Actually haproxy has been running for a few weeks with 100% and i didnt 
notice.. it does keep working it seems..


Anyhow thought i would try and capture the next event if it would happen 
again. It did after a few hours..


After the truss output below the last line keeps repeating fast lots and 
lots of times.


kevent(0,0x0,0,{ },7,{ 1.0 })         = 0 (0x0)
kevent(0,0x0,0,{ },7,{ 1.0 })         = 0 (0x0)
kevent(0,0x0,0,{ },7,{ 1.0 })         = 0 (0x0)
kevent(0,0x0,0,{ 1,EVFILT_READ,EV_EOF,0x0,0x0,0x0 },7,{ 0.99400 }) = 
1 (0x1)

recvfrom(1,0x8024ed972,16290,0,NULL,0x0)     = 0 (0x0)
kevent(0,{ 1,EVFILT_READ,EV_DELETE,0x0,0x0,0x0 },1,0x0,0,0x0) = 0 (0x0)
kevent(0,0x0,0,{ },7,{ 0.0 })         = 0 (0x0)
kevent(0,0x0,0,{ },7,{ 0.0 })         = 0 (0x0)
kevent(0,0x0,0,{ },7,{ 0.0 })         = 0 (0x0)
kevent(0,0x0,0,{ },7,{ 0.0 })         = 0 (0x0)
kevent(0,0x0,0,{ },7,{ 0.0 })         = 0 (0x0)

I tried to gather all possible relevant info in attached file. Not using 
much special configuration options.. but i am using lua to server a 
small simple static response.. I'm not sure if its a problem that might 
be related to LUA, or perhaps there is some other issue.?.
I've got tcpdump and complete truss output from before and while it 
happened after a few hours, but actually just a few request (+- 29).. 
But i would prefer to send these off list though, Willy if you desire i 
send em to your mail address? But maybe i have overlooked it on the 
mailinglist and its a known issue already..? Last connection which i 
think caused/triggered the issue is in screenshot(if it attaches right 
to the mail..)basically just a GET request which gets a ack, followed by 
a FYN,ACK packet from the client 30 seconds later again followed by a ack..


The LetsEncrypt backend that is part of the configuration never got a 
single request according to stats..


Is it a known issue?
Are tcpdump/truss output desired ..? (where should i send em?)
Is there any other output that can try to gather next time?

Regards,
PiBa-NL

HA-Proxy version 1.7.9 2017/08/18
  TARGET  = freebsd

[2.4.2-DEVELOPMENT][admin@pfsense.local]/root: 
/usr/local/pkg/haproxy/haproxy_socket.sh show sess all
show sess all 0x80242b800: [08/Nov/2017:19:40:18.868158] id=15 
proto=tcpv4 source=45.76.a.b:53752

  flags=0x48a, conn_retries=0, srv_conn=0x0, pend_pos=0x0
  frontend=www (id=3 mode=http), listener=37.97.x.y:80 (id=1) 
addr=37.97.x.y:80

  backend= (id=-1 mode=-)
  server= (id=-1)
  task=0x80248f380 (state=0x04 nice=0 calls=4 exp= age=4h23m)
  txn=0x802421800 flags=0x820 meth=1 status=-1 req.st=MSG_BODY 
rsp.st=MSG_RPBEFORE waiting=0
  si[0]=0x80242ba38 (state=EST flags=0x08 endp0=CONN:0x8024ca480 
exp=, et=0x000)
  si[1]=0x80242ba60 (state=EST flags=0x4010 endp1=APPCTX:0x8024ca600 
exp=, et=0x000)

  co0=0x8024ca480 ctrl=tcpv4 xprt=RAW data=STRM target=LISTENER:0x8024ca300
  flags=0x0025b300 fd=1 fd.state=22 fd.cache=0 updt=0
  app1=0x8024ca600 st0=0 st1=0 st2=0 applet=
  req=0x80242b810 (f=0x80c020 an=0x0 pipe=0 tofwd=-1 total=94)
  an_exp= rex= wex=
  buf=0x8024ed900 data=0x8024ed914 o=94 p=94 req.next=94 i=0 size=16384
  res=0x80242b850 (f=0x8040 an=0xa0 pipe=0 tofwd=0 total=0)
  an_exp= rex= wex=
  buf=0x783160 data=0x783174 o=0 p=0 rsp.next=0 i=0 size=0
0x80242ac00: [09/Nov/2017:00:04:24.403636] id=31 proto=unix_stream 
source=unix:1

  flags=0x88, conn_retries=0, srv_conn=0x0, pend_pos=0x0
  frontend=GLOBAL (id=0 mode=tcp), listener=? (id=1) addr=unix:1
  backend= (id=-1 mode=-)
  server= (id=-1)
  task=0x80248f4d0 (state=0x0a nice=-64 calls=1 exp=10s age=?)
  si[0]=0x80242ae38 (state=EST flags=0x08 endp0=CONN:0x8024ca900 
exp=, et=0x000)
  si[1]=0x80242ae60 (state=EST flags=0x4018 endp1=APPCTX:0x8024ca780 
exp=, et=0x000)
  co0=0x8024ca900 ctrl=unix_stream xprt=RAW data=STRM 
target=LISTENER:0x8024ca000

  flags=0x0020b306 fd=2 fd.state=25 fd.cache=0 updt=0
  app1=0x8024ca780 st0=7 st1=0 st2=3 applet=
  req=0x80242ac10 (f=0xc08200 an=0x0 pipe=0 tofwd=-1 total=15)
  an_exp= rex=10s wex=
  buf=0x8024e7dc0 data=0x8024e7dd4 o=0 p=0 req.next=0 i=0 size=16384
  res=0x80242ac50 (f=0x80008002 an=0x0 pipe=0 tofwd=-1 total=1198)
  an_exp= rex= wex=
  buf=0x8025603c0 data=0x8025603d4 o=1198 p=1198 rsp.next=0 i=0 
size=16384


FreeBSD pfsense.local 11.1-RELEASE-p3 FreeBSD 11.1-RELEASE-p3 #362 
r313908+9cf44ec5484(RELENG_2_4): Fri Nov  3 08:23:14 CDT 2017

[2.4.2-DEVELOPMENT][admin@pfsense.local]/root: haproxy -vv
HA-Proxy version 1.7.9 2017/08/18
Copyright 2000-2017 Willy Tarreau 

Build options :
  TARGET  = freebsd
  CPU = generic
  CC  = cc
  CFLAGS  = -O2 -pipe 

Re: Error in `haproxy': munmap_chunk(): invalid pointer:

[Andrew Smalley]

Hi Tim

Can you try a make install first please or mkdir -p
'/etc/haproxy/state/ so the state directory exists and then re-test.

The above is a guess, can you supply the build commands and clarify
this line in the config " bind :::80 v4v6" ? Dont you want to "bind
*:80" and use IPv4 only
Andruw Smalley

Loadbalancer.org Ltd.

www.loadbalancer.org
+1 888 867 9504 / +44 (0)330 380 1064
asmal...@loadbalancer.org

Leave a Review | Deployment Guides | Blog


On 9 November 2017 at 00:00, Tim Düsterhus  wrote:
> Hi
>
> I get the following crash when running:
>
> [timwolla@/t/h/haproxy-1.8-rc2]./haproxy -V
> HA-Proxy version 1.8-rc2-a8d8d6e 2017/11/03
> Copyright 2000-2017 Willy Tarreau 
>
> with the configuration at the bottom of this email as follows:
>
> root@node42:/tmp/haproxy/haproxy-1.8-rc2# ./haproxy -W -f
> /tmp/haproxy/haproxy-1.8-rc2/haproxy.cfg
> [WARNING] 312/004845 (31835) : Can't open server state file
> '/etc/haproxy/state/global': No such file or directory
> [WARNING] 312/004845 (31835) : Can't open server state file
> '/etc/haproxy/state/global': No such file or directory
>
> and then killing the master process using a SIGHUP:
>
>> *** Error in `./haproxy': munmap_chunk(): invalid pointer: 
>> 0x00515028 ***
>> === Backtrace: =
>> /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fda72bbc7e5]
>> /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1a8)[0x7fda72bc9698]
>> ./haproxy[0x4a28d0]
>> ./haproxy[0x4a2c4e]
>> ./haproxy[0x4a2f57]
>> ./haproxy[0x40b16f]
>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fda72b65830]
>> ./haproxy[0x40c149]
>> === Memory map: 
>> 0040-00532000 r-xp  08:03 2765518
>> /tmp/haproxy/haproxy-1.8-rc2/haproxy
>> 00731000-00732000 r--p 00131000 08:03 2765518
>> /tmp/haproxy/haproxy-1.8-rc2/haproxy
>> 00732000-00749000 rw-p 00132000 08:03 2765518
>> /tmp/haproxy/haproxy-1.8-rc2/haproxy
>> 00749000-0074e000 rw-p  00:00 0
>> 01d7f000-01de3000 rw-p  00:00 0  
>> [heap]
>> 7fda7272b000-7fda72741000 r-xp  08:03 528903 
>> /lib/x86_64-linux-gnu/libgcc_s.so.1
>> 7fda72741000-7fda7294 ---p 00016000 08:03 528903 
>> /lib/x86_64-linux-gnu/libgcc_s.so.1
>> 7fda7294-7fda72941000 rw-p 00015000 08:03 528903 
>> /lib/x86_64-linux-gnu/libgcc_s.so.1
>> 7fda72941000-7fda72944000 r-xp  08:03 533578 
>> /lib/x86_64-linux-gnu/libdl-2.23.so
>> 7fda72944000-7fda72b43000 ---p 3000 08:03 533578 
>> /lib/x86_64-linux-gnu/libdl-2.23.so
>> 7fda72b43000-7fda72b44000 r--p 2000 08:03 533578 
>> /lib/x86_64-linux-gnu/libdl-2.23.so
>> 7fda72b44000-7fda72b45000 rw-p 3000 08:03 533578 
>> /lib/x86_64-linux-gnu/libdl-2.23.so
>> 7fda72b45000-7fda72d05000 r-xp  08:03 533590 
>> /lib/x86_64-linux-gnu/libc-2.23.so
>> 7fda72d05000-7fda72f05000 ---p 001c 08:03 533590 
>> /lib/x86_64-linux-gnu/libc-2.23.so
>> 7fda72f05000-7fda72f09000 r--p 001c 08:03 533590 
>> /lib/x86_64-linux-gnu/libc-2.23.so
>> 7fda72f09000-7fda72f0b000 rw-p 001c4000 08:03 533590 
>> /lib/x86_64-linux-gnu/libc-2.23.so
>> 7fda72f0b000-7fda72f0f000 rw-p  00:00 0
>> 7fda72f0f000-7fda72f7d000 r-xp  08:03 541796 
>> /lib/x86_64-linux-gnu/libpcre.so.3.13.2
>> 7fda72f7d000-7fda7317d000 ---p 0006e000 08:03 541796 
>> /lib/x86_64-linux-gnu/libpcre.so.3.13.2
>> 7fda7317d000-7fda7317e000 r--p 0006e000 08:03 541796 
>> /lib/x86_64-linux-gnu/libpcre.so.3.13.2
>> 7fda7317e000-7fda7317f000 rw-p 0006f000 08:03 541796 
>> /lib/x86_64-linux-gnu/libpcre.so.3.13.2
>> 7fda7317f000-7fda73399000 r-xp  08:03 529753 
>> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>> 7fda73399000-7fda73598000 ---p 0021a000 08:03 529753 
>> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>> 7fda73598000-7fda735b4000 r--p 00219000 08:03 529753 
>> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>> 7fda735b4000-7fda735c rw-p 00235000 08:03 529753 
>> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>> 7fda735c-7fda735c3000 rw-p  00:00 0
>> 7fda735c3000-7fda73621000 r-xp  08:03 524697 
>> /lib/x86_64-linux-gnu/libssl.so.1.0.0
>> 7fda73621000-7fda73821000 ---p 0005e000 08:03 524697 
>> /lib/x86_64-linux-gnu/libssl.so.1.0.0
>> 7fda73821000-7fda73825000 r--p 0005e000 08:03 524697 
>> /lib/x86_64-linux-gnu/libssl.so.1.0.0
>> 7fda73825000-7fda7382c000 rw-p 00062000 08:03 524697 
>> /lib/x86_64-linux-gnu/libssl.so.1.0.0
>> 7fda7382c000-7fda73844000 r-xp  

Error in `haproxy': munmap_chunk(): invalid pointer:

[Tim Düsterhus]

Hi

I get the following crash when running:

[timwolla@/t/h/haproxy-1.8-rc2]./haproxy -V
HA-Proxy version 1.8-rc2-a8d8d6e 2017/11/03
Copyright 2000-2017 Willy Tarreau 

with the configuration at the bottom of this email as follows:

root@node42:/tmp/haproxy/haproxy-1.8-rc2# ./haproxy -W -f
/tmp/haproxy/haproxy-1.8-rc2/haproxy.cfg
[WARNING] 312/004845 (31835) : Can't open server state file
'/etc/haproxy/state/global': No such file or directory
[WARNING] 312/004845 (31835) : Can't open server state file
'/etc/haproxy/state/global': No such file or directory

and then killing the master process using a SIGHUP:

> *** Error in `./haproxy': munmap_chunk(): invalid pointer: 0x00515028 
> ***
> === Backtrace: =
> /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fda72bbc7e5]
> /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1a8)[0x7fda72bc9698]
> ./haproxy[0x4a28d0]
> ./haproxy[0x4a2c4e]
> ./haproxy[0x4a2f57]
> ./haproxy[0x40b16f]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fda72b65830]
> ./haproxy[0x40c149]
> === Memory map: 
> 0040-00532000 r-xp  08:03 2765518
> /tmp/haproxy/haproxy-1.8-rc2/haproxy
> 00731000-00732000 r--p 00131000 08:03 2765518
> /tmp/haproxy/haproxy-1.8-rc2/haproxy
> 00732000-00749000 rw-p 00132000 08:03 2765518
> /tmp/haproxy/haproxy-1.8-rc2/haproxy
> 00749000-0074e000 rw-p  00:00 0 
> 01d7f000-01de3000 rw-p  00:00 0  
> [heap]
> 7fda7272b000-7fda72741000 r-xp  08:03 528903 
> /lib/x86_64-linux-gnu/libgcc_s.so.1
> 7fda72741000-7fda7294 ---p 00016000 08:03 528903 
> /lib/x86_64-linux-gnu/libgcc_s.so.1
> 7fda7294-7fda72941000 rw-p 00015000 08:03 528903 
> /lib/x86_64-linux-gnu/libgcc_s.so.1
> 7fda72941000-7fda72944000 r-xp  08:03 533578 
> /lib/x86_64-linux-gnu/libdl-2.23.so
> 7fda72944000-7fda72b43000 ---p 3000 08:03 533578 
> /lib/x86_64-linux-gnu/libdl-2.23.so
> 7fda72b43000-7fda72b44000 r--p 2000 08:03 533578 
> /lib/x86_64-linux-gnu/libdl-2.23.so
> 7fda72b44000-7fda72b45000 rw-p 3000 08:03 533578 
> /lib/x86_64-linux-gnu/libdl-2.23.so
> 7fda72b45000-7fda72d05000 r-xp  08:03 533590 
> /lib/x86_64-linux-gnu/libc-2.23.so
> 7fda72d05000-7fda72f05000 ---p 001c 08:03 533590 
> /lib/x86_64-linux-gnu/libc-2.23.so
> 7fda72f05000-7fda72f09000 r--p 001c 08:03 533590 
> /lib/x86_64-linux-gnu/libc-2.23.so
> 7fda72f09000-7fda72f0b000 rw-p 001c4000 08:03 533590 
> /lib/x86_64-linux-gnu/libc-2.23.so
> 7fda72f0b000-7fda72f0f000 rw-p  00:00 0 
> 7fda72f0f000-7fda72f7d000 r-xp  08:03 541796 
> /lib/x86_64-linux-gnu/libpcre.so.3.13.2
> 7fda72f7d000-7fda7317d000 ---p 0006e000 08:03 541796 
> /lib/x86_64-linux-gnu/libpcre.so.3.13.2
> 7fda7317d000-7fda7317e000 r--p 0006e000 08:03 541796 
> /lib/x86_64-linux-gnu/libpcre.so.3.13.2
> 7fda7317e000-7fda7317f000 rw-p 0006f000 08:03 541796 
> /lib/x86_64-linux-gnu/libpcre.so.3.13.2
> 7fda7317f000-7fda73399000 r-xp  08:03 529753 
> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
> 7fda73399000-7fda73598000 ---p 0021a000 08:03 529753 
> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
> 7fda73598000-7fda735b4000 r--p 00219000 08:03 529753 
> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
> 7fda735b4000-7fda735c rw-p 00235000 08:03 529753 
> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
> 7fda735c-7fda735c3000 rw-p  00:00 0 
> 7fda735c3000-7fda73621000 r-xp  08:03 524697 
> /lib/x86_64-linux-gnu/libssl.so.1.0.0
> 7fda73621000-7fda73821000 ---p 0005e000 08:03 524697 
> /lib/x86_64-linux-gnu/libssl.so.1.0.0
> 7fda73821000-7fda73825000 r--p 0005e000 08:03 524697 
> /lib/x86_64-linux-gnu/libssl.so.1.0.0
> 7fda73825000-7fda7382c000 rw-p 00062000 08:03 524697 
> /lib/x86_64-linux-gnu/libssl.so.1.0.0
> 7fda7382c000-7fda73844000 r-xp  08:03 530442 
> /lib/x86_64-linux-gnu/libpthread-2.23.so
> 7fda73844000-7fda73a43000 ---p 00018000 08:03 530442 
> /lib/x86_64-linux-gnu/libpthread-2.23.so
> 7fda73a43000-7fda73a44000 r--p 00017000 08:03 530442 
> /lib/x86_64-linux-gnu/libpthread-2.23.so
> 7fda73a44000-7fda73a45000 rw-p 00018000 08:03 530442 
> /lib/x86_64-linux-gnu/libpthread-2.23.so
> 7fda73a45000-7fda73a49000 rw-p  00:00 0 
> 7fda73a49000-7fda73a62000 r-xp  08:03 524466 
> /lib/x86_64-linux-gnu/libz.so.1.2.8
> 7fda73a62000-7fda73c61000 

AFM in Focus Dailies November 8

[FilmFestivals.com at AFM]

**Follow us on the social networks and filmfestivals.com. **

 

 

 

 

 

 

 

 

** **

** **

**Read this online

I Promote Film to Festivals

I Email us

I Advertize in our AFM Newsletters and eBlasts

I **

Filmfestivals.com was established in 1995, over the years we became the
premier link between film and festivals with 6.000 festivals listed, over
14 000 festival organizers contacts in our database, 75.000 articles. 162
000 Industry subscribers and 930 000 unique visitors per year...

** **

**Promote your line up during AFM

**

AFM 2017 runs November 1 - 8, 2017 We would like to contribute to your
success in AFM Can we help? Promote your line up to media, buyers and
attendees. Boost your screenings' attendance. Shoot interviews or video
captur... 

**AFM Screenings & Events Update | Friday, November 3

**

Added Screenings Film: Time: Theatre: Company: &...

**Wine tasting Party for Festival Directors at AFM

**

The hosts of the filmfestivals.com wine tasting for Festival Directors at
AFM: from left to right: Laurie Kirby (Fest Forums), Bruno Chatelin
(filfestivals.com), Stuart MacNaught (Festforums) and Lise Romanoff (Vision
Films). The party was held November 3rd at Vision Filmssuite gathering some
45 festival directors from around the world. Next wine tasting will be in
Cannes. Fine wines by Bernardus ...

08.11.2017 | American Film Market Dailies's blog


**The end is coming to AFM...The Apocalypse War!

**

**Finders Keepers - theatrical release in South Africa from Ster Kinekor

**

**www.summerhillfilms.com 
I Contact 
I Suite 327B I Facebook 
I Twitter: @WatchCoolMovies 
**

**Protagonist Pictures has sold North American rights to FilmRise for Clio
Barnard's drama Dark River

**

Protagonist Pictures has sold North American rights to FilmRise for Clio
Barnard's (The Selfish Giant, The Arbor) drama Dark River. It was announced
today by Protagonist Managing Director of Sales and Distribution Vanessa
Saal and FilmRise CEO Danny Fisher. The film debuted as a world premiere at
the Toronto International Film Festival and screened as an official
selection in the London Film Festival to outstanding reviews. The North
American deal was negotiated between Geor...

**Burt Reynolds stars in "Miami Love Affair" -- New Romantic Comedy at AFM

**

CoCo Movie, LLC Brings on Ginger Knight Entertainment for Worldwide Sales
...

**The 31st Israel Film Festival in Los Angeles celebrated its opening night
before a sold-out crowd

**

JEFFREY TAMBOR, LIOR ASHKENAZI HONORED Israel Film Fest in LA- Meir
Fenigstein, left, Ayelet Zurer, Karin Eliyahu-Pery & Jeffrey Tambor The
31st Israel Film Festival in Los Angeles, which runs through November 21st,
celebrated its opening night before a sold-out crowd at the Saban Theatre
in Beverly Hills on November 5th. Festival founder/executive director Meir
Fenigstein presided over the night, which honored Jeffrey Tambor with the
2017 Israel Film Festival Achieveme...

**Working with 

Re: Diagnose a PD-- status

[Mildis]

Christopher,

We couldn’t had the error reproduced today : both dataset and client tool were 
updated.
However, I keep the configuration options at hand if I ever need to debug this 
issue further again.

Thanks for your time on this.
Regards,
Mildis


> Le 7 nov. 2017 à 10:14, Christopher Faulet  a écrit :
> 
> Le 06/11/2017 à 16:47, Mildis a écrit :
>> Hi Christopher,
>> Configuration is attached.
>> The domain2backend map sends data mostly to bck-traefik.
> 
> Hi Mildis,
> 
> At first glance, I don't see anything strange here. The compression filter is 
> not supposed to fail this way. So there is definitely something strange I 
> don't understand for now.
> 
> Could you reproduce the bug on a single request on an HAProxy instance 
> without load, maybe using a dedicated frontend ? If yes, it could be useful 
> to enable the trace filter adding following lines in your configuration, in 
> the used frontend section:
> 
>  filter trace name BEFORE
>  filter compression
>  filter trace name AFTER
> 
> Then, you can retry, starting HAProxy in foreground. The output is verbose, 
> but it give more information about the message filtering.
> 
> Thanks,
> -- 
> Christopher Faulet

Re: [Working update]: Request rate limiting on the backend section

[Krishna Kumar (Engineering)]

To remove the reported "margin of error", the config needed
a fix:

   acl within_limit sc2_gpc0_rate() lt 1000

since the first request was at rate==0, and last one is at 999.


On Wed, Nov 8, 2017 at 3:11 PM, Krishna Kumar (Engineering) <
krishna...@flipkart.com> wrote:

> I finally got the backend rate limiting working pretty well. Here is the
> configuration settings in case it helps anyone else do the same:
>
> frontend http-fe
> bind 
> default_backend http-be
>
> backend http-be
> http-request track-sc2 fe_id
> stick-table type integer size 1k expire 30s store
>   http_req_rate(1s),gpc0,gpc0_rate(1s)
> acl within_limit sc2_gpc0_rate() le 1000
> acl increment_gpc0 sc2_inc_gpc0 ge 0
> http-request allow if within_limit increment_gpc0
> http-request deny deny_status 429 if !within_limit
> server my-server 
>
> During the test, the stick table contents were:
> 0x16e593c: key=3 use=98 exp=2 gpc0=44622 gpc0_rate(1000)=1000
> http_req_rate(1000)=69326
>
> Test results:
> # wrk -t 40 -c 4000 -d 30s 
> RPS: 1003.05 (Total requests: 2031922 Good: 30192 Errors: 2001730 Time:
> 30.10)
>
> Margin of error: 0.3%
>
> Thanks,
> - Krishna
>
>
> On Wed, Nov 8, 2017 at 10:02 AM, Krishna Kumar (Engineering) <
> krishna...@flipkart.com> wrote:
>
>> On Tue, Nov 7, 2017 at 11:57 PM, Lukas Tribus  wrote:
>>
>> Hi Lukas,
>>
>> > Yes, in 1.7 you can change server maxconn values in real time using
>> > the admin socket:
>> > https://cbonte.github.io/haproxy-dconv/1.7/management.html
>> #9.3-set%20maxconn%20server
>>
>> Thanks, will take a look at if we can use this. The only issue is that we
>> want to
>> be able to change rps very often, and some backend sections contain upto
>> 500 servers (and much more during sales), and doing that on the fly at
>> high
>> frequency may not scale.
>>
>> > You are reluctant to elaborate on the bigger picture, so I guess
>> > generic advice is not what you are looking for. I just hope you are
>> > not trying to build some kind of distributed rate-limiting
>> > functionality with this.
>>
>> Sorry, not reluctance, I thought giving too much detail would put off
>> people
>> from taking a look :) So you are right, we are trying to build a
>> distributed rate
>> limiting feature, and the control plane is mostly ready (thanks to HAProxy
>> developers for making such a performant/configurable system). The service
>> monitors current http_request_rate and current RPS setting via uds every
>> second, and updates these values to a central repository (zookeeper), and
>> on demand, tries to increase capacity by requesting capacity from other
>> servers so as to keep capacity constant at the configured value (e.g. 1000
>> RPS). Is this something you would not recommend?
>>
>> > I don't have enough experience with stick-tables to comment on this
>> > generally, but I would suggest you upgrade to a current 1.7 release
>> > first of all and retry your tests. There are currently 223 bugs fixed
>> > in releases AFTER 1.6.3:
>> > http://www.haproxy.org/bugs/bugs-1.6.3.html
>>
>> Thanks, we are considering moving to this version.
>>
>> > Maybe someone more stick-table savvy can comment on your specific
>> question.
>>
>> If anyone else has done something similar, would really like to hear from
>> you on
>> how to control RPS in the backend.
>>
>> Regards,
>> - Krishna
>>
>>
>

[Working update]: Request rate limiting on the backend section

[Krishna Kumar (Engineering)]

I finally got the backend rate limiting working pretty well. Here is the
configuration settings in case it helps anyone else do the same:

frontend http-fe
bind 
default_backend http-be

backend http-be
http-request track-sc2 fe_id
stick-table type integer size 1k expire 30s store
  http_req_rate(1s),gpc0,gpc0_rate(1s)
acl within_limit sc2_gpc0_rate() le 1000
acl increment_gpc0 sc2_inc_gpc0 ge 0
http-request allow if within_limit increment_gpc0
http-request deny deny_status 429 if !within_limit
server my-server 

During the test, the stick table contents were:
0x16e593c: key=3 use=98 exp=2 gpc0=44622 gpc0_rate(1000)=1000
http_req_rate(1000)=69326

Test results:
# wrk -t 40 -c 4000 -d 30s 
RPS: 1003.05 (Total requests: 2031922 Good: 30192 Errors: 2001730 Time:
30.10)

Margin of error: 0.3%

Thanks,
- Krishna


On Wed, Nov 8, 2017 at 10:02 AM, Krishna Kumar (Engineering) <
krishna...@flipkart.com> wrote:

> On Tue, Nov 7, 2017 at 11:57 PM, Lukas Tribus  wrote:
>
> Hi Lukas,
>
> > Yes, in 1.7 you can change server maxconn values in real time using
> > the admin socket:
> > https://cbonte.github.io/haproxy-dconv/1.7/management.html
> #9.3-set%20maxconn%20server
>
> Thanks, will take a look at if we can use this. The only issue is that we
> want to
> be able to change rps very often, and some backend sections contain upto
> 500 servers (and much more during sales), and doing that on the fly at high
> frequency may not scale.
>
> > You are reluctant to elaborate on the bigger picture, so I guess
> > generic advice is not what you are looking for. I just hope you are
> > not trying to build some kind of distributed rate-limiting
> > functionality with this.
>
> Sorry, not reluctance, I thought giving too much detail would put off
> people
> from taking a look :) So you are right, we are trying to build a
> distributed rate
> limiting feature, and the control plane is mostly ready (thanks to HAProxy
> developers for making such a performant/configurable system). The service
> monitors current http_request_rate and current RPS setting via uds every
> second, and updates these values to a central repository (zookeeper), and
> on demand, tries to increase capacity by requesting capacity from other
> servers so as to keep capacity constant at the configured value (e.g. 1000
> RPS). Is this something you would not recommend?
>
> > I don't have enough experience with stick-tables to comment on this
> > generally, but I would suggest you upgrade to a current 1.7 release
> > first of all and retry your tests. There are currently 223 bugs fixed
> > in releases AFTER 1.6.3:
> > http://www.haproxy.org/bugs/bugs-1.6.3.html
>
> Thanks, we are considering moving to this version.
>
> > Maybe someone more stick-table savvy can comment on your specific
> question.
>
> If anyone else has done something similar, would really like to hear from
> you on
> how to control RPS in the backend.
>
> Regards,
> - Krishna
>
>

Re: lua socket api settimeout in seconds vs. milliseconds

[Adis Nezirovic]

On 11/07/2017 08:49 PM, Nick Galbreath wrote:
> I have a question regarding the socket:settimeout function.
> 
> It appears to only accept an integer value for seconds, but internally
> converts to milliseconds (see below).
> 
> Is there a reason to enforce whole seconds, or could this be relaxed to
> accept a lua number (float/double).?

Hi Nick,

HAProxy uses Lua 5.3 which has proper integers. There is already a
similar situation where we have both, seconds and milliseconds
(hlua_sleep/hlua_msleep), so maybe that's the best solution here too,
just add another variant using milliseconds.

Best regards,
Adis