Quote Request

[Chris Rorie]

Hi Sales,

Family Dollar will be very grateful if you can  provide or  source with a quote 
of  the following items:

Wholemilk Powder 28% fat 50lbs 850 bags
Non Fat Grade A Powder Milk 50lbs 850 bags

Typical Composition
Lactose 36.0% - 38.5%
Fat 26.0% - 40.0%
Ash 5.5% - 6.5%
Moisture2.0% - 4.5%


We're interested in maintaining a long-lasting relationship with your company 
if your quotes match our budget. With this in mind, we ask that you give us the 
best prices for these items for our payment terms is Net30. Kindly flourish us 
with a credit application form.

Please feel free to email us for further clarification.

Note quote must be valid for 10 days




Warm regards,
Chris Rorie
Director, Corporate Procurement
Telephone: +1 210-998-1040
E-mail: chris.ro...@familydollarz.com
Web Site: www.familydollarz.com

[no subject]

[Chris Cheltenham]

haproxy.org : Improve your mobile website for better rankings!!

[Chris Hayes]

Hello haproxy.org,

Hope you are doing well.

I was looking for Keywords and found your website on pages 3-4 of Google.
If you were on page #1, you'd get so many prospects/new clients every day.
Most of your targeted keywords are on pages #3-5. You're so close! Do you
mind if I sent an *SEO Audit* of your site to see why you're not on page #1?

It's totally free. I can send the audit report in a couple of hours. If you
have any questions, feel free to ask me. I have more than 10 years of
experience in digital marketing.

Can I send it over?

Look forward to your reply.

Kind Regards

*Thanks!*
*Chris Hayes**! Business Development Manager*



*Note:* - Our next conversation will be on my corporate Email ID. If this
is something you are interested in, please respond to this email. If this
is not your interest, don't worry, we will not email you again.

haproxy.com : Improve your mobile website for better rankings!!

[Chris Hayes]

Hello haproxy.com,

Hope you are doing well.

I was looking for Keywords and found your website on pages 3-4 of Google.
If you were on page #1, you'd get so many prospects/new clients every day.
Most of your targeted keywords are on pages #3-5. You're so close! Do you
mind if I sent an *SEO Audit* of your site to see why you're not on page #1?

It's totally free. I can send the audit report in a couple of hours. If you
have any questions, feel free to ask me. I have more than 10 years of
experience in digital marketing.

Can I send it over?

Look forward to your reply.

Kind Regards

*Thanks!*
*Chris Hayes**! Business Development Manager*



*Note:* - Our next conversation will be on my corporate Email ID. If this
is something you are interested in, please respond to this email. If this
is not your interest, don't worry, we will not email you again.

Patch backport request: __USE_GNU breaks uclibc in the 2.1 branch

[Chris]

Hello everybody,

I am one of the maintainers of the haproxy package for the OpenWRT
project. I am reaching out to you because - as of HAProxy version
2.1.5 - we experience build-issues on some of our build-targets.
We mostly use MUSL and uclibc as our c-libraries because they are more
suitable for embedded devices which are our main focus. Since HAProxy
version 2.1.5, the build is broken for all of our uclibc-targets:

src/standard.c: In function 'dladdr_and_size':
src/standard.c:4356:8: warning: implicit declaration of function
'dladdr1'; did you mean 'dladdr'? [-Wimplicit-function-declaration]
  ret = dladdr1(addr, dli, (void **), RTLD_DL_SYMENT);
^~~
dladdr
src/standard.c:4356:42: error: 'RTLD_DL_SYMENT' undeclared (first use
in this function); did you mean 'DT_SYMENT'?
  ret = dladdr1(addr, dli, (void **), RTLD_DL_SYMENT);
  ^~
  DT_SYMENT
src/standard.c:4356:42: note: each undeclared identifier is reported
only once for each function it appears in



The problem lies in the #ifdef in src/standard.c:

static int dladdr_and_size(const void *addr, Dl_info *dli, size_t *size)
{
int ret;
#ifdef __USE_GNU // most detailed one
const ElfW(Sym) *sym;

ret = dladdr1(addr, dli, (void **), RTLD_DL_SYMENT);
if (ret)
*size = sym ? sym->st_size : 0;
#else
ret = dladdr(addr, dli);
*size = 0;
#endif
return ret;
}



Neither MUSL nor uclibc support dladdr1() so both must fall back to
using dladdr(). However, __USE_GNU is defined in uclibc making it use
dladdr1() resulting in the compilation failure. Using __USE_GNU is
generally not recommended so I wrote a patch which changed the #ifdef
to check for GLIBC. When I was preparing for submitting the patch for
the haproxy dev-branch I realized that Willy already did basically the
exact same thing there (here:
http://git.haproxy.org/?p=haproxy.git;a=commit;h=62af9c83f9ed2b25e0061798e29e3cccfce5fbdc).

So in conclusion, my request is to backport Willy's change to the 2.1
tree, please.

Thanks a lot,
Christian

[PATCH v3] MINOR: build: add aix72-gcc build TARGET and power{8,9} CPUs

[Chris]

Hi Илья,

I agree! I just made a new patch which adds the missing documentation
for the new build-TARGET as well as the two new CPU-types.

thanks,
Chris

On Tue, Feb 11, 2020 at 8:29 PM Илья Шипицин  wrote:
>
> there's AIX related build guide
>
> https://github.com/haproxy/haproxy/blob/master/INSTALL#L512-L517
>
> maybe it should be updated as well
>
> пн, 10 февр. 2020 г. в 16:45, Chris :
>>
>> Hi Willy,
>>
>> thanks a lot for your input and sorry for the delay. Work is pretty
>> rough right now...
>>
>> > Thank you for doing this work. I'm having a few questions below.
>> Sure, I am glad I can help :).
>>
>> > > +# AIX 7.2 and above
>> > > +ifeq ($(TARGET),aix72-gcc)
>> > > +  set_target_defaults = $(call default_opts, \
>> > > +USE_POLL USE_THREAD USE_LIBCRYPT USE_OBSOLETE_LINKER
>> > > USE_GETADDRINFO USE_TFO)
>> >
>> > Are you really really sure about USE_TFO ? I suspect you might have
>> > accidently borrowed it from another line. It stands for TCP FastOpen
>> > and is really not broadly deployed, I was only aware of Linux and
>> > FreeBSD, but there are likely a few other ones. And the fact that it
>> > looks OK is possibly just that it's causing a different TCP socket
>> > option to be set on the connection, so if you're not certain we'd
>> > rather avoid it.
>> I think you are right. I kept USE_TFO in there because it neither
>> introduced any build-issues nor did it cause any runtime issues.
>> However, it probably does nothing and should be removed as long as we
>> are not certain it is actually beneficial. I will have a look at the
>> corresponding AIX internals later at some point.
>>
>> > > src/hlua.c: In function 'hlua_panic_ljmp':
>> > > src/hlua.c:128:1: warning: no return statement in function returning
>> > > non-void [-Wreturn-type]
>> > >  static int hlua_panic_ljmp(lua_State *L) { longjmp(safe_ljmp_env, 1); }
>> > >  ^~
>> >
>> > Don't worry about this one, I'll handle it. I suspect that on linux
>> > platforms the longjmp() function prototype is decorated with
>> > __attribute__((noreturn)) which makes the compiler happy, but that's
>> > likely not the case on any system not relying on a gcc-compatible
>> > compiler by default.
>> Thanks for handling this one. If there is something I shoult test
>> please let me know.
>>
>> > Just two small extra requests :
>> >   - please rebase it on top of the development branch. If you really
>> > need it in 2.1, just indicate it. Given that it's well isolated,
>> > I'm fine with having it backported.
>> >
>> >   - please have a look at CONTRIBUTING to get guidance to write a
>> > subject and a commit message. It doesn't need to be very long
>> > but at least indicating what options you chosed to enable/disable
>> > and on what system you tested it will be enough if we need to
>> > reconsider parts of it later.
>> I already made a new patch based on the current development-HEAD and
>> tried to follow all the guidelines from CONTRIBUTING. I would love to
>> have it backported to 2.1 as the patch is pretty much identical and
>> should not cause any regressions. The patch will follow shortly!
>>
>> > Do you have a permanent and durable access to this machine, with the
>> > ability to occasionally re-run a build test in case we ask you (likely
>> > no more than 2-3 times a year in the worst case) ? I'm asking because
>> > I'm still keeping a very old IBM server running 5.2 on a Power3 just
>> > for the sake of revalidating new releases once in a while. Given that
>> > I could not upgrade it to latest OpenSSL, it cannot even be used to
>> > provide complete binaries to those needing them, so it's getting very
>> > obsolete and knowing that there's a better solution somewhere would
>> > allow me to get rid of it.
>> Well, I am not the primary administrator of this machine. However, I
>> have permanent access and no problem doing the occasional build on it.
>> I will have a chat with the server admin in regards to the longtime
>> prospects of this server and report back to you.
>>
>> > thanks!
>> > Willy
>> You are welcome - I am happy I can contribute!
>>
>> thanks,
>> Christian
>>
>> On Thu, Feb 6, 2020 at 3:36 PM Willy Tarreau  wrote:
>> >
>> > Hello Christian,
>> >
>> > On Mon, Feb 03, 2020 at 12:09:46PM +0100, Chris wrote:

Re: [PATCH v2] MINOR: build: add aix72-gcc build TARGET and power{8,9} CPUs

[Chris]

Hi Willy,

I am really sorry - it feel like I fell into every newby trap. I
attached the proposed patch to this mail. Also, I will never use Gmail
to send patches again.

Thank you for your patience,
Chris

On Tue, Feb 11, 2020 at 8:07 AM Willy Tarreau  wrote:
>
> On Mon, Feb 10, 2020 at 01:19:38PM +0100, Chris wrote:
> > Hello,
> >
> > this is the reworked patch - rebased on the current development-HEAD
> > and formatted according to CONTRIBUTIONS.
>
> Thank you Chris. That's perfect, except for one thing, which is that
> your mailer mangled it by wrapping some long lines and changing tabs
> for spaces. I tried a few minutes to fix it by hand but failed and
> gave up. Please re-send it as an attachment then, it should work fine.
>
> Thanks,
> Willy


0001-MINOR-build-add-aix72-gcc-build-TARGET-and-power-8-9.patch
Description: Binary data

Re: [PATCH v2] MINOR: build: add aix72-gcc build TARGET and power{8,9} CPUs

[Chris]

;  linux-glibc, linux-glibc-legacy, solaris, freebsd,
netbsd, osx,"; \
- echo "  openbsd, aix51, aix52, cygwin, haiku, generic, custom"; \
+ echo "  openbsd, aix51, aix52, aix72-gcc, cygwin, haiku,
generic, custom"; \
fi
 $(Q)echo;echo "Enabled features for TARGET '$(TARGET)' (disable
with 'USE_xxx=') :"
 $(Q)set -- $(foreach opt,$(patsubst USE_%,%,$(use_opts)),$(if
$(USE_$(opt)),$(opt),)); echo "  $$*" | (fmt || cat) 2>/dev/null
-- 
2.25.0


Best regards,
Christian

On Mon, Feb 10, 2020 at 12:41 PM Chris  wrote:
>
> Hi Willy,
>
> thanks a lot for your input and sorry for the delay. Work is pretty
> rough right now...
>
> > Thank you for doing this work. I'm having a few questions below.
> Sure, I am glad I can help :).
>
> > > +# AIX 7.2 and above
> > > +ifeq ($(TARGET),aix72-gcc)
> > > +  set_target_defaults = $(call default_opts, \
> > > +USE_POLL USE_THREAD USE_LIBCRYPT USE_OBSOLETE_LINKER
> > > USE_GETADDRINFO USE_TFO)
> >
> > Are you really really sure about USE_TFO ? I suspect you might have
> > accidently borrowed it from another line. It stands for TCP FastOpen
> > and is really not broadly deployed, I was only aware of Linux and
> > FreeBSD, but there are likely a few other ones. And the fact that it
> > looks OK is possibly just that it's causing a different TCP socket
> > option to be set on the connection, so if you're not certain we'd
> > rather avoid it.
> I think you are right. I kept USE_TFO in there because it neither
> introduced any build-issues nor did it cause any runtime issues.
> However, it probably does nothing and should be removed as long as we
> are not certain it is actually beneficial. I will have a look at the
> corresponding AIX internals later at some point.
>
> > > src/hlua.c: In function 'hlua_panic_ljmp':
> > > src/hlua.c:128:1: warning: no return statement in function returning
> > > non-void [-Wreturn-type]
> > >  static int hlua_panic_ljmp(lua_State *L) { longjmp(safe_ljmp_env, 1); }
> > >  ^~
> >
> > Don't worry about this one, I'll handle it. I suspect that on linux
> > platforms the longjmp() function prototype is decorated with
> > __attribute__((noreturn)) which makes the compiler happy, but that's
> > likely not the case on any system not relying on a gcc-compatible
> > compiler by default.
> Thanks for handling this one. If there is something I shoult test
> please let me know.
>
> > Just two small extra requests :
> >   - please rebase it on top of the development branch. If you really
> > need it in 2.1, just indicate it. Given that it's well isolated,
> > I'm fine with having it backported.
> >
> >   - please have a look at CONTRIBUTING to get guidance to write a
> > subject and a commit message. It doesn't need to be very long
> > but at least indicating what options you chosed to enable/disable
> > and on what system you tested it will be enough if we need to
> > reconsider parts of it later.
> I already made a new patch based on the current development-HEAD and
> tried to follow all the guidelines from CONTRIBUTING. I would love to
> have it backported to 2.1 as the patch is pretty much identical and
> should not cause any regressions. The patch will follow shortly!
>
> > Do you have a permanent and durable access to this machine, with the
> > ability to occasionally re-run a build test in case we ask you (likely
> > no more than 2-3 times a year in the worst case) ? I'm asking because
> > I'm still keeping a very old IBM server running 5.2 on a Power3 just
> > for the sake of revalidating new releases once in a while. Given that
> > I could not upgrade it to latest OpenSSL, it cannot even be used to
> > provide complete binaries to those needing them, so it's getting very
> > obsolete and knowing that there's a better solution somewhere would
> > allow me to get rid of it.
> Well, I am not the primary administrator of this machine. However, I
> have permanent access and no problem doing the occasional build on it.
> I will have a chat with the server admin in regards to the longtime
> prospects of this server and report back to you.
>
> > thanks!
> > Willy
> You are welcome - I am happy I can contribute!
>
> thanks,
> Christian
>
> On Thu, Feb 6, 2020 at 3:36 PM Willy Tarreau  wrote:
> >
> > Hello Christian,
> >
> > On Mon, Feb 03, 2020 at 12:09:46PM +0100, Chris wrote:
> > > Hello everybody,
> > >
> > > I spent some time making haproxy compile and run successfully on AIX
> > > 7.2 using

Re: [PATCH] AIX 7.2 support

[Chris]

Hi Willy,

thanks a lot for your input and sorry for the delay. Work is pretty
rough right now...

> Thank you for doing this work. I'm having a few questions below.
Sure, I am glad I can help :).

> > +# AIX 7.2 and above
> > +ifeq ($(TARGET),aix72-gcc)
> > +  set_target_defaults = $(call default_opts, \
> > +USE_POLL USE_THREAD USE_LIBCRYPT USE_OBSOLETE_LINKER
> > USE_GETADDRINFO USE_TFO)
>
> Are you really really sure about USE_TFO ? I suspect you might have
> accidently borrowed it from another line. It stands for TCP FastOpen
> and is really not broadly deployed, I was only aware of Linux and
> FreeBSD, but there are likely a few other ones. And the fact that it
> looks OK is possibly just that it's causing a different TCP socket
> option to be set on the connection, so if you're not certain we'd
> rather avoid it.
I think you are right. I kept USE_TFO in there because it neither
introduced any build-issues nor did it cause any runtime issues.
However, it probably does nothing and should be removed as long as we
are not certain it is actually beneficial. I will have a look at the
corresponding AIX internals later at some point.

> > src/hlua.c: In function 'hlua_panic_ljmp':
> > src/hlua.c:128:1: warning: no return statement in function returning
> > non-void [-Wreturn-type]
> >  static int hlua_panic_ljmp(lua_State *L) { longjmp(safe_ljmp_env, 1); }
> >  ^~
>
> Don't worry about this one, I'll handle it. I suspect that on linux
> platforms the longjmp() function prototype is decorated with
> __attribute__((noreturn)) which makes the compiler happy, but that's
> likely not the case on any system not relying on a gcc-compatible
> compiler by default.
Thanks for handling this one. If there is something I shoult test
please let me know.

> Just two small extra requests :
>   - please rebase it on top of the development branch. If you really
> need it in 2.1, just indicate it. Given that it's well isolated,
> I'm fine with having it backported.
>
>   - please have a look at CONTRIBUTING to get guidance to write a
> subject and a commit message. It doesn't need to be very long
> but at least indicating what options you chosed to enable/disable
> and on what system you tested it will be enough if we need to
> reconsider parts of it later.
I already made a new patch based on the current development-HEAD and
tried to follow all the guidelines from CONTRIBUTING. I would love to
have it backported to 2.1 as the patch is pretty much identical and
should not cause any regressions. The patch will follow shortly!

> Do you have a permanent and durable access to this machine, with the
> ability to occasionally re-run a build test in case we ask you (likely
> no more than 2-3 times a year in the worst case) ? I'm asking because
> I'm still keeping a very old IBM server running 5.2 on a Power3 just
> for the sake of revalidating new releases once in a while. Given that
> I could not upgrade it to latest OpenSSL, it cannot even be used to
> provide complete binaries to those needing them, so it's getting very
> obsolete and knowing that there's a better solution somewhere would
> allow me to get rid of it.
Well, I am not the primary administrator of this machine. However, I
have permanent access and no problem doing the occasional build on it.
I will have a chat with the server admin in regards to the longtime
prospects of this server and report back to you.

> thanks!
> Willy
You are welcome - I am happy I can contribute!

thanks,
Christian

On Thu, Feb 6, 2020 at 3:36 PM Willy Tarreau  wrote:
>
> Hello Christian,
>
> On Mon, Feb 03, 2020 at 12:09:46PM +0100, Chris wrote:
> > Hello everybody,
> >
> > I spent some time making haproxy compile and run successfully on AIX
> > 7.2 using GCC 8.3 and wanted to contribute my patch in the hope that
> > it could be merged. The patch is based on the current haproxy 2.1 head
> > revision. I can make one for the development branch too - but it
> > should be basically identical.
>
> Thank you for doing this work. I'm having a few questions below.
>
> > +# AIX 7.2 and above
> > +ifeq ($(TARGET),aix72-gcc)
> > +  set_target_defaults = $(call default_opts, \
> > +USE_POLL USE_THREAD USE_LIBCRYPT USE_OBSOLETE_LINKER
> > USE_GETADDRINFO USE_TFO)
>
> Are you really really sure about USE_TFO ? I suspect you might have
> accidently borrowed it from another line. It stands for TCP FastOpen
> and is really not broadly deployed, I was only aware of Linux and
> FreeBSD, but there are likely a few other ones. And the fact that it
> looks OK is possibly just that it's causing a different TCP socket
> option to be set on the connection, so if you're not certain we'd
> rather avoi

[PATCH] AIX 7.2 support

[Chris]

Hello everybody,

I spent some time making haproxy compile and run successfully on AIX
7.2 using GCC 8.3 and wanted to contribute my patch in the hope that
it could be merged. The patch is based on the current haproxy 2.1 head
revision. I can make one for the development branch too - but it
should be basically identical.



--- Makefile.old 2020-02-03 10:56:25.0 +
+++ Makefile 2020-02-03 11:32:00.735254403 +
@@ -144,13 +144,13 @@
 # Use TARGET= to optimize for a specifc target OS among the
 # following list (use the default "generic" if uncertain) :
 #linux-glibc, linux-glibc-legacy, solaris, freebsd, openbsd, netbsd,
-#cygwin, haiku, aix51, aix52, osx, generic, custom
+#cygwin, haiku, aix51, aix52, aix72-gcc, osx, generic, custom
 TARGET =

  TARGET CPU
 # Use CPU= to optimize for a particular CPU, among the following
 # list :
-#generic, native, i586, i686, ultrasparc, custom
+#generic, native, i586, i686, ultrasparc, power8, power9, custom
 CPU = generic

  Architecture, used when not building for native architecture
@@ -257,6 +257,8 @@
 CPU_CFLAGS.i586   = -O2 -march=i586
 CPU_CFLAGS.i686   = -O2 -march=i686
 CPU_CFLAGS.ultrasparc = -O6 -mcpu=v9 -mtune=ultrasparc
+CPU_CFLAGS.power8 = -O2 -mcpu=power8 -mtune=power8
+CPU_CFLAGS.power9 = -O2 -mcpu=power9 -mtune=power9
 CPU_CFLAGS= $(CPU_CFLAGS.$(CPU))

  ARCH dependant flags, may be overridden by CPU flags
@@ -381,7 +383,7 @@
   DEBUG_CFLAGS=
 endif

-# AIX 5.2 and above
+# AIX 5.2
 ifeq ($(TARGET),aix52)
   set_target_defaults = $(call default_opts, \
 USE_POLL USE_LIBCRYPT USE_OBSOLETE_LINKER)
@@ -389,6 +391,14 @@
   DEBUG_CFLAGS=
 endif

+# AIX 7.2 and above
+ifeq ($(TARGET),aix72-gcc)
+  set_target_defaults = $(call default_opts, \
+USE_POLL USE_THREAD USE_LIBCRYPT USE_OBSOLETE_LINKER
USE_GETADDRINFO USE_TFO)
+  TARGET_CFLAGS   = -D_H_XMEM -D_H_VAR
+  TARGET_LDFLAGS  = -latomic
+endif
+
 # Cygwin
 ifeq ($(TARGET),cygwin)
   set_target_defaults = $(call default_opts, \
@@ -754,7 +764,7 @@
  @echo "Please choose the target among the following supported list :"
  @echo
  @echo "   linux-glibc, linux-glibc-legacy, solaris, freebsd, openbsd, netbsd,"
- @echo "   cygwin, haiku, aix51, aix52, osx, generic, custom"
+ @echo "   cygwin, haiku, aix51, aix52, aix72-gcc, osx, generic, custom"
  @echo
  @echo "Use \"generic\" if you don't want any optimization, \"custom\" if you"
  @echo "want to precisely tweak every option, or choose the target which"
@@ -832,7 +842,7 @@
else \
  echo "TARGET not set, you may pass 'TARGET=xxx' to set one among :";\
  echo "  linux-glibc, linux-glibc-legacy, solaris, freebsd,
netbsd, osx,"; \
- echo "  openbsd, aix51, aix52, cygwin, haiku, generic, custom"; \
+ echo "  openbsd, aix51, aix52, aix72-gcc, cygwin, haiku,
generic, custom"; \
fi
  $(Q)echo;echo "Enabled features for TARGET '$(TARGET)' (disable with
'USE_xxx=') :"
  $(Q)set -- $(foreach opt,$(patsubst USE_%,%,$(use_opts)),$(if
$(USE_$(opt)),$(opt),)); echo "  $$*" | (fmt || cat) 2>/dev/null






The patch implements a new TARGET called aix72-gcc and also adds 2
CPUs (power8 and power9). Here is my proof-of-work:




-bash-4.4$ gmake -j8 TARGET=aix72-gcc CPU=power8 USE_ZLIB=1
ZLIB_LIB=/opt/haproxy/lib ZLIB_INC=/opt/haproxy/include USE_OPENSSL=1
SSL_LIB=/opt/haproxy/lib SSL_INC=/opt/haproxy/include USE_PCRE2=1
USE_PCRE2_JIT=1 PCREDIR=/opt/haproxy USE_LUA=1
LUA_LIB=/opt/haproxy/lib LUA_INC=/opt/haproxy/include/lua
LUA_LIB_NAME=lua LUA_LD_FLAGS=
EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"
  CC  src/ev_poll.o
  CC  src/ssl_sock.o
  CC  src/hlua.o
  CC  src/hlua_fcn.o
  CC  src/mux_h2.o
  CC  src/stream.o
  CC  src/mux_fcgi.o
  CC  src/cfgparse-listen.o
src/hlua.c: In function 'hlua_panic_ljmp':
src/hlua.c:128:1: warning: no return statement in function returning
non-void [-Wreturn-type]
 static int hlua_panic_ljmp(lua_State *L) { longjmp(safe_ljmp_env, 1); }
 ^~
  CC  src/http_ana.o
  CC  src/stats.o
  CC  src/mux_h1.o
  CC  src/flt_spoe.o
  CC  src/server.o
  CC  src/cfgparse.o
  CC  src/checks.o
  CC  src/backend.o
  CC  src/log.o
  CC  src/peers.o
  CC  src/cli.o
  CC  src/haproxy.o
  CC  src/stick_table.o
  CC  src/standard.o
  CC  src/sample.o
  CC  src/proxy.o
  CC  src/stream_interface.o
  CC  src/pattern.o
  CC  src/dns.o
  CC  src/proto_tcp.o
  CC  src/listener.o
  CC  src/cfgparse-global.o
  CC  src/h1.o
  CC  src/http_rules.o
  CC  src/http_fetch.o
  CC  src/cache.o
  CC  src/session.o
  CC  src/fcgi-app.o
  CC  src/connection.o
  CC  src/tcp_rules.o
  CC  src/filters.o
  CC  src/task.o
  CC  src/mworker.o
  CC  src/map.o
  CC  src/h1_htx.o
  CC  src/trace.o
  CC  src/flt_trace.o
  CC  src/acl.o
  CC  src/http_htx.o
  CC  

Re: New haproxy tags v2.1.1 and v2.1.2 not pushed to github

[Chris Software]

OK, thank you.

On Fri, Jan 10, 2020 at 11:07 AM Tim Düsterhus  wrote:

> Chris,
>
> Am 10.01.20 um 16:17 schrieb Chris Software:
> > I am looking at the tags v2.1.1 and v2.1.2 on your main git repository
> > listed here:  http://git.haproxy.org/?p=haproxy-2.1.git and don't think
> > they have been synchronized to your github repository here:
> > https://github.com/haproxy/haproxy/
> >
> > Is there an automatic sychronization process that has failed and needs to
> > be restarted?  Perhaps they need to be pushed manually?
>
> The GitHub repository is a mirror of the development repository
> (http://git.haproxy.org/?p=haproxy.git). It does not contain the
> backported commits within the stable branches and by extension it does
> not contain the tags either.
>
> Best regards
> Tim Düsterhus
>

New haproxy tags v2.1.1 and v2.1.2 not pushed to github

[Chris Software]

Hello,

I am looking at the tags v2.1.1 and v2.1.2 on your main git repository
listed here:  http://git.haproxy.org/?p=haproxy-2.1.git and don't think
they have been synchronized to your github repository here:
https://github.com/haproxy/haproxy/

Is there an automatic sychronization process that has failed and needs to
be restarted?  Perhaps they need to be pushed manually?

Thank you,

Chris

Re: customize format of haproxy X-ForwardedFor ssl_c_s_dn during SSL termination

[Chris Software]

Hello haproxy,

My team member has completed a patch which allows someone to* configure* the
format of the proxied certificate's DN and Issuer DN.

I have attached the patch here.  How can we submit this for inclusion to
haproxy?

Chris



On Fri, Dec 20, 2019 at 12:53 PM Chris Software 
wrote:

> Hello,
>
> This is an update on the offchance that some diligent team member is
> spinning their wheels on this.
>
> Some team members of mine are modifying the haproxy ssl.c file to make the
> format of the ssl_c_s_dn  variable configurable, and editing for simplicity
> to use standard openssl function calls.
>
> There is discussion about submitting this change back as a patch.
>
> Chris
>
>
>
>
>


0001-Adds-support-for-generating-RFC2253-DNs-using-config.patch
Description: Binary data

RE: customize format of haproxy X-ForwardedFor ssl_c_s_dn during SSL termination

[Chris Software]

Hello,

This is an update on the offchance that some diligent team member is
spinning their wheels on this.

Some team members of mine are modifying the haproxy ssl.c file to make the
format of the ssl_c_s_dn  variable configurable, and editing for simplicity
to use standard openssl function calls.

There is discussion about submitting this change back as a patch.

Chris

customize format of haproxy X-ForwardedFor ssl_c_s_dn during SSL termination

[Chris Software]

I am running haproxy in an Alpine Docker container. It is doing SSL
termination for https and injecting the client DN into the X-ForwardedFor
HTTP Header. But the format it uses for the client DN is not one that my
application supports.

Can I change the format somehow, perhaps using openssl.cnf? People
apparently change encryption algorithm settings that way. Can I change my
DN format also?

This is the format that I need my certificate DN to look like in the
X-ForwardedFor header. It is rather LDAP-ey.

CN=Jane Smith,OU=org1,OU=org2,O=myorg,C=AU

But haproxy is injecting a rather ASN.1 looking format

/C=AU/O=myorg/OU=org2/OU=org1/CN=Jane Smith

These are the verisons of the software. They can be changed, as I am
compiling haproxy from source. I can also set any flags at compilation time:

   1. Haproxy 2.0
   2. Alpine 3.10
   3. openssl 1.1.1

Here are what I think are the relevant parts of the haproxy.cfg file .

frontend fe
mode http
bind *:443 ssl no-sslv3 no-tls10 no-tlsv11 crt /certs/mycert
ca-file /certs/myca
option forwardfor
http-request set-header X-ForwardedFor %{+Q+E}[ssl_c_s_dn]
default_backend be

backend be
   balance source
   mode http
   server server1 IP:PORT ca-file /certs/myca crt /certs/mycert ssl verify none

Is there something I can do to change the format? I have tried using the
documented structure like this: %{+Q+E}[ssl_c_s_dn(CN)] but the format of
my certificate DNs is very disorganized. There is no way to predict how
many OU, C, O, etc there might be, and sometimes they are missing. So I
don't think that is a viable solution.

I have also looked at how this question: haproxy tls hash algorithm

customizes
haproxy behavior with openssl settings. Can I do that to get the DN into a
different format? If so, how? I am not sure what steps to follow. Do I need
to modify openssl.cnf at compile time, or have it changed at runtime on the
server? What sections and values?

Thank you!

Re: [PATCH] BUILD: common: Add __ha_cas_dw fallback for single threaded builds

[Chris Packham]

On 12/05/19 5:29 AM, Willy Tarreau wrote:
> On Fri, May 10, 2019 at 11:52:31AM +0200, Willy Tarreau wrote:
>>> Actually I think there's an additional change needed in my patch. By
>>> passing the parameters to HA_ATOMIC_CAS we end up attempting to
>>> dereference a void *. So this should needs to cast to a proper type. For
>>> what it's worth I'll send a v2 that does this.
>>
>> OK, but since it's already merged, please send an incremental patch.
> 
> Don't waste your time, finally I fixed it definitely.
> 
> Willy
> 

Thanks Willy.

Re: [PATCH] BUILD: common: Add __ha_cas_dw fallback for single threaded builds

[Chris Packham]

On 10/05/19 8:57 PM, Willy Tarreau wrote:
> On Thu, May 09, 2019 at 05:07:40PM +1200, Chris Packham wrote:
>> __ha_cas_dw() is used in fd_rm_from_fd_list() and when built without
>> USE_THREADS=1 the linker fails to find __ha_cas_dw(). Add a definition
>> of __ha_cas_dw() for the #ifndef USE_THREADS case.
> 
> Just found your patch, I think it's indeed OK to fall back to
> HA_ATOMIC_CAS in this case since we won't use atomic instructions.
> I'd like that we do a bit of tidying in this area so that it's
> clearer which functions are always atomic and which ones possibly
> are not, but for now that's OK. I've merged it now.

Actually I think there's an additional change needed in my patch. By 
passing the parameters to HA_ATOMIC_CAS we end up attempting to 
dereference a void *. So this should needs to cast to a proper type. For 
what it's worth I'll send a v2 that does this.

[PATCHv2] BUILD: common: Add __ha_cas_dw fallback for single threaded builds

[Chris Packham]

__ha_cas_dw() is used in fd_rm_from_fd_list() and when built without
USE_THREADS=1 the linker fails to find __ha_cas_dw(). Add a definition
of __ha_cas_dw() for the #ifndef USE_THREADS case.

Signed-off-by: Chris Packham 
---
Changes in v2:
- cast to int * to avoid dereferencing void *

 include/common/hathreads.h | 5 +
 1 file changed, 5 insertions(+)

diff --git a/include/common/hathreads.h b/include/common/hathreads.h
index cae6eabe..7314293a 100644
--- a/include/common/hathreads.h
+++ b/include/common/hathreads.h
@@ -140,6 +140,11 @@ static inline void __ha_barrier_full(void)
 {
 }
 
+static inline int __ha_cas_dw(void *target, void *compare, void *set)
+{
+   return HA_ATOMIC_CAS((int *)target, (int *)compare, (int *)set);
+}
+
 static inline void thread_harmless_now()
 {
 }
-- 
2.21.0

Re: Link error building haproxy-1.9.7

[Chris Packham]

On 9/05/19 9:50 PM, William Lallemand wrote:
> On Thu, May 09, 2019 at 03:52:45AM +0000, Chris Packham wrote:
>> Hi,
>>
>> I'm encountering the following linker error when building haproxy-1.9.7
>>
>> make CC=arm-softfloat-linux-gnueabi USE_OPENSSL=1
>> ...
>> LD  haproxy
>>   
>> /usr/bin/../lib/gcc/arm-softfloat-linux-gnueabi/8.3.0/../../../../arm-softfloat-linux-gnueabi/bin/ld:
>> src/fd.o: in function `fd_rm_from_fd_list':
>>haproxy-1.9.7/src/fd.c:267: undefined reference to `__ha_cas_dw'
>>collect2: error: ld returned 1 exit status
>>Makefile:994: recipe for target 'haproxy' failed
>>make: *** [haproxy] Error 1
>>
>> Eyeballing the code I think it's because USE_THREAD is not defined and
>> __ha_cas_dw is only defined when USE_THREAD is defined
>>
>>
> 
> HAProxy is not supposed to build without a TARGET argument, I can't reproduce
> your problem, what is your complete make line?
> 

Here's the full make invocation (MUA wrapped unfortunately)

make -j32 -l16 CC=arm-unknown-linux-gnueabihf-gcc 
LD=arm-unknown-linux-gnueabihf-gcc 
DESTDIR=output/armv7/haproxy/new/install PREFIX=/usr CFLAGS=-"O2 -g2 
-mtune=cortex-a9 -march=armv7-a -mabi=aapcs-linux 
--sysroot=output/armv7/haproxy/staging 
LDFLAGS=--sysroot=output/armv7/haproxy/staging USE_OPENSSL=1 
SSL_INC=output/armv7/haproxy/staging/usr/include 
SSL_LIB=output/armv7/haproxy/staging/usr/lib TARGET=linux26

[PATCH] BUILD: common: Add __ha_cas_dw fallback for single threaded builds

[Chris Packham]

__ha_cas_dw() is used in fd_rm_from_fd_list() and when built without
USE_THREADS=1 the linker fails to find __ha_cas_dw(). Add a definition
of __ha_cas_dw() for the #ifndef USE_THREADS case.

Signed-off-by: Chris Packham 
---
 include/common/hathreads.h | 5 +
 1 file changed, 5 insertions(+)

diff --git a/include/common/hathreads.h b/include/common/hathreads.h
index cae6eabe..516a66d0 100644
--- a/include/common/hathreads.h
+++ b/include/common/hathreads.h
@@ -140,6 +140,11 @@ static inline void __ha_barrier_full(void)
 {
 }
 
+static inline int __ha_cas_dw(void *target, void *compare, void *set)
+{
+   return HA_ATOMIC_CAS(target, compare, set);
+}
+
 static inline void thread_harmless_now()
 {
 }
-- 
2.21.0

Link error building haproxy-1.9.7

[Chris Packham]

Hi,

I'm encountering the following linker error when building haproxy-1.9.7

   make CC=arm-softfloat-linux-gnueabi USE_OPENSSL=1
   ...
   LD  haproxy
 
/usr/bin/../lib/gcc/arm-softfloat-linux-gnueabi/8.3.0/../../../../arm-softfloat-linux-gnueabi/bin/ld:
 
src/fd.o: in function `fd_rm_from_fd_list':
  haproxy-1.9.7/src/fd.c:267: undefined reference to `__ha_cas_dw'
  collect2: error: ld returned 1 exit status
  Makefile:994: recipe for target 'haproxy' failed
  make: *** [haproxy] Error 1

Eyeballing the code I think it's because USE_THREAD is not defined and 
__ha_cas_dw is only defined when USE_THREAD is defined

[PATCH] CONTRIB: log: emit warning when -sf/-sd cannot parse argument <2nd>

[Chris Lane]

Previously, -sf and -sd command line parsing used atol which cannot
detect errors.  I had a problem where I was doing -sf "$pid1 $pid2 $pid"
and it was sending the gracefully terminate signal only to the first pid.
The change uses strtol and checks endptr and errno to see if the parsing
worked.  It will exit when the pid list is not parsed.
---
 src/haproxy.c | 16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/src/haproxy.c b/src/haproxy.c
index f1a2fb9..68a606d 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -1445,13 +1445,27 @@ static void init(int argc, char **argv)
else
oldpids_sig = SIGTERM; /* terminate 
immediately */
while (argc > 1 && argv[1][0] != '-') {
+   char * endptr = NULL;
oldpids = realloc(oldpids, (nb_oldpids 
+ 1) * sizeof(int));
if (!oldpids) {
ha_alert("Cannot allocate old 
pid : out of memory.\n");
exit(1);
}
argc--; argv++;
-   oldpids[nb_oldpids] = atol(*argv);
+   errno = 0;
+   oldpids[nb_oldpids] = strtol(*argv, 
, 10);
+   if (errno) {
+   ha_alert("-%2s option: failed 
to parse {%s}: %s\n",
+flag,
+*argv, 
strerror(errno));
+   exit(1);
+   } else if (endptr && strlen(endptr)) {
+   while (isspace(*endptr)) 
endptr++;
+   if (*endptr != 0) 
+   ha_alert("-%2s option: 
some bytes unconsumed in PID list {%s}\n",
+flag, endptr);
+   exit(1);
+   }
if (oldpids[nb_oldpids] <= 0)
usage(progname);
nb_oldpids++;
-- 
2.1.1

[PATCH] CONTRIB: log: exit with warning when -sf/-sd cannot parse argument

[Chris Lane]

(from git format-patch -1 as per mail thread [PATCH] CONTRIB: log: emit warning 
when -sf/-sd cannot parse argument)

Previously, -sf and -sd command line parsing used atol which cannot
detect errors.  I had a problem where I was doing -sf "$pid1 $pid2 $pid"
and it was sending the gracefully terminate signal only to the first pid.
The change uses strtol and checks endptr and errno to see if the parsing
worked.  It will exit when the pid list is not parsed.
---
 src/haproxy.c | 15 ++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/src/haproxy.c b/src/haproxy.c
index f1a2fb9..30490b3 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -1445,13 +1445,26 @@ static void init(int argc, char **argv)
else
oldpids_sig = SIGTERM; /* terminate 
immediately */
while (argc > 1 && argv[1][0] != '-') {
+   char * endptr = NULL;
oldpids = realloc(oldpids, (nb_oldpids 
+ 1) * sizeof(int));
if (!oldpids) {
ha_alert("Cannot allocate old 
pid : out of memory.\n");
exit(1);
}
argc--; argv++;
-   oldpids[nb_oldpids] = atol(*argv);
+   errno = 0;
+   oldpids[nb_oldpids] = strtol(*argv, 
, 10);
+   if (errno) {
+   ha_alert("-%2s option: failed 
to parse {%s}: %s\n",
+   (char *)*argv, (char 
*)strerror(errno));
+   exit(1);
+   } else if (endptr && strlen(endptr)) {
+   while (isspace(*endptr)) 
endptr++;
+   if (*endptr != 0) 
+   ha_alert("-%2s option: 
some bytes unconsumed in PID list {%s}\n",
+   flag, endptr);
+   exit(1);
+   }
if (oldpids[nb_oldpids] <= 0)
usage(progname);
nb_oldpids++;
-- 
2.1.1

[PATCH] CONTRIB: log: emit warning when -sf/-sd cannot parse argument

[Chris Lane]

Previously, -sf and -sd command line parsing used atol which cannot
detect errors.  I had a problem where I was doing -sf "$pid1 $pid2 $pid"
and it was sending the gracefully terminate signal only to the first pid.
The change uses strtol and checks endptr and errno to see if the parsing
worked.  It doesn't exit so as not to cause failures but will allow 
trouble-shooting
to be faster.
---
 src/haproxy.c | 13 -
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/src/haproxy.c b/src/haproxy.c
index eb5e65b..3185a2e 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -1412,13 +1412,24 @@ static void init(int argc, char **argv)
else
oldpids_sig = SIGTERM; /* terminate 
immediately */
while (argc > 1 && argv[1][0] != '-') {
+   char * endptr = NULL;
oldpids = realloc(oldpids, (nb_oldpids 
+ 1) * sizeof(int));
if (!oldpids) {
ha_alert("Cannot allocate old 
pid : out of memory.\n");
exit(1);
}
argc--; argv++;
-   oldpids[nb_oldpids] = atol(*argv);
+   errno = 0;
+   oldpids[nb_oldpids] = strtol(*argv, 
, 10);
+   if (errno) {
+   ha_alert("-%2s option: failed 
to parse {%s}: %s\n",
+(char *)*argv, 
strerror(errno));
+   } else if (endptr && strlen(endptr)) {
+   while (isspace(*endptr)) 
endptr++;
+   if (*endptr != 0) 
+   ha_alert("-%2s option: 
some bytes unconsumed in PID list {%s}\n",
+flag, endptr);
+   }
if (oldpids[nb_oldpids] <= 0)
usage(progname);
nb_oldpids++;
-- 
2.1.1

Re: CVE-2016-2183 vulnerability

[Chris Short]

Try this to help you build a good config:
https://mozilla.github.io/server-side-tls/ssl-config-generator/

On Thursday, October 20, 2016, Ed Guizar <guiza...@securecloudsystems.com>
wrote:

> We are running HAProxy version 1.5.14.  Our external PCI can failed due to
> CVE-2016-2183 vulnerability.  Apparently we need to disable DES and 3DES,
> at least that is what I understand from reading the description.  How can I
> accomplish this?
>
>
>
> I have the following entry in the Global configuration:
>
>
>
> ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:
> ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:
> ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-
> DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-
> SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:
> ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-
> AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-
> SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-
> SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-
> AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-
> SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:
> DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!
> aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
>
>
>
> ssl-default-server-ciphers ECDHE-RSA-AES128-GCM-SHA256:
> ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:
> ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-
> DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-
> SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:
> ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-
> AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-
> SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-
> SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-
> AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-
> SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:
> DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!
> aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
>
>
>
>
>
> *Ed Guizar*
>
>
>


-- 
Thanks,

Chris Short
https://chrisshort.net
Public PGP Key: http://cshort.co/pub_key
Keybase: http://cshort.co/keybaseio

Trying to stop health check messages from being logged

[Chris White]

I am trying to stop backend health check error messages from being logged to 
the system console.

I have tried using "no option log-health-checks” in my configuration


defaults

  log global

  no option log-health-checks

  mode http

  option  dontlognull


But I am still seeing error messages logged when a backend server goes down


e.g.

2016 May  4 23:11:14 devtest0 backend web has no server available!

2016 May  4 23:12:06 devtest0 backend photos has no server available!


What can I do to prevent these messages from going to the console.

Re: Add servers without disruption

[Chris Warren]

> On 29 Mar 2016, at 18:22, Paul Draper  wrote:
> 
> As I understand it, there seems to no way to add a server to HAProxy without 
> incurring significant disruption. Adding a server requires reloading 
> configuration. This loses all statistics, all health check information, etc. 
> So, for instance, after adding a server, HAProxy will send traffic to 
> unhealthy servers.

1.6.x's server-state-file feature helps haproxy reloads be a lot less 
disruptive (stats and state are maintained across reoads).

We've started using a patch which adds an "initial-state up/down" option to 
each server line (or to the default) to stop any new servers getting traffic 
directed at them before the first healthcheck:

https://github.com/beamly/haproxy-1.6/commit/9e7ad68a0c6582a38591eb27626fdb31bb5f8c18

For scaling-down/removing instances we also use "option redispatch" so any 
requests sent to backends which are shutting down (but not yet failed a 
healthcheck) get resent to a good server.

This seems to give us the behaviour we require in an autoscaled environment.

IDEA: initial-state up/down option for servers

[Chris Warren]

Hi,

We use haproxy in an auto-scaling environment. On an auto-scaling event, the 
haproxy configuration is rewritten to list all existing servers for each 
proxied service. A graceful reload is then performed.

The issue is that by default haproxy assumes a server is UP (going down) until 
the first healthcheck tells it otherwise. If one of the servers for a 
proxy/backend is not yet actually healthy (e.g. in the initial moments after a 
new instance is booted), then some requests will be forwarded to the new 
instance, resulting in 503 responses.

This was a must before 1.6’s server-state-file feature as we’d not want 
everything to be marked down for a few seconds after a reload. However when 
using a server-state-file we know the state of every server other than any new 
ones. We’d like these new ones to be marked DOWN (going up) so they do not 
receive requests until the first healthcheck is passed.

We’re currently testing a patch which adds an “initial-state up/down” option to 
each server (and the default-server option) - the default behaviour remains 
unchanged:
https://github.com/beamly/haproxy-1.6/commit/9e7ad68a0c6582a38591eb27626fdb31bb5f8c18

I’m wondering if this is something that could be considered for a future 
haproxy release?

Many thanks,
Chris

Re: IDEA: initial-state up/down option for servers

[Chris Warren]

On 18 Mar 2016, at 03:03, Igor Cicimov 
<ig...@encompasscorporation.com<mailto:ig...@encompasscorporation.com>> wrote:

On Fri, Mar 18, 2016 at 10:38 AM, Chris Warren 
<ch...@beamly.com<mailto:ch...@beamly.com>> wrote:
Hi,

We use haproxy in an auto-scaling environment. On an auto-scaling event, the 
haproxy configuration is rewritten to list all existing servers for each 
proxied service. A graceful reload is then performed.

The issue is that by default haproxy assumes a server is UP (going down) until 
the first healthcheck tells it otherwise. If one of the servers for a 
proxy/backend is not yet actually healthy (e.g. in the initial moments after a 
new instance is booted), then some requests will be forwarded to the new 
instance, resulting in 503 responses.

I use "slowstart 60s" to give the new instance enough time to boot up before 
HAP starts the health checks. Not sure though if I understood your issue 
correctly.



Thanks Igor,

We tried slowstart, but unfortunately it doesn’t seem to be applied to servers 
when haproxy starts (the docs agree).

I also double-checked the code - and unless I’m missing something, slowstart 
only applies when an instance is in SRV_ST_STARTING state 
(http://git.haproxy.org/?p=haproxy-1.6.git;a=blob;f=src/queue.c;h=1f27c498b0a08cafee61e816dada2f4dc1a425f7;hb=HEAD#l53)
 but new instances on startup default to SRV_ST_RUNNING 
(http://git.haproxy.org/?p=haproxy-1.6.git;a=blob;f=src/server.c;h=4891d2dcda049b3fb4317d09c5f2e4362cb9bbe9;hb=HEAD#l899)
 until the first check is performed.

Thanks again,
Chris

Re: Limiting the size of the backend queues

[Chris White]

Ideally I would like to return 429 Too Many Requests.
 





-Original Message-
From: Willy Tarreau <w...@1wt.eu>
Date: Tuesday, February 9, 2016 at 11:27 PM
To: Chris White <chris.wh...@illumio.com>
Cc: "haproxy@formilux.org" <haproxy@formilux.org>
Subject: Re: Limiting the size of the backend queues

>On Tue, Feb 09, 2016 at 03:45:15PM +, Chris White wrote:
>> I have been looking at the documentation for haproxy and I can???t seem to
>> find anyway of limiting the size of each backend proxy queue. I can see how
>> to limit the queue size and number of connections handle by each of the
>> backend servers.  I can see how to configure the backend queue timeout, but
>> no way of actually limiting the total number of requests queued per backend
>> proxy. I really want to put limits on how many requests are queued per
>> backend proxy, is there anyway of doing this?
>
>Limiting is one thing, but what matters is what happens once the limit is
>reached. Are you sure you want to return a 503 in such a case ?
>
>Willy
>

Limiting the size of the backend queues

[Chris White]

I have been looking at the documentation for haproxy and I can’t seem to find 
anyway of limiting the size of each backend proxy queue. I can see how to limit 
the queue size and number of connections handle by each of the backend servers. 
 I can see how to configure the backend queue timeout, but no way of actually 
limiting the total number of requests queued per backend proxy. I really want 
to put limits on how many requests are queued per backend proxy, is there 
anyway of doing this?

Chris White

BUG/MINOR: examples: Fixing haproxy.spec to remove references to .cfg files

[Chris Short]

Building RPMs from the provided haproxy.spec fails due to references to
config files that do not exist. Please see attached patch for suggested fix.

I should point out, this patch will inevitably create an empty /etc/haproxy
dir and I'm not sure that's desirable/intended.

Thanks,

Chris Short
https://chrisshort.net
Public PGP Key: http://cshort.co/pub_key
Keybase: http://cshort.co/keybaseio


0001-BUG-MINOR-examples-Fixing-haproxy.spec-to-remove-ref.patch
Description: Binary data

Re: ip_nonlocal_bind=1 set but sometimes get "cannot bind socket" on reload (-sf)

[Chris Riley]

Hi Vincent,

> SigIgn is correct (SIGPIPE) is ignored. However, SigBlk seems
> incorrect. HAProxy only blocks signals when dequeuing them. However, > no 
> signal
is pending either, so they should be delivered? Maybe it was
> bad luck? If you try again, does SigBlk become 0?

No matter how many times I send signals to this process, SigBlk remains as
fffe7bfa7a26.

I did find something that seems odd though when I was looking at the file
descriptors that this PID has open.

When I run lsof I see this (output filtered for brevity, excludes linked
libraries):

COMMAND PIDUSER   FD  TYPE DEVICE SIZE/OFF
NODE NAME
haproxy   11537 haproxy0u 0,90
5350 anon_inode
haproxy   11537 haproxy4u unix 0x88023306d200  0t0
126464 /tmp/haproxy.sock.11536.tmp
haproxy   11537 haproxy5u IPv4 126465  0t0
 TCP 192.168.10.15:http (LISTEN)
haproxy   11537 haproxy6u IPv4 126466  0t0
 UDP *:58171
haproxy   11537 haproxy7u IPv4 126467  0t0
 TCP 192.168.200.100:http (LISTEN)
haproxy   11537 haproxy8u IPv4 126468  0t0
 TCP 192.168.200.120:http (LISTEN)
haproxy   11537 haproxy9u IPv4 126469  0t0
 TCP 192.168.200.110:http (LISTEN)

The first odd thing is that /tmp/haproxy.sock.11536.tmp does not exist. The
socket file that stats socket uses is defined as /tmp/haproxy.sock in
haproxy.cfg. And it's either odd or a coincidence that the number in the
filename is 1 digit lower than the PID.

After running lsof I took a look at /proc/11537/fd

ls -l /proc/11537/fd
total 0
lrwx-- 1 root root 64 Oct 30 15:29 0 -> anon_inode:[eventpoll]
lrwx-- 1 root root 64 Oct 30 15:29 4 -> socket:[126464]
lrwx-- 1 root root 64 Oct 30 15:29 5 -> socket:[126465]
lrwx-- 1 root root 64 Oct 30 15:29 6 -> socket:[126466]
lrwx-- 1 root root 64 Oct 30 15:29 7 -> socket:[126467]
lrwx-- 1 root root 64 Oct 30 15:29 8 -> socket:[126468]
lrwx-- 1 root root 64 Oct 30 15:29 9 -> socket:[126469]

It won't show here but all of those symlinks are non-existent (flashing
red). And each one of those symlinks corresponds to FD listed in the output
of lsof that shows haproxy actively listening on those sockets. (?) And if
I connect to any of those IPs:ports they pass traffic.

In looking at tcp_bind_listener() in proto_tcp.c (comment on line 782) it
indicates that the desired behavior is to reuse file descriptors instead of
creating a new socket. Would these orphaned file descriptors indicate that
a new socket was created instead of the file descriptors being reused? I'm
wondering if that is the case due to the behavior I saw with kernel 2.6 and
the "cannot bind socket" message on reloads and that due to the
SO_REUSEPORT in kernel 3.9 and later, additional processes are allowed to
bind to these same IPs:ports and that may be masking the issue on 3.9+
kernels.

The other thing I'm wondering is if signal_unregister_handler()
in sig_soft_stop in haproxy.c has removed all of the signal handlers that
this PID would otherwise be listening for and that's why it is unresponsive
to anything other than -SIGKILL. soft_stop() is called right
before signal_unregister_handler() so is it possible that something went
sideways while executing soft_stop(), leaving this PID sort of in limbo?

Regards,
Chris


On Fri, Oct 30, 2015 at 3:28 PM, Vincent Bernat <ber...@luffy.cx> wrote:

>  ❦ 30 octobre 2015 15:14 -0400, Chris Riley <cri...@rentpath.com> :
>
> > SigQ: 3/63840
> > SigPnd: 
> > SigBlk: fffe7bfa7a26
> > SigIgn: 1000
> > SigCgt: 000180300205
>
> SigIgn is correct (SIGPIPE) is ignored. However, SigBlk seems
> incorrect. HAProxy only blocks signals when dequeuing them. However, no
> signal is pending either, so they should be delivered? Maybe it was bad
> luck? If you try again, does SigBlk become 0?
> --
> Don't stop with your first draft.
> - The Elements of Programming Style (Kernighan & Plauger)
>

Re: ip_nonlocal_bind=1 set but sometimes get "cannot bind socket" on reload (-sf)

[Chris Riley]

Hi Willy,

The permissions where one of the first things I checked. consul-template
runs as root in order to be able to reload/restart daemon and it's using
the same init script that the system uses on startup. Not all of the
reloads fail, the first few initial ones are successful. What's odd is that
the behavior goes away when I failover all IPs to one server and
set net.ipv4.ip_nonlocal_bind=0. After that all reloads are successful, no
matter how many times in a row reload is called. The issue remains at bay
even after failing half of the IPs back over to the secondary server and
setting net.ipv4.ip_nonlocal_bind=1 again. That is until the servers
reboot, then the behavior returns. Vincent got me thinking about the 2.6.32
kernel that is part of CentOS 6.4. I'm wondering if
net.ipv4.ip_nonlocal_bind behaves oddly in 2.6.x with respect to the status
of existing socket file descriptors. I'm going to try kernel 3.10 from
CentOS 7 to see if I can reproduce it in 3.10 in order to rule out or
confirm an issue with the kernel.

However, I'm not sure that's the issue. When a reload fails there is
nothing in the log file that indicates that haproxy saw SIGTTOU or SIGUSR1
("Pausing %s %s." and "Stopping %s %s in %d ms."). I can reproduce this
behavior if I don't provide a PID to -sf. When looking at the code
in proxy.c it looks like pause_proxy() is either not being called
by pause_proxies in haproxy.c (due to the missed SIGTTOU) or in
pause_proxy() the proxy state check is returning 1 at the top of the
pause_proxy() function. I'm going to add some additional logging statements
to see if I can isolate what's happening.

Regards,
Chris


On Fri, Oct 30, 2015 at 3:11 AM, Willy Tarreau <w...@1wt.eu> wrote:

> On Fri, Oct 30, 2015 at 08:04:48AM +0100, Vincent Bernat wrote:
> >  ??? 30 octobre 2015 00:34 -0400, Chris Riley <cri...@rentpath.com> :
> >
> > > The kernel version is 2.6.32-358.23.2.el6.x86_64, the OS is CentOS
> > > 6.4.
> >
> > With this version of the kernel, the previous instance of HAProxy has to
> > release the port before the new one can bind. It seems that in your
> > case, this doesn't happen. Nothing suspicious in the logs of the
> > previous instance?
>
> It would be nice to ensure the process is reloaded with appropriate
> permissions. The new process indeed needs to send a signal to the old
> process, and bind to the ports. If any of these operations fail, it will
> not be able to start.
>
> Willy
>
>

Re: ip_nonlocal_bind=1 set but sometimes get "cannot bind socket" on reload (-sf)

[Chris Riley]

Hi Vincent,

What's odd is that if I failover all virtual IPs to one server and
set net.ipv4.ip_nonlocal_bind=0 on that server the issue goes away. The
issue remains "fixed" when I fail half of the virtual IPs back to the
secondary server and set net.ipv4.ip_nonlocal_bind=1. However, after a
reboot of both servers the initial behavior comes back. This seems to be
something related to the way the 2.6.32 kernel handles
net.ipv4.ip_nonlocal_bind and how it relates to the sockets' file
descriptors.

The logs don't show anything suspicious. When a reload is successful I see
the expected output in the logs:

Oct 30 09:49:53 127.0.0.1 haproxy[26191]: Proxy haproxy-stats started.
Oct 30 09:50:22 127.0.0.1 haproxy[26192]: Pausing proxy haproxy-stats.
Oct 30 09:50:22 127.0.0.1 haproxy[26215]: Proxy haproxy-stats started.
Oct 30 09:50:22 127.0.0.1 haproxy[26192]: Stopping proxy haproxy-stats in 0
ms.
Oct 30 09:50:22 127.0.0.1 haproxy[26192]: Proxy haproxy-stats stopped (FE:
0 conns, BE: 0 conns).

When a reload is unsuccessful the code that pauses, starts a new proxy, and
stops the original proxy isn't called so there is no output in the logs.
Instead the Alert (cannot bind socket) is sent to stderr and is logged by
consul-template.

I'm going to compile the 3.10 kernel from CentOS 7 for CentOS 6 and see if
the behavior persists and report back.

Thanks,
Chris


On Fri, Oct 30, 2015 at 3:04 AM, Vincent Bernat <ber...@luffy.cx> wrote:

>  ❦ 30 octobre 2015 00:34 -0400, Chris Riley <cri...@rentpath.com> :
>
> > The kernel version is 2.6.32-358.23.2.el6.x86_64, the OS is CentOS
> > 6.4.
>
> With this version of the kernel, the previous instance of HAProxy has to
> release the port before the new one can bind. It seems that in your
> case, this doesn't happen. Nothing suspicious in the logs of the
> previous instance?
> --
> Let us endeavor so to live that when we come to die even the undertaker
> will be
> sorry.
> -- Mark Twain, "Pudd'nhead Wilson's Calendar"
>

Re: ip_nonlocal_bind=1 set but sometimes get "cannot bind socket" on reload (-sf)

[Chris Riley]

Hi Willy,

Thanks for your quick reply.

> It should work better but will very likely hide the root cause. I suspect
> you'll find two processes running after a reload because the old one
> doesn't stop then.

Yep, that's exactly what I'm seeing with 3.10. I've got a bunch of haproxy
processes stacked up, all with an -sf flag and each being passed the PID of
the previous process. The only one not in the process list is the initial
haproxy instance created when 'service haproxy start' was first run.

lb-01 ~ [qa] # ps ax | grep hap
 2834 ?Ss 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg
-p /var/run/haproxy.pid -sf 2822
 2871 ?Ss 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg
-p /var/run/haproxy.pid -sf 2859
 2883 ?Ss 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg
-p /var/run/haproxy.pid -sf 2871
 2910 ?Ss 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg
-p /var/run/haproxy.pid -sf 2896
 2922 ?Ss 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg
-p /var/run/haproxy.pid -sf 2910
 2947 ?Ss 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg
-p /var/run/haproxy.pid -sf 2934
 2959 ?Ss 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg
-p /var/run/haproxy.pid -sf 2947
 2962 pts/1S+ 0:00 grep --colour=auto hap


> That would confirm the possibility that the signal is not sent at all,
> or at least not to the right process. Could you check the exact command
> that is started, to ensure the pids are correct (or present at all) ?
> Can you also try by hand to first send SIGUSR1 to the old process,
> then perform the reload, then send SIGTTOU by hand to the old one ?
> If it works, it would confirm an issue with the ability to send a
> signal to the old process from the new one.

'service haproxy start' invokes this:

daemon $exec -D -f /etc/$prog/$prog.cfg -p /var/run/$prog.pid

which produces:

usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid

ps ax shows:

2822 ?Ss 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg
-p /var/run/haproxy.pid

cat /var/run/haproxy.pid shows:
2822

'service haproxy reload' invokes this:

$exec -D -f /etc/$prog/$prog.cfg -p /var/run/$prog.pid -sf $(cat
/var/run/$prog.pid)

which produces:

2834 ?Ss 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg
-p /var/run/haproxy.pid -sf 2822

cat /var/run/haproxy.pid then shows:
2834

I'll try manually sending SIGUSR1 and SIGTTOU as you suggested and see if I
can determine what's happening.

Any chance that this issue is related to or that same as this one?

https://github.com/haproxy/haproxy/issues/48

Regards,
Chris

On Fri, Oct 30, 2015 at 11:50 AM, Willy Tarreau <w...@1wt.eu> wrote:

> Hi Chris,
>
> On Fri, Oct 30, 2015 at 11:18:30AM -0400, Chris Riley wrote:
> > Hi Willy,
> >
> > The permissions where one of the first things I checked. consul-template
> > runs as root in order to be able to reload/restart daemon and it's using
> > the same init script that the system uses on startup. Not all of the
> > reloads fail, the first few initial ones are successful. What's odd is
> that
> > the behavior goes away when I failover all IPs to one server and
> > set net.ipv4.ip_nonlocal_bind=0.
>
> That's really strange.
>
> > After that all reloads are successful, no
> > matter how many times in a row reload is called. The issue remains at bay
> > even after failing half of the IPs back over to the secondary server and
> > setting net.ipv4.ip_nonlocal_bind=1 again. That is until the servers
> > reboot, then the behavior returns. Vincent got me thinking about the
> 2.6.32
> > kernel that is part of CentOS 6.4. I'm wondering if
> > net.ipv4.ip_nonlocal_bind behaves oddly in 2.6.x with respect to the
> status
> > of existing socket file descriptors.
>
> No, nonlocal_bind hasn't changed for a while, what was brought later (3.9)
> was SO_REUSEPORT which haproxy always uses, so that makes it easier to
> rebind regardless of the presence of an old process.
>
> > I'm going to try kernel 3.10 from
> > CentOS 7 to see if I can reproduce it in 3.10 in order to rule out or
> > confirm an issue with the kernel.
>
> It should work better but will very likely hide the root cause. I suspect
> you'll find two processes running after a reload because the old one
> doesn't
> stop then.
>
> > However, I'm not sure that's the issue. When a reload fails there is
> > nothing in the log file that indicates that haproxy saw SIGTTOU or
> SIGUSR1
> > ("Pausing %s %s." and "Stopping %s %s in %d ms."). I can reproduce this
> > behavior if I don't provide a PID to -sf. When looking at the code
> > in proxy.c it looks like pause_proxy() is eit

Re: ip_nonlocal_bind=1 set but sometimes get "cannot bind socket" on reload (-sf)

[Chris Riley]

Hi Vincent,

Thanks for your quick reply. I tried with 3.10 and the reloads don't "fail"
as they do with 2.6 but the stacking up of haproxy processes that Willy
mentioned might occur does happen. It looks like for some reason the new
process is having an issue sending signals to the old process. I'm
following Willy's suggestions to try to determine what may be happening.

Regards,
Chris

On Fri, Oct 30, 2015 at 10:54 AM, Vincent Bernat <ber...@luffy.cx> wrote:

>  ❦ 30 octobre 2015 10:50 -0400, Chris Riley <cri...@rentpath.com> :
>
> > I'm going to compile the 3.10 kernel from CentOS 7 for CentOS 6 and
> > see if the behavior persists and report back.
>
> With a 3.10, you are unlikely to get the same behaviour as two processes
> are allowed to listen to the same IP/port. So, if it's a possibility for
> you, it is likely to solve the problem.
> --
> Harp not on that string.
> -- William Shakespeare, "Henry VI"
>

Re: ip_nonlocal_bind=1 set but sometimes get "cannot bind socket" on reload (-sf)

[Chris Riley]

Hi Vincent,

> It stays in the epoll loop even while you are sending the signals? Could
> you also check what you get with "grep '^Sig' /proc/PID/status"?

Yes, it stays in the epoll loop when sending any signal other than
-SIGKILL. I'm poking around with gdb to see if I can see anything. This is
the backtrace:

(gdb) bt
#0  0x003bf84e8f23 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1  0x0049e886 in _do_poll ()
#2  0x00406a3b in run_poll_loop ()
#3  0x00409e61 in main ()

And here is the output of grep '^Sig' /proc/11537/status

SigQ:   3/63840
SigPnd: 
SigBlk: fffe7bfa7a26
SigIgn: 1000
SigCgt: 000180300205

Regards,
Chris

On Fri, Oct 30, 2015 at 3:09 PM, Vincent Bernat <ber...@luffy.cx> wrote:

>  ❦ 30 octobre 2015 14:50 -0400, Chris Riley <cri...@rentpath.com> :
>
> > Good idea. I just tried and it appears to be in an epoll_wait loop.
> > This is after sending the PID SIGTTOU and SIGUSR1. SIGTERM also has no
> > effect, the process stays in this epoll_wait loop.
> >
> > strace -p11537
> > Process 11537 attached - interrupt to quit
> > epoll_wait(0, {}, 200, 1000) = 0
> > epoll_wait(0, {}, 200, 1000) = 0
> > epoll_wait(0, {}, 200, 1000) = 0
> > epoll_wait(0, {}, 200, 1000) = 0
> > epoll_wait(0, {}, 200, 1000) = 0
> > epoll_wait(0, {}, 200, 1000) = 0
> > epoll_wait(0, ^C 
> > Process 11537 detached
>
> It stays in the epoll loop even while you are sending the signals? Could
> you also check what you get with "grep '^Sig' /proc/PID/status"?
> --
> When one burns one's bridges, what a very nice fire it makes.
> -- Dylan Thomas
>

Re: ip_nonlocal_bind=1 set but sometimes get "cannot bind socket" on reload (-sf)

[Chris Riley]

gdb shows this when attached to the same process:

0x003bf84e8f23 in __epoll_wait_nocancel () from /lib64/libc.so.6

Regards,
Chris

On Fri, Oct 30, 2015 at 2:38 PM, Vincent Bernat <ber...@luffy.cx> wrote:

>  ❦ 30 octobre 2015 14:36 -0400, Chris Riley <cri...@rentpath.com> :
>
> > When the processes stack up, the old ones don't respond to anything
> > other than 'kill -9'.
>
> You could try to strace them to check where they currently are.
> --
> If more of us valued food and cheer and song above hoarded gold, it would
> be a merrier world.
> -- J.R.R. Tolkien
>

Re: ip_nonlocal_bind=1 set but sometimes get "cannot bind socket" on reload (-sf)

[Chris Riley]

Hi Willy,

I tried manually sending SIGUSR1 and SIGTTOU as you suggested but I had
mixed results. Sometimes the procs would do what I expected and sometimes
they wouldn't.

> Looks similar indeed. RHEL has selinux enabled by default I believe, I
> don't know if that could prevent haproxy from sending a signal to
> another process. Maybe you can try to stop it (if enabled at all).

selinux is disabled on these hosts. I tried with an infinite while loop
like in the github post but I can't force it to happen. I tried rapid fire,
with a 1 second pause, and with a small random rewrite to haproxy.cfg
before the reload (to mimic a change written out by consul-template) and no
matter how long I let the while loop run, the previous process always
cleanly exists - with and without traffic passing through it (stats page
continually refreshing)

while :; do service haproxy reload; done

while :; do service haproxy reload; sleep 1; done

while :; do /etc/haproxy/test.sh; service haproxy reload; sleep 1; done

test.sh:

#!/bin/bash
sed "s/<>/$RANDOM/" /etc/haproxy/haproxy.cfg.tmpl >
/etc/haproxy/haproxy.cfg

I'm going to take a closer look at the source for consul-template (
https://github.com/hashicorp/consul-template) to see if it manipulates the
command provided in the template configuration or if it blindly passes it
to the shell. This is what the config looks like for consul-template:

template {
  source = "/etc/consul-template/templates/haproxy.cfg.ctmpl"
  destination = "/etc/haproxy/haproxy.cfg"
  command = "/sbin/service haproxy reload || true"
}

I will let you know if I find anything of interest.

Regards,
Chris

On Fri, Oct 30, 2015 at 1:20 PM, Willy Tarreau <w...@1wt.eu> wrote:

> On Fri, Oct 30, 2015 at 12:23:57PM -0400, Chris Riley wrote:
> > Hi Willy,
> >
> > Thanks for your quick reply.
> >
> > > It should work better but will very likely hide the root cause. I
> suspect
> > > you'll find two processes running after a reload because the old one
> > > doesn't stop then.
> >
> > Yep, that's exactly what I'm seeing with 3.10. I've got a bunch of
> haproxy
> > processes stacked up, all with an -sf flag and each being passed the PID
> of
> > the previous process. The only one not in the process list is the initial
> > haproxy instance created when 'service haproxy start' was first run.
> >
> > lb-01 ~ [qa] # ps ax | grep hap
> >  2834 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2822
> >  2871 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2859
> >  2883 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2871
> >  2910 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2896
> >  2922 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2910
> >  2947 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2934
> >  2959 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2947
> >  2962 pts/1S+ 0:00 grep --colour=auto hap
>
> These ones don't all match, in fact only the last one is fine. That's
> strange.
>
> > > That would confirm the possibility that the signal is not sent at all,
> > > or at least not to the right process. Could you check the exact command
> > > that is started, to ensure the pids are correct (or present at all) ?
> > > Can you also try by hand to first send SIGUSR1 to the old process,
> > > then perform the reload, then send SIGTTOU by hand to the old one ?
> > > If it works, it would confirm an issue with the ability to send a
> > > signal to the old process from the new one.
> >
> > 'service haproxy start' invokes this:
> >
> > daemon $exec -D -f /etc/$prog/$prog.cfg -p /var/run/$prog.pid
> >
> > which produces:
> >
> > usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
> >
> > ps ax shows:
> >
> > 2822 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid
> >
> > cat /var/run/haproxy.pid shows:
> > 2822
> >
> > 'service haproxy reload' invokes this:
> >
> > $exec -D -f /etc/$prog/$prog.cfg -p /var/run/$prog.pid -sf $(cat
> > /var/run/$prog.pid)
>
> OK so there is no reason for it not to work.
>
> > which produces:
> >
> > 2834 ?Ss

Re: ip_nonlocal_bind=1 set but sometimes get "cannot bind socket" on reload (-sf)

[Chris Riley]

When the processes stack up, the old ones don't respond to anything other
than 'kill -9'.

On Fri, Oct 30, 2015 at 1:20 PM, Willy Tarreau <w...@1wt.eu> wrote:

> On Fri, Oct 30, 2015 at 12:23:57PM -0400, Chris Riley wrote:
> > Hi Willy,
> >
> > Thanks for your quick reply.
> >
> > > It should work better but will very likely hide the root cause. I
> suspect
> > > you'll find two processes running after a reload because the old one
> > > doesn't stop then.
> >
> > Yep, that's exactly what I'm seeing with 3.10. I've got a bunch of
> haproxy
> > processes stacked up, all with an -sf flag and each being passed the PID
> of
> > the previous process. The only one not in the process list is the initial
> > haproxy instance created when 'service haproxy start' was first run.
> >
> > lb-01 ~ [qa] # ps ax | grep hap
> >  2834 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2822
> >  2871 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2859
> >  2883 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2871
> >  2910 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2896
> >  2922 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2910
> >  2947 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2934
> >  2959 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2947
> >  2962 pts/1S+ 0:00 grep --colour=auto hap
>
> These ones don't all match, in fact only the last one is fine. That's
> strange.
>
> > > That would confirm the possibility that the signal is not sent at all,
> > > or at least not to the right process. Could you check the exact command
> > > that is started, to ensure the pids are correct (or present at all) ?
> > > Can you also try by hand to first send SIGUSR1 to the old process,
> > > then perform the reload, then send SIGTTOU by hand to the old one ?
> > > If it works, it would confirm an issue with the ability to send a
> > > signal to the old process from the new one.
> >
> > 'service haproxy start' invokes this:
> >
> > daemon $exec -D -f /etc/$prog/$prog.cfg -p /var/run/$prog.pid
> >
> > which produces:
> >
> > usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
> >
> > ps ax shows:
> >
> > 2822 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid
> >
> > cat /var/run/haproxy.pid shows:
> > 2822
> >
> > 'service haproxy reload' invokes this:
> >
> > $exec -D -f /etc/$prog/$prog.cfg -p /var/run/$prog.pid -sf $(cat
> > /var/run/$prog.pid)
>
> OK so there is no reason for it not to work.
>
> > which produces:
> >
> > 2834 ?Ss 0:00 /usr/sbin/haproxy -D -f
> /etc/haproxy/haproxy.cfg
> > -p /var/run/haproxy.pid -sf 2822
> >
> > cat /var/run/haproxy.pid then shows:
> > 2834
> >
> > I'll try manually sending SIGUSR1 and SIGTTOU as you suggested and see
> if I
> > can determine what's happening.
>
> Yes that will definitely help.
> >
> > Any chance that this issue is related to or that same as this one?
> >
> > https://github.com/haproxy/haproxy/issues/48
>
> Looks similar indeed. RHEL has selinux enabled by default I believe, I
> don't know if that could prevent haproxy from sending a signal to another
> process. Maybe you can try to stop it (if enabled at all).
>
> Regards,
> Willy
>
>

ip_nonlocal_bind=1 set but sometimes get "cannot bind socket" on reload (-sf)

[Chris Riley]

Hello,

I'm seeing some inconsistent/strange behavior with HAProxy (1.5.14 and
1.6.1) not being able to bind to a socket despite
'net.ipv4.ip_nonlocal_bind = 1' being set. HAProxy starts up without issue
initially but after several reloads, the command 'service haproxy reload'
starts failing and reports that HAProxy "cannot bind socket" for each of
the listens/frontends, even for IPs that ARE on that server. The existing
HAProxy process contiues to run without picking up the new changes.

After the reload initially fails all subsequent 'service haproxy reload'
commands also fail. Running 'service haproxy restart' restarts and
immediately binds to the IPs:ports specified in each listen/frontend that
it just complained that it could not bind to.

Here's some background info. There are two servers (lb-01 and lb-02).
Virtual IPs are managed by keepalived (v1.2.19) in two vrrp_instances. Each
vrrp_instance contains half of the total virtual IPs. The first
vrrp_instance has lb-01 defined as MASTER and lb-02 as BACKUP and the
second vrrp_instance has lb-02 defined as MASTER and lb-01 as BACKUP. This
allows each server to act as failover for the other server. This was tested
extensively while I was adding support for ip rules to keepalived and works
without issue. All of HAProxy's configuration is stored in consul (v0.5.2).
consul-template (v0.11.1) writes out /etc/haproxy/haproxy.cfg using the
data in consul and then consul-template calls 'service haproxy reload'. The
OS is CentOS 6.4 and the kernel version is 2.6.32-358.23.2.el6.x86_64.

Here is an example of what I'm seeing (actual IPs have been substituted).
192.168.10.0/24 IPs are assigned to eth0 and 192.168.200.0/24 IPs are
assigned to eth1. (output is from lb-02)

Reloading haproxy: [ALERT] 301/141300 (25939) : Starting proxy
haproxy-stats: cannot bind socket [192.168.10.27:80]
[ALERT] 301/141300 (25939) : Starting proxy haproxy-fe1: cannot bind socket
[192.168.200.100:80]
[ALERT] 301/141300 (25939) : Starting proxy haproxy-fe2: cannot bind socket
[192.168.200.120:80]
[ALERT] 301/141300 (25939) : Starting proxy haproxy-fe3: cannot bind socket
[192.168.200.110:80]

What's strange is that HAProxy is already listening to these IPs:port so it
seems to be some kind of race condition. Of these IPs, 192.168.10.27 is
statically assigned to eth0 and is the only IP assigned to that interface.
192.168.200.110 and 192.168.200.120 are assigned to eth1 on lb-02.
192.168.200.100 is assigned to eth1 on lb-01. Without setting
'net.ipv4.ip_nonlocal_bind = 1' I would expect to see "cannot bind socket"
for 192.168.200.100 but it doesn't make any sense that HAProxy also reports
that it cannot bind on IPs:ports that are assigned to that server.

Does anyone have ideas as to why this might occur?

Best Regards,
Chris Riley

Re: Failed tcp connection attempts after upgrading to 1.5

[Chris Gilmore]

Amazon Linux (kernel version 3.14)
In addition to the failed connection attempts, I also noticed that I was
getting an equal number of resets received for embryonic SYN_RECV sockets.


On Sat, May 16, 2015 at 4:40 AM, Willy Tarreau w...@1wt.eu wrote:

 On Wed, May 13, 2015 at 10:19:56AM -0500, Chris Gilmore wrote:
  I've found the commit that caused this change in behavior.
  From 1.4.23:
  - MEDIUM: checks: avoid accumulating TIME_WAITs during checks
 
  commit fd29cc537b8511db6e256529ded625c8e7f856d0
 
  So, it appears it is a feature and not a bug.

 Yes indeed, otherwise you cannot connect anymore from the local ports to
 the servers. What OS are you running on ? At least on Linux we can
 disable the TCP quick-ack which allows us to close the connection
 before the connection's ACK is sent, and it's clean because the
 server sees SYN, SYN-ACK, RST and doesn't even create a socket nor
 notifies the application.

 Willy

Re: Failed tcp connection attempts after upgrading to 1.5

[Chris Gilmore]

I've found the commit that caused this change in behavior.
From 1.4.23:
- MEDIUM: checks: avoid accumulating TIME_WAITs during checks

commit fd29cc537b8511db6e256529ded625c8e7f856d0

So, it appears it is a feature and not a bug.



On Thu, May 7, 2015 at 3:30 AM, Chris Gilmore chris.gilm...@gmail.com
wrote:

 I've recently upgraded from haproxy 1.4.22 to 1.5.2.
 Ever since then, I have noticed that I am now getting failed tcp
 connection attempts on the remote servers.  The rate is equal to the
 'inter' value (for example, 1 failed connection attempt every 2 seconds).
 Should I be concerned about this?  Other than that, everything appears to
 be working properly.

 The haproxy configuration sets up a tcp health check for a set of mysql
 servers (note: I am not using the mysql-check option).
 Below is my haproxy.cfg

 global
 log 127.0.0.1 local2

 chroot  /var/lib/haproxy
 pidfile /var/run/haproxy.pid
 maxconn 4000
 userhaproxy
 group   haproxy
 daemon

 stats socket /var/lib/haproxy/stats

 defaults
 modetcp
 log global
 option  dontlognull
 option  redispatch
 retries 3
 timeout queue   1m
 timeout connect 10s
 timeout client  1m
 timeout server  1m
 timeout check   10s
 maxconn 3000

 listen mysql
 bind :3306
 mode tcp
 balance leastconn
 server first 10.###.###.###:3306 check
 server second 10.###.###.###:3306 check
 server firstbackup 10.###.###.###:3306 check backup
 server secondbackup 10.###.###.###:3306 check backup

Failed tcp connection attempts after upgrading to 1.5

[Chris Gilmore]

I've recently upgraded from haproxy 1.4.22 to 1.5.2.
Ever since then, I have noticed that I am now getting failed tcp connection
attempts on the remote servers.  The rate is equal to the 'inter' value
(for example, 1 failed connection attempt every 2 seconds).  Should I be
concerned about this?  Other than that, everything appears to be working
properly.

The haproxy configuration sets up a tcp health check for a set of mysql
servers (note: I am not using the mysql-check option).
Below is my haproxy.cfg

global
log 127.0.0.1 local2

chroot  /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
userhaproxy
group   haproxy
daemon

stats socket /var/lib/haproxy/stats

defaults
modetcp
log global
option  dontlognull
option  redispatch
retries 3
timeout queue   1m
timeout connect 10s
timeout client  1m
timeout server  1m
timeout check   10s
maxconn 3000

listen mysql
bind :3306
mode tcp
balance leastconn
server first 10.###.###.###:3306 check
server second 10.###.###.###:3306 check
server firstbackup 10.###.###.###:3306 check backup
server secondbackup 10.###.###.###:3306 check backup

Re: Complete rewrite of HAProxy in Lua

[Chris Allen]

Great one :)

I was 10 seconds away from forwarding this to our web teams :)



On 01/04/2015 09:43, Willy Tarreau wrote:

Hi,

As some might have noticed, HAProxy development is progressively slowing
down over time. I have analyzed the situation and came to the following
conclusions :

   - the code base is increasing and is becoming slower to build day


Due to the huge amount of work, we'll postpone the 1.6 release to 1st
April 2016, which leaves us exactly 366 days to complete this task. I
hope everyone understands that we have no other choice.

Now I have to go back to work, there's no time to waste.

Say us good luck!
Willy

Re: Peer causing segfault?

[Chris Kopp]

Thank you very much!

Chris



 On Feb 14, 2015, at 05:19, Willy Tarreau w...@1wt.eu wrote:
 
 Hi Chris,
 
 On Fri, Feb 13, 2015 at 01:02:22AM +0100, Lukas Tribus wrote:
 service haproxy status returns:
 haproxy dead but pid file exists
 
 and /var/log/messages shows:
 Feb 12 22:46:20 ip-10-72-128-136 kernel: [100695.296333] haproxy[32373]:
 segfault at 8 ip 0046b030 sp 7fff2372f4c0 error 6 in
 haproxy[40+b5000]
 
 Can you provide the output of haproxy -vv, coredump or backtraces
 (see http://permalink.gmane.org/gmane.comp.web.haproxy/7992)?
 
 Thanks for the backtraces that you passed to Lukas, they were very
 helpful. The bug is stupid and is the result of an incomplete change
 introduced in 1.6. I've fixed it and pushed the fix upstream. Here
 comes the patch attached if you're interested.
 
 Best regards,
 Willy
 
 0001-BUG-MAJOR-peers-initialize-s-buffer_wait-when-creati.patch

Peer causing segfault?

[Chris]

I've been trying to get peers to work with HAProxy.  The following works
(meaning that a single server starts and runs):
  peers hapeers
  #  peer ip-10-72-128-49 10.72.128.49:1024
peer ip-10-72-128-136 10.72.128.136:1024

  backend common_rate_table
stick-table type ip size 500k peers hapeers expire 5m store conn_rate(1m)


as soon as I uncomment the other haproxy server in the peers section:

  peers hapeers
peer ip-10-72-128-49 10.72.128.49:1024
peer ip-10-72-128-136 10.72.128.136:1024

  backend common_rate_table
stick-table type ip size 500k peers hapeers expire 5m store conn_rate(1m)


service haproxy status returns:
haproxy dead but pid file exists

and /var/log/messages shows:
Feb 12 22:46:20 ip-10-72-128-136 kernel: [100695.296333] haproxy[32373]:
segfault at 8 ip 0046b030 sp 7fff2372f4c0 error 6 in
haproxy[40+b5000]

If it makes a difference, the stick-table in the dummy common_rate_table
backend is shared between several backends.

Thank you,

Chris

Re: tcp-response inspect-delay with WAIT_END

[Chris]

Baptiste bedis9@... writes:

 Hi Chris,
 
 Could you let us know why exactly you need to delay responses???
 
 Because here you propose a response (which doesn't work) to a problem
 you're facing without explaining us the problem.
 So it's hard to help.

Baptiste,

You say I propose a response which doesn't work..  The statement is
ambiguous.  Do you mean that HAProxy is *not* intended to delay responses
and will not do so?  ..or are you simply referring to the fact that I have
not gotten it to work?

I am essentially trying to find out if tcp-response inspect-delay works
with WAIT_END.  Can I delay a response?  Am I using it correctly
syntactically?  Or do I need to find a way to delay responses outside of
HAProxy?

I do not know if I should continue waiting for a response or if you feel you
have answered my question adequately.

Thank you,

Chris

Re: tcp-response inspect-delay with WAIT_END

[Chris]

 Could you let us know why exactly you need to delay responses???

This is an API.

Unfortunately, the client behavior we are looking to address here cannot be
identified by client IP, ID, or anything else in the request.  In fact, it
cannot be identified until the server has gone through considerable work to
building up the response.

We still want to answer the client, but we know that as soon as we do the
client will come back with other, similar requests.  We cannot typically
slow subsequent requests by IP as this will affect large numbers of clients
behind proxy servers.  We have found that the only reasonable way to slow
the client's behavior is to slow the response.  This has proven to be quite
effective in our particular situation.

Currently the server delays the response after it is built, but instead we
would like to be able to add a magic header to the response that directs
HAProxy to delay it.

Is this possible?  Playing in a sandbox environment I have not been able to
get HAProxy to delay a response using tcp-response inspect-delay.  It
either answers immediately or the client times out.


 
acl trigger_delay res.hdr(response-delay) -m found
tcp-response inspect-delay 100ms
tcp-response content accept unless trigger_delay
tcp-response content accept if WAIT_END
 

tcp-response inspect-delay with WAIT_END

[Chris]

Hello,

We have some complex logic in our application that will at times determine
that the response to a specific query should be delayed.  Currently this is
handled in the application with a short (~100ms) sleep.  

We would like to move this delay in response to the load balancer.  I have
tried to do this by adding a response header as a flag for HAProxy to act on
and adding configuration like the following to the backend:

  acl trigger_delay res.hdr(response-delay) -m found
  tcp-response inspect-delay 100ms
  tcp-response content accept unless trigger_delay
  tcp-response content accept if WAIT_END

With the above configuration, the response is delayed until the client times
out (2 minutes) regardless of how trigger_delay evaluates.

The following configurations exhibit the same behavior:

  tcp-response inspect-delay 100ms
  tcp-response content accept if WAIT_END

- or - 

  acl trigger_delay res.hdr(response-delay) -m found
  tcp-response inspect-delay 100ms
  tcp-response content accept unless trigger_delay
  

It seems that either a header-based ACL or WAIT_END cause any tcp-response
inpect-delay to timeout.  It does not seem to matter if the header-based ACL
returns true or false.  Are they not compatible with a response delay?


Ideally when we encounter the delay flag in the response of the app server,
we would also add the src to a stick-table for reference in delaying
subsequent incoming connections from that IP (maybe the next, say 5 minutes
or so).  Is this possible/reasonable?

Thank you,

Chris

Re: Better understanding of nbproc vs distributing interrupts for cpu load management

[Chris Allen]

On 26/11/2014 00:19, Willy Tarreau wrote:

On Tue, Nov 25, 2014 at 09:33:30PM +, Chris Allen wrote:

On 25/11/2014 18:08, Lukas Tribus wrote:

I think SSL/TLS termination is the only use case where HAProxy
saturates a CPU core of a current generation 3,4Ghz+ CPU, which is why
scaling SSL/TLS is more complex, requiring nbproc 1.
Lukas

Ok that's strange then because we don't have a very complex config.

Does your config use a lot of regex ? This can cost a lot. Also if this
is the case, there's a bug in 1.5.8 which magnifies the time spent
processing regex when they're loaded from files (fixed in 1.5.9). Could
you please also post the output of haproxy -vv in case some optims are
missing.


No, there's one regexp on a rarely-used backend.

Here's our (redacted) config: 
https://gist.github.com/chrisotherwise/cfdd04216d622437f5fe


Here's the output from haproxy -vv: 
https://gist.github.com/chrisotherwise/23067a39fd59449d52f1


Here's the output from sar when we were not too busy: 
https://gist.github.com/chrisotherwise/b2f22379eb5192ea5b45


Here's the output from a pidstat of haproxy: 
https://gist.github.com/acme/02f065091d9f246ceaa3





About 8 front-ends of which only 3 are busy. About 10 back-end pools
each containing
30-60 servers. Things start getting hot when we're serving about 15,000
requests per second (about 1Gb/s bandwidth).

That's not huge.


Is there any way we can find out why we're using so much cpu?

You need to check first how your CPU usage is spread between user, system
and softirq. If you see a lot of user, something is wrong either in haproxy
or in the configuration.

If you can post your config (without the private parts, you can replace IP
addresses, domain names and passwords with ), that will definitely help.


We used
perf to create a flame graph which is here:
http://www.cjx.com/perf-kernel.svg - taken when the system was not very
loaded - perhaps somebody with better
knowledge of haproxy than I can tell if something is awry...

OK that shows a reasonably loaded system. If you're close to 1 Gbps on a
Gig NIC (tg3), you can experience some losses which further increase the
softirq usage.


We have 3 x 1Gb NIC. Things get hot at around 1.1Gb/s





So first, we really need to see this CPU usage breakdown.

Regards,
Willy

Re: Better understanding of nbproc vs distributing interrupts for cpu load management

[Chris Allen]

On 25/11/2014 15:46, Emeric Brun wrote:

On 11/24/2014 05:46 PM, Chris Allen wrote:

We have a load balancer running haproxy 1.5.8. At times of heavy load
we're getting dangerously close
to running out of CPU. I'd be really grateful for some definitive
opinions on the relative merits
of the two possible solutions detailed below - as I'm having trouble
finding detailed and consistent
information on them.


1. Nbproc

Our server has 8 cores. We have two primary frontends. We could run
Nbproc=8 assigning
cores 1,2,3,4 to the first frontend and 5,6,7,8 to the other one. But
several documents
say that running Nbproc is not such a great idea (we totally understand
the issue with
stats and don't do admin socket updates or any of the dangerous
stuff). If we're ok
with that, is Nbproc a reasonable choice?

Related to nbproc:

- If we do run with nbproc, should we be pinning network interrupts or
just letting irqbalance
handle them? The loadbalancer has 3 bonded NICS generating 15 IRQs - but
no particular NIC
is related to any particular frontend.

- If I bind a frontend to multiple processes, only one of those
processes listens on the TCP
port that I have specified. Yet the other processes seem to receive
requests too. How does
that work? And is the primary process any more loaded for doing the
listening?



2. Distributing interrupts

One of our techops team found an old haproxy.com blog that suggested a
good solution for
balancing cpu load was simply by distributing network interrupts across
available cores
but leaving haproxy to run as a single process. We have tried this and
indeed the *average*
cpu load appears shared across all proceses - but logic tells me that
haproxy must still be
loading each cpu on which it runs and will saturate them (even though
the overall average
looks good). Can anybody comment on this approach as our team member is
convinced
he has a silver bullet for performance.


Many thanks for any help/insight into this!





Hi,

Indeed on haproxy.com appliances, we made the choice to dedicate several
cores/CPUs for NICs interrupts (manual bind, no usage of irqbalance) and
to dedicate one other for the haproxy process with nbproc=1.

This configuration offers the most stable performances.

That's also why we recommend for HAPEE users (haproxy entreprise 
edition) to choose CPUs with the maximum freq instead numerous cores.


Regards,
Brun Emeric



Hi,

Thanks for your reply. We have tried this approach and while it gives 
some benefit, the haproxy process itself
remains cpu-bound, with no idle time at all - with both pidstat and perf 
reporting that it uses close to 100%

of available cpu while running.

Re: Better understanding of nbproc vs distributing interrupts for cpu load management

[Chris Allen]

On 25/11/2014 18:08, Lukas Tribus wrote:
 I think SSL/TLS termination is the only use case where HAProxy 
saturates a CPU core of a current generation 3,4Ghz+ CPU, which is why 
scaling SSL/TLS is more complex, requiring nbproc 1.



Lukas


Ok that's strange then because we don't have a very complex config. 
About 8 front-ends of which only 3 are busy. About 10 back-end pools 
each containing
30-60 servers. Things start getting hot when we're serving about 15,000 
requests per second (about 1Gb/s bandwidth).


Is there any way we can find out why we're using so much cpu? We used 
perf to create a flame graph which is here:
http://www.cjx.com/perf-kernel.svg - taken when the system was not very 
loaded - perhaps somebody with better

knowledge of haproxy than I can tell if something is awry...

Better understanding of nbproc vs distributing interrupts for cpu load management

[Chris Allen]

We have a load balancer running haproxy 1.5.8. At times of heavy load 
we're getting dangerously close
to running out of CPU. I'd be really grateful for some definitive 
opinions on the relative merits
of the two possible solutions detailed below - as I'm having trouble 
finding detailed and consistent

information on them.


1. Nbproc

Our server has 8 cores. We have two primary frontends. We could run 
Nbproc=8 assigning
cores 1,2,3,4 to the first frontend and 5,6,7,8 to the other one. But 
several documents
say that running Nbproc is not such a great idea (we totally understand 
the issue with
stats and don't do admin socket updates or any of the dangerous 
stuff). If we're ok

with that, is Nbproc a reasonable choice?

Related to nbproc:

- If we do run with nbproc, should we be pinning network interrupts or 
just letting irqbalance
handle them? The loadbalancer has 3 bonded NICS generating 15 IRQs - but 
no particular NIC

is related to any particular frontend.

- If I bind a frontend to multiple processes, only one of those 
processes listens on the TCP
port that I have specified. Yet the other processes seem to receive 
requests too. How does
that work? And is the primary process any more loaded for doing the 
listening?




2. Distributing interrupts

One of our techops team found an old haproxy.com blog that suggested a 
good solution for
balancing cpu load was simply by distributing network interrupts across 
available cores
but leaving haproxy to run as a single process. We have tried this and 
indeed the *average*
cpu load appears shared across all proceses - but logic tells me that 
haproxy must still be
loading each cpu on which it runs and will saturate them (even though 
the overall average
looks good). Can anybody comment on this approach as our team member is 
convinced

he has a silver bullet for performance.


Many thanks for any help/insight into this!

AUTO: Chris Spitzlay ist außer Haus (Rückkehr am 08.09.2014)

[Chris Spitzlay]

Ich kehre zurück am 08.09.2014.


Bei dringenden Angelegenheiten wenden Sie sich bitte an den Pallas
Helpdesk.

Montag-Freitag 09:00 bis 18:00 Uhr
02232-1896 96
helpd...@pallas.com



Hinweis: Dies ist eine automatische Antwort auf Ihre Nachricht
Notificação de devolucao de cheque 25/08  08:07 (20497) gesendet am
25.08.2014 04:47:03.

Diese ist die einzige Benachrichtigung, die Sie empfangen werden, während
diese Person abwesend ist.



 pallas
Pallas GmbH / Hermülheimer Str. 8a / 50321 Brühl
Geschäftsführer: Dr. Kurt Brand, Stephan Sachweh 
HR B 52019 Amtsgericht Köln

Managed Security Service - Einfach sicher
http://www.xing.com/companies/pallasgmbh
- Save the Date: Firmenfest Palladium 2014: 26.09.14, 17:00 -

Re: haproxy 1.5-dev24: 100% CPU Load or Core Dumped

[Chris Burroughs]

On 05/07/2014 12:35 PM, Vincent Bernat wrote:

  ❦  7 mai 2014 11:15 +0200, Willy Tarreau w...@1wt.eu :


haproxy does not include DTrace probes by any chance right? :)


No, and I have no idea how this works either. But if you feel like it
can provide some value and be done without too much effort, feel free
to try :-)


Here is a proof of concept. To test, use `make TARGET=linux2628
USE_DTRACE=1`. On Linux, you need systemtap-sdt-dev or something like
that. Then, there is a quick example in example/haproxy.stp. You can try
it like this:

#+begin_src sh
sudo stap  ./examples/haproxy.stp
#+end_src



This is a really cool proof of concept.  Thank you for sharing Vincent.

Re: [ANNOUNCE] haproxy-1.5-dev25

[Chris Burroughs]

On 05/10/2014 09:44 AM, Willy Tarreau wrote:

Now the bind-process mess is fixed so that we now support per-listener
process binding using the process bind keyword, which ensures that
we won't need to change the config format during the stable release if
we want to slightly improve it. And that allows us to have one stats
socket per process, finally!


Huzzah!

mode tcp connections in FIN_WAIT2

[chris somme]

Hello,

We're running 1.5-dev19 2013/06/17 and running ssh connections through
haproxy in mode tcp.

Recently we ran into a problem where we had a lot of client connections to
haproxy sitting around in FIN_WAIT2 and the corresponding connections to
the backend in in CLOSE_WAIT. It appears that our backend has closed the
connections but the socket isn't being shudown by the client, and haproxy
is holding the connection to the backend open until the client connection
is shut down. Those sockets eventually prevented us from accepting new
connections, the sockets that are in the CLOSE_WAIT state count against
maxconn.

Once the connection reaches the client timeout haproxy closes the
connection to the backend and the socket in CLOSE_WAIT goes away.

We're on linux and have have tried setting tcp_fin_timeout, hoping that it
would shutdown the connections in FIN_WAIT2 and in turn shutdown the
sockets in CLOSE_WAIT. The tcp_fin_timeout value doesn't seem to have an
effect -- it's currently set to 30 seconds, but socket in FIN_WAIT2 linger
beyond that.

Is there anyway besides the client timeout (which impacts all client
interactions, not just close) to get the backend connections int mode tcp
freed up sooner?

Thanks,
Chris

hdr_ip occ?

[Chris Burroughs]

http://comments.gmane.org/gmane.comp.web.haproxy/5856

'''Since we have not yet reworked the ACLs to rely on the pattern 
subsystem, it's still not possible to make use of hdr_ip(X-f-f,-1) as 
we do on the balance or source keywords.'''


Is the ACL rework mentioned for hddr_ip and a specific occurance (ie 
only the last one to avoid trivial spoofing) part of 1.5?

cR, Tq, timeout http-request

[Chris Burroughs]

We are running 1.4.24 for an application that sees almost entirely small 
http requests.  We have the following timeouts:

timeout client   7s
timeout server   4s
timeout connect  4s
timeout http-request 7s

There are a significant number of cR/http-408 responses in the logs. 
The docs say that should represent timeouts from timeout http-request. 
 My expectation was that in those cases the total request time would be 
equal to  timeout http-request.  However, a full logging trace shows 
some are shorter, and many are longer, than the timeout.


# NumSamples = 143006; Min = 221.00; Max = 13230.00
# Mean = 10706.382585; Variance = 9305084.888424; SD = 3050.423723
# each * represents a count of 1141
  221. -  1521.9000 [ 2]:
 1521.9000 -  2822.8000 [  7402]: **
 2822.8000 -  4123.7000 [  1899]: *
 4123.7000 -  5424.6000 [  6737]: *
 5424.6000 -  6725.5000 [  4155]: ***
 6725.5000 -  8026.4000 [  4391]: ***
 8026.4000 -  9327.3000 [  7942]: **
 9327.3000 - 10628.2000 [  9794]: 
10628.2000 - 11929.1000 [ 15101]: *
11929.1000 - 13230. [ 85583]: 
***

Keep-alive and websocket connections

[Chris Yang]

Dear all,

In the latest HAProxy 1.5 release (dev22), it is indicated that
keep-alive is now enabled by default for both client and server sides.
I have some questions regarding its use in the following scenario.

I use HAProxy in front of an array of servers: one nginx for
delivering static files, and the others being application servers. One
of the application servers exclusively deals with websocket (or in the
event of ws failure, switching to streaming) connections. Currently, I
am using 'http-server-close' by default for all servers, but I think
it'd be better to switch to 'http-keep-alive' for the nginx and keep
'http-server-close' for the websockets server.

Is this a correct setup? Thanks.

Best,

Chris

Re: Keep-alive and websocket connections

[Chris Yang]

Thanks for your suggestion, Lukas.

For my own understanding, are you saying that there is no difference
between having http-keep-alive and having http-server-close to a
backend server once websocket connection to that server is establish,
and both settings allow for establishing websocket connection
perfectly.

So is there any advantage of having http-keep-alive to a websocket backend?


On Thu, Feb 6, 2014 at 4:56 PM, Lukas Tribus luky...@hotmail.com wrote:
 Hi,


 In the latest HAProxy 1.5 release (dev22), it is indicated that
 keep-alive is now enabled by default for both client and server sides.
 I have some questions regarding its use in the following scenario.

 I use HAProxy in front of an array of servers: one nginx for
 delivering static files, and the others being application servers. One
 of the application servers exclusively deals with websocket (or in the
 event of ws failure, switching to streaming) connections. Currently, I
 am using 'http-server-close' by default for all servers, but I think
 it'd be better to switch to 'http-keep-alive' for the nginx and keep
 'http-server-close' for the websockets server.

 You can just default to http-keep-alive everywhere.

 HAProxy recognizes the upgrade headers and switches to TCP mode
 automatically [1].

 Recognizing the upgrade in a HTTP transaction is possible with all modes
 expect tcp mode (of course) and (the pre-dev22 default) http tunnel mode [2].



 Is this a correct setup? Thanks.

 It is, but you may as well simplify it with http-keep-alive on all sections.

 I don't see any advantage by configuring http-server-close on the websocket
 backend.

 Of course you should test this, before putting it in production.



 Regards,

 Lukas



 [1] 
 http://blog.exceliance.fr/2012/11/07/websockets-load-balancing-with-haproxy/
 [2] http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4

Re: HAProxy Next?

[Chris Burroughs]

On 2013-12-17 03:14, Annika Wickert wrote:
 - accesslist for statssocket or ldap authentication for stats socket

For ldap auth I presume you mean the web ui.  You could accomplish this
today by proxying through httpd (or equivalent).

Re: Improving behaviour when nbproc 1

[Chris Burroughs]

On 12/04/2013 11:24 AM, Chris Burroughs wrote:

On 12/03/2013 04:07 PM, Chris Burroughs wrote:

This could just be me not being adept at email patches.  Sorry if this
is obvious but is this supposed to apply against 1.4 or 1.5?


To answer my own question this applies against 1.5.  I'm not sure of the
feasibility or desirability of backporting to 1.4.

I ran with this in a production load test and as far I can tell it
worked as advertised.  We were able to run with nbproc and still have
useful looking stats sockets for haptop, ganglia etc.  At least in our
use case, stats sockets with this patch solve the primary objection to
running with nbproc.



Happpy to test additional patches or variants if further review is 
needed.  Is there anything blockng this getting accepted into 1.5?

Re: [ANNOUNCE] haproxy-1.5-dev20

[Chris Burroughs]

On 12/15/2013 09:41 PM, Willy Tarreau wrote:

   - several hash algorithms are provided, and it is possible to select them
 per backend. This high quality work was done at Tumblr by Bhaskar Maddala.


This is an interesting development.  How do the included functions 
compare to the currenlty en vogue Murmur or CityHash?

Re: Improving behaviour when nbproc 1

[Chris Burroughs]

On 12/09/2013 05:02 PM, James Hogarth wrote:

To answer yes it is against 1.5 ... The caveats are the peers don't work
and the session table and load balancing can get messed up due to the lack
of shared information between processes but if you just need to utilise
multiple stat sockets and the rest doesn't matter so much then it works
nicely.


But thesse are the same caveats that nbproc has always had, not new 
ones, correct?

Re: Multiprocess Loadsharing

[Chris Burroughs]

On 11/28/2013 03:10 AM, Annika Wickert wrote:

Is this a normal behaviour?


http://imgur.com/I7sRWy2

A graph of similar behavior at nbproc=3.  Anecdotally the variance seems 
to be higher under lower loads.

Re: Improving behaviour when nbproc 1

[Chris Burroughs]

On 12/03/2013 04:07 PM, Chris Burroughs wrote:

This could just be me not being adept at email patches.  Sorry if this
is obvious but is this supposed to apply against 1.4 or 1.5?


To answer my own question this applies against 1.5.  I'm not sure of the 
feasibility or desirability of backporting to 1.4.


I ran with this in a production load test and as far I can tell it 
worked as advertised.  We were able to run with nbproc and still have 
useful looking stats sockets for haptop, ganglia etc.  At least in our 
use case, stats sockets with this patch solve the primary objection to 
running with nbproc.

Re: %usr/sys nbproc

[Chris Burroughs]

On 12/04/2013 02:10 AM, Willy Tarreau wrote:

We happen to have another CPU we purchased to be good with highly
threaded Java apps: Intel Xeon CPU E5-2670 0 @ 2.60GHz

It also has a L2 cache per core.  This CPU has performed significantly
better in both many and a few threaded workloads.  Somewhat
surprisingly with a single haproxy I'm only able to get to around 23 k
req/s (vs 20k with the much older Opteron).

Could you please run top during the test, and press 1 to have the
per-cpu measures ? Simply copy-paste it and send it here so that we can
check what to improve.


This is with the simple no nbproc setup on the E5-2670.  So cpu0 has 
interprets, and haproxy is pinned cpu1.  SSL terminators and whatnot run 
on the other cores.  top snapshot was taken at the point the lb could 
handle no more load and the Queue was growing.


top - 11:51:42 up  1:55,  3 users,  load average: 0.68, 0.55, 0.39
Tasks: 212 total,   2 running, 210 sleeping,   0 stopped,   0 zombie
Cpu0  : 19.7%us,  1.7%sy,  0.0%ni, 10.5%id,  0.0%wa,  0.0%hi, 68.1%si, 
0.0%st
Cpu1  : 26.9%us, 72.1%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  1.0%si, 
0.0%st
Cpu2  : 24.7%us,  2.0%sy,  0.0%ni, 73.3%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st
Cpu3  : 17.7%us,  1.3%sy,  0.0%ni, 80.9%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st
Cpu4  :  8.4%us,  1.0%sy,  0.0%ni, 90.6%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st
Cpu5  :  3.3%us,  1.0%sy,  0.0%ni, 95.7%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st
Cpu6  :  2.7%us,  0.3%sy,  0.0%ni, 97.0%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st
Cpu7  :  2.0%us,  0.3%sy,  0.0%ni, 97.7%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st

Mem:  32838864k total,  5523920k used, 27314944k free,26876k buffers
Swap: 67108856k total,0k used, 67108856k free,   700888k cached

Re: %usr/sys nbproc

[Chris Burroughs]

On 11/26/2013 07:25 AM, Chris Burroughs wrote:


As far as I can tell from AMD docs and Vincent's handy /sys trick, each
of the 6 cores has a fully independent L2 cache, and the chip has a
single shared L3 cache.

I'm not sure I'm following the part about the same part of the L3
cache.  Are you saying that some cores are closer to each other on
the L3 cache, like NUMA?


These CPUs seem to be designed for VM hosting, or running highly
threaded Java apps which don't need much FPU. I'm not certain they
were optimized for network processing unfortunately, which is sad
considering that their older brothers were extremely fast at that.



Highly threaded Java apps happens to be what most of our servers are
used for and what we benchmarked for purchasing decisions.


We happen to have another CPU we purchased to be good with highly 
threaded Java apps: Intel Xeon CPU E5-2670 0 @ 2.60GHz


It also has a L2 cache per core.  This CPU has performed significantly 
better in both many and a few threaded workloads.  Somewhat 
surprisingly with a single haproxy I'm only able to get to around 23 k 
req/s (vs 20k with the much older Opteron).

Re: Improving behaviour when nbproc 1

[Chris Burroughs]

This is also something that I think we would likely find very helpful in 
1.4.


On 11/20/2013 11:32 AM, Avatar wrote:

Ou, we've been waiting it so much. Really delightful thing.
Thanks.

On Mon, Nov 18, 2013 at 9:49 PM, James Hogarth james.hoga...@gmail.com wrote:

Hi all,

We've been looking at improving the behaviour when nbproc  1

We focused on making the unix control socket have deterministic rather than
almost random behaviour (depending on which process picks up the incoming
request).

There still remains larger issues with nbproc which may affect your decision
to use it (each process maintaining it's own session persistence, maxconns,
separated stats, etc) but this at least allows for some control of the
sessions when doing so.

The general idea was to make use of existing functionality to minimise code
additions or modifications and as such there is very little it touches that
could cause a problem in itself (indeed setting multiple stats socket lines
already created bindings to multiple control sockets but just to all
processes).

To control all processes a wrapper of some nature is still needed to iterate
through all the sockets to gather data or send commands to each one.

To make use of this functionality just have one stats socket line per
process defined in nbproc.

As an example:

global
   nbproc 4
   stats socket /var/run/haproxy.1.sock
   stats socket /var/run/haproxy.2.sock
   stats socket /var/run/haproxy.3.sock
   stats socket /var/run/haproxy.4.sock

To stop sample-frontend only for maintenance on all processes:

for {1..4}
do
echo 'disable fontend sample-frontend' | socat /var/run/haproxy.1.sock
done

Kind regards,

James


___


 From e8c72a98b9b626f5c853f00f82dcfb1b6dec7764 Mon Sep 17 00:00:00 2001
From: James Hogarth james.hoga...@gmail.com
Date: Mon, 18 Nov 2013 17:26:53 +
Subject: [PATCH] Add multiple sockets to stats process for the case nbproc 
1

---
  doc/configuration.txt |  2 ++
  src/cfgparse.c|  2 +-
  src/haproxy.c | 19 +++
  3 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index 0b4844b..0802739 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -632,6 +632,8 @@ stats socket [address:port|path] [param*]
All parameters supported by bind lines are supported, for instance to
restrict access to some users or their access rights. Please consult
section 5.1 for more information.
+
+  Multiple stats socket lines can be in global if nbproc is greater than 1
controlling each process.

  stats timeout timeout, in milliseconds
The default timeout on the stats socket is set to 10 seconds. It is
possible
diff --git a/src/cfgparse.c b/src/cfgparse.c
index 28507dd..5507d89 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -7144,7 +7144,7 @@ out_uri_auth_compat:
   /* Check multi-process mode compatibility */
   if (global.nbproc  1) {
   if (global.stats_fe  !global.stats_fe-bind_proc) {
- Warning(stats socket will not work as expected in multi-process mode
(nbproc  1), you should force process binding using 'stats
bind-process'.\n);
+ Warning(stats socket will not work as expected in multi-process mode
(nbproc  1), you should force process binding using 'stats bind-process' or
provide multiple stats sockets lines - one for each process.\n);
   }
   }

diff --git a/src/haproxy.c b/src/haproxy.c
index bc03a73..a15b81c 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -1563,6 +1563,25 @@ int main(int argc, char **argv)
   px = px-next;
   }

+ /* unbind unix control sockets leaving one on each process for
deterministic control */
+ {
+ int ListenCount = 1;
+ struct proxy * px = proxy;
+
+ while (px != NULL) {
+ if (strcmp(px-id , GLOBAL) == 0) {
+ struct listener * l;
+ list_for_each_entry(l, px-conf.listeners, by_fe) {
+ if(ListenCount != relative_pid  !(px-bind_proc)){
+unbind_listener(l);
+ }
+ ListenCount++;
+ }
+ }
+ px = px-next;
+ }
+ }
+
   if (proc == global.nbproc) {
   if (global.mode  MODE_SYSTEMD) {
   for (proc = 0; proc  global.nbproc; proc++)
--
1.8.4.2

Re: %usr/sys nbproc

[Chris Burroughs]

On 11/23/2013 04:13 AM, Willy Tarreau wrote:

This is 25% user and 75% system. It's on the high side for the user, since
you generally get between 15 and 25% user for 75-85% system, but since you
have logs enabled, it's not really surprizing so yes it's in the norm. You
should be able to slightly improve this by using http-server-close instead
of httpclose. It will actively close server-side connections and save a
few packets.



My understanding was that HAProxy 1.4 does not formally support having 
persistent connections to backends while closing connections to clients. 
 However, if the backend servers used keep alive and HAProxy did not 
force the connection close that this would likely work.  I thought that 
continuing to shove bytes over a small number of existing TCP connection 
ought to be cheaper (in terms of packets, interprets, %sys, etc) than 
setting up and tearing down yet more sockets.




While I don't have an empirical basis for comparison on this
older hardware, 20k req/s also seemed low.


I remember having benchmarked another recent opteron last year (with
many cores, a 7-something) and it performed very poorly, about 18k/s,
much worse than my 4-years old Phenom 9950. One of the reasons was
that it was difficult to share some L3 cache between certain cores.
I found little information on the 4184, except on Wikipedia (to be
taken with a grain of salt) :

http://en.wikipedia.org/wiki/Bulldozer_%28microarchitecture%29

Thus it probably suffers from the same design as the 7xxx, which is
that you need to identify the cores belonging to the same module, so
that they share the same L2 cache and that they are located in the
same part of the share L3 cache, otherwise the inter-core communications
happen via the outside.



As far as I can tell from AMD docs and Vincent's handy /sys trick, each 
of the 6 cores has a fully independent L2 cache, and the chip has a 
single shared L3 cache.


I'm not sure I'm following the part about the same part of the L3 
cache.  Are you saying that some cores are closer to each other on 
the L3 cache, like NUMA?



These CPUs seem to be designed for VM hosting, or running highly
threaded Java apps which don't need much FPU. I'm not certain they
were optimized for network processing unfortunately, which is sad
considering that their older brothers were extremely fast at that.



Highly threaded Java apps happens to be what most of our servers are 
used for and what we benchmarked for purchasing decisions.



Finally assuming the single process performance can not be further
improved I was considering the following setup:
  * core 0: eht0 interrupts
  * core 1: haproxy bound to eth0
  * core 2: eth1 interrupts
  * core 3: haproxy bound to eth1
  * core 4-5: ssl terminator


I definitely agree. I know at least one setup which runs fine this way.
It was a two-socket system, each with its own NIC and process. But here
you're in the same situation, consider that you have 3 independant CPUs
in the same box. The benefit of doing it this way is that you can still
parallelize network interrupts to multiple cores without having the
response traffic come to the wrong core (proxies are a hell to optimize
because of their two sides).



This setup (haproxy per NIC) was able to handle 50% more load than a 
single haproxy. So from about 20k req/s to 30k.  This is very nice bump 
with with what would otherwise be mostly idle cpu cores.  We found this 
to be very complex to setup at the IP layer though (which isn't 
haproxy's fault but in our particular circumstances might not be worth it).





But I could not find too many examples of similar setups and was unsure
if it was a viable long term configuration.


Yes it is viable. The only limit right now is that you'll need to start
two processes. In the future, when listeners reliably support the
bind-process keyword, it will even be possible to centralize
everything and have a dedicated stats socket for each.

In the mean time I suggest that you have two processes with almost the
same config except interfaces. Note that haproxy supports binding to
interfaces.


For reasons that could be completely incidental to our networking, I was 
unable to get bind *:80 interface eth0 to consistently work and had to 
do bind $IP:80 interface eth0.  With the first one the instance bound 
to eth0 would answer requests that were coming on on eth1.




Otherwise, all your config below looks fine.


Thank you for looking.  I and several of my colleagues have found this 
thread most helpful.

understanding client timeouts

[Chris Burroughs]

I'm trying to track down a problem with cR and CR timeouts with 
haproxy 1.4.  A service I thought was nice and stable turns out to have 
# 4xx/# 2xx ~= 0.2 according to hatop (which pulls in data from the 
stats socket).  Close to 20% client timeouts is far higher than I expected.


The logs look like so:

NAME/NOSRV -1/-1/-1/-1/61001 408 212 - - cR-- 31/31/3/0/0 0/0 BADREQ
NAME/NOSRV -1/-1/-1/-1/5650 400 187 - - CR-- 29/29/3/0/0 0/0 BADREQ
NAME/NOSRV -1/-1/-1/-1/5701 400 187 - - CR-- 28/28/3/0/0 0/0 BADREQ
NAME/NOSRV -1/-1/-1/-1/5716 400 187 - - CR-- 24/24/0/0/0 0/0 BADREQ
NAME/NOSRV -1/-1/-1/-1/5559 400 187 - - CR-- 23/23/0/0/0 0/0 BADREQ
NAME/NOSRV -1/-1/-1/-1/5432 400 187 - - CR-- 22/22/0/0/0 0/0 BADREQ

We usually run with configurations like so:
option   httpclose
option   http-server-close
timeout client   3s
timeout server   3s
timeout connect  5s
timeout http-request 2s

but have tried much higher timeouts:
option   httpclose
timeout client   61s
timeout server   21s
timeout connect  21s
timeout http-request 121s

Higher timeouts turn most of the of the cR timeouts to CR.  The docs for 
CR say The client aborted before sending a full HTTP request. But I'm 
not sure exactly what that means.  Did the client send a RST?  Or a FIN 
before sending any data?  The docs also say The HTTP status code is 
likely a 400 here, but I presume it's unlikely the client actually 
received that


I looked at `show errors` but only a few hundred errors were reported 
which could not account for the large number of BADREQ log lines.

code.google.com haproxy-docs

[Chris Burroughs]

A variety of nicely formatted mirrors of the docs used to be at:

https://code.google.com/p/haproxy-docs

But all such urls are now returng 403.  I'm not sure if they are 
official or not, but does anyone know what happened to them?

Re: Haproxy SSL offloading to nginx backend web servers - need persistent connections

[Chris]

Baptiste,

Thanks again for taking the time to read through and respond.  I had done a
lot of troubleshooting and tried many different configuration options, to
the point that I am now confused.  You are correct, with only one web
server, persistence is not necessary, I should have realized this.  I tried
testing the web server by bypassing haproxy and I was getting the same
result, no image showing up so it is a web server problem that needs to be
resolved first.

With that said, the end plan is to indeed put 2 or 3 web servers behind
haproxy to be load balanced too - when that time comes, may I ask you what
the best way to implement session persistence would be, ensuring that when
a client connects to the web servers that they continue to stay on that web
server during their entire session?

Would this be the - option  http-server-close ?  Or, would implementing
some sort of PHPSESSID cookie be needed?

I think for true testing then, they need to get their web servers working
first, and then I need to get another one set up so that I can actually
test out session persistence.

Last question, when I get this into place, and to verify that sessions are
staying persistent, what is the best way to verify this?  Will haproxy logs
show this, or do I need to perform some sort of strace on the haproxy PID
to watch it?

Thanks a million Baptiste, you are a life saver - not only to me, to but
many people on this amazing list

Sincerely,

Chris


-Chris


On Tue, Nov 12, 2013 at 12:12 AM, Baptiste bed...@gmail.com wrote:

 Hi Chris,

 How can you point HAProxy in such problem?
 I mean, you have a single server in your farm, so if the browser can't
 download objects, it may come from an other point.
 And since you have a single server, you don't need persistence ;)

 Can you confirm whether:
 - the application works well in HTTP (clear)
 - the application works well without HAProxy

 Note that SSL offloading may have some impacts on web application:

 http://blog.exceliance.fr/2013/02/26/ssl-offloading-impact-on-web-applications/

 Can you enable logs as described in the link above and check the result?

 Baptiste


 On Mon, Nov 11, 2013 at 8:49 PM, Chris bludge...@gmail.com wrote:
  Hello everyone,
 
  I have a question concerning session persistence with offloading SSL via
  haproxy to backend nginx web servers running a PHP website.
 
  Basically, I have a configuration that is performing the SSL offloading
  successfully, however, it seems that session persistence is not working
  properly as some of the images from the nginx web server are not showing
 up,
  however the log in fields for this web server are showing up.
 
  What needs to happen is that an http request comes into haproxy, http
 calls
  get redirected to https, SSL is offloaded and the connection is handed
 over
  to the relevant back-end - with session persistence in-tact via ACLs
 within
  haproxy.
 
  I have read that http-server-close will preserve the connection and
 keep
  it persistent but I have not been able to get this to work.  I have also
  read that the type of balance used  (such as round robin) can affect the
  persistence of connections.  I have also tried to use
 
  cookie PHPSESSID insert nocache indirect  but I just am having no luck
 as
  there are a slew of configurations that can be used to do this task and I
  have gone over the haproxy documentation again and again and I just am
 not
  sure of the correct way of doing this.  The haproxy documentation is very
  thorough, but it is also very complex.
 
  In an effort to try to get this working, below is the configuration for
 my
  haproxy setup, I have stripped out all of the testing configurations that
  I've been using to try to get it to work.  I am hoping that someone
 might be
  able to assist me with properly getting this configured to make sessions
  persistent.
 
  Your expertise and advice are greatly welcomed and very appreciated - I
  thank you for your time.
 
  ---
  global
log 127.0.0.1 local0
log 127.0.0.1 local1
user haproxy
group haproxy
daemon
 
  defaults
log global
mode http
option httplog
option dontlognull
option redispatch
stats enable
maxconn 512
retries 3
contimeout 6
clitimeout 6
srvtimeout 6
 
 
  ## http frontend to redirect to https frontend
 
   frontend https_frontend
  bind 0.0.0.0:80
  redirect scheme https if !{ ssl_fc }
 
 
  ## https frontend to offload SSL to the backends
 
  frontend haproxy_https
mode http
option http-server-close
bind 0.0.0.0:443 ssl crt /etc/haproxy/psl-wildcard/wildcard.pem
 ca-file
  /etc/haproxy/psl-wildcard/wildcard.ca-bundle
 
acl is_psl_https  hdr_end(host)   -i www.test-site.com
acl is_broker_psl_https   hdr_end(host)   -i broker.test-site.com
acl is_eclose_psl_https   hdr_end(host)   -i eclose.test-site.com
 
use_backend   is_psl_https_backend

Haproxy SSL offloading to nginx backend web servers - need persistent connections

[Chris]

Hello everyone,

I have a question concerning session persistence with offloading SSL via
haproxy to backend nginx web servers running a PHP website.

Basically, I have a configuration that is performing the SSL offloading
successfully, however, it seems that session persistence is not working
properly as some of the images from the nginx web server are not showing
up, however the log in fields for this web server are showing up.

What needs to happen is that an http request comes into haproxy, http calls
get redirected to https, SSL is offloaded and the connection is handed over
to the relevant back-end - with session persistence in-tact via ACLs within
haproxy.

I have read that http-server-close will preserve the connection and keep
it persistent but I have not been able to get this to work.  I have also
read that the type of balance used  (such as round robin) can affect the
persistence of connections.  I have also tried to use

cookie PHPSESSID insert nocache indirect  but I just am having no luck as
there are a slew of configurations that can be used to do this task and I
have gone over the haproxy documentation again and again and I just am not
sure of the correct way of doing this.  The haproxy documentation is very
thorough, but it is also very complex.

In an effort to try to get this working, below is the configuration for my
haproxy setup, I have stripped out all of the testing configurations that
I've been using to try to get it to work.  I am hoping that someone might
be able to assist me with properly getting this configured to make sessions
persistent.

Your expertise and advice are greatly welcomed and very appreciated - I
thank you for your time.

---
global
  log 127.0.0.1 local0
  log 127.0.0.1 local1
  user haproxy
  group haproxy
  daemon

defaults
  log global
  mode http
  option httplog
  option dontlognull
  option redispatch
  stats enable
  maxconn 512
  retries 3
  contimeout 6
  clitimeout 6
  srvtimeout 6


## http frontend to redirect to https frontend

 frontend https_frontend
bind 0.0.0.0:80
redirect scheme https if !{ ssl_fc }


## https frontend to offload SSL to the backends

frontend haproxy_https
  mode http
  option http-server-close
  bind 0.0.0.0:443 ssl crt /etc/haproxy/psl-wildcard/wildcard.pem ca-file
/etc/haproxy/psl-wildcard/wildcard.ca-bundle

  acl is_psl_https  hdr_end(host)   -i www.test-site.com
  acl is_broker_psl_https   hdr_end(host)   -i broker.test-site.com
  acl is_eclose_psl_https   hdr_end(host)   -i eclose.test-site.com

  use_backend   is_psl_https_backendif  is_psl_https
  use_backend   is_broker_https_backend if  is_broker_psl_https
  use_backend   is_eclose_https_backend if  is_eclose_psl_https

  default_backend   is_psl_https_backend



## backends

backend is_psl_https_backend
  mode http
  balance source
  option http-server-close
  server server1 10.10.221.171:80

backend is_broker_https_backend
  mode http
  balance source
  option http-server-close
  server server1 10.10.221.172:80

backend is_eclose_https_backend
  mode http
  balance source
  option http-server-close
  server server1 10.10.221.173:80


listen admin 0.0.0.0:22002
  mode http
  stats uri /



Again, thank you very much.

Sincerely,

Chris

Re: I'm so close (I Think) - need a little help finishing this - Haproxy SSL offloading with ACLs and a wildcard cert

[Chris]

Hello again Baptiste,

Thank you for replying back to my questions, they are very helpful answers
and I really do appreciate your time.

Concerning your question back about persistence or stickiness:


 - Sessions need to stay alive after being offloaded to the backend web
 servers to port 80 on them.  (Session keepalive or stickiness?)

do you mean persistence or connection keepalives ???


I meant for session persistence, and does the following allow persistence
to happen correctly?:

 option http-server-close


If it does, one final question would be how does one control the length of
persistence, is that something controlled with a specific timeout variable?

Thank you so much again, I really appreciate your help a lot.

Chris


-Chris


On Tue, Oct 22, 2013 at 2:13 AM, Baptiste bed...@gmail.com wrote:

 Hi Chris,

 My answers inline.

 On Mon, Oct 21, 2013 at 10:57 PM, Chris bludge...@gmail.com wrote:
  - If any of these domains and subdomains are called with http://, they
 need
  to be redirected to https://

 # redirect http to https when connection is not ciphered
 http-request redirect scheme https if !{ ssl_fc } { hdr(host)
 www.test.com broker.test.com eclose.test.com images.test.com }


  - If a call comes in just to test.com, it needs to be redirected to
 https
  www.test.com:443

 # redirect text.com:80 to https://www.test.com:443
 http-request redirect prefix https://www.test.com if !{ ssl_fc } {
 hdr(host) test.com }


  - Sessions need to stay alive after being offloaded to the backend web
  servers to port 80 on them.  (Session keepalive or stickiness?)

 do you mean persistence or connection keepalives ???


  - images.test.com does not need to have SSL offloaded and I'm wondering
 if I
  should incorporate this into haproxy at all, maybe for load balancing at
  some point, this seems like a good idea)

 well, this is not what you asked in your first question, you wanted to
 offload SSL on all your domains.
 That said, I agree, ciphering images (usually) is useless.

 # redirect http to https when connection is not ciphered
 http-request redirect scheme https if !{ ssl_fc } { hdr(host)
 www.test.com broker.test.com eclose.test.com }

 since images.test.com is not in the list, the traffic will be accepted
 on port 80.

  defaults
  mode http
  maxconn 512
  option dontlognull
  option http-server-close
  retries 3
  contimeout 6
  clitimeout 6
  srvtimeout 6
 
  frontend haproxy_http
  bind 0.0.0.0:80
  timeout client 8640
  acl is_test hdr_end(host) -i www.test.com
  acl is_broker_test hdr_end(host) -i broker.test.com
  acl is_eclose_test  hdr_end(host) -i eclose.test.com
  acl is_images_test  hdr_end(host)  -i images.test.com
 
 
   # redirect http to https when connection is not ciphered
   http-request redirect scheme https if !{ ssl_fc } { hdr(host)
 www.test.com broker.test.com eclose.test.com images.test.com }
# redirect text.com:80 to https://www.test.com:443
   http-request redirect prefix https://www.test.com if !{ ssl_fc }
 { hdr(host) test.com }

  use_backend is_images_test if { hdr(host) images.test.com }

 
 
 
  frontend haproxy_https
  bind 0.0.0.0:443 ssl crt /etc/haproxy/certs/wildcard.pem
  log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\
 %CC\
  %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\
  {%[ssl_c_verify],%{+Q}[ssl_c_s_dn],{+Q[ssl_c_i_dn]}\ %{+Q}r
  acl is_test_https hdr_end(host) -i www.test.com
  acl is_broker_test_https hdr_end(host) -i broker.test.com
  acl is_eclose_test_https hdr_end(host) -i eclose.test.com
  use_backend is_test_https_backend if  is_test_https
  use_backend is_broker_https_backendif
  is_broker_test_https
  use_backend is_eclose_https_backend  if is_eclose_test_https
 
 
 
  # backends
 
  backend is_test_https_backend
  mode http
  balance source
  option http-server-close
  server server1 192.168.33.70
  server server2 192.168.33.170
 
  backend is_broker_https_backend
  mode http
  balance source
  option http-server-close
  server server1 192.168.33.71
  server server2 192.168.33.171
 
  backend is_eclose_https_backend
  mode http
  balance source
  option http-server-close
  server server1 192.168.33.72
  server server2 192.168.33.172
 
  backend is_images_test
  mode http
  balance source
  option http-server-close
  server server1 192.168.33.73
  server server2 192.168.33.13
 
 
  I think I'm close, but just not sure if I'm sanely doing things.  I've
 tried
  to put piece of information together from several different posts around
 the
  Internet, but I have found nothing that is concise enough to really make
 me
  understand what I'm doing wrong.
 
  Thank you SO much,
 
  Chris


 configuration out of my head, with no testing, but should work.

 Baptiste

I'm so close (I Think) - need a little help finishing this - Haproxy SSL offloading with ACLs and a wildcard cert

[Chris]

Hello,

I was hoping someone could help me out here, I'm fairly new to haproxy and
what I think I need should be fairly simple to do for someone that has some
experience with it, but for me,  I am not having the best of luck.

*Here is my scenario:*

I have an haproxy server set up that I want to use for SSL offloading, I
have compiled it with 1.5-dev19 and added the SSL options into the build
out, this part is done.


I simply need this haproxy to redirect port 80 traffic to 443 for 3 of the
4 sub-domains listed below, and then the haproxy server to offload SSL to
some backend web servers s via a wildcard cert.  I have generated a
wildcard cert for testing already and compiled together the PEM file that
haproxy currently sees as OK.



*Said domain:  *  test.com

*Sub-domains: *   www.test.com
 broker.test.com
eclose.test.com
 images.test.com  (globally open to everyone on port 80, no ssl offloading
needed)

*Requirements:   *

- If any of these domains and subdomains are called with http://, they need
to be redirected to https://
 - If a call comes in just to test.com, it needs to be redirected to https
www.test.com:443
 - Sessions need to stay alive after being offloaded to the backend web
servers to port 80 on them.  (Session keepalive or stickiness?)
 - images.test.com does not need to have SSL offloaded and I'm wondering if
I should incorporate this into haproxy at all, maybe for load balancing at
some point, this seems like a good idea)


I want to make it known that I have tried giving this a shot using the ACLs
for haproxy and
attempting to use the hdr_end(host) statements, but I am not certain if
this is correct, maybe I should be using hdr_begin(host)

As humbling as this might be, I will provide my best attempt at getting
this to work, maybe I'm way off, maybe I'm close, but I'm at my wit's end
on this and was hoping the community of professional might be able to steer
me in the right direction, below is my attempt at getting the above
scenario to work.

I thank you very much for your time and help.


*--- haproxy.conf ---*



defaults
mode http
maxconn 512
option dontlognull
option http-server-close  # session stickiness/persistence?
retries 3
contimeout 6
clitimeout 6
srvtimeout 6

frontend haproxy_http
bind 0.0.0.0:80 http://0.0.0.0/
timeout client 8640
acl is_test hdr_end(host) -i www.test.com
acl is_broker_test hdr_end(host) -i broker.test.com
acl is_eclose_test  hdr_end(host) -i eclose.test.com
acl is_images_test  hdr_end(host)  -i images.test.com


redirect location https ://www.test.com   if is_test
redirect location   https ://broker.test.com if is_broker_test
redirect location https ://eclose.test.com if is_eclose_test
use_backend is_images_test if is_images_test



frontend haproxy_https
bind 0.0.0.0:443 ssl crt /etc/haproxy/certs/wildcard.pem
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\
%CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\
{%[ssl_c_verify],%{+Q}[ssl_c_s_dn],{+Q[ssl_c_i_dn]}\ %{+Q}r
acl is_test_https hdr_end(host) -i www.test.com
acl is_broker_test_https hdr_end(host) -i broker.test.com
acl is_eclose_test_https hdr_end(host) -i eclose.test.com
use_backend is_test_https_backend if  is_test_https
use_backend is_broker_https_backendif  is_broker_test_https
use_backend is_eclose_https_backend  if is_eclose_test_https



# backends

backend is_test_https_backend
mode http
balance source
option http-server-close
server server1 192.168.33.70
server server2 192.168.33.170

backend is_broker_https_backend
mode http
balance source
option http-server-close
server server1 192.168.33.71
server server2 192.168.33.171

backend is_eclose_https_backend
mode http
balance source
option http-server-close
server server1 192.168.33.72
server server2 192.168.33.172

backend is_images_test
mode http
balance source
option http-server-close
server server1 192.168.33.73
server server2 192.168.33.13


 I think I'm close, but just not sure if I'm sanely doing things.  I've
tried to put piece of information together from several different posts
around the Internet, but I have found nothing that is concise enough to
really make me understand what I'm doing wrong.

Thank you SO much,

Chris

Re: acl and regex for adding www where needed

[Chris Allison]

Thanks, that has cleared that up.

cheers

Chris

On 25 September 2013 16:15, Baptiste bed...@gmail.com wrote:
 Hi Chris,

 Here is what your configuration doing:
 IF there is no 'www.' at the beginning of the Host header, then add it.
 IF there is no 'www.' at the beginning of the Host header, then
 redirect the user to /url

 Here is how HAProxy works: the request buffer is different from the
 workspace where the rewrite occurs and the rewritten header can't be
 used in next rules.
 Your configuration creates a loop to me: you should be redirected to
 /url as long as you don't have a www. in your Host header. And
 obviously the Host header is never changed.

 You should be redirecting like this:
 redirect prefix http://www.domain.com code 301 if { hdr(Host) domain.com }

 You must create one rule per domain name...

 Baptiste




 On Wed, Sep 25, 2013 at 9:26 AM, Chris Allison
 chris.alli...@hotmail.com wrote:
 Hi,

 To ensure a permanent IP we have a haproxy instance feeding AWS ELB to
 a number of apache instances.

 All of the vhosts are pinned to www.somedomain.com rather than
 somedomain.com, though we have dns setup so that the endpoiont for
 each domain and www.domain is the haproxy box.

 What we need to do is redirect any traffic for somedomain.com to
 www.somedomain.com.

 I have tried a number of different ways with the haproxy frontend
 config but cannot get it to work consistently (that would be my
 failure not haproxys).

 This is the nearest I have got:

 frontend:
 acl www_hdr hdr_beg(host) -i www.
 reqirep ^Host:\ (.*)$ Host:\ www.\1 if !www_hdr
 redirect prefix / code 301 if !www_hdr

 If I then telnet to the haproxy I can see that the host line gets
 rewritten correctly but I still get a 200 response not the expected
 301.

 Can anyone nudge me in the right direction please.

 regards

 Chris Allison





-- 
Dingo: You must spank her well, and after you are done with her, you
may deal with her as you like... and then... spank me.
All: And me. And me too. And me.
Dingo: Yes. Yes, you must give us all a good spanking.
Dingo: And after the spanking, the oral sex.
Galahad: Well, I could stay a bit longer...

Monty Python And The Holy Grail.

acl and regex for adding www where needed

[Chris Allison]

Hi,

To ensure a permanent IP we have a haproxy instance feeding AWS ELB to
a number of apache instances.

All of the vhosts are pinned to www.somedomain.com rather than
somedomain.com, though we have dns setup so that the endpoiont for
each domain and www.domain is the haproxy box.

What we need to do is redirect any traffic for somedomain.com to
www.somedomain.com.

I have tried a number of different ways with the haproxy frontend
config but cannot get it to work consistently (that would be my
failure not haproxys).

This is the nearest I have got:

frontend:
acl www_hdr hdr_beg(host) -i www.
reqirep ^Host:\ (.*)$ Host:\ www.\1 if !www_hdr
redirect prefix / code 301 if !www_hdr

If I then telnet to the haproxy I can see that the host line gets
rewritten correctly but I still get a 200 response not the expected
301.

Can anyone nudge me in the right direction please.

regards

Chris Allison

Haproxy 1.3 - is there a way to display a better error when no hosts are up/in the pool?

[Chris Patti]

It would be great if we could display a more usable error page than simply
the HTTP ERROR 503 users get when they hit a URL served by an haproxy
server group where all the hosts are down.

Is this possible in 1.3? I can't figure out how to do it from the
documentation.

Thanks in advance!
-Chris

Re: Fwd: Very odd behavior with 'cookie' only working intermittently

[Chris Patti]

Thank you *VERY* much for this tidbit Nenad.

With the early version of HAProxy we're using (v1.3.18) the actual syntax
is:

option httpclose

This worked perfectly, session afinity started performing as expected.

(Just wanted to record this for posterity)

-Chris



On Fri, Jun 14, 2013 at 5:39 PM, Nenad Merdanovic ni...@nimzo.info wrote:

 Hello Chris,

 You are missing one of the options to tell HAproxy to examine every
 request and not just first one for each connection.

 Add this to the defaults section:
 option http-server-close (or httpclose/forceclose, whatever suits you)
 timeout http-keep-alive 5s (for example, tune to your needs)

 Regards,
 Nenad


 On 06/14/2013 11:28 PM, Chris Patti wrote:
  Hello, we have two machines with identical haproxy configurations, set
  up to load balance each other, with Tomcat on the back end.
 
  We're seeing the SERVERID cookie only being set intermittently, despite
  the fact that as per our understanding it should be set with every
 response.
 
  When the cookie is NOT set, the request isn't printed when haproxy is
  running in debug mode, and the backend Total Sessions doesn't
  increment on the status page.
 
  Here's our configuration, we'd sincerely appreciate any hints anyone may
  have.
 
  Thanks,
  -Chris
  -
  global
  log 127.0.0.1   local2 info
  maxconn 4096
  ulimit-n 8250
  chroot /home/haproxy
  user haproxy
  group haproxy
  daemon
  quiet
  pidfile /home/haproxy/haproxy.pid
 
  defaults
  log global
  modehttp
  option  httplog
  option  dontlognull
  retries 3
  redispatch
  maxconn 2000
  contimeout  5000
  clitimeout  6
  srvtimeout  6
  stats uri /gz-haproxy-status
  120
  cookie SERVERID insert indirect nocache
  no option transparent
 
 
  frontend haproxy *:85
  default_backend tomcat
 
  backend tomcat
  balance roundrobin
  option httpchk GET /warehouse/static/images/gazelle_logo_tm.png
  server stg_wh_1 10.94.246.222:8181 http://10.94.246.222:8181 cookie
  i-9b333efb check inter 9000 ri
  se 2 fall 3 port 8181 maxconn 1000
  server stg_wh_2 10.223.37.147:8181 http://10.223.37.147:8181 cookie
  i-73b5be11 check inter 9000 ri
  se 2 fall 3 port 8181 maxconn 10
 

 --
 Nenad Merdanovic | PGP: 0x423edcb2 | Web: http://nimzo.info
 Linkedin: http://www.linkedin.com/in/nenadmerdanovic

Re: HAProxy 1.5-dev18 logs messages twice

[Chris Fryer]

Dinko Korunic wrote:
 On 18.06.2013 17:36, Chris Fryer wrote:
 [...]
 
 I notice that each request is logged once, then logged again immediately
 before the next request is logged.  If there is no next request, the
 request is logged a second time after a pause of between 60 and 70 seconds.

 If I comment out the log global line from the frontend configuration,
 only one request is logged.

 This did not used to happen with HAProxy 1.4
 
 Hi,
 
 This is due to 1.5 supporting several log targets, so in your
 configuration that's effectively having same log target twice. I've
 reported this and the explanation was that it's known and intended behavior.

Thanks.  Needless to say, this is not apparent from the documentation,
so perhaps that should be updated to reflect this state of affairs.

Regards,

Chris

Please access the attached hyperlink for an important electronic communications 
disclaimer: http://lse.ac.uk/emailDisclaimer

HAProxy 1.5-dev18 logs messages twice

[Chris Fryer]

Hello

HAProxy 1.5-dev18 seems to log the same message twice when log global
is specified in the defaults section, and again in a frontend section.
 Here are the relevant extracts from our haproxy.cfg:

global

log 127.0.0.1   local4
log 127.0.0.1   local5 notice

[ ... ]

defaults
logglobal
mode   http
option httplog
option dontlognull

[ ... ]

frontend http 0.0.0.0:80
log global
maxconn 25000
option  forwardfor except 127.0.0.1
option  httpclose
capture request header X-Forwarded-For len 40

[ ... ]

frontend https
bind 0.0.0.0:443 ssl crt /etc/haproxy/certfile \
no-sslv3 ciphers HIGH:RC4:!aNULL:!MD5
log global
maxconn 25000
option  forwardfor except 127.0.0.1
option  httpclose
capture request header X-Forwarded-For len 40

and here is the extract from rsyslog.conf:

$template
DailyHaproxyDetailLog,/var/log/haproxy/haproxy-detail-%$year%-%$month%-%$day%.log
if $syslogfacility-text == 'local4' and $programname == 'haproxy' then
-?DailyHaproxyDetailLog
 ~
if $syslogfacility-text == 'local5' and $programname == 'haproxy' then
-/var/log/haproxy/haproxy-info.log
 ~

I notice that each request is logged once, then logged again immediately
before the next request is logged.  If there is no next request, the
request is logged a second time after a pause of between 60 and 70 seconds.

If I comment out the log global line from the frontend configuration,
only one request is logged.

This did not used to happen with HAProxy 1.4

Is this a bug?

Chris


Please access the attached hyperlink for an important electronic communications 
disclaimer: http://lse.ac.uk/emailDisclaimer

Fwd: Very odd behavior with 'cookie' only working intermittently

[Chris Patti]

Hello, we have two machines with identical haproxy configurations, set up
to load balance each other, with Tomcat on the back end.

We're seeing the SERVERID cookie only being set intermittently, despite the
fact that as per our understanding it should be set with every response.

When the cookie is NOT set, the request isn't printed when haproxy is
running in debug mode, and the backend Total Sessions doesn't increment
on the status page.

Here's our configuration, we'd sincerely appreciate any hints anyone may
have.

Thanks,
-Chris
-
global
 log 127.0.0.1   local2 info
maxconn 4096
ulimit-n 8250
 chroot /home/haproxy
user haproxy
group haproxy
 daemon
quiet
pidfile /home/haproxy/haproxy.pid

defaults
log global
modehttp
option  httplog
 option  dontlognull
retries 3
redispatch
maxconn 2000
 contimeout  5000
clitimeout  6
srvtimeout  6
 stats uri /gz-haproxy-status
120
cookie SERVERID insert indirect nocache
no option transparent


frontend haproxy *:85
default_backend tomcat

backend tomcat
balance roundrobin
 option httpchk GET /warehouse/static/images/gazelle_logo_tm.png
server stg_wh_1 10.94.246.222:8181 cookie i-9b333efb check inter 9000 ri
se 2 fall 3 port 8181 maxconn 1000
server stg_wh_2 10.223.37.147:8181 cookie i-73b5be11 check inter 9000 ri
se 2 fall 3 port 8181 maxconn 10

Re: HAProxy maintains connection to proxied server after client disconnects

[Chris Yang]

Hi Lukas,

Thank you for your suggestion. It works! I guess this is a shortcoming of
streaming and eventsource protocols, because there are two separate
connections involved for each client, whereas for websocket and
long-polling haproxy can tell for sure the connection breaks given there is
only one connection.

I guess *option abortonclose* is also good to have in general, since it
helps defend against DDoS attacks? And I do not see much side-effect for
turning it on. Correct me if I'm wrong here.

Thanks again!



Regards,
Chris


On Sun, Jun 9, 2013 at 11:51 AM, Lukas Tribus luky...@hotmail.com wrote:

 Hi Chris,

 you probably need to set the following option to achieve this behavior:
 option abortonclose



 http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-option%20abortonclose



 Regards,

 Lukas

Multiple Switching

[Chris Brazier]

Hi,

I am hoping someone can help me. I am trying to use both path_beg and regrep at 
the same time, but it is not working. My config is as follows:

frontend httpweb
mode http
option http-server-close
bind :80
acl p_static1 path_beg /static1/
acl p_static2 path_beg /static2/
reqrep ^([^\ ]*)\ /static1/(.*)  \1\ /\2
reqrep ^([^\ ]*)\ /static2/(.*)  \1\ /\2
use_backend bk_web_82 if p_static1
use_backend bk_web_83 if p_static2
default_backend bk_web_http

If I go to: http://ip/static1/item.htm , the backend should be 
http://ip:82/item.htm
If I go to: http://ip/static2/item.htm , the backend should be 
http://ip:83/item.htm
If I go to: http://ip/item.htm , the backend should be http://ip/item.htm

I am finding that, the regrep works fine, but when it comes to the use_backend 
statement, it re-evaluates the acl and therefore always goes to the 
default_backend. If I comment out the regrep, I am sent to the correct backend, 
however this breaks what I am trying to achieve.

If someone could help me, that would be great.

Thanks,

Chris

Chris Brazier
Solutions Architect
IT
DD 020 8780 6987
E chris.braz...@capsticks.commailto:chris.braz...@capsticks.com
W www.capsticks.comhttp://www.capsticks.com
capstickssignaturetag


We’re supporting the Royal Hospital for Neuro-disability. Find out more about 
the Charity’s work at www.rhncharity.org.uk


HealthInvestor Awards 2013: Finalists - Legal advisors of the year (public)

HealthInvestor Awards 2012: Winner - Legal advisors of the year (public)
 
The General Practice Awards 2011: Winner - Legal team of the year
 
Laing  Buisson - Independent Healthcare Awards 2011: Winner - Legal adviser of 
the year


If you are not the intended recipient of the email entitled @|subject|@ then 
please delete the email. 

Disclaimer: The information contained in this communication is confidential and 
may be legally privileged. It is intended solely for use by the intended 
recipient(s) and others authorised to receive it. If you have received this 
message in error you are hereby notified that any disclosure, copying, 
distribution or taking action in reliance of the contents of this information 
is strictly prohibited and may be unlawful. Please note that Capsticks does not 
accept service of proceedings by email. Capsticks Solicitors LLP is a limited 
liability partnership registered in England and Wales under registered number 
OC340360 and is authorised and regulated by the Solicitors Regulation 
Authority. A list of members is open to inspection at our registered office, 1 
St Georges Road, St Georges East, Wimbledon, London, SW19 4DR or online at 
www.capsticks.com. The term partner is used to refer to a member of Capsticks 
Solicitors LLP or an employee or consultant with equivalent standing and 
qualifications. 

This footnote confirms this message has been scanned for viruses. 

Please consider the environment before printing this message.

HAProxy maintains connection to proxied server after client disconnects

[Chris Yang]

Hi,

I'm using HAProxy (1.5 dev18) in front of tornado servers to load-balancing
websockets and xhr-streaming connections. I notice something strange but
consistent with xhr-streaming connection (details here:
https://groups.google.com/forum/?fromgroups#!topic/sockjs/LwfgT2qwIT0).

In short, even after the client closes its connection, HAProxy still
maintains the xhr-streaming connection to the proxied backend, until the
timeout kicks in (i.e. timeout server xxx). I've tested this with both
tornado 2.4.1 and the latest 3.0.2, and the results are consistent. Other
transports (e.g. websockets, polling) do not have this problem, but
evensource transport shares the same issue.

Any idea what causes this? Thanks!



Regards,
Chris

SSL offloading configuration

[Chris Sarginson]

Hi,

Are there any plans to allow HAProxy to take the traffic that it can now 
SSL offload, perform header analysis, and then use an SSL encrypted 
connection to the backend server?


I have a situation where I need to be able to use ACLs against SSL 
encrypted traffic, but then continue passing the traffic to the backend 
over an encrypted connection.  This is specifically a security concern, 
rather than an issue with poor code.


Cheers
Chris

Re: SSL offloading configuration

[Chris Sarginson]

That's AWESOME!  Can't believe I didn't think of that, thanks a lot 
guys :)


Chris

On 30/04/2013 13:53, PiBa-NL wrote:

Hi Chriss,

That seams possible already.?.
If you have the configuration for SSL offloading configured already 
all you need to add is the ssl option to your backend servers.


-- 
http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#5.2 
--

*ssl http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#5-ssl*
This option enables SSL ciphering on outgoing connections to the server. At
the moment, server certificates are not checked, so this is prone to man in
the middle attacks. The real intended use is to permit SSL communication
with software which cannot work in other modes over networks that would
otherwise be considered safe enough for clear text communications. When this
option is used, health checks are automatically sent in SSL too unless there
is a port  http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#port or 
anaddr  http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#addr directive 
indicating the check should be sent to a
different location. See the check-ssl  
http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#check-ssl optino to 
force SSL health checks.
--

Op 30-4-2013 14:47, Chris Sarginson schreef:

Hi,

Are there any plans to allow HAProxy to take the traffic that it can 
now SSL offload, perform header analysis, and then use an SSL 
encrypted connection to the backend server?


I have a situation where I need to be able to use ACLs against SSL 
encrypted traffic, but then continue passing the traffic to the 
backend over an encrypted connection.  This is specifically a 
security concern, rather than an issue with poor code.


Cheers
Chris

Re: Sessions when a server doesn't answer anymore

[Chris Sarginson]

You would probably need to look at something like this:

http://technet.microsoft.com/en-us/library/cc753897(v=ws.10).aspx

Storing sessions from IIS in an SQL db will allow them to be accessible 
from any of your servers.


Chris

On 26/03/2013 09:16, Christophe Rahier wrote:

Hi,

I try to be clear with my question :-)

Our haproxy is in front of 3 IIS web servers.

When a server is down, haproxy doesn't send any traffic to this 
server, it's ok. Users needs to log again in our application.


When this server comes alive, users who previously logged (via this 
server) need to log again in our application as if haproxy closed 
their current session and wants to reconnect them to this server.


How can I avoid this?

Thanks for your help.

Regards,

Christophe

Re: Sessions when a server doesn't answer anymore

[Chris Sarginson]

On 26/03/2013 10:11, Baptiste wrote:

Hi Christophe,

This issue should not happen.
Please share your backend configuration here.

Baptiste
Could this not occur in an instance where cookies were being used for 
the primary server, but sessions are not being shared?


Server 1 fails, client moved to Server 2 (backup with no cookies), logs 
in.  Server 1 comes back up, haproxy detects the cookie from the 
original request, directs the client back to Server 1 which doesn't have 
access to session data on Server 2.




On Tue, Mar 26, 2013 at 10:43 AM, Chris Sarginson ch...@sargy.co.uk wrote:

You would probably need to look at something like this:

http://technet.microsoft.com/en-us/library/cc753897(v=ws.10).aspx

Storing sessions from IIS in an SQL db will allow them to be accessible from
any of your servers.

Chris


On 26/03/2013 09:16, Christophe Rahier wrote:

Hi,

I try to be clear with my question :-)

Our haproxy is in front of 3 IIS web servers.

When a server is down, haproxy doesn't send any traffic to this server, it's
ok. Users needs to log again in our application.

When this server comes alive, users who previously logged (via this server)
need to log again in our application as if haproxy closed their current
session and wants to reconnect them to this server.

How can I avoid this?

Thanks for your help.

Regards,

Christophe

Re: Unknown host

[Chris Sarginson]

Amazon have probably removed the DNS after it's been offline for over a 
certain period of time.


Chris

On 06/03/2013 13:06, Matthieu Boret wrote:

Hi,

I have a strange and random problem with Haproxy. Sometimes, I have an 
error when I'm trying to reload the configuration file.


Reloading haproxy: haproxy[ALERT] 064/083311 (12629) : parsing 
[/etc/haproxy/haproxy.cfg:314] : Unknown host in 
''ec2-55-33-22-88.eu-west-1.compute.amazonaws.com:80 
http://ec2-55-33-22-88.eu-west-1.compute.amazonaws.com:80'
[ALERT] 064/083311 (12629) : Error(s) found in configuration file : 
/etc/haproxy/haproxy.cfg


In fact this host is unreachable because it's a server who has been 
remove since 2 or 3 days. But I don't know why I have this error. 
Normally Haproxy accepts any server in a backend even if he is down.


Each day we launch and remove server from a backend. We are in Amazon 
AWS and we use the version 1.5 dev-7 of Haproxy.
This error appear only one time by month(approximately). I suspect too 
long downtime who become unknown host but I'm not sure. Haproxy has 
this mecanism?



Thanks in advance


Matthieu

Re: XHR-streaming and Websocket connections in 1.5-dev17

[Chris Yang]

Hi Willy,

Thanks for your comments.

It's unusual to see the client write both keep-alive and Upgrade in the
 connection header. Thus, I suspect that something between the client and
 the server (eg: a proxy on the client side or the application server
 itself)
 has added it. If it's a proxy on the client side, this proxy might be
 responsible for this. Not all proxies support WebSocket yet. You should
 try to take a capture of the front traffic (between client and haproxy),
 and isolate a session which exhibits this behaviour (based on the IP
 address
 logged as reporting the problem). In such a capture, we'll see if the
 server
 response is valid, and if it is the client which closes. If we see the
 client
 close, we'll know that there's something between the client and haproxy
 which
 doesn't support WebSocket.


The extra keep-alive in connection header seems to be a Firefox thing. For
example, SockJS mentions about it here:
http://sockjs.github.com/sockjs-protocol/sockjs-protocol-0.3.3.html#section-73.
The strange thing is that when I was testing using Firefox, sometimes it
connects websocket fine, but sometimes not. I will try to capture the
traffic as you've suggested, and compare the captures when it works and
when it does not.

I am using

Hmmm if you can easily reproduce it, this becomes more of a concern. Are
 you sure you're directly connected ? Would you happen to be running windows
 on your laptop with some anti-virus/anti-malware or such software ? We've
 seen that in the past, it was Trend Micro's OfficeScan. It would let
 traffic
 to port 8000 pass without issue, but 80 was improperly analysed and the
 connection was frozen. It would be possible that a similar type of software
 is causing the issue.
 There's nothing abnormal in your configuration. I'm seeing a redirect for
 http-https, have you tried without SSL ?


Indeed, I am using SSL exclusively, because encrypted traffic actually
increases the chance of websocket (and other supported protocols in SockJS)
working, since those annoying proxies and antivirus may just let the
traffic pass through without modifying them. In fact, given the same
network environment I was in for testing, if I don't do https, things
barely work, websocket and xhr-streaming just hang.

I also read this article:
http://www.infoq.com/articles/Web-Sockets-Proxy-Servers, which seems to
suggest that, due to the fact many proxies do not support websocket, such
connection failure is not uncommon. The article also suggests to use HTTPS
to increase the success rate, which is exactly what I've done. Still, even
though I haven't count precisely, the overall success rate is definitely
below 90% for me.

What else can I try here? Thanks!



Best,
Chris

On Fri, Mar 1, 2013 at 1:39 AM, Willy Tarreau w...@1wt.eu wrote:

 Hi Chris,

 I'm quickly jumping into this thread because I'm seeing something strange :

 On Wed, Feb 27, 2013 at 10:35:16AM -0500, Chris Yang wrote:
  Hello again,
 
  Regarding the second problem (failure to establishing websocket
 connections
  in supported browsers), I can confirm that it happens on both Chrome and
  Firefox. And below is what I see for the '101 Switching Protocols'
 request
  in Firefox when it happened:
 
  Request URL: https://www.myexample.com/sockjs/673/nw56jyoj/websocket
   https://www.playmymodel.com/sockjs/chat/673/nw56jyoj/websocket
   Request Method: GET
   Status Code: HTTP/1.1 101 Switching Protocols

   *Request Headers
   *User-Agent:Mozilla/5.0 (Windows NT 6.1; rv:18.0) Gecko/20100101
   Firefox/18.0
   Upgrade:websocket
   Sec-WebSocket-Version:13
   Sec-WebSocket-Key:Fg8Z/EASgn6AlVh0hMYLtw==
   Pragma:no-cache
   Origin:https://www.playmymodel.com
   Host:www.playmymodel.com
   Connection:keep-alive, Upgrade
   Cache-Control:no-cache
   Accept-Language:en-US,en;q=0.5
   Accept-Encoding:gzip, deflate
   Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

 It's unusual to see the client write both keep-alive and Upgrade in the
 connection header. Thus, I suspect that something between the client and
 the server (eg: a proxy on the client side or the application server
 itself)
 has added it. If it's a proxy on the client side, this proxy might be
 responsible for this. Not all proxies support WebSocket yet. You should
 try to take a capture of the front traffic (between client and haproxy),
 and isolate a session which exhibits this behaviour (based on the IP
 address
 logged as reporting the problem). In such a capture, we'll see if the
 server
 response is valid, and if it is the client which closes. If we see the
 client
 close, we'll know that there's something between the client and haproxy
 which
 doesn't support WebSocket.

   *Response Headers
   *Upgrade:websocket
   Sec-WebSocket-Accept:NxU0HFQKbhwr7VBCHf23qLXH/Ic=
   Connection:Upgrade
 
 
   and below is what follows from the SockJS side:
 
  Closed transport: websocket SimpleEvent(type=close, code=2007,
   reason=Transport timeouted

Re: XHR-streaming and Websocket connections in 1.5-dev17

[Chris Yang]

Hello again,

Regarding the second problem (failure to establishing websocket connections
in supported browsers), I can confirm that it happens on both Chrome and
Firefox. And below is what I see for the '101 Switching Protocols' request
in Firefox when it happened:

Request URL: https://www.myexample.com/sockjs/673/nw56jyoj/websocket
 https://www.playmymodel.com/sockjs/chat/673/nw56jyoj/websocketRequest
 Method: GET
 Status Code: HTTP/1.1 101 Switching Protocols
 *Request Headers
 *User-Agent:Mozilla/5.0 (Windows NT 6.1; rv:18.0) Gecko/20100101
 Firefox/18.0
 Upgrade:websocket
 Sec-WebSocket-Version:13
 Sec-WebSocket-Key:Fg8Z/EASgn6AlVh0hMYLtw==
 Pragma:no-cache
 Origin:https://www.playmymodel.com
 Host:www.playmymodel.com
 Connection:keep-alive, Upgrade
 Cache-Control:no-cache
 Accept-Language:en-US,en;q=0.5
 Accept-Encoding:gzip, deflate
 Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
 *Response Headers
 *Upgrade:websocket
 Sec-WebSocket-Accept:NxU0HFQKbhwr7VBCHf23qLXH/Ic=
 Connection:Upgrade


 and below is what follows from the SockJS side:

Closed transport: websocket SimpleEvent(type=close, code=2007,
 reason=Transport timeouted, wasClean=false)


This failure of establishing websocket connection happen relatively often
now, with 4 - 5 Chrome processes running on my laptop, I can almost always
get at least 1 Chrome which encounters this. In addition, I have also
tested this with HAProxy 1.5-dev15, and the problem persists. For
completeness, I attach the updated config as below:

global
 log 127.0.0.1   local1 info
 log 127.0.0.1   local1 notice
 #log loghostlocal0 info
 maxconn 4096
 chroot /usr/share/haproxy
 uid 99
 gid 99
 daemon
 #debug
 #quiet

 defaults
 log global
 modehttp
 option  httplog
 option  dontlognull
 retries 3
 option  redispatch
 maxconn 500
 timeout connect 6s
 timeout client  600s
 timeout server  600s
 timeout tunnel  1200s

 frontend public
 modehttp
 bind0.0.0.0:80
 bind0.0.0.0:443 ssl crt /etc/haproxy/ssl_wc/mysite.pem
 ciphers HIGH:!aNULL:!MD5 no-tlsv10
 timeout client  300s
 option  http-server-close
 option  forwardfor except 127.0.0.1
 reqadd  X-Forwarded-Proto:\ https if { ssl_fc }
 reqadd  X-Forwarded-Proto:\ http if !{ ssl_fc }
 # define ACLs below
 acl is_https ssl_fc
 redirect scheme https if !is_https
 # and other ACLs and rules (omitted here)

 backend sockjs_1
 #timeout server 60s
 server sock1 127.0.0.1:8000

 backend sockjs_2
 #timeout server 60s
 server sock2 127.0.0.1:8060

 backend nginx
 #timeout server 20s
 server nginx 127.0.0.1:8484

 backend stats
 stats enable
 stats uri /haproxy/stats
 stats realm Haproxy\ Statistics

 backend www
 #timeout server 300s
 #option forwardfor header X-Client-IP
 #no option httpclose
 #option http-server-close
 server content 127.0.0.1:8080


Again, backends sockjs_1 and sockjs_2 are what is being affected here.

Could anyone comment on what may cause this undesirable websocket failures
here? Thanks a lot!



Best,
Chris

On Fri, Feb 22, 2013 at 8:40 AM, Chris Yang bhp...@gmail.com wrote:


 Hello,

 I upgraded to dev17 from dev15. I am running Tornado servers behind
 HAProxy with SockJS support. Comparing to before the upgrade, I've noticed
 two problematic behaviors:

 1. When a client is using IE with xhr-streaming protocol to connect to
 servers, such connections are closed (seeing 'Connection get reset by peer'
 error on the Python side) more often than before. Sometimes disconnection
 happened soon after connection (half a minute). I guess it might be related
 to timeout, but since I haven't changed any configuration on HAProxy, I
 wonder if this behavior is related to the upgrade of HAProxy itself.

 2. From time to time, Chrome failed to finish the websocket handshake with
 the server, and instead used xhr-streaming. This didn't happen before the
 upgrade, but I've noticed at least 5+ such cases during the testing with
 dev17.

 I can confirm that both client-side library (sockjs-client) and
 server-side library (sockjs-tornado) have stayed the same since their last
 respective upgrade (months before dev17 came out). So the only variable
 which might have produced the observations above could be HAProxy.

 I attach below my HAProxy config (excluding most of the acls), and I'd
 much appreciate that if someone could take a quick look and let me know if
 some setting(s) may not play well with dev17 which might have caused the
 behavior above, or some new setting(s) should be added. Notice that
 backends sockjs_1 and sockjs_2 are the ones

XHR-streaming and Websocket connections in 1.5-dev17

[Chris Yang]

Hello,

I upgraded to dev17 from dev15. I am running Tornado servers behind HAProxy
with SockJS support. Comparing to before the upgrade, I've noticed two
problematic behaviors:

1. When a client is using IE with xhr-streaming protocol to connect to
servers, such connections are closed (seeing 'Connection get reset by peer'
error on the Python side) more often than before. Sometimes disconnection
happened soon after connection (half a minute). I guess it might be related
to timeout, but since I haven't changed any configuration on HAProxy, I
wonder if this behavior is related to the upgrade of HAProxy itself.

2. From time to time, Chrome failed to finish the websocket handshake with
the server, and instead used xhr-streaming. This didn't happen before the
upgrade, but I've noticed at least 5+ such cases during the testing with
dev17.

I can confirm that both client-side library (sockjs-client) and server-side
library (sockjs-tornado) have stayed the same since their last respective
upgrade (months before dev17 came out). So the only variable which might
have produced the observations above could be HAProxy.

I attach below my HAProxy config (excluding most of the acls), and I'd much
appreciate that if someone could take a quick look and let me know if some
setting(s) may not play well with dev17 which might have caused the
behavior above, or some new setting(s) should be added. Notice that
backends sockjs_1 and sockjs_2 are the ones experiencing the problems.

global
log 127.0.0.1   local1 info
log 127.0.0.1   local1 notice
#log loghostlocal0 info
maxconn 4096
chroot /usr/share/haproxy
uid 99
gid 99
daemon
#debug
#quiet

defaults
log global
modehttp
option  httplog
option  dontlognull
retries 3
option  redispatch
maxconn 500
timeout connect 6s

frontend public
modehttp
bind0.0.0.0:80
bind0.0.0.0:443 ssl crt /etc/haproxy/ssl_wc/mysite.pem ciphers
HIGH:!aNULL:!MD5 no-tlsv10
timeout client  300s
option  http-server-close
option  forwardfor except 127.0.0.1
reqadd  X-Forwarded-Proto:\ https if { ssl_fc }
reqadd  X-Forwarded-Proto:\ http if !{ ssl_fc }
#option http-pretend-keepalive
# define ACLs and use them below
acl is_https ssl_fc
redirect scheme https if !is_https
# other acls omitted here

backend sockjs_1
timeout server 60s
server sock_1 127.0.0.1:8000

backend sockjs_2
timeout server 60s
server sock_2 127.0.0.1:8060

backend nginx
timeout server 20s
server nginx 127.0.0.1:8484

backend stats
stats enable
stats uri /haproxy/stats
stats realm Haproxy\ Statistics

backend www
timeout server 300s
#option forwardfor header X-Client-IP
#no option httpclose
#option http-server-close
server content 127.0.0.1:8080

Thanks very much!



Best,
Chris

Re: client keep-alive when servers

[Chris Burroughs]

On 02/01/2013 03:07 AM, Cyril Bonté wrote:
 Hi Willy,
 
 Le 01/02/2013 08:44, Willy Tarreau a écrit :
 I have some vague memories of someone here reporting this on the tomcat
 ML, resulting in a fix one or two years ago, but I may confuse with
 something else. Maybe you should experiment with newer versions ? What
 you need is just the server to emit content-length with every response,
 including close ones, and to fall back to chunked encoding when
 content-length is unknown.

 
 You're right, this was discussed here on the mailing list :
 http://thread.gmane.org/gmane.comp.web.haproxy/2755
 
 Then, Óscar opened a thread on the tomcat one :
 http://marc.info/?t=12701162342r=1w=2
 The issue was fixed in tomcat 6.0.27 and 5.5.29/
 
 Cheers
 

Thank you for the details for the details.  The pieces are all fitting
together -- this particular service happens to be stuck on 6.0.20.

Re: HAProxy on multi-CPU Hardware

[Chris Sarginson]

Search for nbproc in http://haproxy.1wt.eu/download/1.4/doc/configuration.txt, 
which explains how HaProxy handles multiple CPUs in a box.

Chris

On 01/02/2013 15:54, Peter Mellquist wrote:

Hi!

My understanding is that HAProxy is a single process event model which
utilizes a single CPU even if running on SMP / Multi CPU systems?

Have there been any considerations for having HAProxy fork or thread
allowing a single config file to feed proxies across many CPUs on the same
box? There are various design models for doing this but I am interested
what has already been done?

Thanks,
Peter. 

Re: client keep-alive when servers

[Chris Burroughs]

I'm using haproxy 1.4.17 if that's relevant.  I tried replacing
http-server-close with http-pretend-keepalive, which as far as I can
tell had no effect to client side keepalive behaviour.  Responses still
looked something like this:

 HTTP/1.1 200 OK
 Server: Apache-Coyote/1.1
 Cache-Control: max-age=125
 Content-Type: application/json;charset=UTF-8
 Date: Thu, 31 Jan 2013 13:25:54 GMT
 Connection: close

* Closing connection #0

I then tried enabling keepalive on the backend (tomcat), after which
curls directly too the backend looked like:

 HTTP/1.1 200 OK
 Server: Apache-Coyote/1.1
 Cache-Control: max-age=63
 Content-Type: application/json;charset=UTF-8
 Transfer-Encoding: chunked
 Date: Thu, 31 Jan 2013 13:24:13 GMT

* Connection #0 to host HOST left intact

But responses from haproxy still closed with either http-server-close or
http-pretend-keepalive set still close the connection.  I admit these
results are confusing and I feel like I'm missing something.

On 01/31/2013 02:28 AM, Willy Tarreau wrote:
 On Wed, Jan 30, 2013 at 05:16:26PM -0800, Bryan Talbot wrote:
 http-pretend-keepalive still enables keep-alive to the client like
 http-server-close does.

 The difference is that http-server-close sends a Connection: close to the
 backend to indicate it doesn't intend to use keep alive.  This however
 confuses some tomcat versions and causes them to act like the request was
 using HTTP 1.0 and (i think) send a connection: close in the response which
 is what your client is seeing.  http-pretend-keepalive makes haproxy not
 send that Connection: close to the backend but then it closes the
 connection anyway.
 
 It's not just a matter of Connection header or HTTP version, the real issue
 is that the server fails to announce the content-length nor chunked encoding,
 making it impossible to maintain the connection alive with the client because
 there is no other way to end the response than by closing. And you're 
 perfectly
 right, using http-pretend-keepalive forces tomcat to emit these precious
 headers to delimit the response, allowing the front connection to remain
 alive.
 
 Willy