Re: smtpchk when using proxy protocol

[Tom Lanyon]

On 28/05/2013, at 6:58 PM, Baptiste  wrote:
> Your last chance is to capturethe check with tcpdump and send it back to me.
> Might be a bug, either in Haproxy or postfix.

I remember digging into the latest 1.5 code and finding that check-send-proxy 
looked like it wouldn't work at all, but I didn't take note of why and I can't 
remember now...

Tom

Re: smtpchk when using proxy protocol

[Baptiste]

Hi viet

Your last chance is to capturethe check with tcpdump and send it back to me.
Might be a bug, either in Haproxy or postfix.

Baptiste
 Le 28 mai 2013 09:12, "Viet Hoang"  a écrit :

>  Hi Baptiste,
>
> My banner is normal indeed.
>
> I have checked with telnet and HAProxy without Proxy Protocol.
>
> Telnet directly:
>
>   root@postfix01:~/postfix-2.10.0# telnet X.X.X.X 25
> Trying X.X.X.X...
> Connected to X.X.X.X.
> Escape character is '^]'.
> 220 mail.mydomain.com ESMTP Postfix (Ubuntu)
>
> Connect via HAProxy without Proxy Protocol:
>
>   220 mail.mydomain.com ESMTP Postfix (Ubuntu)
>
> Viet
>
> On 05/28/2013 01:30 PM, Baptiste wrote:
>
> Hi Vit,
>
>  You're right about HAProxy's smtp check.
> Furthermore, here is what the SMTP RFC says:
> "Formally, a reply is defined to be the sequence:  a three-digit code,
> , one line of text, and "
>
>  Your welcome banner doesn't follow the RFC. There may be some options in
> Postfix to set to change this behavior.
>
>  Baptiste
>
>
>
>
> On Tue, May 28, 2013 at 5:54 AM, Vit Dua  wrote:
>
>>  Hi,
>>
>> There is an option in HAProxy 1.5 doc:
>>
>>
>> http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#check-send-proxy
>>
>> And this thread:
>>
>> http://comments.gmane.org/gmane.comp.web.haproxy/11551
>>
>> I have set in HAProxy config:
>>
>>   server postfix01 X.X.X.X:1 check check-send-proxy send-proxy
>>
>> and double-check Postfix's main.cf file:
>>
>>   postscreen_upstream_proxy_protocol = haproxy
>>
>> but there is still that error:
>>
>>   May 28 10:47:17 localhost haproxy[9495]: Server ft_postfix/postfix01
>> is DOWN, reason: Layer7 invalid response, info: "220-mail.mydomain.comESMTP 
>> Postfix (Ubuntu)", check duration: 1ms. 0 active and 0 backup servers
>> left. 0 sessions active, 0 requeued, 0 remaining in queue.
>> May 28 10:47:17 localhost haproxy[9495]: proxy ft_postfix has no server
>> available!
>>
>>
>> I have a look at HAProxy source code and see that it checks if the server
>> speaks  (not sure).
>>
>> I have to choose tcp check instead. It works normally.
>>
>> Viet
>>
>>
>> On 05/28/2013 12:02 AM, Tom Lanyon wrote:
>>
>> On 27/05/2013, at 5:04 PM, Vit Dua   
>> wrote:
>>
>>  I have used proxy protocol for SMTP
>>
>>  
>>
>>  It worked successfully.
>>
>> I wanted to do smtpchk so that I added to the config:
>>
>> server postfix01 X.X.X.X:1 send-proxy check
>>
>>  
>>
>> I am running haproxy-1.5_dev18 in front of Postfix 2.10.0 SMTP servers using 
>> the PROXY protocol.
>>
>> As far as I could see in the 1.5_dev18 code, the smtpchk option does not 
>> send the appropriate PROXY protocol chatter during the health check when 
>> send-proxy is enabled.  I tried to patch this but couldn't see an easy way 
>> to do so with the current health check code (I can't remember the details 
>> right now).
>>
>> The two options I came up with were:
>>
>>  1. fallback to TCP checking
>>  2. perform the SMTP check some other way
>>
>> We are now doing #2 and are doing the SMTP check (without the PROXY 
>> protocol) to a different set of listen ports on the Postfix servers.
>>
>> Here's the relevant parts of our config:
>>
>> postfix master.cf:
>>  # Regular SMTP (no PROXY) on port 'smtp' (TCP 25)
>>  smtp  inet  n   -   n   -   -   smtpd
>>  # A custom service which expects haproxy's PROXY protocol on 
>> non-standard port 9025
>>  9025  inet  n   -   n   -   -   smtpd -o 
>> smtpd_upstream_proxy_protocol=haproxy
>>
>> haproxy.cfg:
>>  frontend f-smtp
>>  bind :25
>>  mode tcp
>>  timeout client 1m
>>  default_backend b-smtp
>>
>>  backend b-smtp
>>  mode tcp
>>  option smtpchk
>>  timeout server 1m
>>  # note: data port 9025 (w/PROXY); smtpchk port 25 (no PROXY).
>>  server smtp1 :9025 send-proxy check port 25
>>  server smtp2 :9025 send-proxy check port 25
>>  server smtp3 :9025 send-proxy check port 25
>>
>> Tom
>>
>>
>>
>>
>
>

Re: smtpchk when using proxy protocol

[Viet Hoang]

Hi Baptiste,

My banner is normal indeed.

I have checked with telnet and HAProxy without Proxy Protocol.

Telnet directly:

root@postfix01:~/postfix-2.10.0# telnet X.X.X.X 25
Trying X.X.X.X...
Connected to X.X.X.X.
Escape character is '^]'.
220 mail.mydomain.com ESMTP Postfix (Ubuntu)


Connect via HAProxy without Proxy Protocol:

220 mail.mydomain.com ESMTP Postfix (Ubuntu)


Viet

On 05/28/2013 01:30 PM, Baptiste wrote:

Hi Vit,

You're right about HAProxy's smtp check.
Furthermore, here is what the SMTP RFC says:
"Formally, a reply is defined to be the sequence:  a three-digit code, 
, one line of text, and "


Your welcome banner doesn't follow the RFC. There may be some options 
in Postfix to set to change this behavior.


Baptiste




On Tue, May 28, 2013 at 5:54 AM, Vit Dua > wrote:


Hi,

There is an option in HAProxy 1.5 doc:


http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#check-send-proxy

And this thread:

http://comments.gmane.org/gmane.comp.web.haproxy/11551

I have set in HAProxy config:

server postfix01 X.X.X.X:1 check check-send-proxy send-proxy


and double-check Postfix's main.cf  file:

postscreen_upstream_proxy_protocol = haproxy


but there is still that error:

May 28 10:47:17 localhost haproxy[9495]: Server
ft_postfix/postfix01 is DOWN, reason: Layer7 invalid response,
info: "220-mail.mydomain.com  ESMTP
Postfix (Ubuntu)", check duration: 1ms. 0 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
May 28 10:47:17 localhost haproxy[9495]: proxy ft_postfix has no
server available!


I have a look at HAProxy source code and see that it checks if the
server speaks  (not sure).

I have to choose tcp check instead. It works normally.

Viet


On 05/28/2013 12:02 AM, Tom Lanyon wrote:

On 27/05/2013, at 5:04 PM, Vit Dua  
  wrote:

I have used proxy protocol for SMTP



It worked successfully.

I wanted to do smtpchk so that I added to the config:

server postfix01 X.X.X.X:1 send-proxy check



I am running haproxy-1.5_dev18 in front of Postfix 2.10.0 SMTP servers 
using the PROXY protocol.

As far as I could see in the 1.5_dev18 code, the smtpchk option does not 
send the appropriate PROXY protocol chatter during the health check when 
send-proxy is enabled.  I tried to patch this but couldn't see an easy way to 
do so with the current health check code (I can't remember the details right 
now).

The two options I came up with were:

1. fallback to TCP checking
2. perform the SMTP check some other way

We are now doing #2 and are doing the SMTP check (without the PROXY 
protocol) to a different set of listen ports on the Postfix servers.

Here's the relevant parts of our config:

postfixmaster.cf  :
# Regular SMTP (no PROXY) on port 'smtp' (TCP 25)
smtp  inet  n   -   n   -   -   smtpd
# A custom service which expects haproxy's PROXY protocol on 
non-standard port 9025
9025  inet  n   -   n   -   -   smtpd -o 
smtpd_upstream_proxy_protocol=haproxy

haproxy.cfg:
frontend f-smtp
bind :25
mode tcp
timeout client 1m
default_backend b-smtp

backend b-smtp
mode tcp
option smtpchk
timeout server 1m
# note: data port 9025 (w/PROXY); smtpchk port 25 (no PROXY).
server smtp1 :9025 send-proxy check port 25
server smtp2 :9025 send-proxy check port 25
server smtp3 :9025 send-proxy check port 25

Tom

Re: smtpchk when using proxy protocol

[Baptiste]

Hi Vit,

You're right about HAProxy's smtp check.
Furthermore, here is what the SMTP RFC says:
"Formally, a reply is defined to be the sequence:  a three-digit code,
, one line of text, and "

Your welcome banner doesn't follow the RFC. There may be some options in
Postfix to set to change this behavior.

Baptiste




On Tue, May 28, 2013 at 5:54 AM, Vit Dua  wrote:

>  Hi,
>
> There is an option in HAProxy 1.5 doc:
>
>
> http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#check-send-proxy
>
> And this thread:
>
> http://comments.gmane.org/gmane.comp.web.haproxy/11551
>
> I have set in HAProxy config:
>
>   server postfix01 X.X.X.X:1 check check-send-proxy send-proxy
>
> and double-check Postfix's main.cf file:
>
>   postscreen_upstream_proxy_protocol = haproxy
>
> but there is still that error:
>
>   May 28 10:47:17 localhost haproxy[9495]: Server ft_postfix/postfix01 is
> DOWN, reason: Layer7 invalid response, info: "220-mail.mydomain.com ESMTP
> Postfix (Ubuntu)", check duration: 1ms. 0 active and 0 backup servers left.
> 0 sessions active, 0 requeued, 0 remaining in queue.
> May 28 10:47:17 localhost haproxy[9495]: proxy ft_postfix has no server
> available!
>
>
> I have a look at HAProxy source code and see that it checks if the server
> speaks  (not sure).
>
> I have to choose tcp check instead. It works normally.
>
> Viet
>
>
> On 05/28/2013 12:02 AM, Tom Lanyon wrote:
>
> On 27/05/2013, at 5:04 PM, Vit Dua   
> wrote:
>
>  I have used proxy protocol for SMTP
>
>  
>
>  It worked successfully.
>
> I wanted to do smtpchk so that I added to the config:
>
> server postfix01 X.X.X.X:1 send-proxy check
>
>  
>
> I am running haproxy-1.5_dev18 in front of Postfix 2.10.0 SMTP servers using 
> the PROXY protocol.
>
> As far as I could see in the 1.5_dev18 code, the smtpchk option does not send 
> the appropriate PROXY protocol chatter during the health check when 
> send-proxy is enabled.  I tried to patch this but couldn't see an easy way to 
> do so with the current health check code (I can't remember the details right 
> now).
>
> The two options I came up with were:
>
>   1. fallback to TCP checking
>   2. perform the SMTP check some other way
>
> We are now doing #2 and are doing the SMTP check (without the PROXY protocol) 
> to a different set of listen ports on the Postfix servers.
>
> Here's the relevant parts of our config:
>
> postfix master.cf:
>   # Regular SMTP (no PROXY) on port 'smtp' (TCP 25)
>   smtp  inet  n   -   n   -   -   smtpd
>   # A custom service which expects haproxy's PROXY protocol on 
> non-standard port 9025
>   9025  inet  n   -   n   -   -   smtpd -o 
> smtpd_upstream_proxy_protocol=haproxy
>
> haproxy.cfg:
>   frontend f-smtp
>   bind :25
>   mode tcp
>   timeout client 1m
>   default_backend b-smtp
>
>   backend b-smtp
>   mode tcp
>   option smtpchk
>   timeout server 1m
>   # note: data port 9025 (w/PROXY); smtpchk port 25 (no PROXY).
>   server smtp1 :9025 send-proxy check port 25
>   server smtp2 :9025 send-proxy check port 25
>   server smtp3 :9025 send-proxy check port 25
>
> Tom
>
>
>
>

Re: smtpchk when using proxy protocol

[Vit Dua]

Hi,

There is an option in HAProxy 1.5 doc:

http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#check-send-proxy

And this thread:

http://comments.gmane.org/gmane.comp.web.haproxy/11551

I have set in HAProxy config:

server postfix01 X.X.X.X:1 check check-send-proxy send-proxy


and double-check Postfix's main.cf file:

postscreen_upstream_proxy_protocol = haproxy


but there is still that error:

May 28 10:47:17 localhost haproxy[9495]: Server ft_postfix/postfix01 is 
DOWN, reason: Layer7 invalid response, info: "220-mail.mydomain.com 
ESMTP Postfix (Ubuntu)", check duration: 1ms. 0 active and 0 backup 
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
May 28 10:47:17 localhost haproxy[9495]: proxy ft_postfix has no server 
available!



I have a look at HAProxy source code and see that it checks if the 
server speaks  (not sure).


I have to choose tcp check instead. It works normally.

Viet

On 05/28/2013 12:02 AM, Tom Lanyon wrote:

On 27/05/2013, at 5:04 PM, Vit Dua  wrote:

I have used proxy protocol for SMTP



It worked successfully.

I wanted to do smtpchk so that I added to the config:

server postfix01 X.X.X.X:1 send-proxy check



I am running haproxy-1.5_dev18 in front of Postfix 2.10.0 SMTP servers using 
the PROXY protocol.

As far as I could see in the 1.5_dev18 code, the smtpchk option does not send 
the appropriate PROXY protocol chatter during the health check when send-proxy 
is enabled.  I tried to patch this but couldn't see an easy way to do so with 
the current health check code (I can't remember the details right now).

The two options I came up with were:

1. fallback to TCP checking
2. perform the SMTP check some other way

We are now doing #2 and are doing the SMTP check (without the PROXY protocol) 
to a different set of listen ports on the Postfix servers.

Here's the relevant parts of our config:

postfix master.cf:
# Regular SMTP (no PROXY) on port 'smtp' (TCP 25)
smtp  inet  n   -   n   -   -   smtpd
# A custom service which expects haproxy's PROXY protocol on 
non-standard port 9025
9025  inet  n   -   n   -   -   smtpd -o 
smtpd_upstream_proxy_protocol=haproxy

haproxy.cfg:
frontend f-smtp
bind :25
mode tcp
timeout client 1m
default_backend b-smtp

backend b-smtp
mode tcp
option smtpchk
timeout server 1m
# note: data port 9025 (w/PROXY); smtpchk port 25 (no PROXY).
server smtp1 :9025 send-proxy check port 25
server smtp2 :9025 send-proxy check port 25
server smtp3 :9025 send-proxy check port 25

Tom

Re: smtpchk when using proxy protocol

[Tom Lanyon]

On 27/05/2013, at 5:04 PM, Vit Dua  wrote:
> I have used proxy protocol for SMTP

> It worked successfully.
> 
> I wanted to do smtpchk so that I added to the config:
> 
> server postfix01 X.X.X.X:1 send-proxy check


I am running haproxy-1.5_dev18 in front of Postfix 2.10.0 SMTP servers using 
the PROXY protocol.

As far as I could see in the 1.5_dev18 code, the smtpchk option does not send 
the appropriate PROXY protocol chatter during the health check when send-proxy 
is enabled.  I tried to patch this but couldn't see an easy way to do so with 
the current health check code (I can't remember the details right now).

The two options I came up with were:

1. fallback to TCP checking
2. perform the SMTP check some other way

We are now doing #2 and are doing the SMTP check (without the PROXY protocol) 
to a different set of listen ports on the Postfix servers.

Here's the relevant parts of our config:

postfix master.cf:
# Regular SMTP (no PROXY) on port 'smtp' (TCP 25)
smtp  inet  n   -   n   -   -   smtpd
# A custom service which expects haproxy's PROXY protocol on 
non-standard port 9025
9025  inet  n   -   n   -   -   smtpd -o 
smtpd_upstream_proxy_protocol=haproxy

haproxy.cfg:
frontend f-smtp
bind :25
mode tcp
timeout client 1m 
default_backend b-smtp

backend b-smtp
mode tcp
option smtpchk
timeout server 1m
# note: data port 9025 (w/PROXY); smtpchk port 25 (no PROXY).
server smtp1 :9025 send-proxy check port 25
server smtp2 :9025 send-proxy check port 25
server smtp3 :9025 send-proxy check port 25

Tom

Re: smtpchk when using proxy protocol

[Jonathan Matthews]

On 27 May 2013 08:40, Vit Dua  wrote:
>
> The log also said:
>
> May 27 14:39:11 localhost haproxy[1278]: Proxy ft_postfix started.
> May 27 14:39:11 localhost haproxy[1278]: Server ft_postfix/postfix01 is DOWN, 
> reason: Layer7 invalid response, info: "220-mail.mydomain.com ESMTP Postfix 
> (Ubuntu)", check duration: 1ms. 0 active and 0 backup servers left. 0 
> sessions active, 0 requeued, 0 remaining in queue.
> May 27 14:39:11 localhost haproxy[1278]: proxy ft_postfix has no server 
> available!

This is just a guess (it's been a while since I've run SMTP in anger!) but:

You see the layer 7 response is "220-..."? Well, that hyphen in the
4th character usually means that this is a response that's going to
spill over to the next line. Check out the example in
http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol#Optional_extensions
- see the difference between the 220 and the 250 responses? Only the
/last/ 250 response is without a hyphen.

I wonder if using the PROXY protocol is making the server respond with
>1 line, which is making the smtpchk fail because the first reply it
sees doesn't match "[0-9][0-9][0-9]..." any more.

Is the server definitely set up to accept the PROXY protocol? Remember
that it's generally an SMTP protocol violation for the client to talk
before receiving an SMTP banner - which is what (I /believe/) the
PROXY protocol does. Which would suggest the server has to explicitly
support the PROXY protocol.

Either way - if it's just something being tickled in the postfix code
which replies with a multi-line response or if it doesn't understand
PROXY messages entirely - I'm afraid I don't have any suggestions for
fixing it. You might need to dig further and let us know what you
find!

HTH,
Jonathan
--
Jonathan Matthews // Oxford, London, UK
http://www.jpluscplusm.com/contact.html

Re: smtpchk when using proxy protocol

[Vit Dua]

The log also said:

May 27 14:39:11 localhost haproxy[1278]: Proxy ft_postfix started.
May 27 14:39:11 localhost haproxy[1278]: Server ft_postfix/postfix01 is 
DOWN, reason: Layer7 invalid response, info: "220-mail.mydomain.com 
ESMTP Postfix (Ubuntu)", check duration: 1ms. 0 active and 0 backup 
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
May 27 14:39:11 localhost haproxy[1278]: proxy ft_postfix has no server 
available!




On 05/27/2013 02:34 PM, Vit Dua wrote:

I have used proxy protocol for SMTP as this tutorial:

http://blog.exceliance.fr/2012/06/30/efficient-smtp-relay-infrastructure-with-postfix-and-load-balancers/

HAProxy config:

# this config needs haproxy-1.1.28 or haproxy-1.2.1

global
log 127.0.0.1   local0
log 127.0.0.1   local1
#log loghostlocal0 info
maxconn 4096
#chroot /usr/share/haproxy
user haproxy
group haproxy
daemon
#debug
#quiet

defaults
log global
option  dontlognull
retries 3
option redispatch
maxconn 2000
contimeout  5000
clitimeout  5
srvtimeout  5

listen ft_postfix 0.0.0.0:25
mode tcp
balance roundrobin
option tcplog
option smtpchk HELO haproxy01.mydomain.com
server postfix01 X.X.X.X:1 send-proxy #check


It worked successfully.

I wanted to do smtpchk so that I added to the config:

server postfix01 X.X.X.X:1 send-proxy check


Then the log told that the server was down though it is not:

May 27 14:07:51 localhost haproxy[6890]: Proxy ft_postfix started.
May 27 14:07:51 localhost haproxy[6890]: Server ft_postfix/postfix01 
is DOWN, reason: Initializing, check duration: 0ms. 0 active and 0 
backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
May 27 14:07:51 localhost haproxy[6890]: proxy ft_postfix has no 
server available!



I also tried

server postfix01 X.X.X.X:1 check send-proxy


but the result is the same.

I could do smtpchk normally without proxy protocol:

May 27 14:30:02 localhost haproxy[24755]: Server ft_postfix/postfix01 
is UP, reason: Layer7 check passed, code: 220, info: 
"mail.mydomain.com ESMTP Postfix (Ubuntu)", check duration: 2ms. 1 
active and 0 backup servers online. 0 sessions requeued, 0 total in queue.



I am using 
http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev18.tar.gz 
on Ubuntu 12.04.


Viet