Hi Andrey,
On Tue, Jul 08, 2014 at 02:21:47PM +0000, Andrey Zakabluk wrote:
> Hi!I use haproxy for balancing http request - this is better LB)! And now I
> tru ballancing EDIFACT files. 1st step I try just forwarding request between
> server GMD and server COMARCH via Haproxy.
> ----------Plan(How this work?)
>
> server GMD send EDIFACT file to --> haproxy:7446, haproxy send --> to server
> COMARCH:7446(processed EDIFACT file) and send response to HAPROXY-->
> Haproxy:3038, haproxy send to --> GMD:3038
> -----------------------------
> --version Haproxy
> HA-Proxy version 1.5.1 2014/06/24
> Copyright 2000-2014 Willy Tarreau <[email protected]>
>
> Build options :
> TARGET = linux2628
> CPU = generic
> CC = gcc
> CFLAGS = -O2 -g -fno-strict-aliasing
> OPTIONS =
>
> Default settings :
> maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
>
> Encrypted password support via crypt(3): yes
> Built without zlib support (USE_ZLIB not set)
> Compression algorithms supported : identity
> Built without OpenSSL support (USE_OPENSSL not set)
> Built without PCRE support (using libc's regex instead)
> Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
> IP_FREEBIND
>
> Available polling systems :
> epoll : pref=300, test result OK
> poll : pref=200, test result OK
> select : pref=150, test result OK
> Total: 3 (3 usable), will use epoll.
> -- Config haproxy
>
> global
> # daemon
> maxconn 10240
> debug
> log 127.0.0.1 local1 debug
> stats socket /opt/haproxy/gmd_bscs_prod/socket_s1cm1/haproxy.sock mode 06
> 00 level admin
> log-tag haproxy
> #nbproc 4
> defaults
> mode tcp
> log global
> option tcplog
> timeout connect 50s
> timeout client 30s
> timeout server 30s
>
> frontend gmdcom1
> bind 10.254.13.100:7446
> default_backend comarch
> mode tcp
> log global
> option tcplog
>
> #
>
> frontend gmdrrs1
> bind 10.254.13.100:3038
> default_backend gmdrrs
> mode tcp
> log global
> option tcplog
>
>
> backend comarch
> balance roundrobin
> mode tcp
> server mdsp_7446 10.254.13.40:7446 check port 7446
>
> backend gmdrrs
> balance roundrobin
> mode tcp
> server rrs_3038 s1cm1t:3038 check port 3038
>
>
>
> -------------- Debug log Haproxy
> Available polling systems :
> epoll : pref=300, test result OK
> poll : pref=200, test result OK
> select : pref=150, test result FAILED
> Total: 3 (2 usable), will use epoll.
> .........................
> 0 sessions requeued, 0 total in queue.
> 00000000:gmdcom1.accept(0005)=0008 from [10.254.62.75:60287]
> 00000000:comarch.srvcls[0008:0009]
> 00000000:comarch.clicls[0008:0009]
> 00000000:comarch.closed[0008:0009]
> 00000001:gmdrrs1.accept(0007)=0008 from [10.254.13.40:21481]
> 00000001:gmdrrs.srvcls[0008:0009]
> 00000001:gmdrrs.clicls[0008:0009]
> 00000001:gmdrrs.closed[0008:0009]
> 00000002:gmdcom1.accept(0005)=0008 from [10.254.62.75:60292]
> 00000002:comarch.srvcls[0008:0009]
> 00000002:comarch.clicls[0008:0009]
> 00000002:comarch.closed[0008:0009]
> 00000003:gmdrrs1.accept(0007)=0008 from [10.254.13.40:21482]
> 00000003:gmdrrs.srvcls[0008:0009]
> 00000003:gmdrrs.clicls[0008:0009]
> 00000003:gmdrrs.closed[0008:0009]
> 00000004:gmdcom1.accept(0005)=0008 from [10.254.62.75:60340]
> 00000004:comarch.srvcls[0008:0009]
> 00000004:comarch.clicls[0008:0009]
> 00000004:comarch.closed[0008:0009]
> 00000005:gmdrrs1.accept(0007)=0008 from [10.254.13.40:21484]
> 00000005:gmdrrs.srvcls[0008:0009]
> 00000005:gmdrrs.clicls[0008:0009]
> 00000005:gmdrrs.closed[0008:0009]
> 00000006:gmdcom1.accept(0005)=0008 from [10.254.62.75:60467]
> 00000006:comarch.srvcls[0008:0009]
> 00000006:comarch.clicls[0008:0009]
> 00000006:comarch.closed[0008:0009]
> 00000007:gmdrrs1.accept(0007)=0008 from [10.254.13.40:21503]
> 00000007:gmdrrs.srvcls[0008:0009]
> 00000007:gmdrrs.clicls[0008:0009]
> 00000007:gmdrrs.closed[0008:0009]
> 00000008:gmdcom1.accept(0005)=0008 from [10.254.62.75:60700]
> 00000008:comarch.srvcls[0008:0009]
> 00000008:comarch.clicls[0008:0009]
> 00000008:comarch.closed[0008:0009]
> 00000009:gmdrrs1.accept(0007)=0008 from [10.254.13.40:21521]
> 00000009:gmdrrs.srvcls[0008:0009]
> 00000009:gmdrrs.clicls[0008:0009]
> 00000009:gmdrrs.closed[0008:0009]
> 0000000a:gmdcom1.accept(0005)=0008 from [10.254.62.75:60729]
> 0000000a:comarch.srvcls[0008:0009]
> 0000000a:comarch.clicls[0008:0009]
> 0000000a:comarch.closed[0008:0009]
> 0000000b:gmdrrs1.accept(0007)=0008 from [10.254.13.40:21522]
> 0000000b:gmdrrs.srvcls[0008:0009]
> 0000000b:gmdrrs.clicls[0008:0009]
> 0000000b:gmdrrs.closed[0008:0009]
> 0000000c:gmdcom1.accept(0005)=0008 from [10.254.62.75:63061]
> 0000000c:comarch.srvcls[0008:0009]
> 0000000c:comarch.clicls[0008:0009]
> 0000000c:comarch.closed[0008:0009]
> 0000000d:gmdrrs1.accept(0007)=0008 from [10.254.13.40:21604]
> 0000000d:gmdrrs.srvcls[0008:0009]
> 0000000d:gmdrrs.clicls[0008:0009]
> 0000000d:gmdrrs.closed[0008:0009]
> 0000000e:gmdcom1.accept(0005)=0008 from [10.254.62.75:64621]
> 0000000e:comarch.srvcls[0008:0009]
> 0000000e:comarch.clicls[0008:0009]
> 0000000e:comarch.closed[0008:0009]
> 0000000f:gmdrrs1.accept(0007)=0008 from [10.254.13.40:21758]
> 0000000f:gmdrrs.srvcls[0008:0009]
> 0000000f:gmdrrs.clicls[0008:0009]
> 0000000f:gmdrrs.closed[0008:0009]
>
> -------------------------log GMD server (application what send EDIFACT files)
> - sometimes I have errors, number errors grow with grow number requests from
> CMD.
> *******************************************************************************
> module: GMDIPC date/time: Wed Jul 02 13:49:50 2014
> group: MEDIATION pid: 14785 line: 939
> error number: -1021 severity: WARNING
> -------------------------------------------------------------------------------
> /main/f4/y12/y132/0;lhsj_main/bscs/batch/src/gmd3g/common/gmdipc.c, ,
> J_BSCSIX_R3_VEL06.00, J_BSCSIX_R3_VEL06.00_140606;2014/01/29
> -------------------------------------------------------------------------------
> foiGMDSendEDIandRDRMessage [50134553]: Can't connect to server process COMARC2
> on host <10.254.13.100> and port <7446>.
> Systemerror[146] = <Connection refused>
> *******************************************************************************
>
> *******************************************************************************
> module: GMDIPC date/time: Wed Jul 02 13:49:50 2014
> group: MEDIATION pid: 14785 line: 939
> error number: -1021 severity: WARNING
> -------------------------------------------------------------------------------
> /main/f4/y12/y132/0;lhsj_main/bscs/batch/src/gmd3g/common/gmdipc.c, ,
> J_BSCSIX_R3_VEL06.00, J_BSCSIX_R3_VEL06.00_140606;2014/01/29
> -------------------------------------------------------------------------------
> foiGMDSendEDIandRDRMessage [50134553]: Can't connect to server process COMARC2
> on host <10.254.13.100> and port <7446>.
> Systemerror[146] = <Connection refused>
> *******************************************************************************
>
> *******************************************************************************
> module: GMDIPC date/time: Wed Jul 02 13:49:50 2014
> group: MEDIATION pid: 14785 line: 966
> error number: -1021 severity: WARNING
> -------------------------------------------------------------------------------
> /main/f4/y12/y132/0;lhsj_main/bscs/batch/src/gmd3g/common/gmdipc.c, ,
> J_BSCSIX_R3_VEL06.00, J_BSCSIX_R3_VEL06.00_140606;2014/01/29
> -------------------------------------------------------------------------------
> foiGMDSendEDIandRDRMessage [50134553]: Was not able to connect to any valid
> VMD instance!
> *******************************************************************************
>
> *******************************************************************************
> module: GMDCOM date/time: Wed Jul 02 13:49:50 2014
> group: MEDIATION pid: 14785 line: 953
> error number: severity: WARNING
> -------------------------------------------------------------------------------
> /main/f4/f6/y12/y132/1;lhsj_main/bscs/batch/src/gmd3g/gmdcom/gmdcom.c, ,
> J_BSCSIX_R3_VEL06.00, J_BSCSIX_R3_VEL06.00_140606;2014/06/04
> -------------------------------------------------------------------------------
> foiGMDSendDispatcher [50134553]: Couldn't send EDIFACT message to VMD.
>
> *******************************************************************************
>
> --------------------------------sysctl
>
> # Controls the default maxmimum size of a mesage queue
> kernel.msgmnb = 65536
>
> # Controls the maximum size of a message, in bytes
> kernel.msgmax = 65536
>
> # Controls the maximum shared segment size, in bytes
> kernel.shmmax = 68719476736
>
> # Controls the maximum number of shared memory segments, in pages
> kernel.shmall = 4294967296
> net.ipv4.ip_nonlocal_bind=1
>
> Percentage requests from GMD processed but percentage request in status fail,
> because I have time out connection to host with haproxy on the port 7446.
> After fixed this is issue I will try use Load Balancing for more instance
> GMD... I hope!
I'm wondering what could cause these "Connection refused" because the
destination is the address where your haproxy is listening.
Could you check 3 things
1) that you have no other machine with the same IP address on the network
Use "arping" for this
2) that iptables is not running on the haproxy machine : it could be that
someone has written overzealous rules that cause some packets to be
rejected, which would lead to a "connection refused" on the client.
3) disable health checks on the servers in the haproxy config. One
possibility could be that when servers go down, incoming connections
are immediately rejected, fast enough for the client to get the RST
before sending anything, and that the client would log a Connection
Refused despite having received the SYN-ACK prior to this reset.
I tend to think that #2 is the most likely to be the cause. I even assisted
to something similar very recently again.
Regards,
Willy
