Travis - What are the client failures you are seeing? What Riak client library are you using, and are you using the PB or HTTP interface to Riak?
The error message you provided indicates that the ping request returned from Riak after haproxy closed the socket for the request. One cause would be very high server load causing timeouts. -- Luke Bakken Engineer lbak...@basho.com On Wed, Sep 14, 2016 at 7:23 AM, Travis Kirstine <tkirst...@firstbasesolutions.com> wrote: > Hi all, > > > > I’m using haproxy with a riak 2.1.4 in a 5 node cluster. I’m getting fairly > consistent enotconn errors in riak which happen to coincide with client > failures. We’ve setup haproxy as recommended > (https://gist.github.com/gburd/1507077) see below. I’m running a leveldb > backend with 9 GB max memory (I can go higher if needed). I’m not sure at > this point if I have a network issue or leveldb / riak issue. > > > > 2016-09-14 08:10:16 =CRASH REPORT==== > > crasher: > > initial call: mochiweb_acceptor:init/3 > > pid: <0.28104.442> > > registered_name: [] > > exception error: > {function_clause,[{webmachine_request,peer_from_peername,[{error,enotconn},{webmachine_request,{wm_reqstate,#Port<0.6539192>,[],undefined,undefined,undefined,{wm_reqdata,'GET',http,{1,0},"defined_in_wm_req_srv_init","defined_in_wm_req_srv_init",defined_on_call,defined_in_load_dispatch_data,"/ping","/ping",[],defined_in_load_dispatch_data,"defined_in_load_dispatch_data",500,1073741824,67108864,[],[],{0,nil},not_fetched_yet,false,{0,nil},<<>>,follow_request,undefined,undefined,[]},undefined,undefined,undefined}}],[{file,"src/webmachine_request.erl"},{line,150}]},{webmachine_request,get_peer,1,[{file,"src/webmachine_request.erl"},{line,124}]},{webmachine,new_request,2,[{file,"src/webmachine.erl"},{line,69}]},{webmachine_mochiweb,loop,2,[{file,"src/webmachine_mochiweb.erl"},{line,49}]},{mochiweb_http,headers,5,[{file,"src/mochiweb_http.erl"},{line,96}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]} > > ancestors: ['http://192.168.18.64:8098_mochiweb',riak_api_sup,<0.319.0>] > > messages: [] > > links: [<0.325.0>,#Port<0.6539192>] > > dictionary: [] > > trap_exit: false > > status: running > > heap_size: 987 > > stack_size: 27 > > reductions: 963 > > neighbours: > > > > > > # haproxy.cfg > > global > > log 127.0.0.1 local2 info > > chroot /var/lib/haproxy > > pidfile /var/run/haproxy.pid > > maxconn 256000 > > user haproxy > > group haproxy > > spread-checks 5 > > daemon > > quiet > > stats socket /var/lib/haproxy/stats > > defaults > > log global > > option httplog > > option dontlognull > > option redispatch > > timeout connect 5000 > > maxconn 256000 > > > > frontend main *:80 > > mode http > > acl url_static path_beg -i /static /images /javascript > /stylesheets > > acl url_static path_end -i .jpg .gif .png .css .js > > > > backend static > > balance roundrobin > > server static 127.0.0.1:4331 check > > > > backend app > > mode http > > balance roundrobin > > server wmts1riak 192.168.18.72:80 check > > server wmts2riak 192.168.18.73:80 check > > > > backend riak_rest_backend > > mode http > > balance roundrobin > > option httpchk GET /ping > > option httplog > > server riak1 192.168.18.64:8098 weight 1 maxconn 1024 check > > server riak2 192.168.18.65:8098 weight 1 maxconn 1024 check > > server riak3 192.168.18.66:8098 weight 1 maxconn 1024 check > > server riak4 192.168.18.67:8098 weight 1 maxconn 1024 check > > server riak5 192.168.18.68:8098 weight 1 maxconn 1024 check > > > > frontend riak_rest > > bind *:8098 > > mode http > > option contstats > > default_backend riak_rest_backend > > > > backend riak_protocol_buffer_backend > > balance leastconn > > mode tcp > > option tcpka > > option srvtcpka > > server riak1 192.168.18.64:8087 weight 1 maxconn 1024 check > > server riak2 192.168.18.65:8087 weight 1 maxconn 1024 check > > server riak3 192.168.18.66:8087 weight 1 maxconn 1024 check > > server riak4 192.168.18.67:8087 weight 1 maxconn 1024 check > > server riak5 192.168.18.68:8087 weight 1 maxconn 1024 check > > > > frontend riak_protocol_buffer > > bind *:8087 > > mode tcp > > option tcplog > > option contstats > > mode tcp > > option tcpka > > option srvtcpka > > default_backend riak_protocol_buffer_backend > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
