So after more testing, it does a appear to be the keep-alive with the thrift server. I am currently running Hive 0.13.0 which doesn't support a keep-alive, I am hoping when I am able to upgrade, that the keep alive helps me through this issue. Any HAProxy experts out there have ideas for dealing with an app that doesn't have a TCP keep alive?
On Wed, Mar 25, 2015 at 12:17 PM, Ben Whitehead <[email protected]> wrote: > Awesome glad to hear it. > > On Tue, Mar 24, 2015 at 10:00 PM, xiaokun <[email protected]> wrote: >> >> After set the timeout value, the tcp request become stable. >> >> 2015-03-23 23:26 GMT+08:00 John Omernik <[email protected]>: >>> >>> So per my HA-Proxy logs, I am getting a condition where the client >>> side time expired while waiting for the client to send or receive >>> data, and this is occurring during the DATA phase. (the two letter >>> HAProxy code is cD . To test, I expanded the timeout (on all things) >>> to 100000. >>> >>> This was interesting in that it showed me the problem a bit clearer. >>> Now, as long as I submit and results for a query in the 100 second >>> range, my connection stays open and happy. If I let it sit for 100 >>> seconds, and then try a query, then it breaks again. >>> >>> It appears HA proxy gets irritable when things aren't being sent over >>> a connection for a long period of time. For services like Minecraft, >>> this is not an issue, when a client is connected, there is always data >>> there, and no issues occur. For Thrift, there may be multiple long >>> pauses as an analyst thinks through the next query etc. I wonder what >>> would be the best way to handle that on a per container level. (not >>> all containers need this sort of customization...) >>> >>> >>> >>> On Mon, Mar 23, 2015 at 5:44 AM, xiaokun <[email protected]> wrote: >>> > I have a similar error when launch jenkins in mesos cluster. And I use >>> > HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from >>> > time >>> > to time, connection between jenkins master and jenkins slave will lost, >>> > while will cause jobs fail. >>> > >>> > I will try to set timeout value for HAProxy >>> > >>> > 2015-03-23 4:18 GMT+08:00 John Omernik <[email protected]>: >>> >> >>> >> Yep all three are going through HAProxy and seem to be working well. >>> >> I should do some testing to determine if I run a thrift server that >>> >> connects to metastore in HAProxy to determine if that's the broken >>> >> link, but I believe it's between the thrift server. I will check out >>> >> the link you posted to get a better idea out of how it works. I am >>> >> running this all on a MapR cluster which is Great for Docker/Mesos as >>> >> I used the MapR FS, mounted locally on each node's NFS to put the data >>> >> for MySQL, the logs for the metastore/thift server, and my world files >>> >> for Minecraft. So far it's all really slick. >>> >> >>> >> On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead >>> >> <[email protected]> wrote: >>> >> > It sounds like you may be running into the connection timeout that >>> >> > is >>> >> > configured for the HAProxy instance. >>> >> > >>> >> > You can see an instance of this in Marathon's >>> >> > HAProxy-marathon-bridge[1]. >>> >> > You may need to raise the timeout values. >>> >> > >>> >> > Are you accessing MySql, Hive metastore and minecraft through >>> >> > HAProxy as >>> >> > well? I know that Minecraft running in docker on mesos works well, >>> >> > though >>> >> > I've always used a direct connection not a proxied one. >>> >> > >>> >> > >>> >> > [1] >>> >> > >>> >> > >>> >> > https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119 >>> >> > >>> >> > On Sat, Mar 21, 2015 at 1:50 PM, John Omernik <[email protected]> >>> >> > wrote: >>> >> >> >>> >> >> I have a nice setup with a Hive thrift server running in a docker >>> >> >> container on Mesos. It works pretty well, but something, I believe >>> >> >> in >>> >> >> how HAProxy works with the connection, is causing the thrift server >>> >> >> connection to die after a time. Basically, I can run a few >>> >> >> queries, >>> >> >> but after 2 or 3, or specifically after a longer query, I get the >>> >> >> error below indicating End of File on the connection. Then no more >>> >> >> connections work until I reestablish the connection to the thrift >>> >> >> server. I've tried looking in logs. the thrift server std err logs >>> >> >> show no issues. I am guess I need to dig into haproxy logs, but I >>> >> >> am >>> >> >> not seeing any issues in syslog so far. I'd love any pointers on >>> >> >> how >>> >> >> to trouble shoot this. By the way, I have MySQl, hive metastore, >>> >> >> and >>> >> >> a minecraft server all running on Mesos/Docker with no issues, not >>> >> >> sure why the thrift server is so sensitive. >>> >> >> >>> >> >> :) >>> >> >> >>> >> >> >>> >> >> pyhs2 connection string: >>> >> >> >>> >> >> hs2 = >>> >> >> >>> >> >> >>> >> >> pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default') >>> >> >> >>> >> >> >>> >> >> Error >>> >> >> >>> >> >> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc >>> >> >> in >>> >> >> read(self, sz) >>> >> >> 118 if len(buff) == 0: >>> >> >> 119 raise >>> >> >> TTransportException(type=TTransportException.END_OF_FILE, >>> >> >> --> 120 message='TSocket read 0 >>> >> >> bytes') >>> >> >> 121 return buff >>> >> >> 122 >>> >> >> >>> >> >> TTransportException: TSocket read 0 bytes >>> >> > >>> >> > >>> > >>> > >> >> >

