So after more testing, it does a appear to be the keep-alive with the
thrift server. I am currently running Hive 0.13.0 which doesn't
support a keep-alive, I am hoping when I am able to upgrade, that the
keep alive helps me through this issue.  Any HAProxy experts out there
have ideas for dealing with an app that doesn't have a TCP keep alive?

On Wed, Mar 25, 2015 at 12:17 PM, Ben Whitehead
<[email protected]> wrote:
> Awesome glad to hear it.
>
> On Tue, Mar 24, 2015 at 10:00 PM, xiaokun <[email protected]> wrote:
>>
>> After set the timeout value, the tcp request become stable.
>>
>> 2015-03-23 23:26 GMT+08:00 John Omernik <[email protected]>:
>>>
>>> So per my HA-Proxy logs, I am getting a condition where the client
>>> side time expired while waiting for the client to send or receive
>>> data, and this is occurring during the DATA phase. (the two letter
>>> HAProxy code is cD . To test, I expanded the timeout (on all things)
>>> to 100000.
>>>
>>> This was interesting in that it showed me the problem a bit clearer.
>>> Now, as long as I submit and results for a query in the 100 second
>>> range, my connection stays open and happy.  If I let it sit for 100
>>> seconds, and then try a query, then it breaks again.
>>>
>>> It appears HA proxy gets irritable when things aren't being sent over
>>> a connection for a long period of time.  For services like Minecraft,
>>> this is not an issue, when a client is connected, there is always data
>>> there, and no issues occur. For Thrift, there may be multiple long
>>> pauses as an analyst thinks through the next query etc.  I wonder what
>>> would be the best way to handle that on a per container level. (not
>>> all containers need this sort of customization...)
>>>
>>>
>>>
>>> On Mon, Mar 23, 2015 at 5:44 AM, xiaokun <[email protected]> wrote:
>>> > I have a similar error when launch jenkins in mesos cluster. And I use
>>> > HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from
>>> > time
>>> > to time, connection between jenkins master and jenkins slave will lost,
>>> > while will cause jobs fail.
>>> >
>>> > I will try to set timeout value for HAProxy
>>> >
>>> > 2015-03-23 4:18 GMT+08:00 John Omernik <[email protected]>:
>>> >>
>>> >> Yep all three are going through HAProxy and seem to be working well.
>>> >> I should do some testing to determine if I run a thrift server that
>>> >> connects to metastore in HAProxy to determine if that's the broken
>>> >> link, but I believe it's between the thrift server. I will check out
>>> >> the link you posted to get a better idea out of how it works. I am
>>> >> running this all on a MapR cluster which is Great for Docker/Mesos as
>>> >> I used the MapR FS, mounted locally on each node's NFS to put the data
>>> >> for MySQL, the logs for the metastore/thift server, and my world files
>>> >> for Minecraft.  So far it's all really slick.
>>> >>
>>> >> On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead
>>> >> <[email protected]> wrote:
>>> >> > It sounds like you may be running into the connection timeout that
>>> >> > is
>>> >> > configured for the HAProxy instance.
>>> >> >
>>> >> > You can see an instance of this in Marathon's
>>> >> > HAProxy-marathon-bridge[1].
>>> >> > You may need to raise the timeout values.
>>> >> >
>>> >> > Are you accessing MySql, Hive metastore and minecraft through
>>> >> > HAProxy as
>>> >> > well? I know that Minecraft running in docker on mesos works well,
>>> >> > though
>>> >> > I've always used a direct connection not a proxied one.
>>> >> >
>>> >> >
>>> >> > [1]
>>> >> >
>>> >> >
>>> >> > https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119
>>> >> >
>>> >> > On Sat, Mar 21, 2015 at 1:50 PM, John Omernik <[email protected]>
>>> >> > wrote:
>>> >> >>
>>> >> >> I have a nice setup with a Hive thrift server running in a docker
>>> >> >> container on Mesos. It works pretty well, but something, I believe
>>> >> >> in
>>> >> >> how HAProxy works with the connection, is causing the thrift server
>>> >> >> connection  to die after a time.  Basically, I can run a few
>>> >> >> queries,
>>> >> >> but after 2 or 3, or specifically after a longer query, I get the
>>> >> >> error below indicating End of File on the connection.  Then no more
>>> >> >> connections work until I reestablish the connection to the thrift
>>> >> >> server. I've tried looking in logs. the thrift server std err logs
>>> >> >> show no issues. I am guess I need to dig into haproxy logs, but I
>>> >> >> am
>>> >> >> not seeing any issues in syslog so far.  I'd love any pointers on
>>> >> >> how
>>> >> >> to trouble shoot this.  By the way, I have MySQl, hive metastore,
>>> >> >> and
>>> >> >> a minecraft server all running on Mesos/Docker with no issues, not
>>> >> >> sure why the thrift server is so sensitive.
>>> >> >>
>>> >> >> :)
>>> >> >>
>>> >> >>
>>> >> >> pyhs2 connection string:
>>> >> >>
>>> >> >> hs2 =
>>> >> >>
>>> >> >>
>>> >> >> pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default')
>>> >> >>
>>> >> >>
>>> >> >> Error
>>> >> >>
>>> >> >> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc
>>> >> >> in
>>> >> >> read(self, sz)
>>> >> >>     118     if len(buff) == 0:
>>> >> >>     119       raise
>>> >> >> TTransportException(type=TTransportException.END_OF_FILE,
>>> >> >> --> 120                                 message='TSocket read 0
>>> >> >> bytes')
>>> >> >>     121     return buff
>>> >> >>     122
>>> >> >>
>>> >> >> TTransportException: TSocket read 0 bytes
>>> >> >
>>> >> >
>>> >
>>> >
>>
>>
>

Reply via email to