Re: HAProxy for Hive Thrift Server on Mesos
Awesome glad to hear it. On Tue, Mar 24, 2015 at 10:00 PM, xiaokun xiaokun...@gmail.com wrote: After set the timeout value, the tcp request become stable. 2015-03-23 23:26 GMT+08:00 John Omernik j...@omernik.com: So per my HA-Proxy logs, I am getting a condition where the client side time expired while waiting for the client to send or receive data, and this is occurring during the DATA phase. (the two letter HAProxy code is cD . To test, I expanded the timeout (on all things) to 10. This was interesting in that it showed me the problem a bit clearer. Now, as long as I submit and results for a query in the 100 second range, my connection stays open and happy. If I let it sit for 100 seconds, and then try a query, then it breaks again. It appears HA proxy gets irritable when things aren't being sent over a connection for a long period of time. For services like Minecraft, this is not an issue, when a client is connected, there is always data there, and no issues occur. For Thrift, there may be multiple long pauses as an analyst thinks through the next query etc. I wonder what would be the best way to handle that on a per container level. (not all containers need this sort of customization...) On Mon, Mar 23, 2015 at 5:44 AM, xiaokun xiaokun...@gmail.com wrote: I have a similar error when launch jenkins in mesos cluster. And I use HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from time to time, connection between jenkins master and jenkins slave will lost, while will cause jobs fail. I will try to set timeout value for HAProxy 2015-03-23 4:18 GMT+08:00 John Omernik j...@omernik.com: Yep all three are going through HAProxy and seem to be working well. I should do some testing to determine if I run a thrift server that connects to metastore in HAProxy to determine if that's the broken link, but I believe it's between the thrift server. I will check out the link you posted to get a better idea out of how it works. I am running this all on a MapR cluster which is Great for Docker/Mesos as I used the MapR FS, mounted locally on each node's NFS to put the data for MySQL, the logs for the metastore/thift server, and my world files for Minecraft. So far it's all really slick. On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead ben.whiteh...@mesosphere.io wrote: It sounds like you may be running into the connection timeout that is configured for the HAProxy instance. You can see an instance of this in Marathon's HAProxy-marathon-bridge[1]. You may need to raise the timeout values. Are you accessing MySql, Hive metastore and minecraft through HAProxy as well? I know that Minecraft running in docker on mesos works well, though I've always used a direct connection not a proxied one. [1] https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119 On Sat, Mar 21, 2015 at 1:50 PM, John Omernik j...@omernik.com wrote: I have a nice setup with a Hive thrift server running in a docker container on Mesos. It works pretty well, but something, I believe in how HAProxy works with the connection, is causing the thrift server connection to die after a time. Basically, I can run a few queries, but after 2 or 3, or specifically after a longer query, I get the error below indicating End of File on the connection. Then no more connections work until I reestablish the connection to the thrift server. I've tried looking in logs. the thrift server std err logs show no issues. I am guess I need to dig into haproxy logs, but I am not seeing any issues in syslog so far. I'd love any pointers on how to trouble shoot this. By the way, I have MySQl, hive metastore, and a minecraft server all running on Mesos/Docker with no issues, not sure why the thrift server is so sensitive. :) pyhs2 connection string: hs2 = pyhs2.connect(host='marathonmaster',port=1,authMechanism='PLAIN',user='bestuser',password='removed',database='default') Error /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc in read(self, sz) 118 if len(buff) == 0: 119 raise TTransportException(type=TTransportException.END_OF_FILE, -- 120 message='TSocket read 0 bytes') 121 return buff 122 TTransportException: TSocket read 0 bytes
Re: HAProxy for Hive Thrift Server on Mesos
After set the timeout value, the tcp request become stable. 2015-03-23 23:26 GMT+08:00 John Omernik j...@omernik.com: So per my HA-Proxy logs, I am getting a condition where the client side time expired while waiting for the client to send or receive data, and this is occurring during the DATA phase. (the two letter HAProxy code is cD . To test, I expanded the timeout (on all things) to 10. This was interesting in that it showed me the problem a bit clearer. Now, as long as I submit and results for a query in the 100 second range, my connection stays open and happy. If I let it sit for 100 seconds, and then try a query, then it breaks again. It appears HA proxy gets irritable when things aren't being sent over a connection for a long period of time. For services like Minecraft, this is not an issue, when a client is connected, there is always data there, and no issues occur. For Thrift, there may be multiple long pauses as an analyst thinks through the next query etc. I wonder what would be the best way to handle that on a per container level. (not all containers need this sort of customization...) On Mon, Mar 23, 2015 at 5:44 AM, xiaokun xiaokun...@gmail.com wrote: I have a similar error when launch jenkins in mesos cluster. And I use HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from time to time, connection between jenkins master and jenkins slave will lost, while will cause jobs fail. I will try to set timeout value for HAProxy 2015-03-23 4:18 GMT+08:00 John Omernik j...@omernik.com: Yep all three are going through HAProxy and seem to be working well. I should do some testing to determine if I run a thrift server that connects to metastore in HAProxy to determine if that's the broken link, but I believe it's between the thrift server. I will check out the link you posted to get a better idea out of how it works. I am running this all on a MapR cluster which is Great for Docker/Mesos as I used the MapR FS, mounted locally on each node's NFS to put the data for MySQL, the logs for the metastore/thift server, and my world files for Minecraft. So far it's all really slick. On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead ben.whiteh...@mesosphere.io wrote: It sounds like you may be running into the connection timeout that is configured for the HAProxy instance. You can see an instance of this in Marathon's HAProxy-marathon-bridge[1]. You may need to raise the timeout values. Are you accessing MySql, Hive metastore and minecraft through HAProxy as well? I know that Minecraft running in docker on mesos works well, though I've always used a direct connection not a proxied one. [1] https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119 On Sat, Mar 21, 2015 at 1:50 PM, John Omernik j...@omernik.com wrote: I have a nice setup with a Hive thrift server running in a docker container on Mesos. It works pretty well, but something, I believe in how HAProxy works with the connection, is causing the thrift server connection to die after a time. Basically, I can run a few queries, but after 2 or 3, or specifically after a longer query, I get the error below indicating End of File on the connection. Then no more connections work until I reestablish the connection to the thrift server. I've tried looking in logs. the thrift server std err logs show no issues. I am guess I need to dig into haproxy logs, but I am not seeing any issues in syslog so far. I'd love any pointers on how to trouble shoot this. By the way, I have MySQl, hive metastore, and a minecraft server all running on Mesos/Docker with no issues, not sure why the thrift server is so sensitive. :) pyhs2 connection string: hs2 = pyhs2.connect(host='marathonmaster',port=1,authMechanism='PLAIN',user='bestuser',password='removed',database='default') Error /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc in read(self, sz) 118 if len(buff) == 0: 119 raise TTransportException(type=TTransportException.END_OF_FILE, -- 120 message='TSocket read 0 bytes') 121 return buff 122 TTransportException: TSocket read 0 bytes
Re: HAProxy for Hive Thrift Server on Mesos
I have a similar error when launch jenkins in mesos cluster. And I use HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from time to time, connection between jenkins master and jenkins slave will lost, while will cause jobs fail. I will try to set timeout value for HAProxy 2015-03-23 4:18 GMT+08:00 John Omernik j...@omernik.com: Yep all three are going through HAProxy and seem to be working well. I should do some testing to determine if I run a thrift server that connects to metastore in HAProxy to determine if that's the broken link, but I believe it's between the thrift server. I will check out the link you posted to get a better idea out of how it works. I am running this all on a MapR cluster which is Great for Docker/Mesos as I used the MapR FS, mounted locally on each node's NFS to put the data for MySQL, the logs for the metastore/thift server, and my world files for Minecraft. So far it's all really slick. On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead ben.whiteh...@mesosphere.io wrote: It sounds like you may be running into the connection timeout that is configured for the HAProxy instance. You can see an instance of this in Marathon's HAProxy-marathon-bridge[1]. You may need to raise the timeout values. Are you accessing MySql, Hive metastore and minecraft through HAProxy as well? I know that Minecraft running in docker on mesos works well, though I've always used a direct connection not a proxied one. [1] https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119 On Sat, Mar 21, 2015 at 1:50 PM, John Omernik j...@omernik.com wrote: I have a nice setup with a Hive thrift server running in a docker container on Mesos. It works pretty well, but something, I believe in how HAProxy works with the connection, is causing the thrift server connection to die after a time. Basically, I can run a few queries, but after 2 or 3, or specifically after a longer query, I get the error below indicating End of File on the connection. Then no more connections work until I reestablish the connection to the thrift server. I've tried looking in logs. the thrift server std err logs show no issues. I am guess I need to dig into haproxy logs, but I am not seeing any issues in syslog so far. I'd love any pointers on how to trouble shoot this. By the way, I have MySQl, hive metastore, and a minecraft server all running on Mesos/Docker with no issues, not sure why the thrift server is so sensitive. :) pyhs2 connection string: hs2 = pyhs2.connect(host='marathonmaster',port=1,authMechanism='PLAIN',user='bestuser',password='removed',database='default') Error /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc in read(self, sz) 118 if len(buff) == 0: 119 raise TTransportException(type=TTransportException.END_OF_FILE, -- 120 message='TSocket read 0 bytes') 121 return buff 122 TTransportException: TSocket read 0 bytes
HAProxy for Hive Thrift Server on Mesos
I have a nice setup with a Hive thrift server running in a docker container on Mesos. It works pretty well, but something, I believe in how HAProxy works with the connection, is causing the thrift server connection to die after a time. Basically, I can run a few queries, but after 2 or 3, or specifically after a longer query, I get the error below indicating End of File on the connection. Then no more connections work until I reestablish the connection to the thrift server. I've tried looking in logs. the thrift server std err logs show no issues. I am guess I need to dig into haproxy logs, but I am not seeing any issues in syslog so far. I'd love any pointers on how to trouble shoot this. By the way, I have MySQl, hive metastore, and a minecraft server all running on Mesos/Docker with no issues, not sure why the thrift server is so sensitive. :) pyhs2 connection string: hs2 = pyhs2.connect(host='marathonmaster',port=1,authMechanism='PLAIN',user='bestuser',password='removed',database='default') Error /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc in read(self, sz) 118 if len(buff) == 0: 119 raise TTransportException(type=TTransportException.END_OF_FILE, -- 120 message='TSocket read 0 bytes') 121 return buff 122 TTransportException: TSocket read 0 bytes
