Re: Having a Connections Leak with the Hive Server
Scott, after re-reading your original email, I'm thinking maybe we didn't have the same problem. Hive crashed for us when it ran out of file descriptors, it didn't hang. Nonetheless, an upgrade may help. On Wed, Sep 1, 2010 at 10:24 AM, Scott Whitecross swhitecr...@gmail.comwrote: Thanks Dave. We've been investigating moving to .6, and this may help make the decision easier. On Tue, Aug 31, 2010 at 5:02 PM, Dave Brondsema dbronds...@geek.netwrote: We had that problem on 0.4 also. 0.6 seems to be working better now, but we only switched a day or two ago. See https://issues.apache.org/jira/browse/HIVE-1181 and http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201001.mbox/%3cd35d2f55-d770-422d-b418-6bee212a6...@forward.co.uk%3e On Mon, Aug 30, 2010 at 3:50 PM, Scott Whitecross swhitecr...@gmail.comwrote: Hi all - I'm running the Hive Server to allow multiple JDBC queries and inserts over the course of a day. What I've noticed is that there seems to be a pretty bad leak with connections over the course of a week, to the point of making the machine unreachable. Looking through the code being used, it appears to be closing Hive connections appropriately. Is there a known problem with Hive and connections now? (I'm currently running Hive .4). Thanks. -- Dave Brondsema Software Engineer Geeknet www.geek.net -- Dave Brondsema Software Engineer Geeknet www.geek.net
RE: Having a Connections Leak with the Hive Server
We had some leaking file descriptors which ended up being a problem in hadoop. They fixed it on 0.21 but not on older versions. There is a workaround for hive which we successfully use. By adding this to your hive-site.xml: !-- workaround for connection leak problem fixed in HADOOP-5476 but only commited to hadoop 0.21.0 -- property namehive.fileformat.check/name valuefalse/value /property Bennie. From: Dave Brondsema [mailto:dbronds...@geek.net] Sent: Thursday, September 02, 2010 3:13 PM To: hive-user@hadoop.apache.org Subject: Re: Having a Connections Leak with the Hive Server Scott, after re-reading your original email, I'm thinking maybe we didn't have the same problem. Hive crashed for us when it ran out of file descriptors, it didn't hang. Nonetheless, an upgrade may help. On Wed, Sep 1, 2010 at 10:24 AM, Scott Whitecross swhitecr...@gmail.commailto:swhitecr...@gmail.com wrote: Thanks Dave. We've been investigating moving to .6, and this may help make the decision easier. On Tue, Aug 31, 2010 at 5:02 PM, Dave Brondsema dbronds...@geek.netmailto:dbronds...@geek.net wrote: We had that problem on 0.4 also. 0.6 seems to be working better now, but we only switched a day or two ago. See https://issues.apache.org/jira/browse/HIVE-1181 and http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201001.mbox/%3cd35d2f55-d770-422d-b418-6bee212a6...@forward.co.uk%3e On Mon, Aug 30, 2010 at 3:50 PM, Scott Whitecross swhitecr...@gmail.commailto:swhitecr...@gmail.com wrote: Hi all - I'm running the Hive Server to allow multiple JDBC queries and inserts over the course of a day. What I've noticed is that there seems to be a pretty bad leak with connections over the course of a week, to the point of making the machine unreachable. Looking through the code being used, it appears to be closing Hive connections appropriately. Is there a known problem with Hive and connections now? (I'm currently running Hive .4). Thanks. -- Dave Brondsema Software Engineer Geeknet www.geek.nethttp://www.geek.net -- Dave Brondsema Software Engineer Geeknet www.geek.nethttp://www.geek.net
Re: Having a Connections Leak with the Hive Server
Thanks Dave. We've been investigating moving to .6, and this may help make the decision easier. On Tue, Aug 31, 2010 at 5:02 PM, Dave Brondsema dbronds...@geek.net wrote: We had that problem on 0.4 also. 0.6 seems to be working better now, but we only switched a day or two ago. See https://issues.apache.org/jira/browse/HIVE-1181 and http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201001.mbox/%3cd35d2f55-d770-422d-b418-6bee212a6...@forward.co.uk%3e On Mon, Aug 30, 2010 at 3:50 PM, Scott Whitecross swhitecr...@gmail.comwrote: Hi all - I'm running the Hive Server to allow multiple JDBC queries and inserts over the course of a day. What I've noticed is that there seems to be a pretty bad leak with connections over the course of a week, to the point of making the machine unreachable. Looking through the code being used, it appears to be closing Hive connections appropriately. Is there a known problem with Hive and connections now? (I'm currently running Hive .4). Thanks. -- Dave Brondsema Software Engineer Geeknet www.geek.net
Re: Having a Connections Leak with the Hive Server
We had that problem on 0.4 also. 0.6 seems to be working better now, but we only switched a day or two ago. See https://issues.apache.org/jira/browse/HIVE-1181 and http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201001.mbox/%3cd35d2f55-d770-422d-b418-6bee212a6...@forward.co.uk%3e On Mon, Aug 30, 2010 at 3:50 PM, Scott Whitecross swhitecr...@gmail.comwrote: Hi all - I'm running the Hive Server to allow multiple JDBC queries and inserts over the course of a day. What I've noticed is that there seems to be a pretty bad leak with connections over the course of a week, to the point of making the machine unreachable. Looking through the code being used, it appears to be closing Hive connections appropriately. Is there a known problem with Hive and connections now? (I'm currently running Hive .4). Thanks. -- Dave Brondsema Software Engineer Geeknet www.geek.net
