Hi! I'm using Hive 0.7.0 and Hadoop 0.20, both from Cloudera's cdh3u0. Jets3t is used from jets3t-0.6.1.jar. I've just found a post (https://forums.aws.amazon.com/thread.jspa?threadID=19076&tstart=0) that describes this issue and I'm trying to figure out if this bug is in this version.
// Wouter --Wouter de Bie Developer Business Intelligence, Spotify wou...@spotify.com +46 72 018 0777 This e-mail (including any attachments) may contain information that is confidential and/or privileged. It is intended only for the recipient(s). If you have reason to believe that you are not the intended recipient of this e-mail, please contact the sender immediately and delete the e-mail from your computer. On Wednesday, July 6, 2011 at 7:39 PM, Aggarwal, Vaibhav wrote: > Could you please tell us which Hadoop and Hive version are you using? > Looks like you might be using an older version of Hadoop (more specifically > one which ships with old version of jets3t). > > Thanks > Vaibhav > > From: Wouter de Bie [mailto:wou...@spotify.com] > Sent: Wednesday, July 06, 2011 9:07 AM > To: user@hive.apache.org (mailto:user@hive.apache.org) > Subject: Hive session locking up after 4 queries using S3 > > Hi all, > > > > I'm using Hive with the s3native FS. Today, I noticed that hive locks up > after 4 queries that directly access S3 (select * from mytable limit 10). > With debug logging on, I get the following output: > > > > 2011-07-06 15:54:31,459 DEBUG s3native.NativeS3FileSystem > (NativeS3FileSystem.java:getFileStatus(393)) - getFileStatus retrieving > metadata for key > 'tmp/hive-mapred/hive_2011-07-06_15-54-29_881_4253697128840334916/-mr-10000' > > 2011-07-06 15:54:31,459 DEBUG httpclient.RestS3Service > (RestS3Service.java:getObjectImpl(1511)) - Retrieving Head information for > bucket XXXXXXXX and object > tmp/hive-mapred/hive_2011-07-06_15-54-29_881_4253697128840334916/-mr-10000 > > 2011-07-06 15:54:31,460 DEBUG service.Jets3tProperties > (Jets3tProperties.java:getBoolProperty(314)) - > s3service.disable-dns-buckets=false > > 2011-07-06 15:54:31,460 DEBUG httpclient.RestS3Service > (RestS3Service.java:setupConnection(811)) - S3 URL: > https://XXXXXXXX:443/tmp%2Fhive-mapred%2Fhive_2011-07-06_15-54-29_881_4253697128840334916%2F-mr-10000 > > 2011-07-06 15:54:31,460 DEBUG httpclient.RestS3Service > (RestS3Service.java:performRequest(334)) - Performing HEAD request for > 'https://XXXXXXXXXXXXXX/tmp%2Fhive-mapred%2Fhive_2011-07-06_15-54-29_881_4253697128840334916%2F-mr-10000', > expecting response code 200 > > 2011-07-06 15:54:31,461 DEBUG httpclient.RestS3Service > (RestS3Service.java:buildAuthorizationString(872)) - Adding authorization for > AWS Access Key 'XXXXXXXXXXXXX'. > > 2011-07-06 15:54:31,461 DEBUG httpclient.RestS3Service > (RestS3Service.java:buildAuthorizationString(922)) - Canonical string ('|' is > a newline): HEAD|||Wed, 06 Jul 2011 15:54:31 > GMT|/XXXXXXXXX/tmp%2Fhive-mapred%2Fhive_2011-07-06_15-54-29_881_4253697128840334916%2F-mr-10000 > > 2011-07-06 15:54:31,461 DEBUG httpclient.HttpClient > (HttpClient.java:executeMethod(322)) - enter > HttpClient.executeMethod(HttpMethod) > > 2011-07-06 15:54:31,462 DEBUG httpclient.HttpClient > (HttpClient.java:executeMethod(373)) - enter > HttpClient.executeMethod(HostConfiguration,HttpMethod,HttpState) > > 2011-07-06 15:54:31,462 DEBUG httpclient.MultiThreadedHttpConnectionManager > (MultiThreadedHttpConnectionManager.java:getConnectionWithTimeout(383)) - > enter HttpConnectionManager.getConnectionWithTimeout(HostConfiguration, long) > > 2011-07-06 15:54:31,462 DEBUG httpclient.MultiThreadedHttpConnectionManager > (MultiThreadedHttpConnectionManager.java:getConnectionWithTimeout(390)) - > HttpConnectionManager.getConnection: config = > HostConfiguration[host=https://XXXXXXXXX.s3.amazonaws.com], timeout = 0 > > 2011-07-06 15:54:31,462 DEBUG httpclient.MultiThreadedHttpConnectionManager > (MultiThreadedHttpConnectionManager.java:getHostPool(775)) - enter > HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration) > > 2011-07-06 15:54:31,463 DEBUG httpclient.MultiThreadedHttpConnectionManager > (MultiThreadedHttpConnectionManager.java:doGetConnection(494)) - Unable to > get a connection, waiting..., > hostConfig=HostConfiguration[host=https://XXXXXXXXXX.s3.amazonaws.com] > > > > Does anyone know if I can do anything to prevent this? It looks like > connections are not returned correctly to the pool.. > > > > // Wouter > > > > > > > > > >
