Hi , I am using hive version 0.7 ( as suggested by lib/hive-hwi-0.7.0.war) . I checked out code from svn two weeks back . I use thrift api's to load data into hive programmatically .
Vaibhav Negi On Sat, Feb 12, 2011 at 6:06 AM, Viral Bajaria <viral.baja...@gmail.com>wrote: > We use hive through the thrift api too. I have seen the above error when > someone accidentally connects to the thrift server port through multiple > instances of our hive-wrapper code. We open a bunch of hive thrift ports to > achieve the automation of running multiple hive queries in parallel i.e. > continuous aggregations. > > you need to make sure you are not facing a similar issue because the hive > thrift server as far as i know is single threaded up until hive 0.5.0 (the > version that i use). Not too sure if that's been changed in 0.6.0 or higher. > > -Viral > > On Fri, Feb 11, 2011 at 7:06 AM, Ajo Fod <ajo....@gmail.com> wrote: > >> Are you using hive 0.6? ... may be fixed in the latest version. >> >> Also I wonder why these thrift libraries are being used ... is this normal >> hive operation, or can you do something to avoid using thrift? >> >> -Ajo >> >> >> On Fri, Feb 11, 2011 at 12:05 AM, vaibhav negi >> <sssssssena...@gmail.com>wrote: >> >>> >>> Hi all, >>> >>> I am loading data into hive tables by connecting to hiveserver through >>> thrift api using "load data local inpath ... " query . >>> Hive server is running as a background process for days . After some 4-5 >>> days hiveserver throws this error : >>> >>> 11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error occurred >>> during acceptance of message. >>> org.apache.thrift.transport.TTransportException: >>> java.net.SocketException: Too many open files >>> at >>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124) >>> at >>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35) >>> at >>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31) >>> at >>> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184) >>> at >>> org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >>> Caused by: java.net.SocketException: Too many open files >>> at java.net.PlainSocketImpl.socketAccept(Native Method) >>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) >>> at java.net.ServerSocket.implAccept(ServerSocket.java:453) >>> at java.net.ServerSocket.accept(ServerSocket.java:421) >>> at >>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119) >>> ... 9 more >>> >>> I checked network connections using netstat and it show about 1000 active >>> network connections, most of them to hadoop datanode . Output is attached as >>> netstat.out >>> >>> after i restarted hiveserver network connections falls to about 20 . >>> output is attached as netstat_after_HS_restart.out >>> >>> Is this a problem with hive or hadoop configuration ? >>> >>> Vaibhav Negi >>> >> >> >
