In short, hiveserver has leaks, the upcoming hiveserver2 (and the changes made in hive trunk due to it should prevent many of these. We run our hiveserver's behind a load balancer (4 of them) basically this increases the mean time to failure due to leaks. It would be good if hive-server had some jmx or some method to determine # of connected clients and then an admin can script a restart in a dead time.
IF you are using zk locking turn that off, zklocking+hiveserver leaks much more then just plan hiveserver. On Fri, Feb 8, 2013 at 7:04 AM, Jakub Holy <jakub.h...@iterate.no> wrote: > Hi everybody, I hope you'll be able to help me understand why our Hive > server occasionally starts throwing "HiveException: Unable to fetch table > <same table name>" and subsequently noe JDBC calls get through. > > I have observed this mostly after having started the server so I guess there > could be some timing issue in Hive initialization. Most of the time Hive > works all right so there no defect in our configuration/infrastructure per > se. > > Any idea how this could have been happening and how to prevent/fix it? We > use Hive 0.9.0 and connect to it via JDBC over Thrift. When I restart Hive, > the problem disappears. > > Thanks a lot! > > Here is the full log with some context around: > > 2013-02-08 09:28:43,943 WARN mapred.JobClient > (JobClient.java:copyAndConfigureFiles(667)) - Use GenericOptionsParser for > parsing the arguments. Applications should implement Tool for the same. > 2013-02-08 09:28:53,399 ERROR ql.Driver (SessionState.java:printError(400)) > - FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.MapRedTask > 2013-02-08 09:29:36,774 ERROR DataNucleus.Plugin > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > "org.eclipse.core.resources" but it cannot be resolved. > 2013-02-08 09:29:36,774 ERROR DataNucleus.Plugin > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > "org.eclipse.core.resources" but it cannot be resolved. > 2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > "org.eclipse.core.runtime" but it cannot be resolved. > 2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > "org.eclipse.core.runtime" but it cannot be resolved. > 2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > "org.eclipse.text" but it cannot be resolved. > 2013-02-08 09:29:36,777 ERROR DataNucleus.Plugin > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires > "org.eclipse.text" but it cannot be resolved. > 2013-02-08 09:29:51,997 ERROR parse.SemanticAnalyzer > (SemanticAnalyzer.java:getMetaData(1128)) - > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table > playerevents > at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:896) > at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:830) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:954) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7524) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) > at > org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:191) > at > org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:629) > at > org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:617) > at > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: javax.jdo.JDOException: Attempt to instantiate insert request > gave an error > NestedThrowables: > java.lang.reflect.InvocationTargetException > at > org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:434) > at > org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:539) > at > org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:551) > at > org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:560) > at > org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:800) > at > org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:733) > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111) > at $Proxy0.getTable(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:974) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:735) > at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:888) > ... 16 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:588) > at > org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:324) > at > org.datanucleus.store.rdbms.RDBMSPersistenceHandler.getFetchRequest(RDBMSPersistenceHandler.java:267) > at > org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:239) > at > org.datanucleus.jdo.state.JDOStateManagerImpl.loadFieldsFromDatastore(JDOStateManagerImpl.java:1929) > at > org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields(JDOStateManagerImpl.java:1597) > at > org.datanucleus.jdo.state.Hollow.transitionRetrieve(Hollow.java:168) > at > org.datanucleus.state.AbstractStateManager.retrieve(AbstractStateManager.java:470) > at > org.datanucleus.ObjectManagerImpl.retrieveObject(ObjectManagerImpl.java:1131) > at > org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:534) > ... 28 more > Caused by: java.lang.IllegalStateException: Table object has not been been > initialised : `SDS` > at > org.datanucleus.store.rdbms.table.AbstractTable.assertIsInitialized(AbstractTable.java:625) > at > org.datanucleus.store.rdbms.table.ClassTable.getMemberMapping(ClassTable.java:2904) > at > org.datanucleus.store.rdbms.sql.SQLStatementHelper.selectMemberOfSourceInStatement(SQLStatementHelper.java:512) > at > org.datanucleus.store.rdbms.sql.SQLStatementHelper.selectFetchPlanOfSourceClassInStatement(SQLStatementHelper.java:441) > at > org.datanucleus.store.rdbms.sql.SQLStatementHelper.selectMemberOfSourceInStatement(SQLStatementHelper.java:567) > at > org.datanucleus.store.rdbms.request.FetchRequest.processMembersOfClass(FetchRequest.java:468) > at > org.datanucleus.store.rdbms.request.FetchRequest.<init>(FetchRequest.java:166) > ... 42 more > 2013-02-08 09:29:52,016 ERROR ql.Driver (SessionState.java:printError(400)) > - FAILED: Error in semantic analysis: Unable to fetch table playerevents > org.apache.hadoop.hive.ql.parse.SemanticException: Unable to fetch table > playerevents > -- > A world without pair-programming is a sad, lonely place. > > (Vær så snill og hjelp meg med å korrigere min norsk, skriftlig og muntlig. > Takk!) > > Jakub Holy > Solutions Engineer | +47 966 23 666 > Iterate AS | www.iterate.no > The Lean Software Development Consultancy > - http://theholyjava.wordpress.com/ -
