Re: Possible bug loading data in Hive.
Hive 0.9.0 with CDH4.1 --- This is very old release. I would recommend to upgrade to hive-0.13 or at least 0.12 and see. Error you are seeing is on loading data into a partition and metastore alter/add partition is failing. Can you try upgrading and see if that resolves your issue? If not can you share your datanuclues related settings in hive On Tue, Jun 10, 2014 at 2:16 PM, Fernando Agudo fag...@pragsis.com wrote: Hello, I'm working with Hive 0.9.0 with CDH4.1. I have a process which it's loading data in Hive every minute. It creates the partition if it's necessary. I have been monitoring this process for three days and I realize that there's a method (*listStorageDescriptorsWithCD*) which increases the execution time. First execution this method lasted about 15 millisencond and in the end it took more than 3 seconds (three days later), after that, Hive throws an exception and starts working again. I have checking this method but I haven't figured out any suspicious, could it be a bug? *2014-06-05 09:58:20,921* DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2036)) - Executing listStorageDescriptorsWithCD *2014-06-05 09:58:20,928* DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2045)) - Done executing query for listStorageDescriptorsWithCD *2014-06-08 20:15:33,867* DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2036)) - Executing listStorageDescriptor sWithCD *2014-06-08 20:15:36,134* DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2045)) - Done executing query for listSt orageDescriptorsWithCD 2014-06-08 20:16:34,600 DEBUG metastore.ObjectStore (ObjectStore.java: removeUnusedColumnDescriptor(1989)) - execute removeUnusedColumnDescr iptor *2014-06-08 20:16:34,600 DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2036)) - Executing listStorageDescriptor** **sWithCD* 2014-06-08 20:16:34,805 ERROR metadata.Hive (Hive.java:getPartition(1453)) - org.apache.hadoop.hive.ql.metadata.HiveException: Unable to al ter partition. at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition( Hive.java:429) at org.apache.hadoop.hive.ql.metadata.Hive.getPartition( Hive.java:1446) at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition( Hive.java:1158) at org.apache.hadoop.hive.ql.exec.MoveTask.execute( MoveTask.java:304) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential( TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950) at org.apache.hadoop.hive.service.HiveServer$ HiveServerHandler.execute(HiveServer.java:191) at org.apache.hadoop.hive.service.ThriftHive$Processor$ execute.getResult(ThriftHive.java:630) at org.apache.hadoop.hive.service.ThriftHive$Processor$ execute.getResult(ThriftHive.java:618) at org.apache.thrift.ProcessFunction.process( ProcessFunction.java:32) at org.apache.thrift.TBaseProcessor.process( TBaseProcessor.java:34) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run( TThreadPoolServer.java:176) at java.util.concurrent.ThreadPoolExecutor$Worker. runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: MetaException(message:The transaction for alter partition did not commit successfully.) at org.apache.hadoop.hive.metastore.ObjectStore. alterPartition(ObjectStore.java:1927) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingRawStore. invoke(RetryingRawStore.java:111) at $Proxy0.alterPartition(Unknown Source) at org.apache.hadoop.hive.metastore.HiveAlterHandler. alterPartition(HiveAlterHandler.java:254) at org.apache.hadoop.hive.metastore.HiveMetaStore$ HMSHandler.rename_partition(HiveMetaStore.java:1816) at org.apache.hadoop.hive.metastore.HiveMetaStore$ HMSHandler.rename_partition(HiveMetaStore.java:1788) at org.apache.hadoop.hive.metastore.HiveMetaStore$ HMSHandler.alter_partition(HiveMetaStore.java:1771) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient. alter_partition(HiveMetaStoreClient.java:834) at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition( Hive.java:425) ... 17 more 2014-06-08 20:16:34,827 ERROR exec.Task (SessionState.java:printError(403)) -
Re: Possible bug loading data in Hive.
I have problems to upgrade to hive-0.13 or 0.12 because is in production. Only have this configuration of the datanuclues: property namedatanucleus.fixedDatastore/name valuetrue/value /property property namedatanucleus.autoCreateSchema/name valuefalse/value /property This is relevant for the problem? Thanks, On 10/06/14 10:53, Nitin Pawar wrote: Hive 0.9.0 with CDH4.1 --- This is very old release. I would recommend to upgrade to hive-0.13 or at least 0.12 and see. Error you are seeing is on loading data into a partition and metastore alter/add partition is failing. Can you try upgrading and see if that resolves your issue? If not can you share your datanuclues related settings in hive On Tue, Jun 10, 2014 at 2:16 PM, Fernando Agudo fag...@pragsis.com wrote: Hello, I'm working with Hive 0.9.0 with CDH4.1. I have a process which it's loading data in Hive every minute. It creates the partition if it's necessary. I have been monitoring this process for three days and I realize that there's a method (*listStorageDescriptorsWithCD*) which increases the execution time. First execution this method lasted about 15 millisencond and in the end it took more than 3 seconds (three days later), after that, Hive throws an exception and starts working again. I have checking this method but I haven't figured out any suspicious, could it be a bug? *2014-06-05 09:58:20,921* DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2036)) - Executing listStorageDescriptorsWithCD *2014-06-05 09:58:20,928* DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2045)) - Done executing query for listStorageDescriptorsWithCD *2014-06-08 20:15:33,867* DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2036)) - Executing listStorageDescriptor sWithCD *2014-06-08 20:15:36,134* DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2045)) - Done executing query for listSt orageDescriptorsWithCD 2014-06-08 20:16:34,600 DEBUG metastore.ObjectStore (ObjectStore.java: removeUnusedColumnDescriptor(1989)) - execute removeUnusedColumnDescr iptor *2014-06-08 20:16:34,600 DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2036)) - Executing listStorageDescriptor** **sWithCD* 2014-06-08 20:16:34,805 ERROR metadata.Hive (Hive.java:getPartition(1453)) - org.apache.hadoop.hive.ql.metadata.HiveException: Unable to al ter partition. at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition( Hive.java:429) at org.apache.hadoop.hive.ql.metadata.Hive.getPartition( Hive.java:1446) at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition( Hive.java:1158) at org.apache.hadoop.hive.ql.exec.MoveTask.execute( MoveTask.java:304) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential( TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950) at org.apache.hadoop.hive.service.HiveServer$ HiveServerHandler.execute(HiveServer.java:191) at org.apache.hadoop.hive.service.ThriftHive$Processor$ execute.getResult(ThriftHive.java:630) at org.apache.hadoop.hive.service.ThriftHive$Processor$ execute.getResult(ThriftHive.java:618) at org.apache.thrift.ProcessFunction.process( ProcessFunction.java:32) at org.apache.thrift.TBaseProcessor.process( TBaseProcessor.java:34) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run( TThreadPoolServer.java:176) at java.util.concurrent.ThreadPoolExecutor$Worker. runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: MetaException(message:The transaction for alter partition did not commit successfully.) at org.apache.hadoop.hive.metastore.ObjectStore. alterPartition(ObjectStore.java:1927) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingRawStore. invoke(RetryingRawStore.java:111) at $Proxy0.alterPartition(Unknown Source) at org.apache.hadoop.hive.metastore.HiveAlterHandler. alterPartition(HiveAlterHandler.java:254) at org.apache.hadoop.hive.metastore.HiveMetaStore$ HMSHandler.rename_partition(HiveMetaStore.java:1816) at org.apache.hadoop.hive.metastore.HiveMetaStore$ HMSHandler.rename_partition(HiveMetaStore.java:1788) at
Re: Possible bug loading data in Hive.
The error you see is with hive metastore and these issues were kind of related to two sided 1) Load on metastore 2) datanuclueas related For now if possible, see if you can restart hive metastore and that resolves your issue. On Tue, Jun 10, 2014 at 3:27 PM, Fernando Agudo fag...@pragsis.com wrote: I have problems to upgrade to hive-0.13 or 0.12 because is in production. Only have this configuration of the datanuclues: property namedatanucleus.fixedDatastore/name valuetrue/value /property property namedatanucleus.autoCreateSchema/name valuefalse/value /property This is relevant for the problem? Thanks, On 10/06/14 10:53, Nitin Pawar wrote: Hive 0.9.0 with CDH4.1 --- This is very old release. I would recommend to upgrade to hive-0.13 or at least 0.12 and see. Error you are seeing is on loading data into a partition and metastore alter/add partition is failing. Can you try upgrading and see if that resolves your issue? If not can you share your datanuclues related settings in hive On Tue, Jun 10, 2014 at 2:16 PM, Fernando Agudo fag...@pragsis.com wrote: Hello, I'm working with Hive 0.9.0 with CDH4.1. I have a process which it's loading data in Hive every minute. It creates the partition if it's necessary. I have been monitoring this process for three days and I realize that there's a method (*listStorageDescriptorsWithCD*) which increases the execution time. First execution this method lasted about 15 millisencond and in the end it took more than 3 seconds (three days later), after that, Hive throws an exception and starts working again. I have checking this method but I haven't figured out any suspicious, could it be a bug? *2014-06-05 09:58:20,921* DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2036)) - Executing listStorageDescriptorsWithCD *2014-06-05 09:58:20,928* DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2045)) - Done executing query for listStorageDescriptorsWithCD *2014-06-08 20:15:33,867* DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2036)) - Executing listStorageDescriptor sWithCD *2014-06-08 20:15:36,134* DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2045)) - Done executing query for listSt orageDescriptorsWithCD 2014-06-08 20:16:34,600 DEBUG metastore.ObjectStore (ObjectStore.java: removeUnusedColumnDescriptor(1989)) - execute removeUnusedColumnDescr iptor *2014-06-08 20:16:34,600 DEBUG metastore.ObjectStore (ObjectStore.java: listStorageDescriptorsWithCD(2036)) - Executing listStorageDescriptor** **sWithCD* 2014-06-08 20:16:34,805 ERROR metadata.Hive (Hive.java:getPartition(1453)) - org.apache.hadoop.hive.ql.metadata.HiveException: Unable to al ter partition. at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition( Hive.java:429) at org.apache.hadoop.hive.ql.metadata.Hive.getPartition( Hive.java:1446) at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition( Hive.java:1158) at org.apache.hadoop.hive.ql.exec.MoveTask.execute( MoveTask.java:304) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task. java:153) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential( TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java: 1331) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950) at org.apache.hadoop.hive.service.HiveServer$ HiveServerHandler.execute(HiveServer.java:191) at org.apache.hadoop.hive.service.ThriftHive$Processor$ execute.getResult(ThriftHive.java:630) at org.apache.hadoop.hive.service.ThriftHive$Processor$ execute.getResult(ThriftHive.java:618) at org.apache.thrift.ProcessFunction.process( ProcessFunction.java:32) at org.apache.thrift.TBaseProcessor.process( TBaseProcessor.java:34) at org.apache.thrift.server.TThreadPoolServer$ WorkerProcess.run( TThreadPoolServer.java:176) at java.util.concurrent.ThreadPoolExecutor$Worker. runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: MetaException(message:The transaction for alter partition did not commit successfully.) at org.apache.hadoop.hive.metastore.ObjectStore. alterPartition(ObjectStore.java:1927) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingRawStore. invoke(RetryingRawStore.java:111) at
