I have problems to upgrade to hive-0.13 or 0.12 because is in production.
Only have this configuration of the datanuclues:
<property>
<name>datanucleus.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateSchema</name>
<value>false</value>
</property>
This is relevant for the problem?
Thanks,
On 10/06/14 10:53, Nitin Pawar wrote:
Hive 0.9.0 with CDH4.1 <--- This is very old release.
I would recommend to upgrade to hive-0.13 or at least 0.12 and see.
Error you are seeing is on loading data into a partition and metastore
alter/add partition is failing.
Can you try upgrading and see if that resolves your issue?
If not can you share your datanuclues related settings in hive
On Tue, Jun 10, 2014 at 2:16 PM, Fernando Agudo <fag...@pragsis.com> wrote:
Hello,
I'm working with Hive 0.9.0 with CDH4.1. I have a process which it's
loading data in Hive every minute. It creates the partition if it's
necessary.
I have been monitoring this process for three days and I realize that
there's a method (*listStorageDescriptorsWithCD*) which increases the
execution time. First execution this method lasted about 15 millisencond
and in the end it took more than 3 seconds (three days later), after that,
Hive throws an exception and starts working again.
I have checking this method but I haven't figured out any suspicious,
could it be a bug?
*2014-06-05 09:58:20,921* DEBUG metastore.ObjectStore (ObjectStore.java:
listStorageDescriptorsWithCD(2036)) - Executing
listStorageDescriptorsWithCD
*2014-06-05 09:58:20,928* DEBUG metastore.ObjectStore (ObjectStore.java:
listStorageDescriptorsWithCD(2045)) - Done executing query for
listStorageDescriptorsWithCD
*2014-06-08 20:15:33,867* DEBUG metastore.ObjectStore (ObjectStore.java:
listStorageDescriptorsWithCD(2036)) - Executing listStorageDescriptor
sWithCD
*2014-06-08 20:15:36,134* DEBUG metastore.ObjectStore (ObjectStore.java:
listStorageDescriptorsWithCD(2045)) - Done executing query for listSt
orageDescriptorsWithCD
2014-06-08 20:16:34,600 DEBUG metastore.ObjectStore (ObjectStore.java:
removeUnusedColumnDescriptor(1989)) - execute removeUnusedColumnDescr
iptor
*2014-06-08 20:16:34,600 DEBUG metastore.ObjectStore (ObjectStore.java:
listStorageDescriptorsWithCD(2036)) - Executing listStorageDescriptor**
**sWithCD*
2014-06-08 20:16:34,805 ERROR metadata.Hive (Hive.java:getPartition(1453))
- org.apache.hadoop.hive.ql.metadata.HiveException: Unable to al
ter partition.
at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition(
Hive.java:429)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(
Hive.java:1446)
at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(
Hive.java:1158)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(
MoveTask.java:304)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(
TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950)
at org.apache.hadoop.hive.service.HiveServer$
HiveServerHandler.execute(HiveServer.java:191)
at org.apache.hadoop.hive.service.ThriftHive$Processor$
execute.getResult(ThriftHive.java:630)
at org.apache.hadoop.hive.service.ThriftHive$Processor$
execute.getResult(ThriftHive.java:618)
at org.apache.thrift.ProcessFunction.process(
ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(
TBaseProcessor.java:34)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(
TThreadPoolServer.java:176)
at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: MetaException(message:The transaction for alter partition did
not commit successfully.)
at org.apache.hadoop.hive.metastore.ObjectStore.
alterPartition(ObjectStore.java:1927)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hive.metastore.RetryingRawStore.
invoke(RetryingRawStore.java:111)
at $Proxy0.alterPartition(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveAlterHandler.
alterPartition(HiveAlterHandler.java:254)
at org.apache.hadoop.hive.metastore.HiveMetaStore$
HMSHandler.rename_partition(HiveMetaStore.java:1816)
at org.apache.hadoop.hive.metastore.HiveMetaStore$
HMSHandler.rename_partition(HiveMetaStore.java:1788)
at org.apache.hadoop.hive.metastore.HiveMetaStore$
HMSHandler.alter_partition(HiveMetaStore.java:1771)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.
alter_partition(HiveMetaStoreClient.java:834)
at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition(
Hive.java:425)
... 17 more
2014-06-08 20:16:34,827 ERROR exec.Task (SessionState.java:printError(403))
- Failed with exception org.apache.hadoop.hive.ql.metadata.HiveException:
Unable to alter partition.
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter
partition.
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(
Hive.java:1454)
at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(
Hive.java:1158)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(
MoveTask.java:304)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(
TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950)
at org.apache.hadoop.hive.service.HiveServer$
HiveServerHandler.execute(HiveServer.java:191)
at org.apache.hadoop.hive.service.ThriftHive$Processor$
execute.getResult(ThriftHive.java:630)
at org.apache.hadoop.hive.service.ThriftHive$Processor$
execute.getResult(ThriftHive.java:618)
at org.apache.thrift.ProcessFunction.process(
ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(
TBaseProcessor.java:34)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(
TThreadPoolServer.java:176)
at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
alter partition.
at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition(
Hive.java:429)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(
Hive.java:1446)
... 16 more
Caused by: MetaException(message:The transaction for alter partition did
not commit successfully.)
at org.apache.hadoop.hive.metastore.ObjectStore.
alterPartition(ObjectStore.java:1927)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hive.metastore.RetryingRawStore.
invoke(RetryingRawStore.java:111)
at $Proxy0.alterPartition(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveAlterHandler.
alterPartition(HiveAlterHandler.java:254)
at org.apache.hadoop.hive.metastore.HiveMetaStore$
HMSHandler.rename_partition(HiveMetaStore.java:1816)
at org.apache.hadoop.hive.metastore.HiveMetaStore$
HMSHandler.rename_partition(HiveMetaStore.java:1788)
at org.apache.hadoop.hive.metastore.HiveMetaStore$
HMSHandler.alter_partition(HiveMetaStore.java:1771)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.
alter_partition(HiveMetaStoreClient.java:834)
at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition(
Hive.java:425)
... 17 more
2014-06-08 20:16:34,852 ERROR ql.Driver (SessionState.java:printError(403))
- FAILED: Execution Error, return code 1 from org.apache.hadoop
.hive.ql.exec.MoveTask
--
*Fernando Agudo Tarancón*
/Big Data Software Engineer/
Telf.: +34 917 680 490
Fax: +34 913 833 301
C/ Manuel Tovar, 49-53 - 28034 Madrid - Spain
_http://www.bidoop.es_
--
*Fernando Agudo Tarancón*
/Big Data Software Engineer/
Telf.: +34 917 680 490
Fax: +34 913 833 301
C/ Manuel Tovar, 49-53 - 28034 Madrid - Spain
_http://www.bidoop.es_
