[jira] [Commented] (HIVE-13869) ObjectStore synchronization issue manifested in db notification listener
[ https://issues.apache.org/jira/browse/HIVE-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930656#comment-15930656 ] Vaibhav Gumashta commented on HIVE-13869: - Removing target 1.2.2 (minor upgrade) and adding target 1.3 as there is no patch yet. > ObjectStore synchronization issue manifested in db notification listener > > > Key: HIVE-13869 > URL: https://issues.apache.org/jira/browse/HIVE-13869 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 1.1.1 >Reporter: Nachiket Vaidya >Priority: Blocker > > I used pyhs2 python client to create tables/partitions in hive. I was working > fine until I moved to multithreaded scripts which created 8 connections and > ran DDL queries concurrently. > I got the error as > {noformat} > 2016-05-04 17:49:26,226 ERROR > org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-194]: > HMSHandler Fatal error: Invalid state. Transaction has already started > org.datanucleus.transaction.NucleusTransactionException: Invalid state. > Transaction has already started > at > org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47) > at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131) > at > org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88) > at > org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80) > at > org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:463) > at > org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:7522) > at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114) > at com.sun.proxy.$Proxy10.addNotificationEvent(Unknown Source) > at > org.apache.hive.hcatalog.listener.DbNotificationListener.enqueue(DbNotificationListener.java:261) > at > org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:123) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1483) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1502) > at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) > at > com.sun.proxy.$Proxy14.create_table_with_environment_context(Unknown Source) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:9267) > {noformat} > At one point, I got another stack trace as > {noformat} > 2016-05-26 12:32:27,904 ERROR > org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-7]: > MetaException(message:java.lang.NullPointerException) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5535) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.add_partitions_req(HiveMetaStore.java:2308) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) > at com.sun.proxy.$Proxy14.add_partitions_req(Unknown Source) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_partitions_req.getResult(ThriftHiveMetastore.java:9723) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_partitions_req.getResult(ThriftHiveMetastore.java:9707) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at >
[jira] [Commented] (HIVE-13869) ObjectStore synchronization issue manifested in db notification listener
[ https://issues.apache.org/jira/browse/HIVE-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727463#comment-15727463 ] Alexander Kolbasov commented on HIVE-13869: --- HIVE-1573 goes into more detail of the actual problem. > ObjectStore synchronization issue manifested in db notification listener > > > Key: HIVE-13869 > URL: https://issues.apache.org/jira/browse/HIVE-13869 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 1.1.1 >Reporter: Nachiket Vaidya >Priority: Blocker > > I used pyhs2 python client to create tables/partitions in hive. I was working > fine until I moved to multithreaded scripts which created 8 connections and > ran DDL queries concurrently. > I got the error as > {noformat} > 2016-05-04 17:49:26,226 ERROR > org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-194]: > HMSHandler Fatal error: Invalid state. Transaction has already started > org.datanucleus.transaction.NucleusTransactionException: Invalid state. > Transaction has already started > at > org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47) > at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131) > at > org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88) > at > org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80) > at > org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:463) > at > org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:7522) > at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114) > at com.sun.proxy.$Proxy10.addNotificationEvent(Unknown Source) > at > org.apache.hive.hcatalog.listener.DbNotificationListener.enqueue(DbNotificationListener.java:261) > at > org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:123) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1483) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1502) > at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) > at > com.sun.proxy.$Proxy14.create_table_with_environment_context(Unknown Source) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:9267) > {noformat} > At one point, I got another stack trace as > {noformat} > 2016-05-26 12:32:27,904 ERROR > org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-7]: > MetaException(message:java.lang.NullPointerException) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5535) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.add_partitions_req(HiveMetaStore.java:2308) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) > at com.sun.proxy.$Proxy14.add_partitions_req(Unknown Source) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_partitions_req.getResult(ThriftHiveMetastore.java:9723) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_partitions_req.getResult(ThriftHiveMetastore.java:9707) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at > org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) > at >
[jira] [Commented] (HIVE-13869) ObjectStore synchronization issue manifested in db notification listener
[ https://issues.apache.org/jira/browse/HIVE-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304442#comment-15304442 ] Sushanth Sowmyan commented on HIVE-13869: - Also, [~vaidyand], could you please verify if this problem exists in master? > ObjectStore synchronization issue manifested in db notification listener > > > Key: HIVE-13869 > URL: https://issues.apache.org/jira/browse/HIVE-13869 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 1.1.1 >Reporter: Nachiket Vaidya > > I used pyhs2 python client to create tables/partitions in hive. I was working > fine until I moved to multithreaded scripts which created 8 connections and > ran DDL queries concurrently. > I got the error as > {noformat} > 2016-05-04 17:49:26,226 ERROR > org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-194]: > HMSHandler Fatal error: Invalid state. Transaction has already started > org.datanucleus.transaction.NucleusTransactionException: Invalid state. > Transaction has already started > at > org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47) > at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131) > at > org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88) > at > org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80) > at > org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:463) > at > org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:7522) > at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114) > at com.sun.proxy.$Proxy10.addNotificationEvent(Unknown Source) > at > org.apache.hive.hcatalog.listener.DbNotificationListener.enqueue(DbNotificationListener.java:261) > at > org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:123) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1483) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1502) > at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) > at > com.sun.proxy.$Proxy14.create_table_with_environment_context(Unknown Source) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:9267) > {noformat} > At one point, I got another stack trace as > {noformat} > 2016-05-26 12:32:27,904 ERROR > org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-7]: > MetaException(message:java.lang.NullPointerException) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5535) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.add_partitions_req(HiveMetaStore.java:2308) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) > at com.sun.proxy.$Proxy14.add_partitions_req(Unknown Source) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_partitions_req.getResult(ThriftHiveMetastore.java:9723) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_partitions_req.getResult(ThriftHiveMetastore.java:9707) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at > org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) > at >
[jira] [Commented] (HIVE-13869) ObjectStore synchronization issue manifested in db notification listener
[ https://issues.apache.org/jira/browse/HIVE-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304430#comment-15304430 ] Sushanth Sowmyan commented on HIVE-13869: - Removing fix version of 1.2.2 - fix version is supposed to be marked after commit. I assume the intent was to have target version of 1.2.2. > ObjectStore synchronization issue manifested in db notification listener > > > Key: HIVE-13869 > URL: https://issues.apache.org/jira/browse/HIVE-13869 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 1.1.1 >Reporter: Nachiket Vaidya > > I used pyhs2 python client to create tables/partitions in hive. I was working > fine until I moved to multithreaded scripts which created 8 connections and > ran DDL queries concurrently. > I got the error as > {noformat} > 2016-05-04 17:49:26,226 ERROR > org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-194]: > HMSHandler Fatal error: Invalid state. Transaction has already started > org.datanucleus.transaction.NucleusTransactionException: Invalid state. > Transaction has already started > at > org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47) > at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131) > at > org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88) > at > org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80) > at > org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:463) > at > org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:7522) > at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114) > at com.sun.proxy.$Proxy10.addNotificationEvent(Unknown Source) > at > org.apache.hive.hcatalog.listener.DbNotificationListener.enqueue(DbNotificationListener.java:261) > at > org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:123) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1483) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1502) > at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) > at > com.sun.proxy.$Proxy14.create_table_with_environment_context(Unknown Source) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:9267) > {noformat} > At one point, I got another stack trace as > {noformat} > 2016-05-26 12:32:27,904 ERROR > org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-7]: > MetaException(message:java.lang.NullPointerException) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5535) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.add_partitions_req(HiveMetaStore.java:2308) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) > at com.sun.proxy.$Proxy14.add_partitions_req(Unknown Source) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_partitions_req.getResult(ThriftHiveMetastore.java:9723) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_partitions_req.getResult(ThriftHiveMetastore.java:9707) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at >