[
https://issues.apache.org/jira/browse/HIVE-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039334#comment-14039334
]
Xiaobing Zhou commented on HIVE-7167:
-------------------------------------
Thanks [~sershe],
1) Here's the exception.
2) it's because these three methods are potential places that causes contention
when two metastores both try to initialize DB in the same time.
3) I don't think so. If one tried and failed in the first time, JDOException is
caught for one more time try, if failed again, simply ignored, means another
metastore succeeds.
2014-05-15 21:06:01,659 WARN server.HiveServer2
(HiveServer2.java:startHiveServer2(112)) - Error starting HiveServer2 on
attempt 1, will retry in 60 seconds
java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:347)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:292)
at
org.apache.hive.service.cli.session.SessionManager.applyAuthorizationConfigPolicy(SessionManager.java:88)
at
org.apache.hive.service.cli.session.SessionManager.init(SessionManager.java:63)
at
org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:110)
at
org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:68)
at
org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:100)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149)
Caused by: java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1413)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:341)
... 9 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1411)
... 14 more
Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes
to datastore
NestedThrowables:
java.sql.BatchUpdateException: Violation of UNIQUE KEY constraint
'GLOBALPRIVILEGEINDEX'. Cannot insert duplicate key in object
'dbo.GLOBAL_PRIVS'. The duplicate key value is (admin, ROLE, All, admin, ROLE).
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
at
org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
at
org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:406)
at
org.apache.hadoop.hive.metastore.ObjectStore.grantPrivileges(ObjectStore.java:3877)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
at $Proxy10.grantPrivileges(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultRoles(HiveMetaStore.java:567)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:398)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
... 19 more
Caused by: java.sql.BatchUpdateException: Violation of UNIQUE KEY constraint
'GLOBALPRIVILEGEINDEX'. Cannot insert duplicate key in object
'dbo.GLOBAL_PRIVS'. The duplicate key value is (admin, ROLE, All, admin, ROLE).
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:1160)
at
com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:424)
at
org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:372)
at
org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:628)
at
org.datanucleus.store.rdbms.SQLController.processStatementsForConnection(SQLController.java:596)
at
org.datanucleus.store.rdbms.SQLController$1.transactionFlushed(SQLController.java:683)
at
org.datanucleus.store.connection.AbstractManagedConnection.transactionFlushed(AbstractManagedConnection.java:86)
at
org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionFlushed(ConnectionManagerImpl.java:454)
at org.datanucleus.TransactionImpl.flush(TransactionImpl.java:203)
at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:267)
at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
... 34 more
2014-05-15 21:06:01,667 INFO server.HiveServer2
(HiveServer2.java:startHiveServer2(120)) - Exception caught when calling stop
of HiveServer2 before retrying start
java.lang.NullPointerException
at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:82)
at
org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:116)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149)
> Hive Metastore fails to start with SQLServerException
> -----------------------------------------------------
>
> Key: HIVE-7167
> URL: https://issues.apache.org/jira/browse/HIVE-7167
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.13.0
> Reporter: Xiaobing Zhou
> Assignee: Xiaobing Zhou
> Labels: patch,, test
> Attachments: HIVE-7167.1.patch, HIVE-7167.2.patch
>
>
> In the case that hiveserver2 uses embedded metastore and hiveserver uses
> remote metastore, this exception comes up when hiveserver2 and hiveserver are
> started simultaneously.
> metastore service status is running but when I launch hive cli, I get
> following metastore connection error:
> C:\apps\dist\hive-0.13.0.2.1.2.0-1660\bin>hive.cmd
> 14/05/09 17:40:03 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.*
> no l
> onger has any effect. Use hive.hmshandler.retry.* instead
> Logging initialized using configuration in
> file:/C:/apps/dist/hive-0.13.0.2.1.2.
> 0-1660/conf/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.RuntimeExceptio
> n: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav
> a:347)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.h
> ive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore
> Utils.java:1413)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(Retry
> ingMetaStoreClient.java:62)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret
> ryingMetaStoreClient.java:72)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja
> va:2444)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav
> a:341)
> ... 7 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct
> orAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC
> onstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore
> Utils.java:1411)
> ... 12 more
> Caused by: MetaException(message:Could not connect to meta store using any of
> th
> e URIs provided. Most recent failure:
> org.apache.thrift.transport.TTransportExce
> ption: java.net.ConnectException: Connection refused: connect
> at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaSto
> reClient.java:336)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaS
> toreClient.java:214)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct
> orAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC
> onstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore
> Utils.java:1411)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(Retry
> ingMetaStoreClient.java:62)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret
> ryingMetaStoreClient.java:72)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja
> va:2444)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav
> a:341)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.net.ConnectException: Connection refused: connect
> at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.ja
> va:339)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocket
> Impl.java:200)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java
> :182)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:157)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
> at java.net.Socket.connect(Socket.java:579)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
> ... 19 more
> )
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaSto
> reClient.java:382)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaS
> toreClient.java:214)
> ... 17 more
> Because of this issue, we are seeing many failures in hive related system
> tests.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
