[
https://issues.apache.org/jira/browse/HIVE-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364857#comment-14364857
]
Pavan Srinivas commented on HIVE-9543:
--------------------------------------
Hey [~brocknoland], as [~lijy83] pointed out, when ever there is a
MysqlException, a duplicate entry will be created. The problem here is with
https://db.apache.org/jdo/api20/apidocs/javax/jdo/Query.html#execute%28%29
execute() function of Query.java. "execute()" does not throw any exceptions.
Some reproducible steps:
1) Make sure metastore db is reachable&running.
2) Enable debugger and stop at a breakpoint inside checkSchema().
3) Stop the metastore db or make it unreachable.
4) Continue on the debugger, you would see execute call of getMSchemaVersion()
does not throw exceptions of DB reachability. It just returns an empty array as
if there are no entries in the table. Thus making the getMSchemaVersion() throw
NoSuchObjectException.
This issue happens when we have a flacky network connection to DB.
> MetaException(message:Metastore contains multiple versions)
> -----------------------------------------------------------
>
> Key: HIVE-9543
> URL: https://issues.apache.org/jira/browse/HIVE-9543
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.13.1
> Reporter: Junyong Li
>
> When i run bin/hive command, i got the following exception:
> {noformat}
> Logging initialized using configuration in
> jar:file:/home/hadoop/apache-hive-0.13.1-bin/lib/hive-common-0.13.1.jar!/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
> ... 7 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
> ... 12 more
> Caused by: MetaException(message:Metastore contains multiple versions)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6368)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6330)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6289)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6277)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy9.verifySchema(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:476)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
> ... 17 more
> {noformat}
> And i have found two record in metastore table VERSION. after reading source
> code, i found following code maybe cause the problem:
> In the org.apache.hadoop.hive.metastore.ObjectStore.java:6289:
> {noformat}
> String schemaVer = getMetaStoreSchemaVersion();
> if (schemaVer == null) {
> // metastore has no schema version information
> if (strictValidation) {
> throw new MetaException("Version information not found in
> metastore. ");
> } else {
> LOG.warn("Version information not found in metastore. "
> + HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION.toString() +
> " is not enabled so recording the schema version " +
> MetaStoreSchemaInfo.getHiveSchemaVersion());
>
> setMetaStoreSchemaVersion(MetaStoreSchemaInfo.getHiveSchemaVersion(),
> "Set by MetaStore");
> }
> }
> {noformat}
> If there is exception in the {{getMetaStoreSchemaVersion()}}, Hive will
> insert a duplication record
> And i also found my another exception maybe the same cause as above.the
> exception is :
> {noformat}
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
> ... 7 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
> ... 12 more
> Caused by: javax.jdo.JDODataStoreException: Insert of object
> "org.apache.hadoop.hive.metastore.model.MRoleMap@2fd02622" using statement
> "INSERT INTO `ROLE_MAP` (`ROLE_GRANT_ID`,`GRANTOR`,`A
> DD_TIME`,`ROLE_ID`,`GRANTOR_TYPE`,`GRANT_OPTION`,`PRINCIPAL_NAME`,`PRINCIPAL_TYPE`)
> VALUES (?,?,?,?,?,?,?,?)" failed : Duplicate entry 'hadoop-1-admin-ROLE' for
> key 'USERROLEMAPINDEX'
> NestedThrowables:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Duplicate entry 'hadoop-1-admin-ROLE' for key 'USERROLEMAPINDEX'
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.grantRole(ObjectStore.java:3058)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy9.grantRole(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addAdminUsers(HiveMetaStore.java:611)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:399)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:55)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:60)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
> ... 17 more
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Duplicate entry 'hadoop-1-admin-ROLE' for key 'USERROLEMAPINDEX'
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
> at com.mysql.jdbc.Util.getInstance(Util.java:381)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3515)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3447)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1951)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2101)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2554)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1761)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2046)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1964)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1949)
> at
> com.jolbox.bonecp.PreparedStatementHandle.executeUpdate(PreparedStatementHandle.java:205)
> at
> org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeUpdate(ParamLoggingPreparedStatement.java:399)
> at
> org.datanucleus.store.rdbms.SQLController.executeStatementUpdate(SQLController.java:439)
> at
> org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:410)
> at
> org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:167)
> at
> org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:143)
> at
> org.datanucleus.state.JDOStateManager.internalMakePersistent(JDOStateManager.java:3784)
> at
> org.datanucleus.state.JDOStateManager.makePersistent(JDOStateManager.java:3760)
> at
> org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2219)
> at
> org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:2065)
> at
> org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1913)
> at
> org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727)
> ... 32 more
> {noformat}
> The following code maybe cause the exception, in In the
> org.apache.hadoop.hive.metastore.ObjectStore.java:3018:
> {noformat}
> MRole nameCheck = this.getMRole(roleName);
> if (nameCheck != null) {
> throw new InvalidObjectException("Role " + roleName + " already
> exists.");
> }
> int now = (int)(System.currentTimeMillis()/1000);
> MRole mRole = new MRole(roleName, now,
> ownerName);
> pm.makePersistent(mRole);
> commited = commitTransaction();
> success = true;
> {noformat}
> If there is exception in the {{getMRole(roleName)}} function, Hive will try
> to insert a duplicate record into table ROLE_MAP, this will cause exception
> above.
> So can we just throw the exception instead of ignoring the exception in the
> {{getMRole(ObjectStore.java:3292)}} and in the
> {{getMetaStoreSchemaVersion(ObjectStore.java:6326)}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
