[ https://issues.apache.org/jira/browse/SENTRY-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883550#comment-15883550 ]
kalyan kumar kalvagadda commented on SENTRY-1629: ------------------------------------------------- Issue reported could be resolved by removing AUTHZ_OBJ_ID_OID as primary. But this approach exposes a issue we end up storing duplicate paths when multiple objects use same path. *Example:* Let's say there are two external tables pointing to the same file we would end up having two entries in MAUTHZPATHSMAPPING_PATHS both having same path but different AUTHZ_OBJ_ID_OID. To address this issue JDO and schema definition have to be changes. I will update the details in my next comment. > Current MAuthzPathsMapping table definition may cause error 'Duplicate entry > XX for key PRIMARY' > ------------------------------------------------------------------------------------------------ > > Key: SENTRY-1629 > URL: https://issues.apache.org/jira/browse/SENTRY-1629 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin > Affects Versions: sentry-ha-redesign > Reporter: Hao Hao > Assignee: kalyan kumar kalvagadda > Fix For: sentry-ha-redesign > > > In MAuthzPathsMapping table defines 1-N mapping of authzObjName to HMSPaths. > However, current JDO metadata/SQL definition may cause duplicate entry error. > {noformat} 2017-02-13 07:03:06,909 ERROR > org.apache.sentry.provider.db.service.persistent.TransactionManager: The > transaction has reached max retry numbe, rAdd request failed : INSERT INTO > `MAUTHZPATHSMAPPING_PATHS` (`AUTHZ_OBJ_ID_OID`,`PATHS`) VALUES (?,?) > javax.jdo.JDODataStoreException: Add request failed : INSERT INTO > `MAUTHZPATHSMAPPING_PATHS` (`AUTHZ_OBJ_ID_OID`,`PATHS`) VALUES (?,?) > FailedObject:MSentryPathsUpdate [authzObj=default.web_logs], > paths=[[user/admin/2015_11_21, user/admin/2015_11_20, > user/hive/warehouse/web_logs, user/admin/2015_11_18, user/admin/2015_11_19]], > createTimeMs=[1486969386906] > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:449) > at > org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732) > at > org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752) > at > org.apache.sentry.provider.db.service.persistent.SentryStore.createAuthzPathsMappingCore(SentryStore.java:2200) > at > org.apache.sentry.provider.db.service.persistent.SentryStore.access$2500(SentryStore.java:88) > at > org.apache.sentry.provider.db.service.persistent.SentryStore$31.execute(SentryStore.java:2174) > at > org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:114) > at > org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:183) > at > org.apache.sentry.provider.db.service.persistent.SentryStore.persistFullPathsImage(SentryStore.java:2170) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:263) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > NestedThrowablesStackTrace: > org.datanucleus.store.rdbms.exceptions.MappedDatastoreException: INSERT INTO > `MAUTHZPATHSMAPPING_PATHS` (`AUTHZ_OBJ_ID_OID`,`PATHS`) VALUES (?,?) > at > org.datanucleus.store.rdbms.scostore.JoinSetStore.doInternalAdd(JoinSetStore.java:754) > at > org.datanucleus.store.rdbms.scostore.JoinSetStore.internalAdd(JoinSetStore.java:502) > at > org.datanucleus.store.rdbms.scostore.JoinSetStore.addAll(JoinSetStore.java:433) > at > org.datanucleus.store.rdbms.mapping.java.CollectionMapping.postInsert(CollectionMapping.java:136) > at > org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:519) > at > org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:167) > at > org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:143) > at > org.datanucleus.state.JDOStateManager.internalMakePersistent(JDOStateManager.java:3784) > at > org.datanucleus.state.JDOStateManager.makePersistent(JDOStateManager.java:3760) > at > org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2219) > at > org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:2065) > at > org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1913) > at > org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217) > at > org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727) > at > org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752) > at > org.apache.sentry.provider.db.service.persistent.SentryStore.createAuthzPathsMappingCore(SentryStore.java:2200) > at > org.apache.sentry.provider.db.service.persistent.SentryStore.access$2500(SentryStore.java:88) > at > org.apache.sentry.provider.db.service.persistent.SentryStore$31.execute(SentryStore.java:2174) > at > org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:114) > at > org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:183) > at > org.apache.sentry.provider.db.service.persistent.SentryStore.persistFullPathsImage(SentryStore.java:2170) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:263) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.sql.BatchUpdateException: Duplicate entry '102' for key > 'PRIMARY' > at > com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1815) > at > com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1277) > at > com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469) > at > org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:372) > at > org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:683) > at > org.datanucleus.store.rdbms.SQLController.executeStatementUpdate(SQLController.java:409) > at > org.datanucleus.store.rdbms.scostore.JoinSetStore.doInternalAdd(JoinSetStore.java:733) > ... 28 more > Caused by: > com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: > Duplicate entry '102' for key 'PRIMARY' > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) > at com.mysql.jdbc.Util.getInstance(Util.java:360) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823) > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530) > at > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907) > at > com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141) > at > com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773) > ... 34 more > {noformat} > The possible fix could be either changing JDO metadata definition of 1-N > mapping with adding a new HMSPaths table. Or change the SQL scripts. -- This message was sent by Atlassian JIRA (v6.3.15#6346)