Myroslav Papirkovskyi created AMBARI-15831:
----------------------------------------------
Summary: DB exception when trying to add 700 hosts to cluster
Key: AMBARI-15831
URL: https://issues.apache.org/jira/browse/AMBARI-15831
Project: Ambari
Issue Type: Bug
Components: ambari-server
Affects Versions: 2.2.2
Reporter: Myroslav Papirkovskyi
Assignee: Myroslav Papirkovskyi
Priority: Critical
Fix For: 2.2.2
Had a working cluster with 900 nodes and everything was fine. Proceeded to add
700 nodes (with no external-ip) to the cluster - which registered fine. Added
DataNode, NodeManager and ResourceManager to all 700 nodes and clicked on
Deploy in UI. One of the calls failed and UI wizard was stuck, with the
following in ambari-server.log
{noformat}
06 Apr 2016 17:30:45,172 ERROR [qtp-ambari-client-2713]
AmbariJpaLocalTxnInterceptor:180 - [DETAILED ERROR] Rollback reason:
Local Exception Stack:
Exception [EclipseLink-4002] (Eclipse Persistence Services -
2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key
value violates unique constraint "clusterhostmapping_pkey"
Error Code: 0
Call: INSERT INTO ClusterHostMapping (cluster_id, host_id) VALUES (?, ?)
bind => [2 parameters bound]
at
org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1620)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:900)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633)
at
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
at
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
at
org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
at
org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at
org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
at
org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153)
at
com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at
com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at
org.apache.ambari.server.state.cluster.ClustersImpl$$EnhancerByGuice$$f4eca4b6.mapHostClusterEntities(<generated>)
at
org.apache.ambari.server.state.cluster.ClustersImpl.mapHostToCluster(ClustersImpl.java:592)
at
org.apache.ambari.server.state.cluster.ClustersImpl.mapHostToCluster(ClustersImpl.java:616)
at
org.apache.ambari.server.state.cluster.ClustersImpl.updateHostWithClusterAndAttributes(ClustersImpl.java:492)
at
org.apache.ambari.server.controller.internal.HostResourceProvider.createHosts(HostResourceProvider.java:509)
at
org.apache.ambari.server.controller.internal.HostResourceProvider$1.invoke(HostResourceProvider.java:197)
at
org.apache.ambari.server.controller.internal.HostResourceProvider$1.invoke(HostResourceProvider.java:194)
at
org.apache.ambari.server.controller.internal.AbstractResourceProvider.invokeWithRetry(AbstractResourceProvider.java:450)
at
org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:273)
at
org.apache.ambari.server.controller.internal.HostResourceProvider.createResources(HostResourceProvider.java:194)
at
org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:289)
at
org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:76)
at
org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36)
at
org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)
at
org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
at
org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:106)
at
org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:75)
at
org.apache.ambari.server.api.services.HostService.createHosts(HostService.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
.....
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value
violates unique constraint "clusterhostmapping_pkey"
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:892)
... 115 more
06 Apr 2016 17:30:45,175 ERROR [qtp-ambari-client-2713]
AmbariJpaLocalTxnInterceptor:188 - [DETAILED ERROR] Internal exception (1) :
org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique
constraint "clusterhostmapping_pkey"
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:892)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633)
at
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
at
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
at
org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
at
org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at
org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
at
org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153)
at
com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at
com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at
org.apache.ambari.server.state.cluster.ClustersImpl$$EnhancerByGuice$$f4eca4b6.mapHostClusterEntities(<generated>)
at
org.apache.ambari.server.state.cluster.ClustersImpl.mapHostToCluster(ClustersImpl.java:592)
at
org.apache.ambari.server.state.cluster.ClustersImpl.mapHostToCluster(ClustersImpl.java:616)
at
org.apache.ambari.server.state.cluster.ClustersImpl.updateHostWithClusterAndAttributes(ClustersImpl.java:492)
at
org.apache.ambari.server.controller.internal.HostResourceProvider.createHosts(HostResourceProvider.java:509)
at
org.apache.ambari.server.controller.internal.HostResourceProvider$1.invoke(HostResourceProvider.java:197)
at
org.apache.ambari.server.controller.internal.HostResourceProvider$1.invoke(HostResourceProvider.java:194)
at
org.apache.ambari.server.controller.internal.AbstractResourceProvider.invokeWithRetry(AbstractResourceProvider.java:450)
at
org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:273)
at
org.apache.ambari.server.controller.internal.HostResourceProvider.createResources(HostResourceProvider.java:194)
at
org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:289)
at
org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:76)
at
org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36)
at
org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)
{noformat}
{noformat}
06 Apr 2016 17:30:45,176 ERROR [qtp-ambari-client-2713]
BaseManagementHandler:66 - Caught a runtime exception while attempting to
create a resource: Exception [EclipseLink-4002] (Eclipse Persistence Services -
2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key
value violates unique constraint "clusterhostmapping_pkey"
Error Code: 0
Call: INSERT INTO ClusterHostMapping (cluster_id, host_id) VALUES (?, ?)
bind => [2 parameters bound]
javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse
Persistence Services - 2.6.2.v20151217-774c696):
org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key
value violates unique constraint "clusterhostmapping_pkey"
Error Code: 0
Call: INSERT INTO ClusterHostMapping (cluster_id, host_id) VALUES (?, ?)
bind => [2 parameters bound]
at
org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:159)
at
org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153)
at
org.apache.ambari.server.state.cluster.ClustersImpl.mapHostToCluster(ClustersImpl.java:592)
at
org.apache.ambari.server.state.cluster.ClustersImpl.mapHostToCluster(ClustersImpl.java:616)
at
org.apache.ambari.server.state.cluster.ClustersImpl.updateHostWithClusterAndAttributes(ClustersImpl.java:492)
at
org.apache.ambari.server.controller.internal.HostResourceProvider.createHosts(HostResourceProvider.java:509)
at
org.apache.ambari.server.controller.internal.HostResourceProvider$1.invoke(HostResourceProvider.java:197)
at
org.apache.ambari.server.controller.internal.HostResourceProvider$1.invoke(HostResourceProvider.java:194)
at
org.apache.ambari.server.controller.internal.AbstractResourceProvider.invokeWithRetry(AbstractResourceProvider.java:450)
at
org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:273)
at
org.apache.ambari.server.controller.internal.HostResourceProvider.createResources(HostResourceProvider.java:194)
at
org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:289)
at
org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:76)
at
org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36)
at
org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)
at
org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
at
org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:106)
at
org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:75)
at
org.apache.ambari.server.api.services.HostService.createHosts(HostService.java:114)
....
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services -
2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key
value violates unique constraint "clusterhostmapping_pkey"
Error Code: 0
Call: INSERT INTO ClusterHostMapping (cluster_id, host_id) VALUES (?, ?)
bind => [2 parameters bound]
at
org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1620)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:900)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633)
at
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
at
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
at
org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
at
org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at
org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
... 100 more
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value
violates unique constraint "clusterhostmapping_pkey"
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:892)
... 112 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
