-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53338/
-----------------------------------------------------------
Review request for Ambari, Jonathan Hurley, Nate Cole, and Robert Nettleton.
Bugs: AMBARI-18751
https://issues.apache.org/jira/browse/AMBARI-18751
Repository: ambari
Description
-------
STR:
- Install Ambari 2.4.2 (from branch-2.4)
- Upgrade to Ambari 2.5.0
The following exception fails the upgrade. It appears as though this is from [a
recent
commit|https://github.com/apache/ambari/commit/57116b774c62e5ff6ce22de70458e7c01cccdb07]
into {{branch-2.4}}: AMBARI-18433
When the upgrade runs, it assumes that the role authorizations need to be
added. In existing 2.4.0 and 2.4.1 installations, this is true. But since this
code is now in the 2.4 branch, 2.4.2+ installations will already have the
required data seeded.
```
2016-10-31 11:12:01,193 WARN [main] (Slf4jMLog.java:220) log() - [c3p0] A
PooledConnection that has already signalled a Connection error is still in use!
2016-10-31 11:12:01,193 WARN [main] (Slf4jMLog.java:223) log() - [c3p0]
Another error has occurred [ org.postgresql.util.PSQLException: ERROR: current
transaction is aborted, commands ignored until end of transaction block ] which
will not be reported to listeners!
org.postgresql.util.PSQLException: ERROR: current transaction is aborted,
commands ignored until end of transaction block
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302)
at
com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:353)
at
org.eclipse.persistence.internal.databaseaccess.DatabasePlatform.wasFailureCommunicationBased(DatabasePlatform.java:2914)
at
org.eclipse.persistence.platform.server.ServerPlatformBase.wasFailureCommunicationBased(ServerPlatformBase.java:546)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1616)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:900)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633)
at
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
at
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
at
org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
at
org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at
org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
at
org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153)
at
com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at
com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at
org.apache.ambari.server.orm.dao.PermissionDAO$$EnhancerByGuice$$abe53a1.merge(<generated>)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.addAuthorizationToRole(AbstractUpgradeCatalog.java:888)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.addRoleAuthorization(AbstractUpgradeCatalog.java:848)
at
org.apache.ambari.server.upgrade.UpgradeCatalog250.createRoleAuthorizations(UpgradeCatalog250.java:198)
at
org.apache.ambari.server.upgrade.UpgradeCatalog250.executeDMLUpdates(UpgradeCatalog250.java:130)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:953)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:241)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:363)
[EL Warning]: 2016-10-31 11:12:01.195--UnitOfWork(768406587)--
2016-10-31 11:12:01,197 ERROR [main] (AmbariJpaLocalTxnInterceptor.java:180)
detailedLogForPersistenceError() - [DETAILED ERROR] Rollback reason:
Local Exception Stack:
Exception [EclipseLink-4002] (Eclipse Persistence Services -
2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key
value violates unique constraint "pk_permsn_roleauthorization"
Detail: Key (permission_id, authorization_id)=(1, CLUSTER.RUN_CUSTOM_COMMAND)
already exists.
Error Code: 0
Call: INSERT INTO permission_roleauthorization (authorization_id,
permission_id) VALUES (?, ?)
bind => [2 parameters bound]
at
org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1620)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:900)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633)
at
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
at
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
at
org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
at
org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at
org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
at
org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153)
at
com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at
com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at
org.apache.ambari.server.orm.dao.PermissionDAO$$EnhancerByGuice$$abe53a1.merge(<generated>)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.addAuthorizationToRole(AbstractUpgradeCatalog.java:888)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.addRoleAuthorization(AbstractUpgradeCatalog.java:848)
at
org.apache.ambari.server.upgrade.UpgradeCatalog250.createRoleAuthorizations(UpgradeCatalog250.java:198)
at
org.apache.ambari.server.upgrade.UpgradeCatalog250.executeDMLUpdates(UpgradeCatalog250.java:130)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:953)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:241)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:363)
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value
violates unique constraint "pk_permsn_roleauthorization"
Detail: Key (permission_id, authorization_id)=(1, CLUSTER.RUN_CUSTOM_COMMAND)
already exists.
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)
at
com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:384)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:892)
... 23 more
2016-10-31 11:12:01,198 ERROR [main] (AmbariJpaLocalTxnInterceptor.java:188)
detailedLogForPersistenceError() - [DETAILED ERROR] Internal exception (1) :
org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique
constraint "pk_permsn_roleauthorization"
Detail: Key (permission_id, authorization_id)=(1, CLUSTER.RUN_CUSTOM_COMMAND)
already exists.
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)
at
com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:384)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:892)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633)
at
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
at
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
at
org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
at
org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at
org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
at
org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153)
at
com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at
com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at
org.apache.ambari.server.orm.dao.PermissionDAO$$EnhancerByGuice$$abe53a1.merge(<generated>)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.addAuthorizationToRole(AbstractUpgradeCatalog.java:888)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.addRoleAuthorization(AbstractUpgradeCatalog.java:848)
at
org.apache.ambari.server.upgrade.UpgradeCatalog250.createRoleAuthorizations(UpgradeCatalog250.java:198)
at
org.apache.ambari.server.upgrade.UpgradeCatalog250.executeDMLUpdates(UpgradeCatalog250.java:130)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:953)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:241)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:363)
2016-10-31 11:12:01,198 ERROR [main] (SchemaUpgradeHelper.java:243)
executeDMLUpdates() - Upgrade failed.
javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse
Persistence Services - 2.6.2.v20151217-774c696):
org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key
value violates unique constraint "pk_permsn_roleauthorization"
Detail: Key (permission_id, authorization_id)=(1, CLUSTER.RUN_CUSTOM_COMMAND)
already exists.
Error Code: 0
Call: INSERT INTO permission_roleauthorization (authorization_id,
permission_id) VALUES (?, ?)
bind => [2 parameters bound]
at
org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:159)
at
org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.addAuthorizationToRole(AbstractUpgradeCatalog.java:888)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.addRoleAuthorization(AbstractUpgradeCatalog.java:848)
at
org.apache.ambari.server.upgrade.UpgradeCatalog250.createRoleAuthorizations(UpgradeCatalog250.java:198)
at
org.apache.ambari.server.upgrade.UpgradeCatalog250.executeDMLUpdates(UpgradeCatalog250.java:130)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:953)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:241)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:363)
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services -
2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key
value violates unique constraint "pk_permsn_roleauthorization"
Detail: Key (permission_id, authorization_id)=(1, CLUSTER.RUN_CUSTOM_COMMAND)
already exists.
Error Code: 0
Call: INSERT INTO permission_roleauthorization (authorization_id,
permission_id) VALUES (?, ?)
bind => [2 parameters bound]
at
org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1620)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:900)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633)
at
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
at
org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
at
org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
at
org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at
org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
... 8 more
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value
violates unique constraint "pk_permsn_roleauthorization"
Detail: Key (permission_id, authorization_id)=(1, CLUSTER.RUN_CUSTOM_COMMAND)
already exists.
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)
at
com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:384)
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:892)
... 20 more
2016-10-31 11:12:01,199 ERROR [main] (SchemaUpgradeHelper.java:376) main() -
Exception occurred during upgrade, failed
org.apache.ambari.server.AmbariException: Exception [EclipseLink-4002] (Eclipse
Persistence Services - 2.6.2.v20151217-774c696):
org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key
value violates unique constraint "pk_permsn_roleauthorization"
Detail: Key (permission_id, authorization_id)=(1, CLUSTER.RUN_CUSTOM_COMMAND)
already exists.
Error Code: 0
Call: INSERT INTO permission_roleauthorization (authorization_id,
permission_id) VALUES (?, ?)
bind => [2 parameters bound]
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:244)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:363)
```
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog242.java
31093f1
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
0220464
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog242Test.java
29bf820
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java
b0dc224
Diff: https://reviews.apache.org/r/53338/diff/
Testing
-------
Manually tested upgrades and clean installs for Ambari 2.4.2 and 2.5.0.
# Local test results:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5:17.390s
[INFO] Finished at: Tue Nov 01 08:15:15 EDT 2016
[INFO] Final Memory: 199M/1790M
[INFO] ------------------------------------------------------------------------
# Jenkins test results: PENDING
Thanks,
Robert Levas
