[
https://issues.apache.org/jira/browse/AMBARI-15363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hurley updated AMBARI-15363:
-------------------------------------
Status: Patch Available (was: Open)
> Postgres And c3p0 Queries Can Hang Ambari On Large Queries
> ----------------------------------------------------------
>
> Key: AMBARI-15363
> URL: https://issues.apache.org/jira/browse/AMBARI-15363
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.1.0
> Reporter: Jonathan Hurley
> Assignee: Jonathan Hurley
> Priority: Critical
> Fix For: 2.2.2
>
> Attachments: AMBARI-15363.patch
>
>
> When executing certain JPA queries, the Ambari Server seems to deadlock in
> the c3p0 library.
> {code}
> 07 Mar 2016 18:03:01,313 ERROR [qtp-ambari-client-12980] BaseProvider:142 -
> Timed out waiting for metrics.
> 07 Mar 2016 18:04:36,314 ERROR [qtp-ambari-client-12988] BaseProvider:142 -
> Timed out waiting for metrics.
> 07 Mar 2016 18:07:13,653 ERROR [qtp-ambari-client-12990] BaseProvider:142 -
> Timed out waiting for metrics.
> 07 Mar 2016 18:07:27,287 ERROR [qtp-ambari-client-12990] BaseProvider:142 -
> Timed out waiting for metrics.
> 07 Mar 2016 18:09:13,789 WARN [Timer-0] ThreadPoolAsynchronousRunner:608 -
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@7675360a
> -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending
> tasks!
> 07 Mar 2016 18:11:01,108 WARN [Timer-0] ThreadPoolAsynchronousRunner:624 -
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@7675360a
> -- APPARENT DEADLOCK!!! Complete Status:
> Managed Threads: 3
> Active Threads: 1
> Active Tasks:
>
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@207cca8d
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
> Pending Tasks:
>
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@7d3ee004
>
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3ac3468f
> Pool thread stack traces:
>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
> java.lang.Object.wait(Native Method)
>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
> java.lang.Object.wait(Native Method)
>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
> {code}
> Looks like the problem is twofold. First, we have a thread performing a very
> heavy operation in the database:
> {code}
> "qtp-ambari-client-36" prio=5 tid=0x00007fd6b344f000 nid=0x8203 runnable
> [0x0000700002b3e000]
> java.lang.Thread.State: RUNNABLE
> at java.lang.Thread.currentThread(Native Method)
> at java.lang.ThreadLocal.get(ThreadLocal.java:143)
> at java.lang.StringCoding.deref(StringCoding.java:63)
> at java.lang.StringCoding.decode(StringCoding.java:179)
> at java.lang.String.<init>(String.java:416)
> at org.postgresql.core.Encoding.decode(Encoding.java:191)
> at org.postgresql.core.Encoding.decode(Encoding.java:203)
> at
> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(AbstractJdbc2ResultSet.java:1979)
> at
> com.mchange.v2.c3p0.impl.NewProxyResultSet.getString(NewProxyResultSet.java:3316)
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.getObjectThroughOptimizedDataConversion(DatabaseAccessor.java:1345)
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.getObject(DatabaseAccessor.java:1264)
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.fetchRow(DatabaseAccessor.java:1075)
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processResultSet(DatabaseAccessor.java:768)
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:655)
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:558)
> at
> org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2002)
> at
> org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:570)
> at
> org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:242)
> at
> org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:228)
> at
> org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:299)
> at
> org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:694)
> at
> org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2738)
> at
> org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2691)
> at
> org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:495)
> at
> org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
> at
> org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
> at
> org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
> at
> org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:403)
> at
> org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
> at
> org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896)
> at
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804)
> at
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1786)
> at
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1751)
> at
> org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
> at
> org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:469)
> at
> org.apache.ambari.server.orm.dao.DaoUtils.selectList(DaoUtils.java:62)
> {code}
> And then we see this:
> {code}
> Internal Exception: org.postgresql.util.PSQLException: Ran out of memory
> retrieving query results.
> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
> Mar 08, 2016 9:57:53 PM com.sun.jersey.spi.container.ContainerResponse
> mapMappableContainerException
> SEVERE: The exception contained within MappableContainerException could not
> be mapped to a response, re-throwing to the HTTP container
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:149)
> at java.lang.StringCoding.decode(StringCoding.java:193)
> at java.lang.String.<init>(String.java:416)
> at org.postgresql.core.Encoding.decode(Encoding.java:191)
> at org.postgresql.core.Encoding.decode(Encoding.java:203)
> at
> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(AbstractJdbc2ResultSet.java:1979)
> at
> com.mchange.v2.c3p0.impl.NewProxyResultSet.getString(NewProxyResultSet.java:3316)
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.getObjectThroughOptimizedDataConversion(DatabaseAccessor.java:1345)
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.getObject(DatabaseAccessor.java:1264)
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.fetchRow(DatabaseAccessor.java:1075)
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processResultSet(DatabaseAccessor.java:768)
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:655)
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:558)
> at
> org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2002)
> at
> org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:570)
> at
> org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:242)
> at
> org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:228)
> at
> org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:299)
> at
> org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:694)
> at
> org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2738)
> at
> org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2691)
> at
> org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:495)
> at
> org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
> at
> org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
> at
> org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
> at
> org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:403)
> at
> org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
> at
> org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896)
> at
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804)
> at
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1786)
> at
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1751)
> at
> org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
> {code}
> The deadlock could be caused by how c3p0 is working with the connection pool.
> It could also be something specific to how it's generating the native query
> for Postgres. Investigating some various solutions to this...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
