-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44597/
-----------------------------------------------------------
(Updated March 9, 2016, 11:45 p.m.)
Review request for Ambari, Alejandro Fernandez, Robert Levas, and Sid Wagle.
Bugs: AMBARI-15363
https://issues.apache.org/jira/browse/AMBARI-15363
Repository: ambari
Description
-------
When executing certain JPA queries, the Ambari Server seems to deadlock in the
c3p0 library. It's caused by two issues:
- Problems with c3p0 connection management (updated version to use better
connection pool handling)
- EclipseLink JPA CriteriaBuilder cartesian products (updated how we build
sorts for criteria to prevent duplicate Root<?>)
```
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@7675360a --
APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
07 Mar 2016 18:11:01,108 WARN [Timer-0] ThreadPoolAsynchronousRunner:624 -
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@7675360a --
APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 1
Active Tasks:
```
Looks like the problem is twofold. First, we have a thread performing a very
heavy operation in the database:
```
"qtp-ambari-client-36" prio=5 tid=0x00007fd6b344f000 nid=0x8203 runnable
[0x0000700002b3e000]
java.lang.Thread.State: RUNNABLE
at java.lang.Thread.currentThread(Native Method)
at java.lang.ThreadLocal.get(ThreadLocal.java:143)
at java.lang.StringCoding.deref(StringCoding.java:63)
at java.lang.StringCoding.decode(StringCoding.java:179)
at java.lang.String.<init>(String.java:416)
at org.postgresql.core.Encoding.decode(Encoding.java:191)
at org.postgresql.core.Encoding.decode(Encoding.java:203)
at
org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(AbstractJdbc2ResultSet.java:1979)
at
com.mchange.v2.c3p0.impl.NewProxyResultSet.getString(NewProxyResultSet.java:3316)
...
at
org.apache.ambari.server.orm.dao.DaoUtils.selectList(DaoUtils.java:62)
```
And then we see this:
```
Internal Exception: org.postgresql.util.PSQLException: Ran out of memory
retrieving query results.
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Mar 08, 2016 9:57:53 PM com.sun.jersey.spi.container.ContainerResponse
mapMappableContainerException
SEVERE: The exception contained within MappableContainerException could not be
mapped to a response, re-throwing to the HTTP container
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:149)
at java.lang.StringCoding.decode(StringCoding.java:193)
at java.lang.String.<init>(String.java:416)
at org.postgresql.core.Encoding.decode(Encoding.java:191)
at org.postgresql.core.Encoding.decode(Encoding.java:203)
at
org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(AbstractJdbc2ResultSet.java:1979)
at
com.mchange.v2.c3p0.impl.NewProxyResultSet.getString(NewProxyResultSet.java:3316)
...
at
org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
```
Diffs (updated)
-----
ambari-project/pom.xml ed94004
ambari-server/pom.xml f691fad
ambari-server/src/main/java/org/apache/ambari/server/api/query/JpaPredicateVisitor.java
3a8a631
ambari-server/src/main/java/org/apache/ambari/server/api/query/JpaSortBuilder.java
8021346
ambari-server/src/test/java/org/apache/ambari/server/api/query/JpaSortBuilderTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/44597/diff/
Testing
-------
Executed problematic queries against a massive database. Problem is resolved in
test environment.
Tests run: 3928, Failures: 0, Errors: 0, Skipped: 33
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 28:37 min
[INFO] Finished at: 2016-03-09T18:37:57-05:00
[INFO] Final Memory: 51M/740M
[INFO] ------------------------------------------------------------------------
Thanks,
Jonathan Hurley
