-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44597/
-----------------------------------------------------------

(Updated March 9, 2016, 5:30 p.m.)


Review request for Ambari, Alejandro Fernandez, Robert Levas, and Sid Wagle.


Changes
-------

It also helps to upload the right patch file.


Bugs: AMBARI-15363
    https://issues.apache.org/jira/browse/AMBARI-15363


Repository: ambari


Description
-------

When executing certain JPA queries, the Ambari Server seems to deadlock in the 
c3p0 library. It's caused by two issues:

- Problems with c3p0 connection management (updated version to use better 
connection pool handling)
- EclipseLink JPA CriteriaBuilder cartesian products (updated how we build 
sorts for criteria to prevent duplicate Root<?>)

```
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@7675360a -- 
APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
07 Mar 2016 18:11:01,108  WARN [Timer-0] ThreadPoolAsynchronousRunner:624 - 
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@7675360a -- 
APPARENT DEADLOCK!!! Complete Status:
        Managed Threads: 3
        Active Threads: 1
        Active Tasks:
```

Looks like the problem is twofold. First, we have a thread performing a very 
heavy operation in the database:

```
"qtp-ambari-client-36" prio=5 tid=0x00007fd6b344f000 nid=0x8203 runnable 
[0x0000700002b3e000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.Thread.currentThread(Native Method)
        at java.lang.ThreadLocal.get(ThreadLocal.java:143)
        at java.lang.StringCoding.deref(StringCoding.java:63)
        at java.lang.StringCoding.decode(StringCoding.java:179)
        at java.lang.String.<init>(String.java:416)
        at org.postgresql.core.Encoding.decode(Encoding.java:191)
        at org.postgresql.core.Encoding.decode(Encoding.java:203)
        at 
org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(AbstractJdbc2ResultSet.java:1979)
        at 
com.mchange.v2.c3p0.impl.NewProxyResultSet.getString(NewProxyResultSet.java:3316)
    ...
        at 
org.apache.ambari.server.orm.dao.DaoUtils.selectList(DaoUtils.java:62)
```

And then we see this:

```
Internal Exception: org.postgresql.util.PSQLException: Ran out of memory 
retrieving query results.
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Mar 08, 2016 9:57:53 PM com.sun.jersey.spi.container.ContainerResponse 
mapMappableContainerException
SEVERE: The exception contained within MappableContainerException could not be 
mapped to a response, re-throwing to the HTTP container
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:149)
        at java.lang.StringCoding.decode(StringCoding.java:193)
        at java.lang.String.<init>(String.java:416)
        at org.postgresql.core.Encoding.decode(Encoding.java:191)
        at org.postgresql.core.Encoding.decode(Encoding.java:203)
        at 
org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(AbstractJdbc2ResultSet.java:1979)
        at 
com.mchange.v2.c3p0.impl.NewProxyResultSet.getString(NewProxyResultSet.java:3316)
    ...
        at 
org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
```


Diffs (updated)
-----

  ambari-project/pom.xml ed94004 
  ambari-server/pom.xml f691fad 
  
ambari-server/src/main/java/org/apache/ambari/server/api/query/JpaPredicateVisitor.java
 3a8a631 
  
ambari-server/src/main/java/org/apache/ambari/server/api/query/JpaSortBuilder.java
 8021346 

Diff: https://reviews.apache.org/r/44597/diff/


Testing
-------

Executed problematic queries against a massive database. Problem is resolved in 
test environment.

Unit Tests Pending


Thanks,

Jonathan Hurley

Reply via email to