-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44597/#review122845
-----------------------------------------------------------


Ship it!




Ship It!

- Sid Wagle


On March 9, 2016, 10:30 p.m., Jonathan Hurley wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44597/
> -----------------------------------------------------------
> 
> (Updated March 9, 2016, 10:30 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Robert Levas, and Sid Wagle.
> 
> 
> Bugs: AMBARI-15363
>     https://issues.apache.org/jira/browse/AMBARI-15363
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> When executing certain JPA queries, the Ambari Server seems to deadlock in 
> the c3p0 library. It's caused by two issues:
> 
> - Problems with c3p0 connection management (updated version to use better 
> connection pool handling)
> - EclipseLink JPA CriteriaBuilder cartesian products (updated how we build 
> sorts for criteria to prevent duplicate Root<?>)
> 
> ```
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@7675360a 
> -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending 
> tasks!
> 07 Mar 2016 18:11:01,108  WARN [Timer-0] ThreadPoolAsynchronousRunner:624 - 
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@7675360a 
> -- APPARENT DEADLOCK!!! Complete Status:
>         Managed Threads: 3
>         Active Threads: 1
>         Active Tasks:
> ```
> 
> Looks like the problem is twofold. First, we have a thread performing a very 
> heavy operation in the database:
> 
> ```
> "qtp-ambari-client-36" prio=5 tid=0x00007fd6b344f000 nid=0x8203 runnable 
> [0x0000700002b3e000]
>    java.lang.Thread.State: RUNNABLE
>       at java.lang.Thread.currentThread(Native Method)
>       at java.lang.ThreadLocal.get(ThreadLocal.java:143)
>       at java.lang.StringCoding.deref(StringCoding.java:63)
>       at java.lang.StringCoding.decode(StringCoding.java:179)
>       at java.lang.String.<init>(String.java:416)
>       at org.postgresql.core.Encoding.decode(Encoding.java:191)
>       at org.postgresql.core.Encoding.decode(Encoding.java:203)
>       at 
> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(AbstractJdbc2ResultSet.java:1979)
>       at 
> com.mchange.v2.c3p0.impl.NewProxyResultSet.getString(NewProxyResultSet.java:3316)
>     ...
>       at 
> org.apache.ambari.server.orm.dao.DaoUtils.selectList(DaoUtils.java:62)
> ```
> 
> And then we see this:
> 
> ```
> Internal Exception: org.postgresql.util.PSQLException: Ran out of memory 
> retrieving query results.
> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
> Mar 08, 2016 9:57:53 PM com.sun.jersey.spi.container.ContainerResponse 
> mapMappableContainerException
> SEVERE: The exception contained within MappableContainerException could not 
> be mapped to a response, re-throwing to the HTTP container
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>       at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:149)
>       at java.lang.StringCoding.decode(StringCoding.java:193)
>       at java.lang.String.<init>(String.java:416)
>       at org.postgresql.core.Encoding.decode(Encoding.java:191)
>       at org.postgresql.core.Encoding.decode(Encoding.java:203)
>       at 
> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(AbstractJdbc2ResultSet.java:1979)
>       at 
> com.mchange.v2.c3p0.impl.NewProxyResultSet.getString(NewProxyResultSet.java:3316)
>     ...
>       at 
> org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
> ```
> 
> 
> Diffs
> -----
> 
>   ambari-project/pom.xml ed94004 
>   ambari-server/pom.xml f691fad 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/query/JpaPredicateVisitor.java
>  3a8a631 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/query/JpaSortBuilder.java
>  8021346 
> 
> Diff: https://reviews.apache.org/r/44597/diff/
> 
> 
> Testing
> -------
> 
> Executed problematic queries against a massive database. Problem is resolved 
> in test environment.
> 
> Unit Tests Pending
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Reply via email to