----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44597/#review122824 -----------------------------------------------------------
Ship it! Ship It! - Robert Levas On March 9, 2016, 5:30 p.m., Jonathan Hurley wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44597/ > ----------------------------------------------------------- > > (Updated March 9, 2016, 5:30 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Robert Levas, and Sid Wagle. > > > Bugs: AMBARI-15363 > https://issues.apache.org/jira/browse/AMBARI-15363 > > > Repository: ambari > > > Description > ------- > > When executing certain JPA queries, the Ambari Server seems to deadlock in > the c3p0 library. It's caused by two issues: > > - Problems with c3p0 connection management (updated version to use better > connection pool handling) > - EclipseLink JPA CriteriaBuilder cartesian products (updated how we build > sorts for criteria to prevent duplicate Root<?>) > > ``` > com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@7675360a > -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending > tasks! > 07 Mar 2016 18:11:01,108 WARN [Timer-0] ThreadPoolAsynchronousRunner:624 - > com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@7675360a > -- APPARENT DEADLOCK!!! Complete Status: > Managed Threads: 3 > Active Threads: 1 > Active Tasks: > ``` > > Looks like the problem is twofold. First, we have a thread performing a very > heavy operation in the database: > > ``` > "qtp-ambari-client-36" prio=5 tid=0x00007fd6b344f000 nid=0x8203 runnable > [0x0000700002b3e000] > java.lang.Thread.State: RUNNABLE > at java.lang.Thread.currentThread(Native Method) > at java.lang.ThreadLocal.get(ThreadLocal.java:143) > at java.lang.StringCoding.deref(StringCoding.java:63) > at java.lang.StringCoding.decode(StringCoding.java:179) > at java.lang.String.<init>(String.java:416) > at org.postgresql.core.Encoding.decode(Encoding.java:191) > at org.postgresql.core.Encoding.decode(Encoding.java:203) > at > org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(AbstractJdbc2ResultSet.java:1979) > at > com.mchange.v2.c3p0.impl.NewProxyResultSet.getString(NewProxyResultSet.java:3316) > ... > at > org.apache.ambari.server.orm.dao.DaoUtils.selectList(DaoUtils.java:62) > ``` > > And then we see this: > > ``` > Internal Exception: org.postgresql.util.PSQLException: Ran out of memory > retrieving query results. > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded > Mar 08, 2016 9:57:53 PM com.sun.jersey.spi.container.ContainerResponse > mapMappableContainerException > SEVERE: The exception contained within MappableContainerException could not > be mapped to a response, re-throwing to the HTTP container > java.lang.OutOfMemoryError: GC overhead limit exceeded > at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:149) > at java.lang.StringCoding.decode(StringCoding.java:193) > at java.lang.String.<init>(String.java:416) > at org.postgresql.core.Encoding.decode(Encoding.java:191) > at org.postgresql.core.Encoding.decode(Encoding.java:203) > at > org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(AbstractJdbc2ResultSet.java:1979) > at > com.mchange.v2.c3p0.impl.NewProxyResultSet.getString(NewProxyResultSet.java:3316) > ... > at > org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258) > ``` > > > Diffs > ----- > > ambari-project/pom.xml ed94004 > ambari-server/pom.xml f691fad > > ambari-server/src/main/java/org/apache/ambari/server/api/query/JpaPredicateVisitor.java > 3a8a631 > > ambari-server/src/main/java/org/apache/ambari/server/api/query/JpaSortBuilder.java > 8021346 > > Diff: https://reviews.apache.org/r/44597/diff/ > > > Testing > ------- > > Executed problematic queries against a massive database. Problem is resolved > in test environment. > > Unit Tests Pending > > > Thanks, > > Jonathan Hurley > >
