Hello,
I have noticed that with openjpa 2.2.0 (have not checked earlier versions)
even for short lived transactions caches field _joinsPK and _joins of
org.apache.openjpa.jdbc.schema.ForeignKey for all traversed entities using
the Iterator for the resultset.
The jprofiler dump shows count (154612) of cached keys which corresponds to
the total number of query result entities (classA) traversed with two
additional referenced objects (ClassA -1-to-2-> ClassB, where classB is
referenced by FK) , while each transaction consists of 200 entities. This
amounts to ~ 200MB held in HashMap for previously traversed entities.
Both classA and classB have composite Ids if that matters.
Pseuodcode is like:
divide VLResultset into equiwidth buckets of size i.e. 200
for each bucket {
em.beginTransaction();
List<classA> sourceResults = store.getSourcesResultsByBuckets(run,
bucket, -1); //fetch 200 instances using JPQL
for(classA result : sourceResults)
{
//do some ops, including referencing classB and others
referenced
}
em.commit or rollbackTransaction()
} //each bucket
Is it possible to remove this functionality, what are commandments for very
large read-only operations (except recreating EntityManager) to avoid this
kinds of leaks?
I tried to set AutoDetachType.NON_TRANSACTIONAL_READ and no transactions but
this causes problems for lazily fetched fields, so wanted to avoid it as I
am not sure if this would be relevant for this problem.
PU is attached at the end.
I would appreciate any hints.
Cheers,
Chris
<referenceTree direction="outgoing">
<outgoing referenceType="field" referenceName="field _joins of
org.apache.openjpa.jdbc.schema.ForeignKey" count="154612"
sizeBytes="8658272">
<outgoing referenceType="field" referenceName="field table of
java.util.LinkedHashMap (declared by java.util.HashMap)" count="154612"
sizeBytes="12368960" />
<outgoing referenceType="field" referenceName="field header of
java.util.LinkedHashMap" count="154612" sizeBytes="6184480" />
<outgoing referenceType="map key" referenceName="map key of
java.util.LinkedHashMap" count="108" sizeBytes="11664" />
<outgoing referenceType="map value" referenceName="map value of
java.util.LinkedHashMap" count="30" sizeBytes="3240" />
<outgoing referenceType="constant" referenceName="constant
java.util.LinkedHashMap" count="6" sizeBytes="1920" />
<outgoing referenceType="field" referenceName="field keySet of
java.util.LinkedHashMap (declared by java.util.AbstractMap)" count="88"
sizeBytes="1408" />
<outgoing referenceType="field" referenceName="field values of
java.util.LinkedHashMap (declared by java.util.AbstractMap)" count="70"
sizeBytes="1120" />
<outgoing referenceType="implemented interface"
referenceName="implemented interface of java.util.LinkedHashMap" count="1"
sizeBytes="320" />
<outgoing referenceType="superclass" referenceName="superclass of
java.util.LinkedHashMap" count="1" sizeBytes="320" />
</outgoing>
<outgoing referenceType="field" referenceName="field _joinsPK of
org.apache.openjpa.jdbc.schema.ForeignKey" count="154612"
sizeBytes="8658272">
<outgoing referenceType="field" referenceName="field table of
java.util.LinkedHashMap (declared by java.util.HashMap)" count="154612"
sizeBytes="12368960">
<outgoing referenceType="array content" count="154634"
sizeBytes="6185360">
<outgoing referenceType="field" referenceName="field before of
java.util.LinkedHashMap$Entry" count="154634" sizeBytes="6185360" />
<outgoing referenceType="field" referenceName="field after of
java.util.LinkedHashMap$Entry" count="154634" sizeBytes="6185360" />
<outgoing referenceType="field" referenceName="field next of
java.util.LinkedHashMap$Entry (declared by java.util.HashMap$Entry)"
count="35630" sizeBytes="1425200" />
<outgoing referenceType="field" referenceName="field value of
java.util.LinkedHashMap$Entry (declared by java.util.HashMap$Entry)"
count="104" sizeBytes="11248" />
<outgoing referenceType="field" referenceName="field key of
java.util.LinkedHashMap$Entry (declared by java.util.HashMap$Entry)"
count="29" sizeBytes="3136" />
<outgoing referenceType="constant" referenceName="constant
java.util.LinkedHashMap$Entry" count="3" sizeBytes="960" />
<outgoing referenceType="superclass" referenceName="superclass of
java.util.LinkedHashMap$Entry" count="1" sizeBytes="320" />
</outgoing>
<outgoing referenceType="superclass" referenceName="superclass of
java.util.HashMap$Entry[ ]" count="1" sizeBytes="320">
<outgoing referenceType="constant" referenceName="constant
java.lang.Object" count="11" sizeBytes="2656" />
</outgoing>
</outgoing>
<outgoing referenceType="field" referenceName="field header of
java.util.LinkedHashMap" count="154612" sizeBytes="6184480" />
<outgoing referenceType="map value" referenceName="map value of
java.util.LinkedHashMap" count="108" sizeBytes="11664" />
<outgoing referenceType="map key" referenceName="map key of
java.util.LinkedHashMap" count="30" sizeBytes="3240" />
<outgoing referenceType="constant" referenceName="constant
java.util.LinkedHashMap" count="6" sizeBytes="1920" />
<outgoing referenceType="implemented interface"
referenceName="implemented interface of java.util.LinkedHashMap" count="1"
sizeBytes="320" />
<outgoing referenceType="superclass" referenceName="superclass of
java.util.LinkedHashMap" count="1" sizeBytes="320" />
</outgoing>
Example of one of 180K FK refrences:
<primitiveData referenceType="field" referenceName="_seq" value="0"/>
<primitiveData referenceType="field" referenceName="_delAction"
value="2"/>
<primitiveData referenceType="field" referenceName="_upAction"
value="2"/>
<primitiveData referenceType="field" referenceName="_index" value="1"/>
<primitiveData referenceType="field" referenceName="_deferred"
value="true"/>
<primitiveData referenceType="field" referenceName="_count" value="0"/>
<outgoing leaf="false"
class="org.apache.openjpa.jdbc.identifier.DBIdentifier" objectId="0x8d57f5"
type="instance" referenceType="field" referenceName="_columnName"/>
<outgoing leaf="false" class="org.apache.openjpa.jdbc.schema.Column[ ]"
objectId="0x8d6424" type="<class>[]" referenceType="field"
referenceName="_constCols"/>
<outgoing leaf="false" class="org.apache.openjpa.jdbc.schema.Column[ ]"
objectId="0x8d6424" type="<class>[]" referenceType="field"
referenceName="_constColsPK"/>
<outgoing leaf="false" class="java.util.LinkedHashMap"
objectId="0x8db03a" type="instance" referenceType="field"
referenceName="_joins"/>
<outgoing leaf="false" class="java.util.LinkedHashMap"
objectId="0x8db039" type="instance" referenceType="field"
referenceName="_joinsPK"/>
<outgoing leaf="false" class="org.apache.openjpa.jdbc.schema.Column[ ]"
objectId="0x8db038" type="<class>[]" referenceType="field"
referenceName="_locals"/>
<outgoing leaf="false"
class="org.apache.openjpa.jdbc.identifier.DBIdentifier" objectId="0x8db03b"
type="instance" referenceType="field" referenceName="_name"/>
<outgoing leaf="false"
class="org.apache.openjpa.jdbc.identifier.DBIdentifier" objectId="0x8d57f5"
type="instance" referenceType="field" referenceName="_pkColumnName"/>
<outgoing leaf="false"
class="org.apache.openjpa.jdbc.identifier.DBIdentifier" objectId="0x8d57f5"
type="instance" referenceType="field" referenceName="_pkSchemaName"/>
<outgoing leaf="false" class="org.apache.openjpa.jdbc.schema.Table"
objectId="0x8db036" type="instance" referenceType="field"
referenceName="_pkTable"/>
<outgoing leaf="false"
class="org.apache.openjpa.jdbc.identifier.DBIdentifier" objectId="0x8d57f5"
type="instance" referenceType="field" referenceName="_pkTableName"/>
<outgoing leaf="false" class="org.apache.openjpa.jdbc.schema.Column[ ]"
objectId="0x8db037" type="<class>[]" referenceType="field"
referenceName="_pks"/>
<outgoing leaf="false"
class="org.apache.openjpa.jdbc.identifier.DBIdentifier" objectId="0x8d9c32"
type="instance" referenceType="field" referenceName="_schemaName"/>
<outgoing leaf="false" class="org.apache.openjpa.jdbc.schema.Table"
objectId="0x8dafbd" type="instance" referenceType="field"
referenceName="_table"/>
<outgoing leaf="false"
class="org.apache.openjpa.jdbc.identifier.QualifiedDBIdentifier"
objectId="0x8dafc8" type="instance" referenceType="field"
referenceName="_tableName"/>
PU properties:
<properties>
<property name="openjpa.jdbc.DBDictionary"
value="postgres(batchLimit=1000)" />
<property name="openjpa.jdbc.SynchronizeMappings"
value="buildSchema(SchemaAction='',primaryKeys=true,foreignKeys=true,indexes=true)"
/>
<property name="openjpa.RuntimeUnenhancedClasses"
value="unsupported" />
<property name="openjpa.DynamicEnhancementAgent"
value="false" />
<property name="openjpa.DynamicDataStructs"
value="true" />
<property name="openjpa.MetaDataRepository"
value="Preload=true"/>
<property name="openjpa.ProxyManager"
value="AssertAllowedType=false,TrackChanges=true,Unproxyable=gaia.cu1.tools.time.GaiaTime;weka.classifiers.trees.RandomForest;gaia.cu7.om.ClassifierDefinition.ClassifierWrapper"
/>
<property name="openjpa.BrokerImpl"
value="non-finalizing"/>
<property name="openjpa.jdbc.EagerFetchMode"
value="parallel" />
<property name="openjpa.FetchBatchSize" value="1000" />
<property name="openjpa.jdbc.ResultSetType"
value="forward-only" />
<property name="openjpa.jdbc.FetchDirection"
value="forward" />
<property name="openjpa.jdbc.LRSSize" value="query" />
<property name="openjpa.InverseManager" value="false" />
<property name="openjpa.jdbc.SchemaFactory"
value="native(foreignKeys=true)" />
<property name="openjpa.jdbc.MappingDefaults"
value="ForeignKeyDeleteAction=cascade,
JoinForeignKeyDeleteAction=cascade,IndexLogicalForeignKeys=true,
AddNullIndicator=false" />
<property name="openjpa.ConnectionRetainMode" value =
"on-demand"/>
<property name="openjpa.IgnoreChanges" value="true" />
<property name="openjpa.FlushBeforeQueries"
value="true" />
<property name="openjpa.LockTimeout" value="30000" />
<property name="openjpa.AutoDetach" value="false"/>
<property name="openjpa.Compatibility"
value="checkDatabaseForCascadePersistToDetachedEntity=false,AbstractMappingUniDirectional=false"
/>
<property name="openjpa.Multithreaded" value = "false"
/>
<property name="openjpa.BrokerImpl"
value="allowReferenceToSiblingContext=true" />
<property name="openjpa.DataCache"
value="true(EnableStatistics=false,CacheSize=100,
Types=gaia.cu7.om.ClassifierDefinition;gaia.cu7.om.Run;gaia.cu7.om.DictionaryAttribute;gaia.cu7.om.Survey;gaia.cu7.om.Catalog;gaia.cu7.om.input.TimeSeriesType;gaia.cu7.om.output.TimeSeriesResultType;gaia.cu7.om.input.CadenceType,
ExcludedTypes=gaia.cu7.om.input.PhotometricTimeSeries;gaia.cu7.om.output.SourceResult;gaia.cu7.om.output.StatisticalParameters)"
/>
<property name="openjpa.RemoteCommitProvider"
value="sjvm" />
<property name="openjpa.QueryCache"
value="true(CacheSize=100,
SoftReferenceSize=50)" />
<property name="openjpa.Instrumentation"
value="jmx(Instrument='DataCache')" />
<property name="openjpa.Sequence"
value="org.apache.openjpa.jdbc.kernel.NativeJDBCSeq(Allocate=500,
Increment=1)" />
<property name="openjpa.ConnectionDriverName"
value="org.postgresql.Driver" />
<property name="openjpa.Log"
value="DefaultLevel=TRACE , Runtime=TRACE,
Tool=INFO, SQL=TRACE,
DataCache=INFO, JDBC=ERROR, SQLDiag=INFO" />
<property name="openjpa.Log"
value="File=OM10-surveys.log,
DefaultLevel=INFO, Runtime=INFO,
Tool=INFO, SQL=INFO, DataCache=INFO, JDBC=INFO, SQLDiag=INFO" />
<property name="openjpa.ConnectionFactoryProperties"
value="PrintParameters=true, PrettyPrint=true,
PrettyPrintLineLength=130" />
<property name="openjpa.ConnectionProperties"
value="datatype.hstore=gaia.cu7.om.dal.dictionary.PGhstore,accessToUnderlyingConnectionAllowed=true"
/>
<property name="openjpa.Connection2Properties"
value="datatype.hstore=gaia.cu7.om.dal.dictionary.PGhstore" />
</properties>
--
View this message in context:
http://openjpa.208410.n2.nabble.com/Memory-leak-for-long-lived-read-only-EM-tp7581554.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
