John - It's Friday afternoon, so I haven't really dug into this issue deeply, but for giggles can you try to set openjpa.InverseManager to false? I struggled to recreate your failure, but as soon as I set that property to true I started seeing the same behavior.
Thanks, Rick On Fri, Jun 8, 2012 at 10:43 AM, Kevin Sutter <kwsut...@gmail.com> wrote: > Hi John, > 1. The @InverseLogical annotation shouldn't affect your scenario. I was > just curious as to why you were using it. My ignorance of the usefulness > of this property would shy me away from its usage, but that's just me. > > 2. Thanks for the explanation of your update() method. This sounds like > the proper usage of the merge() processing. > > 3. Thanks. Just curious on your persistence model. > > 4. Your persistence.xml looks fine. Nothing else jumps out at me. > > This leaves us at figuring out a simple testcase to reproduce the problem. > Since we have many JUnits with caching and collections already, I'm > surprised that we haven't discovered the issue prior to this. I'm not sure > if I'll have time today to create or modify or JUnit to reproduce the issue > (taking some vacation). If you have any cycles to simplify your scenario, > it would help. Otherwise, maybe early next week we can get to the bottom > of this... > > I did look into your comments concerning the openjpa.RefreshFromDataCache > property... Unfortunately, this property doesn't seem to be working as you > might expect. First, it's not documented in our OpenJPA manual. And, when > I look at the code, it doesn't look like the value of this property is > being used properly to bypass the cache. So, I wouldn't count on this. > > I then looked at the proper means of specifying a datacache bypass with the > javax.persistence.cache.RetrieveMode properties as defined by JPA 2.0. You > can specify a value of USE or BYPASS that can be used for finds, queries, > and refresh operations. But, then there's this statement in the spec: > > *"The retrieveMode property is ignored for the refresh method,* > *which always causes data to be retrieved from the database, not the > cache." > * > > You have mentioned that you are performing a refresh. Are you calling > em.refresh(object)? If so, this sounds like there might be a bug in that > processing... You could try passing in the RetrieveMode property of BYPASS > on the refresh just to see if it makes a difference. > > That's all I got right now... Have a good weekend! > > Kevin > > On Fri, Jun 8, 2012 at 1:57 AM, Boblitz John <john.bobl...@bertschi.com > >wrote: > > > Hi Kevin, > > > > Thanks for the response. Here some answers: > > > > 1. "Why is the @InverseLogical annotation required?" > > I included this quite early in the development of as the manual > suggested > > that > > using the annotation would cause openJpa to manage the relations for me. > > Are you implying that mappedBy in uneeded if I use @InverseLogical? > > I could remove the annotation and check - but this is prevalent > > throuhout the model, and I have not noticed any problems elsewhere (yet). > > > > 2. "Can you explain the use of the update() method?" > > update receives the detached entities which are then merged and > committed. > > The application is on three tiers in a Java SE Environment- DB -> App -> > > Gui. The GUI > > gets the data in one step and the enitities are detached. At some point > > later, the data is > > passed back to the app. The process is, basically: > > > > Get a manager > > Read the data from a DTO Object > > Validate the data > > Create entity object and fill with DTO Data > > Merge into the context > > Commit > > > > I am stuck with using separate DTO Objects which unfortunately also break > > the references as only the > > uid is passed - but, other than the overhead, I have noticed no problems > > in doing so > > > > 3. "How is the lifecycle of the EM being managed, ...?" > > As I mentioned in 2, since I'm in SE, each request essentially gets it's > > own manager. > > > > 4. "Can you post your whole persistence.xml? " > > Below my persistence.xml. Yes, the QueryCache & QuerySQLCache are > active. > > > > I just reran the scenario and produced a trace (some very nasty sqls in > > there as the apps are > > all still using default fetch plans and the Equipment entity is complex > :D > > ) > > > > Essentially, when the merge() is executed - several selects are executed > > and since > > nothing changed yet, all info is pulled from cache. > > > > Then, several updates / inserts are executed as expected. Just after > > these statements I get another > > list of hits on the cache ... > > > > Then I get this: (413938 is the Equipment Entity and 502801 is the new > > Axle] > > > > - Performing a commit on the cache. Adding [502801], updating [] and > > [414690, 414571, 414570, 414082, 414688, 414689, 414626, 414627, 414624, > > 413938, 414572, 414625, 414681, > > 414621, 414680, 414623, 414683, 414622, 414682, 414685, 414684, 414687, > > 414686, 414679], and removing []. > > > > Just after that, I do a refresh and this shows: > > - Cache hit while looking up key "413938". > > > > Here I expected that the cache would not be used and that the DB would be > > read to resynch the cache since > > I have: openjpa.RefreshFromDataCache: false > > > > > > > > I appreciate the help!!! > > > >  > > > > John > > > > ---- > > > > Who is General Failure, and why is he reading my hard disk? > > > > > > > > <?xml version="1.0" encoding="UTF-8"?> > > <persistence > > version="2.0" > > xmlns="http://java.sun.com/xml/ns/persistence"; > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > > xsi:schemaLocation="http://java.sun.com/xml/ns/persistence > > > > http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd"; > > > > > > > <persistence-unit > > name="g11.persistence" > > transaction-type="RESOURCE_LOCAL" > > > > > > > > > <provider>org.apache.openjpa.persistence.PersistenceProviderImpl > > </provider> > > > > > > <!-- JPQL Named Queries --> > > <mapping-file>META-INF/dbNamedQueries.xml</mapping-file> > > <!-- JPQL Named Queries generated from the database schema.xml --> > > <mapping-file>META-INF/guiNamedQueries.xml</mapping-file> > > <!-- JPQL Named Queries generated for gui selection screens --> > > <mapping-file>META-INF/acNamedQueries.xml</mapping-file> > > <!-- JPQL Named Queries generated for gui selection screens --> > > <mapping-file>META-INF/bdisNamedQueries.xml</mapping-file> > > <!-- JPQL Named Queries generated for general bdis queries --> > > > > > > > > <!-- Class Definitions --> > > <class> base.BaseEntity</class> > > <class> ams.locations.BusinessPartner</class> > > <class> ams.locations.Country</class> > > <class> ams.locations.CountryGroup</class> > > <class> ams.locations.Location</class> > > <class> ams.locations.Subdivision</class> > > <class> ams.locations.ValueAddedTax</class> > > <class> com.codes.Currency</class> > > <class> com.codes.ExchangeRate</class> > > <class> com.codes.ModeOfTransportation</class> > > <class> com.codes.Notes</class> > > <class> com.codes.Period</class> > > <class> com.codes.Text</class> > > <class> com.codes.TranslatedText</class> > > <class> com.codes.UnitOfMeasurement</class> > > <class> com.codes.WritingSystem</class> > > <class> com.company.BusinessUnit</class> > > <class> com.company.Company</class> > > <class> com.company.Department</class> > > <class> com.company.OrganisationalUnit</class> > > <class> com.company.ServiceCenter</class> > > <class> com.system.Printer</class> > > <class> com.system.UserGroup</class> > > <class> com.system.Users</class> > > <class> com.system.Permission</class> > > <class> com.system.EntityExport</class> > > <class> fms.equipment.Equipment</class> > > <class> fms.equipment.EquipmentAttachment</class> > > <class> fms.equipment.EquipmentNote</class> > > <class> fms.equipment.HireOut</class> > > <class> fms.equipment.RentalAgreement</class> > > <class> fms.equipment.TransferAgreement</class> > > <class> fms.equipment.certifications.Certification</class> > > <class> > > fms.equipment.certifications.CertificationType</class> > > <class> > > fms.equipment.certifications.DriverCertification</class> > > <class> > > fms.equipment.certifications.DriverCertificationDocument</class> > > <class> > > fms.equipment.certifications.DriverRequiredCertifications</class> > > <class> > > fms.equipment.certifications.EquipmentCertification</class> > > <class> > > fms.equipment.certifications.EquipmentCertificationDocument</class> > > <class> > > fms.equipment.certifications.EquipmentRequiredCertifications</class> > > <class> fms.equipment.certifications.Issuer</class> > > <class> fms.equipment.components.Axle</class> > > <class> fms.equipment.components.Chamber</class> > > <class> fms.equipment.components.Chassis</class> > > <class> > fms.equipment.components.TankBoxUsageHistory</class> > > <class> fms.equipment.components.Hose</class> > > <class> fms.equipment.components.Motor</class> > > <class> fms.equipment.components.MotorUsage</class> > > <class> fms.equipment.components.TankBox</class> > > <class> fms.equipment.damage.DamageHow</class> > > <class> fms.equipment.damage.DamageWhat</class> > > <class> fms.equipment.damage.DamageWhere</class> > > <class> fms.equipment.damage.EquipmentDamage</class> > > <class> fms.equipment.properties.Characteristic</class> > > <class> > fms.equipment.properties.CharacteristicType</class> > > <class> fms.equipment.properties.EquipmentType</class> > > <class> fms.equipment.properties.Manufacturer</class> > > <class> fms.equipment.properties.Model</class> > > <class> fms.equipment.properties.PropertyType</class> > > <class> fms.equipment.properties.Specification</class> > > <class> fms.equipment.properties.SpecificationType</class> > > <class> > > fms.equipment.properties.TechnicalCharacteristic</class> > > <class> fms.equipment.service.Service</class> > > <class> fms.equipment.service.ServiceHistory</class> > > <class> fms.equipment.service.ServiceType</class> > > <class> fms.equipment.service.ServicesOffered</class> > > <class> hrs.Driver</class> > > <class> hrs.DriverRestriction</class> > > <class> hrs.DriverVacationModel</class> > > <class> hrs.Employee</class> > > <class> hrs.RestrictionType</class> > > > > > > <properties> > > > > <!-- Logging --> > > <property > > name="openjpa.Log" > > value="log4j" /> > > > > <!-- Connection --> > > <property > > name="openjpa.ConnectionDriverName" > > value="org.postgresql.Driver" /> > > <property > > name="openjpa.ConnectionProperties" > > value="MaxActive=100, MaxIdle=5, > MinIdle=2, > > MaxWait=60000" /> > > <property > > name="openjpa.ConnectionFactoryProperties" > > value="QueryTimeOut=5000, > PrettyPrint=true, > > PrettyPrintLineLength=80, PrintParameters=true" /> > > > > <property > > name="openjpa.ConnectionURL" > > value=**** /> > > <property > > name="openjpa.ConnectionUserName" > > value=**** /> > > <property > > name="openjpa.ConnectionPassword" > > value=**** /> > > <property > > name="openjpa.jdbc.DBDictionary" > > > > value="postgres(supportsNullTableForGetImportedKeys=false)" /> > > > > <!-- Caching --> > > <property > > name="openjpa.DataCache" > > value="true(CacheSize=5000, > > EnableStatistics=true)" /> > > <property > > name="openjpa.RemoteCommitProvider" > > value="sjvm" /> > > <property > > name="openjpa.QueryCache" > > value="true(CacheSize=1000, > > SoftReferenceSize=100, EvictPolicy='timestamp')" /> > > <property > > name="openjpa.QueryCompilationCache" > > value="all" /> > > <property > > name="openjpa.jdbc.QuerySQLCache" > > value="true(EnableStatistics=true)" /> > > > > > > <!-- Misc --> > > <property > > name="openjpa.InverseManager" > > value="true" /> > > <property > > name="openjpa.DetachState" > > > > value="fetch-groups(DetachedStateField=true)" /> > > <property > > name="openjpa.jdbc.SchemaFactory" > > value="native(ForeignKeys=true)" /> > > > > </properties> > > </persistence-unit> > > </persistence> > > > > > > > > > > > > > -----Ursprüngliche Nachricht----- > > > Von: Kevin Sutter [mailto:kwsut...@gmail.com] > > > Gesendet: Donnerstag, 7. Juni 2012 20:48 > > > An: users@openjpa.apache.org > > > Betreff: Re: Bug? DataCache after update contains only the > > > updated information from a Collection > > > > > > Hi John, > > > Good background information. Thanks for the detail. > > > Unfortunately, nothing is jumping out at me as an "easy fix". > > > I do have a couple of observations and/or questions... > > > > > > o Why is the @InverseLogical annotation required? It would > > > seem that you have already defined the bidirectional > > > relationship via the mappedBy attribute on the @OneToMany. > > > Is there something else you were trying to model by using the > > > @InverseLogical? And, does the removal of that annotation > > > change the processing in any way? > > > > > > o Can you explain the use of the update() method? Are the > > > entities being passed into this method detached from a > > > persistence context? That's where the merge() method comes > > > into play -- merging detached entities into a persistence > > > context. The merge() method is not used just because there > > > are updates to an Entity. > > > > > > o Your code example doesn't show the actual interaction with > > > the EM. How is the lifecycle of the EM being managed, and is > > > the same EM being used for all of the interactions with > > > transactions, persisting, merging, etc? > > > > > > o Can you post your whole persistence.xml? For example, you > > > didn't mention the use of the QueryCache property, but it > > > looks like this is enabled via the dump of the properties. > > > The QueryCache is no longer automatically enabled when the > > > DataCache is enabled, so it must be explicitly enabled. Not > > > that it should affect this particular scenario, but I'm just > > > curious if there are other property "mismatches" that I didn't catch. > > > > > > o If none of these questions provide any fruit, then you > > > might have found a bug and a JIRA will be required. But, > > > let's do a bit more Q&A first... > > > > > > Thanks, > > > Kevin > > > > > > On Thu, Jun 7, 2012 at 2:05 AM, Boblitz John > > > <john.bobl...@bertschi.com>wrote: > > > > > > > Hello, > > > > > > > > I'm running openJpa 2.2.0 and have a problem with the > > > DataCache function. > > > > I spent the better part of two days localizing the problem > > > and looking > > > > for solutions, but alas ... > > > > > > > > So, hoping some guru out there can spot my error, here the details: > > > > > > > > Environment JavaSE > > > > > > > > Persistence XML: > > > > <property name="openjpa.DataCache" value="true(CacheSize=5000, > > > > EnableStatistics=true)" /> <property > > > > name="openjpa.RemoteCommitProvider" value="sjvm" /> <property > > > > name="openjpa.DetachState" > > > > value="fetch-groups(DetachedStateField=true)" /> > > > > > > > > Entities: > > > > > > > > @MappedSuperclass > > > > @EntityListeners({ EntityManipulationLogger.class, > > > EntityLogger.class > > > > }) public abstract class BaseEntity { @Version > > > > @Column(columnDefinition = "int8") private long versionId; > > > > > > > > @Id > > > > @GeneratedValue(strategy = GenerationType.SEQUENCE) @Column(name = > > > > "uniqueid", columnDefinition = "int8") protected long uniqueId; > > > > > > > > ... > > > > > > > > @Entity > > > > @Table(name = "Equipment") > > > > public class Equipment extends BaseEntity { > > > > > > > > @JsonManagedReference > > > > @OneToMany(cascade = CascadeType.ALL, mappedBy = "equipmentId") > > > > @InverseLogical("equipmentId") > > > > private Set<Axle> axles = new HashSet<Axle>(); > > > > > > > > ... > > > > > > > > @Entity > > > > @Table(name = "Axle") > > > > public class Axle extends BaseEntity { > > > > > > > > @JsonBackReference > > > > @ManyToOne(fetch = FetchType.EAGER) > > > > @JoinColumn(name = "equipmentId", columnDefinition = > > > "int8", nullable > > > > = > > > > false) > > > > private Equipment equipmentId; > > > > > > > > > > > > Code used to Update the DataBase: > > > > > > > > @Override > > > > public <T extends BaseEntity> T update(T pEntity) { > > > > this.lock.lock(); > > > > T object = null; > > > > try { > > > > try { > > > > getTransaction().begin(); > > > > object = merge(pEntity); > > > > } catch (Exception e) { > > > > mTrc.error("Error in update(): ", e); > > > > getTransaction().setRollbackOnly(); > > > > handleException(e); > > > > } finally { > > > > if (getTransaction().isActive()) { > > > > if (getTransaction().getRollbackOnly()) { > > > > getTransaction().rollback(); > > > > } else { > > > > getTransaction().commit(); > > > > } > > > > } > > > > } > > > > } catch (Exception e) { > > > > mTrc.error("Error in update(): ", e); > > > > handleException(e); > > > > } finally { > > > > this.lock.unlock(); > > > > } > > > > return object; > > > > } > > > > > > > > Symptom: > > > > When updating the Equipment entity, if an additional Axle > > > is added to > > > > the Set and the other Axles remain unchanged: > > > > 1. Upon entry into update(), pEntity contains all the Axels (new & > > > > old) including UID & Version. > > > > 2. object = merge(pEntity) - performs as expected, object contains > > > > all the Axels and the new Axel has been assigned a UID 3. After > > > > commit(), the new Axle is added to the Database (good) 4. > > > The entity > > > > is updated in the DataCache (I would assume this is good as > > > > well) > > > > 5. "object" however now only contains one element in the > > > Set<Axle> - > > > > the one we added, the others are no longer there. > > > > 6. A subsequent refresh of the returns the same results as > > > in #5! It > > > > hits in the DataCache, so I assume that the Update of the Cache > > > > contains only the "new" data from object. > > > > > > > > Turning off the DataCache (or excluding the Equipment > > > Entity) solves > > > > the problem. > > > > > > > > Here the complete config: > > > > > > > > openjpa.AutoClear: 0 > > > > openjpa.AutoDetach: [Ljava.lang.String;@60cf710e > > > > openjpa.BrokerFactory: jdbc > > > > openjpa.BrokerImpl: default > > > > openjpa.CacheDistributionPolicy: default > > > > openjpa.Callbacks: default > > > > openjpa.ClassResolver: default > > > > openjpa.Compatibility: default > > > > openjpa.ConnectionDriverName: org.postgresql.Driver > > > > openjpa.ConnectionFactoryMode: false > > > > openjpa.ConnectionFactoryProperties: QueryTimeOut=5000, > > > > PrettyPrint=true, PrettyPrintLineLength=80, PrintParameters=true > > > > openjpa.ConnectionPassword: ****** > > > > openjpa.ConnectionProperties: MaxActive=100, MaxIdle=5, MinIdle=2, > > > > MaxWait=60000 > > > > openjpa.ConnectionRetainMode: 0 > > > > openjpa.ConnectionURL: ****** > > > > openjpa.ConnectionUserName: ***** > > > > openjpa.DataCache: true(CacheSize=5000, EnableStatistics=true) > > > > openjpa.DataCacheManager: default > > > > openjpa.DataCacheTimeout: -1 > > > > openjpa.DetachState: fgs(DetachedStateField=true) > > > > openjpa.DynamicDataStructs: false > > > > openjpa.DynamicEnhancementAgent: true > > > > openjpa.EntityManagerFactory: default > > > > openjpa.FetchBatchSize: -1 > > > > openjpa.FetchGroups: [Ljava.lang.String;@53077fc9 > > > > openjpa.FlushBeforeQueries: 0 > > > > openjpa.Id: g11.persistence > > > > openjpa.IgnoreChanges: false > > > > openjpa.InitializeEagerly: false > > > > openjpa.InstrumentationManager: default > > > > openjpa.InverseManager: true > > > > openjpa.LifecycleEventManager: validating > > > > openjpa.LockManager: mixed > > > > openjpa.Log: log4j > > > > openjpa.ManagedRuntime: auto > > > > openjpa.MaxFetchDepth: -1 > > > > openjpa.MetaDataFactory: *** truncated!! > > > > openjpa.MetaDataRepository: default > > > > openjpa.Multithreaded: false > > > > openjpa.NontransactionalRead: true > > > > openjpa.NontransactionalWrite: true > > > > openjpa.Optimistic: true > > > > openjpa.OrphanedKeyAction: log > > > > openjpa.ProxyManager: default > > > > openjpa.QueryCache: true(CacheSize=1000, SoftReferenceSize=100, > > > > EvictPolicy='timestamp') > > > > openjpa.QueryCompilationCache: all > > > > openjpa.ReadLockLevel: 10 > > > > openjpa.RefreshFromDataCache: false > > > > openjpa.RemoteCommitProvider: sjvm > > > > openjpa.RestoreState: 1 > > > > openjpa.RetainState: true > > > > openjpa.RetryClassRegistration: false > > > > openjpa.RuntimeUnenhancedClasses: 1 > > > > openjpa.SavepointManager: in-mem > > > > openjpa.Sequence: table > > > > openjpa.TransactionMode: false > > > > openjpa.WriteLockLevel: 20 > > > > openjpa.jdbc.DBDictionary: > > > > postgres(supportsNullTableForGetImportedKeys=false) > > > > openjpa.jdbc.DriverDataSource: auto > > > > openjpa.jdbc.EagerFetchMode: 2 > > > > openjpa.jdbc.FetchDirection: 1000 > > > > openjpa.jdbc.FinderCache: true > > > > openjpa.jdbc.IdentifierUtil: default > > > > openjpa.jdbc.LRSSize: 2 > > > > openjpa.jdbc.MappingDefaults: jpa > > > > openjpa.jdbc.QuerySQLCache: true(EnableStatistics=true) > > > > openjpa.jdbc.ResultSetType: 1003 > > > > openjpa.jdbc.SQLFactory: default > > > > openjpa.jdbc.SchemaFactory: native(ForeignKeys=true) > > > > openjpa.jdbc.Schemas: [Ljava.lang.String;@60cf710e > > > > openjpa.jdbc.SubclassFetchMode: 1 > > > > openjpa.jdbc.SynchronizeMappings: null > > > > openjpa.jdbc.TransactionIsolation: -1 > > > > openjpa.jdbc.UpdateManager: default > > > > > > > > > > > > > > > > > > > > > > > >  > > > > > > > > John > > > > > > > > ---- > > > > > > > > Who is General Failure, and why is he reading my hard disk? > > > > > > > > > > > > > > -- *Rick Curtis*
