Hi again.
Found the solution to my problem.
Adding a QueryCache seems to have helped us listing entities in different
threads.
The Eviction with timestamp is crusial for performance and seems to work
well for us.
<property name="openjpa.QueryCache" value="true(CacheSize=1000,
SoftReferenceSize=0, EvictPolicy='timestamp')"/>
The last problem we have is in the high intensive Entities that we write
every request.
In some rare instances 2/100000 request this entity is written to database
using INSERT instead of UPDATE when committing a merge. This will trigger a
duplicate primary key. Could be something architectural so probably not
something for this forum.
Thanks for all your help so far.
Best regards
Daniel
On Wed, May 9, 2012 at 8:22 PM, Daniel Persson <mailto.wo...@gmail.com>wrote:
> Hi again.
>
> Just wanted to add that we have reduced the settings to this
>
>
> <property name="openjpa.DataCache" value="true(CacheSize=1000,
> SoftReferenceSize=0)"/>
> <property name="openjpa.QueryCache" value="false"/>
> <property name="openjpa.Log" value="DefaultLevel=INFO, Runtime=INFO,
> Tool=INFO, SQL=INFO"/>
>
> <property name="openjpa.DetachState"
> value="fetch-groups(DetachedStateField=true)"/>
>
>
>
> Because we have alot of threads doing database calls we have one function
> to handle the fetching of an EntityManager. Threads are both coming from
> Servlet calls and stateful worker threads so we need some sort of
> separation so we don't get transaction locks.
> We solved it with ThreadLocal HashTable with the managers and an static
> Hashtable for the factories. This should give us one factory per database
> and one entity manager per thread. And because we don't have persistent
> data connections we have to check if they are open as well when getting the
> connection.
>
>
> private static transient ThreadLocal<Hashtable<String, EntityManager>>
> entityManagers = new ThreadLocal<Hashtable<String, EntityManager>>();
> private static Hashtable<String, EntityManagerFactory> factories = new
> Hashtable<String, EntityManagerFactory>();
>
> public static EntityManager getEntityManager(String entityName) throws
> Exception {
> if(entityManagers.get() == null) {
> entityManagers.set(new Hashtable<String, EntityManager>());
> }
>
> EntityManager entityManager = entityManagers.get().get(entityName);
>
> if(entityManager != null) {
> OpenJPAEntityManager oem = OpenJPAPersistence.cast(entityManager);
> Connection con = (Connection) oem.getConnection();
> if(con.isClosed()) {
> entityManager = null;
> }
> }
>
> if(entityManager == null || !entityManager.isOpen()) {
> if(factories.get(entityName) == null) {
> factories.put(entityName,
> Persistence.createEntityManagerFactory(entityName, System.getProperties()));
> }
>
> entityManager =
> factories.get(entityName).createEntityManager()
> entityManagers.get().put(entityName, entityManager);
> }
> return entityManager;
> }
>
> We work hard on finding a solution and getting a stable system so any
> assistance is very appreciated.
>
> Best regards
>
> Daniel
>
>
>
> On Wed, May 9, 2012 at 3:44 PM, Daniel Persson <mailto.wo...@gmail.com>wrote:
>
>> Hi Rick.
>>
>> At the moment we have a problem with EntityManagers not being in sync. We
>> have an application where you could update user data. And when we reload
>> the page with the user data listed. These are elements connected to the
>> user data and are them self another entity entirely and we fetch them with
>> "select e from entity e where e.userid = :gaUserId".
>> When removing or adding an entity we could reload the page and depending
>> on which entitymanager deliver the result we see the added or removed
>> entities. So the result set size could differ from 1,2,3,4 with 4 reloads
>> of the web browser. So we need consistancy between EntityManagers.
>>
>> The data saved to database is in sync but the managers in each thread
>> have different result sets saved.
>>
>> I haven't been able to reproduce the problem with TRACE logging. The
>> error occurs from time to time in my simple example but it occurred when I
>> tested without trace.
>> I found another error though with trace log.
>>
>> Caused by: <openjpa-2.1.1-r422266:1148538 nonfatal general error>
>> org.apache.openjpa.persistence.PersistenceException: index = 51
>>
>> at
>> org.apache.openjpa.kernel.BrokerImpl.afterCompletion(BrokerImpl.java:2018)
>> at
>> org.apache.openjpa.kernel.LocalManagedRuntime.commit(LocalManagedRuntime.java:94)
>> at org.apache.openjpa.kernel.BrokerImpl.commit(BrokerImpl.java:1498)
>> at
>> org.apache.openjpa.kernel.DelegatingBroker.commit(DelegatingBroker.java:933)
>> at
>> org.apache.openjpa.persistence.EntityManagerImpl.commit(EntityManagerImpl.java:569)
>> ... 1 more
>> Caused by: java.lang.IndexOutOfBoundsException: index = 51
>> at serp.bytecode.lowlevel.ConstantPool.getEntry(ConstantPool.java:52)
>> at
>> serp.bytecode.lowlevel.ComplexEntry.getNameAndTypeEntry(ComplexEntry.java:85)
>> at
>> serp.bytecode.FieldInstruction.getFieldTypeName(FieldInstruction.java:220)
>> at
>> serp.bytecode.GetFieldInstruction.getStackChange(GetFieldInstruction.java:22)
>> at serp.bytecode.Code.calculateMaxStack(Code.java:214)
>> at
>> org.apache.openjpa.enhance.DynamicStorageGenerator.addSetMethod(DynamicStorageGenerator.java:342)
>> at
>> org.apache.openjpa.enhance.DynamicStorageGenerator.addSetMethods(DynamicStorageGenerator.java:272)
>> at
>> org.apache.openjpa.enhance.DynamicStorageGenerator.generateStorage(DynamicStorageGenerator.java:121)
>> at
>> org.apache.openjpa.enhance.PCDataGenerator.generateStorage(PCDataGenerator.java:116)
>> at
>> org.apache.openjpa.enhance.PCDataGenerator.generatePCData(PCDataGenerator.java:93)
>> at
>> org.apache.openjpa.datacache.DataCacheStoreManager.newPCData(DataCacheStoreManager.java:790)
>> at
>> org.apache.openjpa.datacache.DataCacheStoreManager.updateCaches(DataCacheStoreManager.java:146)
>> at
>> org.apache.openjpa.datacache.DataCacheStoreManager.commit(DataCacheStoreManager.java:89)
>> at
>> org.apache.openjpa.kernel.DelegatingStoreManager.commit(DelegatingStoreManager.java:95)
>> at
>> org.apache.openjpa.kernel.BrokerImpl.endStoreManagerTransaction(BrokerImpl.java:1455)
>> at
>> org.apache.openjpa.kernel.BrokerImpl.endTransaction(BrokerImpl.java:2335)
>> at
>> org.apache.openjpa.kernel.BrokerImpl.afterCompletion(BrokerImpl.java:1994)
>> ... 5 more
>>
>> Best regards
>>
>> Daniel
>>
>>
>> On Tue, May 8, 2012 at 3:09 PM, Rick Curtis <curti...@gmail.com> wrote:
>>
>>> Daniel -
>>>
>>> Yes, I'd be interested in seeing the full exception. That being said,
>>> you're probably safe just turning off the DynamicDataStructs . In our
>>> performance testing, we never observed an improvement when that property
>>> is
>>> enabled.
>>>
>>> Since we're on the topic of properties, I'd also recommend getting rid of
>>> the openjpa.Multithreaded property. Sharing EntityManagers / Entities
>>> across multiple threads isn't something that OpenJPA was designed to do.
>>> openjpa.Multithreaded was supposed to address that concern, but it did so
>>> poorly. There are a number of outstanding JIRAs that document some of the
>>> issues... I can dig them up if you'd like?
>>>
>>> Thanks,
>>> Rick
>>>
>>> On Tue, May 8, 2012 at 1:43 AM, Daniel Persson <mailto.wo...@gmail.com
>>> >wrote:
>>>
>>> > Hi Rick.
>>> >
>>> > These are the actual stacktraces that you get with log level WARN. I
>>> > changed to TRACE and got more information that actually lead me to
>>> beleave
>>> > that the DynamicDataStructs where the problem. For some reason it
>>> could not
>>> > handle special types of data. Without DynamicDataStructs it seems to
>>> work
>>> > at the moment.
>>> >
>>> > Didn't save the stacktrace but I'm sure I got the null pointer
>>> exception on
>>> >
>>> > 365: BCMethod method = bc.declareMethod(name, type, new Class[]{
>>> int.class
>>> > });
>>> >
>>> > in DynamicStorageGenerator.java
>>> >
>>> > It didn't recognize the type supplied. If your interested I could turn
>>> the
>>> > feature on and get a trace for you.
>>> >
>>> > Best regards
>>> >
>>> > Daniel
>>> >
>>> > On Mon, May 7, 2012 at 3:59 PM, Rick Curtis <curti...@gmail.com>
>>> wrote:
>>> >
>>> > > Daniel -
>>> > >
>>> > > Can you post the entire stacktrace? You've snipped out the important
>>> > parts.
>>> > >
>>> > > Thanks,
>>> > > Rick
>>> > >
>>> > > On Mon, May 7, 2012 at 3:19 AM, Daniel Persson <
>>> mailto.wo...@gmail.com
>>> > > >wrote:
>>> > >
>>> > > > Hi.
>>> > > >
>>> > > > I'd used OpenJPA for my projects for a while now and started to
>>> use it
>>> > at
>>> > > > work.
>>> > > >
>>> > > > Our service runs on Tomcat and have a lot of users. So after
>>> > implementing
>>> > > > OpenJPA for a few months we are now in the testing phase.
>>> > > > During the stress test adding a few hundred records using different
>>> > > tomcat
>>> > > > threads I get the same OpenJPA error over and over.
>>> > > >
>>> > > > I've changed the configuration back and forth to resolve the issue.
>>> > > > Sometimes I get it to work for a while and then it breaks again.
>>> > > > So I would like to ask what the error could be a result of and if I
>>> > have
>>> > > > any obvious configuration faults.
>>> > > >
>>> > > > I've written a small test class adding entities using 1000 threads
>>> and
>>> > > this
>>> > > > will generate the errors from time to time.
>>> > > > The code uses one factory and a lot of managers.
>>> > > >
>>> > > > <openjpa-2.1.1-r422266:1148538 fatal store error>
>>> > > > org.apache.openjpa.persistence.RollbackException: null
>>> > > > at
>>> > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.openjpa.persistence.EntityManagerImpl.commit(EntityManagerImpl.java:593)
>>> > > >
>>> > > > Caused by: <openjpa-2.1.1-r422266:1148538 nonfatal general error>
>>> > > > org.apache.openjpa.persistence.PersistenceException: null
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.openjpa.kernel.BrokerImpl.afterCompletion(BrokerImpl.java:2018)
>>> > > > at
>>> > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.openjpa.kernel.LocalManagedRuntime.commit(LocalManagedRuntime.java:94)
>>> > > > at
>>> > > org.apache.openjpa.kernel.BrokerImpl.commit(BrokerImpl.java:1498)
>>> > > > at
>>> > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.openjpa.kernel.DelegatingBroker.commit(DelegatingBroker.java:933)
>>> > > > at
>>> > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.openjpa.persistence.EntityManagerImpl.commit(EntityManagerImpl.java:569)
>>> > > > ... 27 more
>>> > > > Caused by: java.lang.NullPointerException
>>> > > >
>>> > > >
>>> > > > <persistence xmlns="http://java.sun.com/xml/ns/persistence";
>>> > xmlns:xsi="
>>> > > > http://www.w3.org/2001/XMLSchema-instance"; version="1.0">
>>> > > > <persistence-unit name="userDbPersistence"
>>> > > > transaction-type="RESOURCE_LOCAL">
>>> > > >
>>> > > <non-jta-data-source>java:comp/env/jdbc/userDb</non-jta-data-source>
>>> > > >
>>> > > > <class>......</class>
>>> > > >
>>> > > > <properties>
>>> > > > <property name="openjpa.Multithreaded" value="true" />
>>> > > > <property name="openjpa.InverseManager" value="true" />
>>> > > > <property name="openjpa.DynamicDataStructs"
>>> value="true"/>
>>> > > > <property name="openjpa.Compatibility"
>>> > > >
>>> > > >
>>> > >
>>> >
>>> value="QuotedNumbersInQueries=true,CopyOnDetach=true,cascadeWithDetach=true,superclassDiscriminatorStrategyByDefault=false"
>>> > > > />
>>> > > > <property name="openjpa.DataCache"
>>> > value="true(CacheSize=1000,
>>> > > > SoftReferenceSize=0)"/>
>>> > > > <property name="openjpa.QueryCache" value="false"/>
>>> > > > <property name="openjpa.RemoteCommitProvider"
>>> value="sjvm"/>
>>> > > > <property name="openjpa.Log" value="DefaultLevel=WARN,
>>> > > > Runtime=WARN, Tool=WARN, SQL=WARN"/>
>>> > > > <property name="openjpa.FetchBatchSize" value="0"/>
>>> > > > <property name="openjpa.DetachState"
>>> > > > value="fetch-groups(DetachedStateField=true)"/>
>>> > > > <property name="openjpa.LockTimeout" value="3000"/>
>>> > > > <property name="openjpa.jdbc.EagerFetchMode"
>>> value="join" />
>>> > > > <property name="openjpa.jdbc.SubclassFetchMode"
>>> > > > value="parallel" />
>>> > > > </properties>
>>> > > > </persistence-unit>
>>> > > > </persistence>
>>> > > >
>>> > > > Any tips or suggestions are much appreciated. Have tried a lot of
>>> > > different
>>> > > > configurations. One requirement for the configuration as well is
>>> that
>>> > we
>>> > > > want to use some sort of caching because this was one of the
>>> reason for
>>> > > the
>>> > > > shift to openJPA.
>>> > > >
>>> > > > Best regards
>>> > > >
>>> > > > Daniel
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > *Rick Curtis*
>>> > >
>>> >
>>>
>>>
>>>
>>> --
>>> *Rick Curtis*
>>>
>>
>>
>
