Re: [URGENT] performance issues
Hi,
Setting a batchLimit to -1 indicates there is no limit. Normally, this
would not be good since you could run into out of resources (memory,
connections, locks, etc) type situation. So, I wouldn't suggest running
with batchLimit set to -1 for any database. Setting it to 0 turns it off.
By default, we set batchLimit to 100 for DB2 and Oracle. I can't remember
which database you are using, but maybe you could try running with a
batchLimit of 100. Depending on your application, running with statement
batching may help with performance.
Kevin
On Mon, Feb 2, 2009 at 3:58 AM, is_maximum mnr...@gmail.com wrote:
Hi Fay,
I've just found that all the problems have raised from 'batchLimit=-1'
if I set it to 0 everything work fine. now the weird optimistic exception
and that jvm error have gone.
So I think we can't think of this feature (batch statements). do you have
any idea?
Fay Wang wrote:
Hi,
Sometimes the OptimisticLockException is thrown because of the
SQLException during insert. Could you turn on the trace and send us the
log file?
property name=openjpa.Log
value=File=c:\\trace.log,DefaultLevel=TRACE/
Regards,
Fay
--- On Sun, 2/1/09, is_maximum mnr...@gmail.com wrote:
From: is_maximum mnr...@gmail.com
Subject: Re: [URGENT] performance issues
To: users@openjpa.apache.org
Date: Sunday, February 1, 2009, 10:33 PM
Hi Kevin,
Sorry for replying late, I was facing a rush of work.
Well we didn't change the OpenJPA version we still use
1.2.0 version and we
still experiencing the same damned
OptimisticLockException!!!
Since I have changed these configuration the system
responses weired.
I have an entity say Entry with a one-to-many relationship
with Category.
when a new entry is coming I persist them like this:
dao.save(entry);
for (Category c : entry.getCategories()){
//do some work on categories here
c.setEntry(entry);
dao.save(c);
}
this snippet code were working fine before but after
changing the batchLimit
to -1 it won't work at all and a fatal error will be
thrown:
JDWP exit error JVMTI_ERROR_WRONG_PHASE(112): on getting
class status
FATAL ERROR in native method: JDWP on getting class status,
jvmtiError=JVMTI_ERROR_WRONG_PHASE(112)
I don't know why this batch updating affect this simple
insert statements.
However I used OpenJPA 2.0 Snapshot to see if this error
has been cleared
but in that version still we have problem. By the way,
because we have used
the word 'entry' in our JPQL currently we decided
not to use version 2,0
because this is a reserved word in this version! You see
there are lots of
problems just right when we are going to close the
project!!!
Do you have any idea on removing optimistic locking
exception? because this
is a big deal right now and if we couldn't remove it,
that will kill us :)
thanks
Kevin Sutter wrote:
On Thu, Jan 29, 2009 at 2:20 PM, is_maximum
mnr...@gmail.com wrote:
I really don't know why after about 6 months
working with OpenJPA and
inserting near a million records we should face
this problem? what have
changed that cause optimistic locking to be
activated spontaneously!!!
Hmmm... Not sure. What has changed in the last six
months? That is,
what
version of OpenJPA did you start with and what version
are you currently
running with? As you move from release to release,
there are bound to be
changes. And, in some cases, the changes may actually
have closed some
holes that you were accidentally avoiding. That is,
maybe you were
lucky... :-)
We do have quite an extensive test bucket and we do
attempt to avoid any
type of regressions. But, our test bucket (like any
other test bucket) is
not exhaustive. You could be exercising an area of
code that just hasn't
been covered to the extent it should be. There are so
many variables with
databases, configurations, jvms, environmental,
openjpa, etc.
If you can help identify and isolate the changes that
caused your
concerns,
then maybe we can help out.
Thanks,
Kevin
Pinaki Poddar wrote:
Hi,
but using build-time enhancer. I tried
ant-runner for maven but was
unable to enhance my classes
if I don't give any class name I get
this message:
A few minutes of digging with classpath
and other beasts to enhance
at
build-time will save you a lot of pain and
gain performance. Enhancer
fails mostly with classpath issues and
availability of
META-INF/persistence.xml etc.
If you have access to OpenJPA source code,
follow the trail at
./openjpa-persistence-jdbc/pom.xml and
./openjpa-persistence-jdbc/src/main/ant/enhancer.xml
It shows one way to enhance 100's of
Entity classes used for testing.
--
View this message
Re: [URGENT] performance issues
Hi Kevin,
Sorry for replying late, I was facing a rush of work.
Well we didn't change the OpenJPA version we still use 1.2.0 version and we
still experiencing the same damned OptimisticLockException!!!
Since I have changed these configuration the system responses weired.
I have an entity say Entry with a one-to-many relationship with Category.
when a new entry is coming I persist them like this:
dao.save(entry);
for (Category c : entry.getCategories()){
//do some work on categories here
c.setEntry(entry);
dao.save(c);
}
this snippet code were working fine before but after changing the batchLimit
to -1 it won't work at all and a fatal error will be thrown:
JDWP exit error JVMTI_ERROR_WRONG_PHASE(112): on getting class status
FATAL ERROR in native method: JDWP on getting class status,
jvmtiError=JVMTI_ERROR_WRONG_PHASE(112)
I don't know why this batch updating affect this simple insert statements.
However I used OpenJPA 2.0 Snapshot to see if this error has been cleared
but in that version still we have problem. By the way, because we have used
the word 'entry' in our JPQL currently we decided not to use version 2,0
because this is a reserved word in this version! You see there are lots of
problems just right when we are going to close the project!!!
Do you have any idea on removing optimistic locking exception? because this
is a big deal right now and if we couldn't remove it, that will kill us :)
thanks
Kevin Sutter wrote:
On Thu, Jan 29, 2009 at 2:20 PM, is_maximum mnr...@gmail.com wrote:
I really don't know why after about 6 months working with OpenJPA and
inserting near a million records we should face this problem? what have
changed that cause optimistic locking to be activated spontaneously!!!
Hmmm... Not sure. What has changed in the last six months? That is,
what
version of OpenJPA did you start with and what version are you currently
running with? As you move from release to release, there are bound to be
changes. And, in some cases, the changes may actually have closed some
holes that you were accidentally avoiding. That is, maybe you were
lucky... :-)
We do have quite an extensive test bucket and we do attempt to avoid any
type of regressions. But, our test bucket (like any other test bucket) is
not exhaustive. You could be exercising an area of code that just hasn't
been covered to the extent it should be. There are so many variables with
databases, configurations, jvms, environmental, openjpa, etc.
If you can help identify and isolate the changes that caused your
concerns,
then maybe we can help out.
Thanks,
Kevin
Pinaki Poddar wrote:
Hi,
but using build-time enhancer. I tried ant-runner for maven but was
unable to enhance my classes
if I don't give any class name I get this message:
A few minutes of digging with classpath and other beasts to enhance
at
build-time will save you a lot of pain and gain performance. Enhancer
fails mostly with classpath issues and availability of
META-INF/persistence.xml etc.
If you have access to OpenJPA source code, follow the trail at
./openjpa-persistence-jdbc/pom.xml and
./openjpa-persistence-jdbc/src/main/ant/enhancer.xml
It shows one way to enhance 100's of Entity classes used for testing.
--
View this message in context:
http://n2.nabble.com/-URGENT--performance-issues-tp2232295p2241065.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
--
View this message in context:
http://n2.nabble.com/-URGENT--performance-issues-tp2232295p2256731.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: [URGENT] performance issues
On Thu, Jan 29, 2009 at 2:20 PM, is_maximum mnr...@gmail.com wrote: I really don't know why after about 6 months working with OpenJPA and inserting near a million records we should face this problem? what have changed that cause optimistic locking to be activated spontaneously!!! Hmmm... Not sure. What has changed in the last six months? That is, what version of OpenJPA did you start with and what version are you currently running with? As you move from release to release, there are bound to be changes. And, in some cases, the changes may actually have closed some holes that you were accidentally avoiding. That is, maybe you were lucky... :-) We do have quite an extensive test bucket and we do attempt to avoid any type of regressions. But, our test bucket (like any other test bucket) is not exhaustive. You could be exercising an area of code that just hasn't been covered to the extent it should be. There are so many variables with databases, configurations, jvms, environmental, openjpa, etc. If you can help identify and isolate the changes that caused your concerns, then maybe we can help out. Thanks, Kevin Pinaki Poddar wrote: Hi, but using build-time enhancer. I tried ant-runner for maven but was unable to enhance my classes if I don't give any class name I get this message: A few minutes of digging with classpath and other beasts to enhance at build-time will save you a lot of pain and gain performance. Enhancer fails mostly with classpath issues and availability of META-INF/persistence.xml etc. If you have access to OpenJPA source code, follow the trail at ./openjpa-persistence-jdbc/pom.xml and ./openjpa-persistence-jdbc/src/main/ant/enhancer.xml It shows one way to enhance 100's of Entity classes used for testing. -- View this message in context: http://n2.nabble.com/-URGENT--performance-issues-tp2232295p2241065.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: [URGENT] performance issues
Pinaki Poddar on 29/01/09 05:09, wrote: 2. Build-time enhancement i.e. to execute one command after compilation performs best and minimizes complexity of load-time-weaving. I have never understood what really makes people to avoid 'build-time enhancement' - especially when most of the apps are built, packaged and deployed via Ant/Maven. This was one of the things that made EJB2.x so unpopular. Obviously it's not a big deal, but it's one extra step in the build cycle. Talking about Maven, I couldn't actually find a current approach that worked for this. Do you know of one? Adam
Re: [URGENT] performance issues
Hi Pinaki,
Well I am going to disable the locking and see the result if it works better
and I'll feedback the result.
but using build-time enhancer. I tried ant-runner for maven but was unable
to enhance my classes
if I don't give any class name I get this message:
openjpa.Tool - No targets were given. Running on all classes in your
persistent classes List, or all metadata files in classpath directories if
you have not listed your persistent classes.
and if I give the class name it complains:
[java] Exception in thread main
org.apache.commons.lang.exception.NestableRuntimeException: E
rror extracting class information from com.foo.data.Account.class.
[java] at
org.apache.openjpa.lib.meta.ClassArgParser.parseTypeNames(ClassArgParser.java:223
)
[java] at
org.apache.openjpa.lib.meta.ClassArgParser.parseTypes(ClassArgParser.java:161)
[java] at
org.apache.openjpa.enhance.PCEnhancer.run(PCEnhancer.java:4476)
[java] at
org.apache.openjpa.enhance.PCEnhancer.run(PCEnhancer.java:4439)
[java] at
org.apache.openjpa.enhance.PCEnhancer$1.run(PCEnhancer.java:4409)
[java] at
org.apache.openjpa.lib.conf.Configurations.launchRunnable(Configurations.java:708
)
[java] at
org.apache.openjpa.lib.conf.Configurations.runAgainstAllAnchors(Configurations.ja
va:698)
[java] at
org.apache.openjpa.enhance.PCEnhancer.run(PCEnhancer.java:4404)
[java] at
org.apache.openjpa.enhance.PCEnhancer.main(PCEnhancer.java:4395)
[java] Caused by: java.io.FileNotFoundException:
com.kishware.core.gl.data.Account.class (The s
ystem cannot find the file specified)
[java] at java.io.FileInputStream.open(Native Method)
[java] at java.io.FileInputStream.init(FileInputStream.java:106)
[java] at
org.apache.openjpa.lib.util.J2DoPrivHelper$28.run(J2DoPrivHelper.java:630)
[java] at java.security.AccessController.doPrivileged(Native
Method)
[java] at
org.apache.openjpa.lib.meta.ClassArgParser.getFromClassFile(ClassArgParser.java:3
03)
[java] at
org.apache.openjpa.lib.meta.ClassArgParser.parseTypeNames(ClassArgParser.java:214
)
[java] ... 8 more
[java] Java Result: 1
here is the snippet I am used in pom.xml:
plugin
artifactIdmaven-antrun-plugin/artifactId
executions
execution
phaseprocess-classes/phase
configuration
tasks
property
name=compile_classpath
refid=maven.compile.classpath/
property
name=runtime_classpath
refid=maven.runtime.classpath/
property name=test_classpath
refid=maven.test.classpath/
property
name=plugin_classpath
refid=maven.plugin.classpath/
!-- echo message=compile
classpath:
${compile_classpath}/
echo message=runtime
classpath:
${runtime_classpath}/
echo message=test
classpath:
${test_classpath}/
echo message=plugin
classpath:
${plugin_classpath}/--
fileset id=enhance.path.ref
dir=.
include name=target/test-classes/**/*.class /
include
name=target/classes/**/*.class/
/fileset
property
name=enhance_classpath
refid=maven.plugin.classpath/
echo message=Enhancing
classes /
java
classname=org.apache.openjpa.enhance.PCEnhancer
classpath=${compile_classpath};${runtime_classpath};src/main/java;target/classes;E:/my.applications/my.eclipse.workspace/general-ledger/modules/general-ledger-da/src/main/java/resources
dir=target/classes fork=true
arg
value=com.foo.data.Account.class /
arg value=-p
META-INF/persistence.xml /
arg
value=-adc true /
arg
value=-epr true /
/java
Re: [URGENT] performance issues
Hi, but using build-time enhancer. I tried ant-runner for maven but was unable to enhance my classes if I don't give any class name I get this message: A few minutes of digging with classpath and other beasts to enhance at build-time will save you a lot of pain and gain performance. Enhancer fails mostly with classpath issues and availability of META-INF/persistence.xml etc. If you have access to OpenJPA source code, follow the trail at ./openjpa-persistence-jdbc/pom.xml and ./openjpa-persistence-jdbc/src/main/ant/enhancer.xml It shows one way to enhance 100's of Entity classes used for testing. -- View this message in context: http://n2.nabble.com/-URGENT--performance-issues-tp2232295p2239148.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: [URGENT] performance issues
Thanks Pinaki, I was really stuck and abandoned any hope but your useful hints on classpath issues saved me. but today we had another weird day!!! First I create version field for some tables and put @Version on them and awfully everything got worst. even a simple insert with different account (different account number in DB) encountered optimistic locking exception!!! I am wondered why this never happened before. Then I tried to disable optimistic locking and set the LockManager to 'pessimistic' but nothing has changed and we still experience the same exception 'OptimisticLockException'. the problem remained even I set that value to 'none'. After hours of toiling I gain nothing and gave up. I really don't know why after about 6 months working with OpenJPA and inserting near a million records we should face this problem? what have changed that cause optimistic locking to be activated spontaneously!!! Pinaki Poddar wrote: Hi, but using build-time enhancer. I tried ant-runner for maven but was unable to enhance my classes if I don't give any class name I get this message: A few minutes of digging with classpath and other beasts to enhance at build-time will save you a lot of pain and gain performance. Enhancer fails mostly with classpath issues and availability of META-INF/persistence.xml etc. If you have access to OpenJPA source code, follow the trail at ./openjpa-persistence-jdbc/pom.xml and ./openjpa-persistence-jdbc/src/main/ant/enhancer.xml It shows one way to enhance 100's of Entity classes used for testing. -- View this message in context: http://n2.nabble.com/-URGENT--performance-issues-tp2232295p2241065.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: [URGENT] performance issues
Hi Kevin Thanks for your reply How are you invoking the enhancer? First I tried to set the javaagent in my App server but there was a NoClassDefFoundError so I leave it out and in OpenJPA manual I found that the enhancer in Java EE 5 will be automatically enabled. Thus actually I am relying on App server. and container-managed persistence The methods that you mention below are very minor, from a processing viewpoint. each call is minor but when I see the JProbe reports it shows that a single save() method in which we used em.persist() have taken say 20 millis and almost 60 percent of this time is taken cumulatively by methods used by enhancer rather than the call to persist() method. Are your Entities versioned? No. Actually we don't use optimistic locking purposely because the number of update to the records are really high and also the number of requests is really high as well. Let me tell you this is a banking system and this business method we are testing is one of the critical methods that supposed to have high performance. By optimistic locking, all other requests will face problem and we need to ask the users to retry their request and neither the users nor the bank managers will accept this. Are you using Lazy or Eager fetching of attributes and relationships? well generally we are using lazy relationships but in this specific case, we don't need any eager fetching and the entities are single fortunately. I see in your persistence.xml that you have turned off statement batching for your Oracle database Well I didn't get attention to it and I'll ask my colleagues in the team if they had any reason to turn it off. And, turning on the data cache should also help, not cause you to go backwards. You do mention that you are using the 1.2.0 release. Have you tried moving to a later revision in the 1.2.x service stream? Or, even the 1.3.x branch? Just to help narrow down whether something has already been discovered and resolved for your environment. No I didn't try later versions and yes I have to test it But regarding the data cache, I am wondered why using that we got lower performance!!! We set the data cache with a size of 2000 and soft cache enabled with a size of 100. However I am a bit scared of getting unusual results using data cache due to frequent update we make into the tables. finally I appreciate your reply and trying to help me. thanks Kevin Sutter wrote: Hi is_maximum, The results you are experiencing are atypical. Normally, when the Entity classes are enhanced, OpenJPA's performance is one of or the best, depending on the specific benchmark. Let's start with a few questions to see if we can narrow in on the problem. o First off, you mention the enhancer. How are you invoking the enhancer? Are you statically pre-enhancing the Entity classes? Or, are you using the -javaagent jvm parameter? Or, are you relying on the Container classloader integration? I see that you are using the Sun Application Server, but are you relying on the container-managed persistence or are you using application-managed persistence? o The methods that you mention below are very minor, from a processing viewpoint. So, I really don't think the slow-down is due to enhancement. o Are your Entities versioned? That is, are you using an @Version field in your Entities? Due to the optimistic locking that JPA does by default, determining whether a given Entity is different from the database contents is much, much more efficient if we can just check the @Version field instead of comparing all of the individual fields. o Are you using Lazy or Eager fetching of attributes and relationships? Depending on the type of the attribute and/or relationship, the default fetch type may be different. For example, if you have a relationship defined as having a Lazy fetch type, but you are constantly accessed the contents of that relationship, then instead of waiting until you touch that relationship to get the contents from the database (lazy), you should probably change that to eager to allow us to get the data on the first trip to the database. The default values for the fetch type are meant to fit the 80/20 rule, but depending on your application, they may not fit your needs. o I see in your persistence.xml that you have turned off statement batching for your Oracle database. That's kind of surprising since under normal circumstances, statement batching can only help your performance concerns. Unless you are hitting some condition that forced you to turn it off? o And, turning on the data cache should also help, not cause you to go backwards. You do mention that you are using the 1.2.0 release. Have you tried moving to a later revision in the 1.2.x service stream? Or, even the 1.3.x branch? Just to help narrow down whether something has already been discovered and resolved for your environment. Let's start with that and see where
Re: [URGENT] performance issues
More comments inline... On Wed, Jan 28, 2009 at 12:13 PM, is_maximum mnr...@gmail.com wrote: Hi Kevin Thanks for your reply How are you invoking the enhancer? First I tried to set the javaagent in my App server but there was a NoClassDefFoundError so I leave it out and in OpenJPA manual I found that the enhancer in Java EE 5 will be automatically enabled. Thus actually I am relying on App server. and container-managed persistence Right. If you are using Container managed persistence, then allowing the hook between the app server and openjpa is the easiest way to go. I have not confirmed that the Sun application server is doing the proper classloader integration, but let's assume they are for the time being. I suppose if you wanted to, you could turn on the OpenJPA trace and verify that the Entity classes are being enhanced during load. The methods that you mention below are very minor, from a processing viewpoint. each call is minor but when I see the JProbe reports it shows that a single save() method in which we used em.persist() have taken say 20 millis and almost 60 percent of this time is taken cumulatively by methods used by enhancer rather than the call to persist() method. Agree. Every little thing adds up when doing performance evaluations. It's just none of the analysis we have ever done within WebSphere has shown a problem in these persistence capable inserted methods. Nada. Are your Entities versioned? No. Actually we don't use optimistic locking purposely because the number of update to the records are really high and also the number of requests is really high as well. Let me tell you this is a banking system and this business method we are testing is one of the critical methods that supposed to have high performance. By optimistic locking, all other requests will face problem and we need to ask the users to retry their request and neither the users nor the bank managers will accept this. This is interesting. Not sure what you mean that you don't use optimistic locking. JPA use optimistic locking inheritantly. The only thing you can do is override it with pessimistic locking. But, you haven't indicated that you are using OpenJPA's pessimistic lock manager. And, the JPA spec itself doesn't even define pessimistic locking. So, if you are not using Versioned Entities, then you are paying a huge overhead because we are still required to check for optimistic collisions. The only way to do that without the Version field is to read in all of the fields and see if any changes have been done. From the discussion thus far, this would seem to be your area of concern. As a test, you could turn on the SQL trace and see what kind of interactions are happening with the database. Based on the discussion thus far, I would think you are going to see a ton of wasted SQL interactions. Are you using Lazy or Eager fetching of attributes and relationships? well generally we are using lazy relationships but in this specific case, we don't need any eager fetching and the entities are single fortunately. I see in your persistence.xml that you have turned off statement batching for your Oracle database Well I didn't get attention to it and I'll ask my colleagues in the team if they had any reason to turn it off. And, turning on the data cache should also help, not cause you to go backwards. You do mention that you are using the 1.2.0 release. Have you tried moving to a later revision in the 1.2.x service stream? Or, even the 1.3.x branch? Just to help narrow down whether something has already been discovered and resolved for your environment. No I didn't try later versions and yes I have to test it But regarding the data cache, I am wondered why using that we got lower performance!!! We set the data cache with a size of 2000 and soft cache enabled with a size of 100. However I am a bit scared of getting unusual results using data cache due to frequent update we make into the tables. Right. Turning on the caches should not degrade performance. finally I appreciate your reply and trying to help me. thanks Kevin Sutter wrote: Hi is_maximum, The results you are experiencing are atypical. Normally, when the Entity classes are enhanced, OpenJPA's performance is one of or the best, depending on the specific benchmark. Let's start with a few questions to see if we can narrow in on the problem. o First off, you mention the enhancer. How are you invoking the enhancer? Are you statically pre-enhancing the Entity classes? Or, are you using the -javaagent jvm parameter? Or, are you relying on the Container classloader integration? I see that you are using the Sun Application Server, but are you relying on the container-managed persistence or are you using application-managed persistence? o The methods that you mention below are very minor, from a processing viewpoint. So, I really
Re: [URGENT] performance issues
Hi, 1. If locking is less of a priority then performance, then use property name=openjpa.LockManager value=none/ By default, OpenJPA goes for optimistic lock mode. So as Kevin pointed out, without @Version and default Optimistic lock mode -- many wasted SQL to compare state for version checking. However, one should be aware of data integrity risks in such cases. Without any locking and no version field, OpenJPA essentially will follow a last-write-wins straegy. But as updates happen at per-field basis, something like this can happen Transaction 1: Reads a Person P whose first name is 'John' and last name is 'Doe' Transaction 2: Reads the same Person Transaction 1: Modifies first name to 'Harry' and commits Transaction 2: Modified last name to 'Smith' and commits. Both commit will succeed and database will have 'Harry Smith'. 2. Build-time enhancement i.e. to execute one command after compilation performs best and minimizes complexity of load-time-weaving. I have never understood what really makes people to avoid 'build-time enhancement' - especially when most of the apps are built, packaged and deployed via Ant/Maven. is_maximum wrote: Hi Kevin, Well now there are some questions here, first, consider the App Server dosn't enable enhancer how can I tell my App server to activate it? Do I have to use the same javaagent property? second, We just neglected the version field and I didn't know the OpenJPA will use optimistic locking anyway. Is there any way to disable this kind of locking? At the time we call the merge() method I can see the row in target table in Oracle is being locked so this means that we are using pessimistic locking, Am I right? One more question is that if we are using optimistic locking how to handle it? if a request changes an entity say Account, and at the same time there are say 10 requests that are modifying the same account, the first request will commit its changes and all other 10 requests will fail because their fetched entity is no longer valid. As I know using optimistic locking is not a good option for the entities with very frequent update actions. -- View this message in context: http://n2.nabble.com/-URGENT--performance-issues-tp2232295p2237146.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
