Just as an update on this; I've run Cesar's JMeter tests, and it has revealed an issue in Isis core ... my suspicion being a possible synchronized deadlock. [1]
Will investigate with a view to fixing in 1.13.0 Thx Dan [1] https://issues.apache.org/jira/browse/ISIS-1421 On 2 June 2016 at 16:50, César Camilo Lugo Marcos <[email protected]> wrote: > Hello. > > We are looking into several recommended topics to tune up the > application. One of them is the use of JDO Optimistic locking instead > of the Datanucleus default pessimistic locking, to improve concurrency. > I have not found a setting within Datanucleus to change this setting > globally or per transaction, just found this to define the property to > be used as version on the domain object: > > @PersistenceCapable > @Version(strategy=VersionStrategy.VERSION_NUMBER) > public class MyClass > { > ... > } > > Do you know about how to set it globally? > > > On Tue, 2016-05-31 at 18:54 +0000, César Camilo Lugo Marcos wrote: > > Hello Dan. Answers to your questions: > > > > - which version of Isis are you on? > > 1.12.0 > > - how have you implemented the stress test? is it automated, eg JMeter, > or just adhoc manual testing? > > Using JMeter. We recorded a quite simple script with about 6 read > only operations using th Wicket viewer. All samplers are the recorded HTTP. > > - is this stress testing the app running locally on Tomcat, or up in the > cloud (on Amazon Elastic Beanstalk)? > > We have it installed in the AWS Elastic Beanstalk with PostgreSQL. > > We also made a test on a local host with Tomcat and MySQL, obtaining > very similar results. > > > - how much DB traffic does a single request create? Is it all > read-only; or are there also writes? > > Quite low traffic, About 6 read only operations per user. > > - for reads, are you seeing any N+1 issues? if so, have you tried to > prevent them using DataNucleus eager loading hint (the "fetch groups" > concept)? > > Not sure what N+1 is. We are letting Datanucleous handle all foreign > keys and primary keys. We have not configured any fetch groups or any other > configurations than defaults. > > - for writes, are they expected? > > Did not understand this question. Please explain. > > - have you configured any of the various addons that could be writing to > the DB, eg auditing, command or publishing? are these perhaps enabled for > safe/query-only actions? If so for any looping within the app itself, can > you eliminate using the QueryResultsCache? > > We are only using the logging add-on, to log login and logout > operations. No auditing, command or publishing add ons. > > - what data volumes is this on? If large volumes of data, are there > perhaps missing indices? What's the performance like with small volumes of > data? > > No Data Object or table has more that a few tens of records. > > - is this for in-memory HSQLDB or out-of-memory DB (eg Postgres)? > What's the performance like of one versus the other > > This is PostgreSQL. Also tried MySQL with very similar results. Have > not tried in memory HSQLDB. > > > > > > Cesar. > > > > > > On Tue, 2016-05-31 at 18:53 +0100, Dan Haywood wrote: > > > > > > Hi Cesar, > > > > Those action semantics don't play a large part in the transaction > > management; they are mostly used to determine which type of HTTP method > to > > expose the action under (GET, PUT or POST) and there is also the > constraint > > that contributed/mixin properties/collections can only be supplied by > SAFE > > actions. The _ARE_YOU_SURE variants only impact the Wicket viewer. The > > _REQUEST_CACHEABLE is only honoured if the action is invoked via the > > wrapper factory. > > > > As Martin and Jeroen have said, using JProfiler or similar will likely > > yield valuable info. > > > > If the issue is with database concurrency (which does seem like a good > > hypothesis), then for the most part that is under the management of > > DataNucleus itself. There are a bunch of configuration properties that > can > > be specified in persistor_datanucleus.properties (or isis.properties, > > actually); we strip off any "isis.persistor.datanucleus.impl" prefix and > > pass the rest of the key down to DN to configure. > > > > Questions for you: > > - which version of Isis are you on? > > - how have you implemented the stress test? is it automated, eg JMeter, > or > > just adhoc manual testing? > > - is this stress testing the app running locally on Tomcat, or up in the > > cloud (on Amazon Elastic Beanstalk)? > > - how much DB traffic does a single request create? Is it all read-only; > > or are there also writes? > > - for reads, are you seeing any N+1 issues? if so, have you tried to > > prevent them using DataNucleus eager loading hint (the "fetch groups" > > concept)? > > - for writes, are they expected? > > - have you configured any of the various addons that could be writing to > > the DB, eg auditing, command or publishing? are these perhaps enabled > for > > safe/query-only actions? If so > > - for any looping within the app itself, can you eliminate using the > > QueryResultsCache? > > - what data volumes is this on? If large volumes of data, are there > > perhaps missing indices? What's the performance like with small volumes > of > > data? > > - is this for in-memory HSQLDB or out-of-memory DB (eg Postgres)? What's > > the performance like of one versus the other > > > > I think what I would look for is fast response times for a 1-user system > > running locally on Tomcat, with an inmemory database, and low data > > volumes. From there, ramp up in different ways, but change one thing at > a > > time. > > > > HTH > > Dan > > > > > > > > On 31 May 2016 at 15:12, César Camilo Lugo Marcos < > [email protected]<mailto:[email protected]>> > > wrote: > > > > > Thank you both for the hint. I am looking into JProfiler. Still, > > > anything about the concurrency control options? > > > > > > Cesar. > > > > > > On Tue, 2016-05-31 at 11:13 +0200, Jeroen van der Wal wrote: > > > > I agree with Martin that profiling is the only way to go. To > illustrate: > > > we > > > > recently made some code 8 times faster by a few simple code changes > on > > > > bottlenecks revealed by JProfiler. And those were in places that > we've > > > > never guessed. > > > > > > > > On 31 May 2016 at 08:39, Martin Grigorov <[email protected] > <mailto:[email protected]>> wrote: > > > > > > > > > Hi, > > > > > > > > > > To find out where is the problem you will have to use a profiler > like > > > > > JProfiler, Yourkit, JVisualVM, etc. > > > > > Even some thread dumps would help to see what is going on. > > > > > > > > > > Martin Grigorov > > > > > Wicket Training and Consulting > > > > > https://twitter.com/mtgrigorov > > > > > > > > > > On Mon, May 30, 2016 at 9:00 PM, César Camilo Lugo Marcos < > > > > > [email protected]<mailto:[email protected]>> wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > I would like to add to this topic the following: > > > > > > > > > > > > Most of the transactions we are testing in these stress tests > are not > > > > > > bound in ACTIONS, but just plain reads or default transactions > using > > > > > > Apache ISIS wicket viewer defaults. I don't see any place where I > > > could > > > > > > define semantics for default read or write transactions not bound > > > into > > > > > > ACTION methods. Is there any place I should look into for that? > > > > > > > > > > > > Cesar. > > > > > > > > > > > > > > > > > > On Mon, 2016-05-30 at 18:45 +0000, César Camilo Lugo Marcos > wrote: > > > > > > > Hello, > > > > > > > > > > > > > > We have sarted performing some stress tests to our Apache ISIS > > > > > > > application. We have found this behavior: > > > > > > > > > > > > > > - If we run 1 concurrent user, average response times to the > main > > > > > object > > > > > > > reads through the wicket viewer are from 1 to 1.5 seconds. > > > > > > > - If we run 2 concurrent users, same transactions go to average > > > > > response > > > > > > > times up to 16 to 27 seconds. > > > > > > > - If we run 10 concurrent users, the transactions start to slow > > > down > > > > > > > significantly until the app server freezes and we have to > restart > > > it. > > > > > > > > > > > > > > As you can see, this is a very significant increase in response > > > time > > > > > for > > > > > > > such a slight change in user load (from 1 user to 2 users). So > we > > > are > > > > > > > guessing we should look into the concurrency control. > > > > > > > > > > > > > > In the documentation I have found that the only way to > influence > > > the > > > > > way > > > > > > > Apache ISIS manages transactions and concurrency checking is by > > > using > > > > > > > the semantics configuration of the ACTION annotation. > > > > > > > > > > > > > > semantics=SAFE_AND_REQUEST_CACHEABLE > > > > > > > semantics=SAFE > > > > > > > semantics=IDEMPOTENT > > > > > > > semantics=IDEMPOTENT_ARE_YOU_SURE > > > > > > > semantics=NON_IDEMPOTENT > > > > > > > semantics=NON_IDEMPOTENT_ARE_YOU_SURE > > > > > > > > > > > > > > I just wanted to confirm if this is the place to look into, or > are > > > > > there > > > > > > > any other places where we should be looking into too, to solve > the > > > > > > > performance issue. > > > > > > > > > > > > > > Cesar. > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
