A further update re [1]; I've now pushed a commit [2] which seems to have addressed Cesar's issue ... removing a synchronized modifier.
I'm going to do a review of other uses of 'synchronized', to see if they are justified or not [3]. Thx Dan [1] https://issues.apache.org/jira/browse/ISIS-1421 [2] https://github.com/apache/isis/commit/746176991b523fe9e8ab069392d83ff17af08624 [3] https://issues.apache.org/jira/browse/ISIS-1428 On 3 June 2016 at 13:28, Dan Haywood <[email protected]> wrote: > Just as an update on this; I've run Cesar's JMeter tests, and it has > revealed an issue in Isis core ... my suspicion being a possible > synchronized deadlock. [1] > > Will investigate with a view to fixing in 1.13.0 > > Thx > Dan > > > [1] https://issues.apache.org/jira/browse/ISIS-1421 > > On 2 June 2016 at 16:50, César Camilo Lugo Marcos < > [email protected]> wrote: > >> Hello. >> >> We are looking into several recommended topics to tune up the >> application. One of them is the use of JDO Optimistic locking instead >> of the Datanucleus default pessimistic locking, to improve concurrency. >> I have not found a setting within Datanucleus to change this setting >> globally or per transaction, just found this to define the property to >> be used as version on the domain object: >> >> @PersistenceCapable >> @Version(strategy=VersionStrategy.VERSION_NUMBER) >> public class MyClass >> { >> ... >> } >> >> Do you know about how to set it globally? >> >> >> On Tue, 2016-05-31 at 18:54 +0000, César Camilo Lugo Marcos wrote: >> > Hello Dan. Answers to your questions: >> > >> > - which version of Isis are you on? >> > 1.12.0 >> > - how have you implemented the stress test? is it automated, eg >> JMeter, or just adhoc manual testing? >> > Using JMeter. We recorded a quite simple script with about 6 read >> only operations using th Wicket viewer. All samplers are the recorded HTTP. >> > - is this stress testing the app running locally on Tomcat, or up in >> the cloud (on Amazon Elastic Beanstalk)? >> > We have it installed in the AWS Elastic Beanstalk with PostgreSQL. >> > We also made a test on a local host with Tomcat and MySQL, >> obtaining very similar results. >> > > - how much DB traffic does a single request create? Is it all >> read-only; or are there also writes? >> > Quite low traffic, About 6 read only operations per user. >> > - for reads, are you seeing any N+1 issues? if so, have you tried to >> prevent them using DataNucleus eager loading hint (the "fetch groups" >> concept)? >> > Not sure what N+1 is. We are letting Datanucleous handle all >> foreign keys and primary keys. We have not configured any fetch groups or >> any other configurations than defaults. >> > - for writes, are they expected? >> > Did not understand this question. Please explain. >> > - have you configured any of the various addons that could be writing >> to the DB, eg auditing, command or publishing? are these perhaps enabled >> for safe/query-only actions? If so for any looping within the app itself, >> can you eliminate using the QueryResultsCache? >> > We are only using the logging add-on, to log login and logout >> operations. No auditing, command or publishing add ons. >> > - what data volumes is this on? If large volumes of data, are there >> perhaps missing indices? What's the performance like with small volumes of >> data? >> > No Data Object or table has more that a few tens of records. >> > - is this for in-memory HSQLDB or out-of-memory DB (eg Postgres)? >> What's the performance like of one versus the other >> > This is PostgreSQL. Also tried MySQL with very similar results. >> Have not tried in memory HSQLDB. >> > >> > >> > Cesar. >> > >> > >> > On Tue, 2016-05-31 at 18:53 +0100, Dan Haywood wrote: >> > >> > >> > Hi Cesar, >> > >> > Those action semantics don't play a large part in the transaction >> > management; they are mostly used to determine which type of HTTP method >> to >> > expose the action under (GET, PUT or POST) and there is also the >> constraint >> > that contributed/mixin properties/collections can only be supplied by >> SAFE >> > actions. The _ARE_YOU_SURE variants only impact the Wicket viewer. The >> > _REQUEST_CACHEABLE is only honoured if the action is invoked via the >> > wrapper factory. >> > >> > As Martin and Jeroen have said, using JProfiler or similar will likely >> > yield valuable info. >> > >> > If the issue is with database concurrency (which does seem like a good >> > hypothesis), then for the most part that is under the management of >> > DataNucleus itself. There are a bunch of configuration properties that >> can >> > be specified in persistor_datanucleus.properties (or isis.properties, >> > actually); we strip off any "isis.persistor.datanucleus.impl" prefix and >> > pass the rest of the key down to DN to configure. >> > >> > Questions for you: >> > - which version of Isis are you on? >> > - how have you implemented the stress test? is it automated, eg >> JMeter, or >> > just adhoc manual testing? >> > - is this stress testing the app running locally on Tomcat, or up in the >> > cloud (on Amazon Elastic Beanstalk)? >> > - how much DB traffic does a single request create? Is it all >> read-only; >> > or are there also writes? >> > - for reads, are you seeing any N+1 issues? if so, have you tried to >> > prevent them using DataNucleus eager loading hint (the "fetch groups" >> > concept)? >> > - for writes, are they expected? >> > - have you configured any of the various addons that could be writing to >> > the DB, eg auditing, command or publishing? are these perhaps enabled >> for >> > safe/query-only actions? If so >> > - for any looping within the app itself, can you eliminate using the >> > QueryResultsCache? >> > - what data volumes is this on? If large volumes of data, are there >> > perhaps missing indices? What's the performance like with small >> volumes of >> > data? >> > - is this for in-memory HSQLDB or out-of-memory DB (eg Postgres)? >> What's >> > the performance like of one versus the other >> > >> > I think what I would look for is fast response times for a 1-user system >> > running locally on Tomcat, with an inmemory database, and low data >> > volumes. From there, ramp up in different ways, but change one thing >> at a >> > time. >> > >> > HTH >> > Dan >> > >> > >> > >> > On 31 May 2016 at 15:12, César Camilo Lugo Marcos < >> [email protected]<mailto:[email protected]>> >> > wrote: >> > >> > > Thank you both for the hint. I am looking into JProfiler. Still, >> > > anything about the concurrency control options? >> > > >> > > Cesar. >> > > >> > > On Tue, 2016-05-31 at 11:13 +0200, Jeroen van der Wal wrote: >> > > > I agree with Martin that profiling is the only way to go. To >> illustrate: >> > > we >> > > > recently made some code 8 times faster by a few simple code changes >> on >> > > > bottlenecks revealed by JProfiler. And those were in places that >> we've >> > > > never guessed. >> > > > >> > > > On 31 May 2016 at 08:39, Martin Grigorov <[email protected] >> <mailto:[email protected]>> wrote: >> > > > >> > > > > Hi, >> > > > > >> > > > > To find out where is the problem you will have to use a profiler >> like >> > > > > JProfiler, Yourkit, JVisualVM, etc. >> > > > > Even some thread dumps would help to see what is going on. >> > > > > >> > > > > Martin Grigorov >> > > > > Wicket Training and Consulting >> > > > > https://twitter.com/mtgrigorov >> > > > > >> > > > > On Mon, May 30, 2016 at 9:00 PM, César Camilo Lugo Marcos < >> > > > > [email protected]<mailto:[email protected]>> wrote: >> > > > > >> > > > > > Hello, >> > > > > > >> > > > > > I would like to add to this topic the following: >> > > > > > >> > > > > > Most of the transactions we are testing in these stress tests >> are not >> > > > > > bound in ACTIONS, but just plain reads or default transactions >> using >> > > > > > Apache ISIS wicket viewer defaults. I don't see any place where >> I >> > > could >> > > > > > define semantics for default read or write transactions not >> bound >> > > into >> > > > > > ACTION methods. Is there any place I should look into for that? >> > > > > > >> > > > > > Cesar. >> > > > > > >> > > > > > >> > > > > > On Mon, 2016-05-30 at 18:45 +0000, César Camilo Lugo Marcos >> wrote: >> > > > > > > Hello, >> > > > > > > >> > > > > > > We have sarted performing some stress tests to our Apache ISIS >> > > > > > > application. We have found this behavior: >> > > > > > > >> > > > > > > - If we run 1 concurrent user, average response times to the >> main >> > > > > object >> > > > > > > reads through the wicket viewer are from 1 to 1.5 seconds. >> > > > > > > - If we run 2 concurrent users, same transactions go to >> average >> > > > > response >> > > > > > > times up to 16 to 27 seconds. >> > > > > > > - If we run 10 concurrent users, the transactions start to >> slow >> > > down >> > > > > > > significantly until the app server freezes and we have to >> restart >> > > it. >> > > > > > > >> > > > > > > As you can see, this is a very significant increase in >> response >> > > time >> > > > > for >> > > > > > > such a slight change in user load (from 1 user to 2 users). >> So we >> > > are >> > > > > > > guessing we should look into the concurrency control. >> > > > > > > >> > > > > > > In the documentation I have found that the only way to >> influence >> > > the >> > > > > way >> > > > > > > Apache ISIS manages transactions and concurrency checking is >> by >> > > using >> > > > > > > the semantics configuration of the ACTION annotation. >> > > > > > > >> > > > > > > semantics=SAFE_AND_REQUEST_CACHEABLE >> > > > > > > semantics=SAFE >> > > > > > > semantics=IDEMPOTENT >> > > > > > > semantics=IDEMPOTENT_ARE_YOU_SURE >> > > > > > > semantics=NON_IDEMPOTENT >> > > > > > > semantics=NON_IDEMPOTENT_ARE_YOU_SURE >> > > > > > > >> > > > > > > I just wanted to confirm if this is the place to look into, >> or are >> > > > > there >> > > > > > > any other places where we should be looking into too, to >> solve the >> > > > > > > performance issue. >> > > > > > > >> > > > > > > Cesar. >> > > > > > >> > > > > > >> > > > > >> > > >> > > >> > >> > >> >> >
