Dan, In our first tests we experienced a very significant improvement. Before 2 concurrent users significantly decreased response time, while 5 friezed the server. Now we have ran 100 and 400 concurrent users, and response time does not get impacted that much. Very significant improvement, definitely looks like you are heading in the right direction. Any updates on this will be appreciated as per [3].
Cesar. On Sat, 2016-06-04 at 11:56 +0100, Dan Haywood wrote: > A further update re [1]; > > I've now pushed a commit [2] which seems to have addressed Cesar's issue > ... removing a synchronized modifier. > > I'm going to do a review of other uses of 'synchronized', to see if they > are justified or not [3]. > > Thx > Dan > > > [1] https://issues.apache.org/jira/browse/ISIS-1421 > [2] > https://github.com/apache/isis/commit/746176991b523fe9e8ab069392d83ff17af08624 > [3] https://issues.apache.org/jira/browse/ISIS-1428 > > > > > On 3 June 2016 at 13:28, Dan Haywood <[email protected]> wrote: > > > Just as an update on this; I've run Cesar's JMeter tests, and it has > > revealed an issue in Isis core ... my suspicion being a possible > > synchronized deadlock. [1] > > > > Will investigate with a view to fixing in 1.13.0 > > > > Thx > > Dan > > > > > > [1] https://issues.apache.org/jira/browse/ISIS-1421 > > > > On 2 June 2016 at 16:50, César Camilo Lugo Marcos < > > [email protected]> wrote: > > > >> Hello. > >> > >> We are looking into several recommended topics to tune up the > >> application. One of them is the use of JDO Optimistic locking instead > >> of the Datanucleus default pessimistic locking, to improve concurrency. > >> I have not found a setting within Datanucleus to change this setting > >> globally or per transaction, just found this to define the property to > >> be used as version on the domain object: > >> > >> @PersistenceCapable > >> @Version(strategy=VersionStrategy.VERSION_NUMBER) > >> public class MyClass > >> { > >> ... > >> } > >> > >> Do you know about how to set it globally? > >> > >> > >> On Tue, 2016-05-31 at 18:54 +0000, César Camilo Lugo Marcos wrote: > >> > Hello Dan. Answers to your questions: > >> > > >> > - which version of Isis are you on? > >> > 1.12.0 > >> > - how have you implemented the stress test? is it automated, eg > >> JMeter, or just adhoc manual testing? > >> > Using JMeter. We recorded a quite simple script with about 6 read > >> only operations using th Wicket viewer. All samplers are the recorded HTTP. > >> > - is this stress testing the app running locally on Tomcat, or up in > >> the cloud (on Amazon Elastic Beanstalk)? > >> > We have it installed in the AWS Elastic Beanstalk with PostgreSQL. > >> > We also made a test on a local host with Tomcat and MySQL, > >> obtaining very similar results. > >> > > - how much DB traffic does a single request create? Is it all > >> read-only; or are there also writes? > >> > Quite low traffic, About 6 read only operations per user. > >> > - for reads, are you seeing any N+1 issues? if so, have you tried to > >> prevent them using DataNucleus eager loading hint (the "fetch groups" > >> concept)? > >> > Not sure what N+1 is. We are letting Datanucleous handle all > >> foreign keys and primary keys. We have not configured any fetch groups or > >> any other configurations than defaults. > >> > - for writes, are they expected? > >> > Did not understand this question. Please explain. > >> > - have you configured any of the various addons that could be writing > >> to the DB, eg auditing, command or publishing? are these perhaps enabled > >> for safe/query-only actions? If so for any looping within the app itself, > >> can you eliminate using the QueryResultsCache? > >> > We are only using the logging add-on, to log login and logout > >> operations. No auditing, command or publishing add ons. > >> > - what data volumes is this on? If large volumes of data, are there > >> perhaps missing indices? What's the performance like with small volumes of > >> data? > >> > No Data Object or table has more that a few tens of records. > >> > - is this for in-memory HSQLDB or out-of-memory DB (eg Postgres)? > >> What's the performance like of one versus the other > >> > This is PostgreSQL. Also tried MySQL with very similar results. > >> Have not tried in memory HSQLDB. > >> > > >> > > >> > Cesar. > >> > > >> > > >> > On Tue, 2016-05-31 at 18:53 +0100, Dan Haywood wrote: > >> > > >> > > >> > Hi Cesar, > >> > > >> > Those action semantics don't play a large part in the transaction > >> > management; they are mostly used to determine which type of HTTP method > >> to > >> > expose the action under (GET, PUT or POST) and there is also the > >> constraint > >> > that contributed/mixin properties/collections can only be supplied by > >> SAFE > >> > actions. The _ARE_YOU_SURE variants only impact the Wicket viewer. The > >> > _REQUEST_CACHEABLE is only honoured if the action is invoked via the > >> > wrapper factory. > >> > > >> > As Martin and Jeroen have said, using JProfiler or similar will likely > >> > yield valuable info. > >> > > >> > If the issue is with database concurrency (which does seem like a good > >> > hypothesis), then for the most part that is under the management of > >> > DataNucleus itself. There are a bunch of configuration properties that > >> can > >> > be specified in persistor_datanucleus.properties (or isis.properties, > >> > actually); we strip off any "isis.persistor.datanucleus.impl" prefix and > >> > pass the rest of the key down to DN to configure. > >> > > >> > Questions for you: > >> > - which version of Isis are you on? > >> > - how have you implemented the stress test? is it automated, eg > >> JMeter, or > >> > just adhoc manual testing? > >> > - is this stress testing the app running locally on Tomcat, or up in the > >> > cloud (on Amazon Elastic Beanstalk)? > >> > - how much DB traffic does a single request create? Is it all > >> read-only; > >> > or are there also writes? > >> > - for reads, are you seeing any N+1 issues? if so, have you tried to > >> > prevent them using DataNucleus eager loading hint (the "fetch groups" > >> > concept)? > >> > - for writes, are they expected? > >> > - have you configured any of the various addons that could be writing to > >> > the DB, eg auditing, command or publishing? are these perhaps enabled > >> for > >> > safe/query-only actions? If so > >> > - for any looping within the app itself, can you eliminate using the > >> > QueryResultsCache? > >> > - what data volumes is this on? If large volumes of data, are there > >> > perhaps missing indices? What's the performance like with small > >> volumes of > >> > data? > >> > - is this for in-memory HSQLDB or out-of-memory DB (eg Postgres)? > >> What's > >> > the performance like of one versus the other > >> > > >> > I think what I would look for is fast response times for a 1-user system > >> > running locally on Tomcat, with an inmemory database, and low data > >> > volumes. From there, ramp up in different ways, but change one thing > >> at a > >> > time. > >> > > >> > HTH > >> > Dan > >> > > >> > > >> > > >> > On 31 May 2016 at 15:12, César Camilo Lugo Marcos < > >> [email protected]<mailto:[email protected]>> > >> > wrote: > >> > > >> > > Thank you both for the hint. I am looking into JProfiler. Still, > >> > > anything about the concurrency control options? > >> > > > >> > > Cesar. > >> > > > >> > > On Tue, 2016-05-31 at 11:13 +0200, Jeroen van der Wal wrote: > >> > > > I agree with Martin that profiling is the only way to go. To > >> illustrate: > >> > > we > >> > > > recently made some code 8 times faster by a few simple code changes > >> on > >> > > > bottlenecks revealed by JProfiler. And those were in places that > >> we've > >> > > > never guessed. > >> > > > > >> > > > On 31 May 2016 at 08:39, Martin Grigorov <[email protected] > >> <mailto:[email protected]>> wrote: > >> > > > > >> > > > > Hi, > >> > > > > > >> > > > > To find out where is the problem you will have to use a profiler > >> like > >> > > > > JProfiler, Yourkit, JVisualVM, etc. > >> > > > > Even some thread dumps would help to see what is going on. > >> > > > > > >> > > > > Martin Grigorov > >> > > > > Wicket Training and Consulting > >> > > > > https://twitter.com/mtgrigorov > >> > > > > > >> > > > > On Mon, May 30, 2016 at 9:00 PM, César Camilo Lugo Marcos < > >> > > > > [email protected]<mailto:[email protected]>> wrote: > >> > > > > > >> > > > > > Hello, > >> > > > > > > >> > > > > > I would like to add to this topic the following: > >> > > > > > > >> > > > > > Most of the transactions we are testing in these stress tests > >> are not > >> > > > > > bound in ACTIONS, but just plain reads or default transactions > >> using > >> > > > > > Apache ISIS wicket viewer defaults. I don't see any place where > >> I > >> > > could > >> > > > > > define semantics for default read or write transactions not > >> bound > >> > > into > >> > > > > > ACTION methods. Is there any place I should look into for that? > >> > > > > > > >> > > > > > Cesar. > >> > > > > > > >> > > > > > > >> > > > > > On Mon, 2016-05-30 at 18:45 +0000, César Camilo Lugo Marcos > >> wrote: > >> > > > > > > Hello, > >> > > > > > > > >> > > > > > > We have sarted performing some stress tests to our Apache ISIS > >> > > > > > > application. We have found this behavior: > >> > > > > > > > >> > > > > > > - If we run 1 concurrent user, average response times to the > >> main > >> > > > > object > >> > > > > > > reads through the wicket viewer are from 1 to 1.5 seconds. > >> > > > > > > - If we run 2 concurrent users, same transactions go to > >> average > >> > > > > response > >> > > > > > > times up to 16 to 27 seconds. > >> > > > > > > - If we run 10 concurrent users, the transactions start to > >> slow > >> > > down > >> > > > > > > significantly until the app server freezes and we have to > >> restart > >> > > it. > >> > > > > > > > >> > > > > > > As you can see, this is a very significant increase in > >> response > >> > > time > >> > > > > for > >> > > > > > > such a slight change in user load (from 1 user to 2 users). > >> So we > >> > > are > >> > > > > > > guessing we should look into the concurrency control. > >> > > > > > > > >> > > > > > > In the documentation I have found that the only way to > >> influence > >> > > the > >> > > > > way > >> > > > > > > Apache ISIS manages transactions and concurrency checking is > >> by > >> > > using > >> > > > > > > the semantics configuration of the ACTION annotation. > >> > > > > > > > >> > > > > > > semantics=SAFE_AND_REQUEST_CACHEABLE > >> > > > > > > semantics=SAFE > >> > > > > > > semantics=IDEMPOTENT > >> > > > > > > semantics=IDEMPOTENT_ARE_YOU_SURE > >> > > > > > > semantics=NON_IDEMPOTENT > >> > > > > > > semantics=NON_IDEMPOTENT_ARE_YOU_SURE > >> > > > > > > > >> > > > > > > I just wanted to confirm if this is the place to look into, > >> or are > >> > > > > there > >> > > > > > > any other places where we should be looking into too, to > >> solve the > >> > > > > > > performance issue. > >> > > > > > > > >> > > > > > > Cesar. > >> > > > > > > >> > > > > > > >> > > > > > >> > > > >> > > > >> > > >> > > >> > >> > >
