Thank you for sharing and for being so detailed. - Gabriel
On Tue, Feb 9, 2010 at 10:44 PM, Dave <[email protected]> wrote: > Hi Dorioo, > Here are the results of our performance comparisons. > > Firstly the disclaimer, I am not trying to bag out MM here, he has > done a great job on Transfer. > > We run a large Transfer/Mach-II/Coldspring app (3.6k lines of Transfer > XML config). We have run into the typical memory issues that large > apps on this stack have. I believe these issues are being addressed > with the pluggable cache updates MM is doing. We tested against > Transfer 1.1 > > Our production environment is based on 64bit/20 gig/8proc machines. > The tests were modelling a small sub-system of our application, > without mach-II or coldspring. The tests were executed using a > comparably small server (dual procs, 512 mb allocated to JVM). Scale > wise, this machine approximated our production environment taking into > account how small both the database for the tests was, and that the > test classes, object graph, etc are fairly simple. SQL db was on the > same physical test server. JMeter was also running on the same > physical test server to eliminate network latency issues. > > The test code consisted on 5 level (class) One2Many hierarchy, at the > bottom of that hierarchy was a M2M class. (ie, fairly linear object > model) Loading a single root object would cause the loading of around > 50 child objects. A percentage of the M2M’s at the bottom of the > hierarchy were shared between the root objects. > The fairly linear object model allowed us to test something called > Bulk-Loading, ie, use a CFQuery to grab the parent, joined to each > child, then a grouped CfOutput to create/load each object. > > The tests compared Bulk-Loading on CF8, CF9, Transfer on CF8 & CF9 and > Hibernate on CF9. CF9 was a RC version. > > We developed a simple class caching system for both Hibernate/Bulk > loading objects consisting of a manager class which is stored in > application scope. It had an internal struct to store objects and > automatically reaped objects based on MaxObjects and/or low memory. > This only stored the Root objects, with the root object containing > references to all of their child objects, which stored references to > their children, etc. > Although transfer caches the child object individually, we did not > need to do this as our production code for this sub-system would allow > us to cache the root object and let that manage its children. > > For the transfer test, each object had a decorator class (as they do > in production), and the test would execute a method on the root > object, which would execute a method on it’s child objects and so on. > Transfer was set to cache in application scope. The non-transfer > classes had a similar number of methods to the transfer decorator > class (as well as setters/getters) > > Each test was run 2000 iterations (two J-Meter threads running 1000 > times) & repeated 3 times with results averaged. Each config was > tested against a pool of 50 objects (ie, every object ends up in > cache) and 500 objects (a small percentage of objects is able to be > cached and therefore we get a lot of cache thrashing, as in > production) > > As you can see, my tests are very specific and designed to find the > best way for our project to move forward. You should probably create > your own test-rigs to decide which way to move forward. These tests > took around 1 week to build and conduct & were very worthwhile for us. > > The results: > CF8, 500 Root-objects (cache thrashing) > Transfer with transfer cache: 1403ms/request > Transfer with no cache: 1730ms/request > Bulk-Loading with custom cache: 97ms/request > Bulk-Loading with no cache: 106ms/request > > CF8, 50 Root-objects (designed for all objects to end up in cache) > Transfer with transfer cache: 72ms/request > Transfer with no cache: 1671ms/request > Bulk-Loading with custom cache: 28ms/request > Bulk-Loading with no cache: 95ms/request > > CF9, 500 Root-objects (cache thrashing) > Transfer with transfer cache: 4880ms/request > Transfer with no cache: 2455ms/request > Bulk-Loading with custom cache: 91ms/request > Bulk-Loading with no cache: 126ms/request > Hibernate with custom cache: 66ms/request > Hibernate with no cache: 63ms/request > > CF9, 50 Root-objects (designed for all objects to end up in cache) > Transfer with transfer cache: 66ms/request > Transfer with no cache: 2207ms/request > Bulk-Loading with custom cache: 25ms/request > Bulk-Loading with no cache: 114ms/request > Hibernate with custom cache: 23ms/request > Hibernate with no cache: 64ms/request > > > > > Dave > > On Feb 7, 11:53 am, Dorioo <[email protected]> wrote: > > please post results. Thank you > > > > - Gabriel > > > > > > > > On Sat, Feb 6, 2010 at 7:49 PM, Dave <[email protected]> wrote: > > > I did some profiling of Transfer on CF9 vs CF8, and transfer was a lot > > > slower on CF9 in tests caused a lot of new objects to be loaded from > > > the DB (ie, cache exhaustion). > > > > > I suspect this is due to the fact that transfer has some custom code > > > for CF8, and treating is CF9 as CF7. MM confirmed this may be the > > > cause at CFObjective. > > > > > The results were very interesting, with Hibernate performing a lot > > > better then transfer under load. > > > > > I can post comparisons of transfer vs hand-coded objects vs hibernate > > > on CF8/CF9 if you like (obviously no results for hibernate on CF8). > > > > > The clear winner was Hibernate, followed by hand-coded objects, then > > > transfer. > > > > > dave > > > > > On Feb 5, 5:53 am, Brian G <[email protected]> wrote: > > > > Assuming you've upgraded your CF8 JVM beyond the classloader issue, > > > > can anyone estimate the performance improvement for Transfer-based > > > > apps from upgrading to CF9? > > > > > > I'm familiar with tests like Jamie Krug's object creation test but > I'm > > > > more interested in what kind of actual impact that CFC creation > > > > improvement has on your app. E.g., on the same hardware, same app, > > > > same load, etc, what kind of a performance boost, if any, did you see > > > > in upgrading? > > > > > > Ballparks and gut feels are ok with me, thanks! > > > > > > Brian > > > > > -- > > > Before posting questions to the group please read: > > > > >http://groups.google.com/group/transfer-dev/web/how-to-ask-support-qu. > .. > > > > > You received this message because you are subscribed to the Google > Groups > > > "transfer-dev" group. > > > To post to this group, send email to [email protected] > > > To unsubscribe from this group, send email to > > > [email protected]<transfer-dev%[email protected]> > <transfer-dev%2bunsubscr...@google groups.com> > > > For more options, visit this group at > > >http://groups.google.com/group/transfer-dev?hl=en > > -- > Before posting questions to the group please read: > > http://groups.google.com/group/transfer-dev/web/how-to-ask-support-questions-on-transfer > > You received this message because you are subscribed to the Google Groups > "transfer-dev" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected]<transfer-dev%[email protected]> > For more options, visit this group at > http://groups.google.com/group/transfer-dev?hl=en > -- Before posting questions to the group please read: http://groups.google.com/group/transfer-dev/web/how-to-ask-support-questions-on-transfer You received this message because you are subscribed to the Google Groups "transfer-dev" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/transfer-dev?hl=en
