I know you are saying that you are using the same set of test portlets (I think), and so I'm probably way off on this, but could the use of the FSS or the resource server (ResourceServingWebapp) possibly be slowing down requests?
Also, it might be helpful to do YSlow on page loads and focus in on that, if you think that any part of the images or skin that changed between versions may be part of the problem. Finally, have you tried using a profiler to help nail down any performance issues in the framework itself? Identifying performance issues is great and everyone appreciates the work, but it would be much more helpful to provide specifics so that Eric, etc. can focus on what needs to be looked at. On 6/8/10 2:32 AM, Lennard Fuller wrote: > Do the tomcats for all 3 have similar configurations? Am especially > interested in any sort of compression/gzip configs. Have you used the yslow > plugin to compare the versions of uPortal? If so... are there any new > suggestions with the later versions or decreases in ratings for specific > areas? > > With regards to uPortal, have there been any new tweaks/changes/additions to > the JPA code since 3.0? > Are the test users you are using new to uPortal (does a new layout have to be > generated and persisted) or are you reusing the same users for each test? > > What you describe sounds very similar to what we ran into with the original > 2.6.1 vs 3.0 perf testing. Those tests turned up a number of issues ranging > from some functional, but performance killing JPA issues (easily fixed), to a > pluto bug that Eric fixed. Think we saw 5 or 6 issues in all. Same sort of > behavior... all versions were pretty comparable until a tipping point was > reached and 3.0 couldn't handle the load. After the issues were fixed, 3.0 > was actually faster in some respects and was definitely comparable to 2.6.1. > uPortal 3.1 and 3.2 have had a lot of development done and are fully > functional, but I doubt anyone has seriously pushed them as hard as you are > now. The causes are likely going to vary a good deal, but I would focus on > what is new in 3.1/3.2 since the latest point releases of 3.0 proved to be > sufficiently performant with a very similar set of tests. Taking that > approach should reduce the amount of code to look at... since the changes > between 2.6.1 and 3.1 or 3.2 are far greater than those of 3.0 to 3.1 or 3.2. > > -Lennard > > ----- Original Message ----- > From: "Alex Bragg"<[email protected]> > To: [email protected] > Sent: Monday, June 7, 2010 10:13:54 PM GMT -07:00 U.S. Mountain Time (Arizona) > Subject: [uportal-dev] Fwd: uPortal Peformance > > Hello, > > I'm doing some performance testing, and I could use some hints on a couple of > issues. First, I'm looking for some hints on things I can tweak in > 3.1.1/3.2.1 to improve performance under heavy load. Second, I'm hitting a > bug in 2.6.1 that is preventing me from gathering solid baseline performance > numbers, and perhaps someone else has seen it. Let me explain in further > detail. > > We have been preparing for an upgrade of our production systems from uPortal > 2.6.1 to uPortal 3.x. Currently, we're looking at two 3.x versions, 3.1.1 > and 3.2.1. In my development environment, I have installed 2.6.1, 3.1.1, and > 3.2.1. My 2.6.1 install is running out of a 5.5.28 Tomcat, and my 3.x > versions are running in a 6.0.24 Tomcat. All versions are running under Java > version 1.6.0_12-b04, 64-bit, and I have an Oracle 11gR2 database backing > them. > > The layout in each instance is a simple 5-tab layout, with nothing on the > default tab. I have a custom testing portlet that simply executes a SQL > query 5, 10, or 15 times and renders a 3-line text output. On the remaining > four tabs, I have mixtures of two or more of these testing portlets. I run > tests with JMeter, and the click path is get login page, login, click tab 2, > click tab 3, click tab 4, click tab 5, and logout. JMeter verifies each page > renders properly. The tests I run execute this click path 4000 times spread > across 1, 4, 50, and 200 threads, and there are no waits built into the > scripts. > > Here are results from the tests I have run so far. The values are the 90th > percentile page-response time in seconds. Please note that the number for > 2.6.1 in the 200-thread column isn't valid. At the 200-thread level most of > the 200 threads complete their 20 iterations before JMeter starts additional > threads during ramp-up. I end up with no more than 4 or 5 threads running > concurrently. Another thing that skews these numbers is that I can only get > valid results using users that have successfully logged in before. Anything > above 2 threads with users that have not previously logged in results in > channels failing to render (with the message "You are not authorized to view > this channel"). > > version 1 4 50 200 50-lb2 200-lb2 50-lb4 200-lb4 > 2.6.1 0.07 0.08 0.7 *0.08* 0.69 4.56 > 3.1.1 0.09 0.09 1.96 7.81 1.18 6.02 1.12 5.49 > 3.2.1 0.17 0.18 7.04 26.43 6.17 20.22 > > The "lb2" and "lb4" designators signify that I have started multiple Tomcats > on the server, 2 for lb2 and 4 for lb4, and I'm balancing load with HAProxy. > I see much better utilization on the server, and both page-response times and > elapsed test run times (below) both improve significantly even though I have > not added any additional hardware. > > This table shows the elapsed time in seconds to complete the above tests. > > version 1 4 50 200 50-lb2 200-lb2 50-lb4 200-lb4 > 2.6.1 934 454 216 212 209.09 263.43 > 3.1.1 1,537 462 495 813 386.92 660.39 421.41 414.32 > 3.2.1 3,299 862 1,999 3,958 1259.99 2636.8 > > Basically, what I see here is that at low concurrency 2.6.1 and 3.1.1 are > fairly comparable, and 3.2.1 is noticeably slower. At 50 threads and above, > I see that 2.6.1 is much faster than 3.x. I also see that at very high > loads, 3.x seems to have a point where it just falls over the edge of a cliff. > > Part of that I'm sure is the change in page sizes. Here are the page sizes > JMeter reports (this does not include embedded resources). > > 2.6.1 3.1.1 3.2.1 > Avg. Bytes Avg. Bytes Avg. Bytes > Login Page 2014.93 12865 23963 > Login 2958.61 21716 21909 > Tab 1 5950.05 24221.21 24656 > Tab 2 8840.34 26755.38 27430 > Tab 3 8835.95 26753.3 27428 > Tab 4 7380.03 25525.27 26068 > Logout 2014.94 12865 23963 > TOTAL 5427.84 21528.74 25059.57 > > So, back to my two questions. > > 1. What has changed in 3.1.1 that might explain a significant (at least 2x > slowdown under load)? To me it feels like 2.6.1 is caching rendered elements > to a much greater degree than 3.1.1. What can I tweak to improve this? > > 2. Is anyone aware of something I can change to fix the behavior with new > logins in 2.6.1 to prevent this issue with channels not authorized? > > Thanks, > Alex Bragg > Unicon, Inc. > > -- You are currently subscribed to [email protected] as: [email protected] To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/uportal-dev
