Thanks for the update. Based on another email, I have a quick question. Are you using "mod_deflate" on Apache?
I have also been doing some load testing in relation to issue #7490. My results, assumptions, testing environment, and test "scripts" can be found at http://trac.edgewall.org/wiki/Performance. I would be interested in seeing the results of other people duplicating these test as it may help in reproducing problems like the one you have been working on. I have also been doing some profiling of the Trac-0.11-stable code base and have found no surprises, but I still do have a few theories to test with regards to I/O, available memory, and performance; however until I can construct some tests to disprove these theories, I don't have anything definite to share. For those following any of my previous threads on this, I have switched the load testing tool to Apache JMeter so that others can more easily perform similar testing on their systems (as opposed to the proprietary tool that I was using previously). Please let us know if you find anything else. Lance Chris Peterson wrote: > OK, so let me update briefly on some changes I have done. The only bad part > here is, none of the changes *singly* have made a large difference in > performance, but all together it seems to be running much better. > > I set Apache to have 100 process threads rather than 250 > I disabled DEP on Windows (which was enabled by default) > I disabled the source browsing from all hosted trac environments > I disabled all authentication to the /chrome folder, my <Location directive > for the chrome folder looks as such: > > <Location /chrome> > Satisfy Any > Allow from all > ExpiresActive On > ExpiresDefault "access plus 30 days" > </Location> > > I ran some IO tests on the S3 drive vs the host OS drive, and while the S3 > drive did show some high variances on speed, it also had peaks of very quick > performance (as I would expect from a highly shared resource), so I do not > believe that is my sole IO issue. > > Performance seems much better overall now, and normal page requests are no > longer peaking the CPU quite so badly, but the wiki page still consumes 100% > CPU for approx 1 - 2 seconds, then drops back down. I think (at this late > time at night on the weekend) that performance is acceptable now, but I will > need to get it under load before I know for certain. Each one of these > changes have seemed to cause an improvement, and all together it seems > pretty happy now =) > > Thanks to everyone for all your idea's. I believe my next steps here will > be to update to the 1.6.2 release of SVN, and I believe I will also update > the Sqlite databases to either MySQL or PostgreSQL and see if I can find > some additional performance gains that way. If anyone has a recommendation > on which way to go with the Trac database for performance and ease of > upgrading 50 environments or so, please feel free to chime in! > > Hope everyone is having a great evening, > > Chris Peterson > Alagad, Inc. > > > On 6/5/09 12:57 PM, "Christian Boos" <[email protected]> wrote: > > >> Chris Peterson wrote: >> >>> I work with Doug on this issue, and I wanted to provide some >>> additional details about our configuration and what I am seeing. >>> >>> The server is an Amazon EC2 small instance, running on a dual core >>> Opteron 2.6 ghz w/ 1.66 gig of ram >>> The Trac install lives in the default c:\python25 folder, and the >>> trac / svn data lives on an S3 drive. >>> >>> >> Ah well, you're the author of comment:57 - didn't you indicate there >> that changing the log_level was helping? >> Also, did you test on some similar local machine as well? >> >> >> >>> We have 49 trac environments running here, 4 of which have their own v- >>> host configured, the rest are for internal use and share a root url. >>> All of these environments also have SVN setup. The majority of the >>> shared instances have a common configuration base trac.ini file, with >>> individual files for environment specific configurations. >>> >>> I started out with the 0.11 install of Trac, and have since upgraded >>> to the 0.12dev-r8263 release from SVN (using easy_install and pointing >>> to the SVN repo location), to try and address our performance issues. >>> Under 'About Trac', here is my current config: >>> >>> Trac: 0.12dev-r8263 >>> Python: 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 >>> 32 bit (Intel)] >>> setuptools: 0.6c9 >>> SQLite: 3.5.2 >>> pysqlite: 2.4.0 >>> Genshi: 0.6dev-r1052 >>> Babel: - >>> mod_python: 3.3.1 >>> Pygments: 1.0 >>> Subversion: 1.5.4 (r33841) >>> jQuery: 1.1.3.1 >>> >>> >> Side note, should be jQuery: 1.3.2, no? (that's what is checked in into >> trunk, AFAICT). >> >>> I have Apache connecting using mod_python, but this slow performance >>> and 100% CPU peak usage persists even if I load a single environment >>> using tracd and shut-down the apache instance. >>> >> Ok, that's very interesting: you're able to reproduce the perfomance >> issue with tracd serving a single environment. >> Let's forget Apache for now and focus on tracd. How exactly do you start >> it, btw? >> >> >>> I have gone through the >>> site-packages and, one by one, removed each plugin from the folder and >>> re-started the instance, and I still saw the performance issues (I was >>> trying to isolate the issue to a single plugin, without luck). >>> >>> >> That's also interesting: Trac trunk alone /without any plugin/ shows the >> same performance issue. >> Did you try to disable the browser component as well? ([components] >> trac.versioncontrol.* = disabled) >> I see you're using svn 1.6.0, which might be an issue, as it was mostly >> untested - I use 1.6.2. >> >> >> >>> I applied the patch as suggested in >>> http://trac.edgewall.org/ticket/7785#comment:13 >>> and have seen no improvement. I also have logging currently disabled >>> completely in the hopes of improving performance, still without luck. >>> >>> I did a directory output from my site-packages folder, if it will >>> provide better detail as to plugins I have installed, that you can >>> view here: http://pastebin.com/fea11695 >>> >>> The worst performance seems to be on the wiki pages, but viewing >>> tickets spins the CPU up quite high as well. >>> >> I wonder if that could be related to wiki processing? Are you really >> positive that /no/ plugins were used? (btw, you should also post a log >> file at DEBUG level, starting from the startup of tracd till viewing a >> few of those pages where the CPU spikes). >> >> We did have some performance issues in the past for some specific >> "pathological" markups, maybe you could post some typical wiki content >> (unless the performance problem also happens for unmodified Trac pages >> from the TracGuide). >> >> >>> At one point I attempted >>> to implement the Psyco package (http://psyco.sourceforge.net/) to >>> improve performance, but all plugins have been updated without it >>> again as I saw no performance gains from using it (the package is >>> still in my site-packages, but the code is not in use) >>> >>> At this point I am (sadly) thinking that I may need to re-configure a >>> new server with a clean / new install of python and trac and see how >>> it runs, I really do not know what else to investigate to resolve this >>> problem. Any idea's, as wild as they may be, would be greatly >>> appreciated! >>> >>> >> Well, maybe it's due to something specific to Amazon EC2, so please try >> first to reproduce the issue on a single isolated environment using a >> bare bone tracd from trunk running on a "normal" XP or Vista system of >> yours, then do the same minimalistic test on Amazon EC2. >> >> -- Christian >> >> > > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Trac Users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/trac-users?hl=en -~----------~----~----~----~------~----~------~--~---
