Hi Tom, First off, thanks for sharing your details, and some more specifics about the memory issues you are seeing. Much appreciated.
I've got a few followup questions, if you or Simon don't mind answering them. Just trying to get a better understanding of where the core problem(s) reside, so that we can make useful suggestions to you (and find ways to resolve issues in DSpace itself). I've also noted a few areas where you might be able to achieve some temporary benefits (if you need something temporarily, at least until we can fix any issues in DSpace). On 10/7/2010 5:32 AM, Tom De Mulder wrote: > DSpace scalability issues report, per wiki template: > > 1. dsp...@cambridge, The University of Cambridge, UK. > Technical contacts: Tom De Mulder, [email protected] (systems manager) > Simon Brown [email protected] (DSpace developer) > > 2. a. DSpace version 1.6.2 with extensive local patches, using JSPUI > Size: 137 communities, 258 collections,>200k items, 12TB, 436k > bitstreams (excluding licenses) > > b. PostgreSQL 8.4.4 In order to better understand your PostgreSQL configs, would you be willing to share how your "work_mem" and "shared_buffers" are configured? Or, if you could share the whole Postgres config, that could also help. I just know there are sometimes ways to performance tune PostgreSQL for larger sized databases (which yours surely is, based on the amount of content). If you've already investigated PostgreSQL performance tuning, it'd be good to know as well. Here's some very basic info from our wiki (and much more off the PostgreSQL site as well), which shows settings we'd be looking to tune for potentially better DB performance: https://wiki.duraspace.org/display/DSPACE/PostgresPerformanceTuning http://wiki.postgresql.org/wiki/Performance_Optimization > > c. Tomcat 6.0.24 standalone > > d. Separate servers for webapp, DB, storage and ancillary functions > Webapp/DB servers are HT 8-core Intel servers running Ubuntu Linux > with 16GB of memory and fast local storage > Java memory: -Xmx2048M -Xms2048M Obviously, throwing more memory at this issue may not be a long term solution. But, you could think about (even temporarily) providing more memory to Java (beyond the 2GB) to see if that can temporarily lessen the frequency of these memory errors (likely, they may still occur at some point though). I'm not sure if this is possible temporary fix or not (depends on whether the rest of that 16GB of memory is already allocated to other applications) > 3. a. - Unless Tomcat is restarted, it will consistently fail due to lack of > memory in less than 48 hours. > - Batch importer: will fail on large batch imports (order of thousands > of items), performance degrades with size of repository and of batch. > - Search indexer: fails on large repositories, slowing down and > eventually running out of memory. > - Assetstore: random structure causes large overhead on filesystem for > no real gain > > See also our poster, presented in Gothenburg: > http://tools.dspace.cam.ac.uk/DSUG09%20A2%20poster.pdf Could you actually send us a few of the out of memory error messages you are seeing? You may be seeing out of memory errors either from PermGen ("OutOfMemoryError: PermGen space") or from Heap ("OutOfMemoryError: Java heap space"). So, it'd be good to determine which type(s) of memory error it is. We might be able to help you tweak your Java settings to at least temporarily avoid these errors being thrown. (Again, likely not a permanent fix, but may help temporarily until we can fix any memory usage issues in DSpace.) Also, knowing the type(s) of memory error may be able to help us determine what the cause in DSpace may be. > > b. Installed vanilla DSpace 1.6.2, imported 200k randomly generated > items, ran siege against it, watched it not cope. > We've done profiling in the past, but not for 1.6. However, we've not > noticed significant changes in the code that has issues. It'd be good to get an idea of what wasn't coping well in the vanilla DSpace 1.6.2 tests you ran. For instance, did accessing DSpace suddenly become extremely slow (e.g. browsing/searching), or was it the import script that slowed down (or maybe it was both)? I guess the question is what did you have the SIEGE program (http://www.joedog.org/index/siege-home) testing? Was it just doing random accesses of the website and noting very poor performance or receiving more out-of-memory errors? If you could, it would also would be wonderful if you would share your scripts for randomly generating 200K DSpace items. This would allow us to do the same testing locally to replicate exactly what you have seen. Plus, this seems like it could be a great testing tool for the whole community (we are always looking for a decent set of test data). Sure, we could probably rewrite the code from scratch, but it's always better to just grab a copy of code if you can :) > c. We have patches for the indexer; batch importer; thumbnail and PDF > text extraction; assetstore structure; dark item masking in OAI and browse > code I'd already mentioned this before, and there's not a huge rush. But, when you find some time, it'd be good to add some/all of these patches to JIRA (especially those you feel are applicable to 1.6.x). That way we can get more eyes on them, and do a full review and (hopefully, if all works out in our 1.7 timeline) maybe even get some/many into 1.7.0. For more on the DSpace 1.7 timeline, see this wiki page https://wiki.duraspace.org/display/DSPACE/DSpace+Release+1.7.0+Notes > 4. We can't commit to volunteering unless this can be incorporated into the > work we need to undertake in our primary capacity of running the University's > Institutional Repository. However, we would be willing to try and make this > happen. No problem & definitely understood. Providing this sort of performance feedback is already much appreciated. If you are able to share your patches, a little more specifics on the OutOfMemoryErrors, and potentially share your 'randomly generated items' script, that'd be a huge boost for us. It'd definitely help us to be able to get a better sense of exactly what is happening (and where) in DSpace, so that we can hopefully get some immediate fixes ready in time for the 1.7.0 release. Thanks! - Tim ------------------------------------------------------------------------------ Beautiful is writing same markup. Internet Explorer 9 supports standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3. Spend less time writing and rewriting code and more time creating great experiences on the web. Be a part of the beta today. http://p.sf.net/sfu/beautyoftheweb _______________________________________________ DSpace-tech mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dspace-tech
