The bottleneck seems CPU bound to me. Maybe trying the external xpdf extractor will help a lot here...
On Fri, Aug 20, 2010 at 5:09 PM, Marcelo Henrique Gomes <[email protected]> wrote: > Hi Stuart, > > Thanks a lot. I changed the Postgre database and search index to a local > disk and the result was better. > > The filter-media spend 40% less and the search is so faster. > > I forgot to mention that the version of my DSpace is 1.5.2. > > I attached informations of cpu and memory when I run the filter-media > execution. The pid from filter-media was #77476. > > Please, one more question :D > It seems that indexing time is increasing as database size increases. If it > process only new items, why is it happening? > > > > > On Sat, Aug 14, 2010 at 4:18 PM, Stuart Lewis <[email protected]> > wrote: >> >> Hi Marcelo, >> >> > Today we have approx. 400.000 items in the repository >> > >> > Consider three PDF files per day, with a variable number os pages (100 ~ >> > 5000) each. >> > >> > We create the hierarchy Year (community) -> Month (Community) -> Day >> > (Community) -> File (collection) for each file. For instance, 2008 -> Aug >> > -> >> > 13 -> My file 1; 2008 -> Aug -> 13 -> My file 2; 2008 -> Aug -> 13 -> My >> > file 3. >> > We upload each file page (a PDF of approx. 100Kb) as a different item. >> > In short, we have to index hundreds of small items per day. >> > >> > The server configuration is: >> > >> > CPU: Intel(R) Xeon(TM) CPU 3.00GHz (2992.52-MHz K8-class CPU) >> > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >> > real memory = 2147483648 (2048 MB) >> > avail memory = 2056413184 (1961 MB) >> > >> > Postgres db running in the same machine >> > >> > Disk (for DSpace and database) is a sorage mounted with nfs >> >> Depending on where the bottleneck is (possible in several places) you >> could try a few changes. Of course all this advice should be taken with a >> pinch of salt as everyone's system is different. >> >> The first change I would try is to move the postgres database (data >> directory) and search index files (dspace/search/) to a local (or locally >> attached) disk. This should be a lot faster. NFS is naturally slower than >> local disk, and operates slightly differently. Both Postgres and Lucene >> recommend that you don't run them over NFS. Since the indexing is primarily >> building search indexes in the database and in lucene, you should see some >> speed improvements. >> >> NFS should be fine for the assetstore though. Of course if you move the >> search index and postgres database files to a local disk, you'll need to >> remember to ensure they are included in your backup schedule. (Some people >> don't backup their DSpace lucene search index as it changes often and can be >> regenerated from DSpace - but that is up to you). >> >> If you still need a speed improvement, then depending on your budget you >> could try a few options: >> >> - Databases thrive well with a lot of RAM. The more data they can cache >> in memory, the faster they will be. Luckily RAM is quite cheap. Also read >> the documentation online about tuning Postgres to make sure it uses the RAM. >> >> - If you have a bit more money, try buying a second server, and running >> postgres on that. >> >> If you do find any of these give a useful improvement, please could you >> report back - that way we'll be able to give useful advice to other people >> in a similar situation. >> >> I hope that helps, >> >> >> Stuart Lewis >> IT Innovations Analyst and Developer >> Te Tumu Herenga The University of Auckland Library >> Auckland Mail Centre, Private Bag 92019, Auckland 1142, New Zealand >> Ph: +64 (0)9 373 7599 x81928 >> > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by > > Make an app they can't live without > Enter the BlackBerry Developer Challenge > http://p.sf.net/sfu/RIM-dev2dev > _______________________________________________ > DSpace-tech mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/dspace-tech > > ------------------------------------------------------------------------------ This SF.net email is sponsored by Make an app they can't live without Enter the BlackBerry Developer Challenge http://p.sf.net/sfu/RIM-dev2dev _______________________________________________ DSpace-tech mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dspace-tech

