Dear Jose, On Tue, 2007-06-19 at 07:10, Jose Blanco wrote:
> 3. We will also have the need to update these records periodically, > and so it seems like following a similar architecture as the one > DSpace uses, it would take a very unreasonable amount of time to > update 12 million records. Just last week I used the ItemImporter to > load 3,600 records and I believe it took about 7 hours for the load to > complete. I'm assuming that the reason it took so long was because my > repository already has about 35,000 items and the inserts to the > database were taking time, more than anything related to lucene. When > I loaded the same number of records in my development instance it took > less than 2 hours and I have very few records there ( probably about > 1000 ). Any thoughts on this? This issue has been raised many times on the various DSpace lists, but I am yet to see any substantive action on the part of the core developers to address it. Actually setting up some dedicated test servers with a decent amount of representative and scalable test data would be a start. One would take this for granted with any well organised test and release cycle, but to my knowledge DSpace releases are not subjected to serious performance profiling, scalability or stress testing. I have been hoping for some time that this deficiency would be corrected, but I am beginning to doubt that it will be addressed in the medium term. > 4. Has any one out there had to do something like this, and if so what > have you found that works. One solution that comes to mind is Zebra. > It is suppose to handle large repositories quite well. Are there any > users of Zebra out there that might have an opinion on this? I definitely suggest that you mail your requirements to the IndexData list. http://www.indexdata.dk/zebra/ http://lists.indexdata.dk/cgi-bin/mailman/listinfo/zebralist Over the years I have found IndexData's developers to be extremely helpful and responsive. An anecdote: Only the other day I found a bug in `yaz-client'. Not only was it fixed within a couple of days, but after consultation, new functionality was added. Just splendid. If only this attitude was more widespread. Best regards, Richard MAHONEY -- Richard MAHONEY | internet: http://indica-et-buddhica.org/ Littledene | telephone/telefax (man.): +64 3 312 1699 Bay Road | cellular: +64 27 482 9986 OXFORD, NZ | email: [EMAIL PROTECTED] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Indica et Buddhica: Materials for Indology and Buddhology Repositorium: http://indica-et-buddhica.org/repositorium/ Philologica: http://indica-et-buddhica.org/philologica/ Subscriptions: http://subscriptions.indica-et-buddhica.org/ ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ DSpace-tech mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dspace-tech
