Hello, I'm working on a project based on full-text XML documents. Each document is a digital representation of a book, containing a short metadata header but consisting mainly of <page> elements, each of which contains a single text node containing a blob of OCR'ed text. We now have 50,000 such documents and are hitting performance/scaling problems. After some profiling, we've concluded that searching isn't the problem but that snippeting is a major culprit. At ML World, my colleagues and I asked Mary Holstege about this, and she indicated that most likely I/O is the bottleneck, because each time a document is processed it must be pulled from disk in its entirety, since of course the <page> elements are not amenable to indexing. She suggested breaking the monolithic documents into pieces, perhaps one XML document per page. At that point, performing a search would involve the search itself but also post-processing to assemble the results back into book-based results for display to th e user.
I'm sure that would work well if I knew how to collate the page-level search results into book-level results. I'm sure there are readers of this list -- perhaps especially in publishing or government -- who work not just with smaller, data-centric documents but also with large, full-text documents. Is there a way to retain the integrity of the documents as complete books and still have good performance? If not, and if we were to split the books into pages, what's a good strategy for collating the page-level search results so they still appear to be book-level results to the user? Many thanks, Greg Gregory Murray Digital Library Application Developer Princeton Theological Seminary _______________________________________________ General mailing list [email protected] http://community.marklogic.com/mailman/listinfo/general
