That can be a tricky question to answer. The bottleneck may be disk or could also be CPU, since highlighting several MB of XML can use quite a few cycles. If you can establish that the bottleneck is CPU, it may be enough to extend your highlight code to set $cts:action to 'skip' or 'break' after the first few matches.
Otherwise your goal is to create documents or fragments that average something like the 1-100 kB. XML is messy so we cannot hit that number precisely, so do not take that target too literally. At the same time, make sure that the fragments are well-aligned with search and update requirements. The simplest thing that might possibly work would be to fragment by page. I do not much like that answer myself: the sizes are reasonable, but paragraphs and sentences may cross page boundaries. Since search terms are indexed by fragment, that may cause problems. Imagine searching for "now is the time", but one of the potential matches crosses a page boundary. If the book is a single document and the page boundary is a phrase-through, there is no problem. But if each page is its own fragment root, then neither page will match the full query. Still, this option is worth considering and does not require much extra work because each book still looks like a single XML document. Another common approach is to treat each book as a directory (http://blakeley.com/blogofile/2012/03/19/directory-assistance/), which contains one document for every top-level structural element. Usually these are chapters or something at a similar level. That limits the highlighting problem to a reasonable amount of XML, and searching chapters independently makes sense for most applications. Some chapters may be larger than others, but still smaller than the entire book. Crucially, they should be small enough to highlight comfortably. Reassembling the entire book can be as simple as having a toc.xml or manifest.xml in each directory. Most book-oriented applications use one or the other of those approaches. There are also some clever algorithmic tricks involving cts:walk and cts:highlight together, and approaches using duplicated content for passage-oriented or sentence-oriented search. But these can be challenging to implement, and most applications can use one of the simpler designs outlined above. -- Mike On 18 May 2012, at 13:48 , Murray, Gregory wrote: > Hello, > > I'm working on a project based on full-text XML documents. Each document is a > digital representation of a book, containing a short metadata header but > consisting mainly of <page> elements, each of which contains a single text > node containing a blob of OCR'ed text. We now have 50,000 such documents and > are hitting performance/scaling problems. After some profiling, we've > concluded that searching isn't the problem but that snippeting is a major > culprit. At ML World, my colleagues and I asked Mary Holstege about this, and > she indicated that most likely I/O is the bottleneck, because each time a > document is processed it must be pulled from disk in its entirety, since of > course the <page> elements are not amenable to indexing. She suggested > breaking the monolithic documents into pieces, perhaps one XML document per > page. At that point, performing a search would involve the search itself but > also post-processing to assemble the results back into book-based results for > display to th > e user. > > I'm sure that would work well if I knew how to collate the page-level search > results into book-level results. I'm sure there are readers of this list -- > perhaps especially in publishing or government -- who work not just with > smaller, data-centric documents but also with large, full-text documents. Is > there a way to retain the integrity of the documents as complete books and > still have good performance? If not, and if we were to split the books into > pages, what's a good strategy for collating the page-level search results so > they still appear to be book-level results to the user? > > Many thanks, > Greg > > Gregory Murray > Digital Library Application Developer > Princeton Theological Seminary > > _______________________________________________ > General mailing list > [email protected] > http://community.marklogic.com/mailman/listinfo/general > _______________________________________________ General mailing list [email protected] http://community.marklogic.com/mailman/listinfo/general
