Greg, I suspect your algorithms are still searching for and finding snippets in entire documents/books, rather than pages. Fragmenting on page allows you to more efficiently load a smaller number of pages from a book, but you have to explicitly work with pages to see the benefit. If you load a full book document you'll get all the pages, and no benefit.
Adding a searchable-expression to the search:search options may help: <searchable-expression xmlns:lib="http://digital.library.ptsem.edu/ia";> /lib:page </searchable-expression> (above code not tested) This tells search:search to look for *and return* pages. Yours, Damon -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Murray, Gregory Sent: Tuesday, May 22, 2012 11:58 AM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Efficient full-text searching and snippeting Mike, Thanks very much for your quick reply. Clearly there are lots of good ideas here. Our first plan of attack has been to set <page> as a fragment root and reindex the database. Our 50,000 documents now consist of 18 million fragments. Strangely, the situation appears to be even worse than before. The query resolution time seems reasonable, but snippet processing is now several times slower than it was before. We're using search:search(), and running a typical query in QC gives these metrics: <search:metrics> <search:query-resolution-time>PT0.093S</search:query-resolution-time> <search:facet-resolution-time>PT0.109S</search:facet-resolution-time> <search:snippet-resolution-time>PT6.582S</search:snippet-resolution-time> <search:total-time>PT6.926S</search:total-time> </search:metrics> which makes me wonder if the snippet process is still bringing in the entire document (book) rather than an individual fragment (page). In case they're relevant, I should clarify a few things: Our document structure includes a metadata element that stands outside of any <page> element. Also, a typical page will contain about 1000 UTF-8 characters, which is smaller than the optimal size range. And there are inevitably some blank pages and thus some empty fragments. For example: <doc xmlns="http://digital.library.ptsem.edu/ia";> <metadata> <id>1edhumanimmortalitjameuoft</id> <name>James, William, 1842-1910.</name> <title>Human immortality : two supposed objections to the doctrine</title> <date>1898</date> ... </metadata> <text type="ocr"> <page id="1edhumanimmortalitjameuoft_0001" absSeq="1"/> <page id="1edhumanimmortalitjameuoft_0002" absSeq="2"/> ... <page id="1edhumanimmortalitjameuoft_0011" absSeq="11" xmlns="http://digital.library.ptsem.edu/ia";> HUMAN IMMORTALITY Tis a matter unfortunately too often seen in history to call for much remark, ... </page> ... </text> </doc> Given that additional info -- do you have any thoughts on why snippeting would be even slower than before fragmenting on <page>? Many thanks, Greg On May 18, 2012, at 5:23 PM, Michael Blakeley wrote: > That can be a tricky question to answer. The bottleneck may be disk or could > also be CPU, since highlighting several MB of XML can use quite a few cycles. > If you can establish that the bottleneck is CPU, it may be enough to extend > your highlight code to set $cts:action to 'skip' or 'break' after the first > few matches. > > Otherwise your goal is to create documents or fragments that average > something like the 1-100 kB. XML is messy so we cannot hit that number > precisely, so do not take that target too literally. At the same time, make > sure that the fragments are well-aligned with search and update requirements. > > The simplest thing that might possibly work would be to fragment by page. I > do not much like that answer myself: the sizes are reasonable, but paragraphs > and sentences may cross page boundaries. Since search terms are indexed by > fragment, that may cause problems. Imagine searching for "now is the time", > but one of the potential matches crosses a page boundary. If the book is a > single document and the page boundary is a phrase-through, there is no > problem. But if each page is its own fragment root, then neither page will > match the full query. Still, this option is worth considering and does not > require much extra work because each book still looks like a single XML > document. > > Another common approach is to treat each book as a directory > (http://blakeley.com/blogofile/2012/03/19/directory-assistance/), which > contains one document for every top-level structural element. Usually these > are chapters or something at a similar level. That limits the highlighting > problem to a reasonable amount of XML, and searching chapters independently > makes sense for most applications. Some chapters may be larger than others, > but still smaller than the entire book. Crucially, they should be small > enough to highlight comfortably. Reassembling the entire book can be as > simple as having a toc.xml or manifest.xml in each directory. > > Most book-oriented applications use one or the other of those approaches. > There are also some clever algorithmic tricks involving cts:walk and > cts:highlight together, and approaches using duplicated content for > passage-oriented or sentence-oriented search. But these can be challenging to > implement, and most applications can use one of the simpler designs outlined > above. > > -- Mike > > On 18 May 2012, at 13:48 , Murray, Gregory wrote: > >> Hello, >> >> I'm working on a project based on full-text XML documents. Each document is >> a digital representation of a book, containing a short metadata header but >> consisting mainly of <page> elements, each of which contains a single text >> node containing a blob of OCR'ed text. We now have 50,000 such documents and >> are hitting performance/scaling problems. After some profiling, we've >> concluded that searching isn't the problem but that snippeting is a major >> culprit. At ML World, my colleagues and I asked Mary Holstege about this, >> and she indicated that most likely I/O is the bottleneck, because each time >> a document is processed it must be pulled from disk in its entirety, since >> of course the <page> elements are not amenable to indexing. She suggested >> breaking the monolithic documents into pieces, perhaps one XML document per >> page. At that point, performing a search would involve the search itself but >> also post-processing to assemble the results back into book-based results >> for display to > th >> e user. >> >> I'm sure that would work well if I knew how to collate the page-level search >> results into book-level results. I'm sure there are readers of this list -- >> perhaps especially in publishing or government -- who work not just with >> smaller, data-centric documents but also with large, full-text documents. Is >> there a way to retain the integrity of the documents as complete books and >> still have good performance? If not, and if we were to split the books into >> pages, what's a good strategy for collating the page-level search results so >> they still appear to be book-level results to the user? >> >> Many thanks, >> Greg >> >> Gregory Murray >> Digital Library Application Developer >> Princeton Theological Seminary >> >> _______________________________________________ >> General mailing list >> [email protected] >> http://community.marklogic.com/mailman/listinfo/general >> > > _______________________________________________ > General mailing list > [email protected] > http://community.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://community.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://community.marklogic.com/mailman/listinfo/general
