Damon, Adding <searchable-expression> appears to have a big impact. Searching without it and looking at xdmp:query-meters() output, I get this:
<qm:link-cache-hits>8327</qm:link-cache-hits> <qm:link-cache-misses>9921</qm:link-cache-misses> But if I add <searchable-expression>, I get this: <qm:link-cache-hits>123</qm:link-cache-hits> <qm:link-cache-misses>6</qm:link-cache-misses> Unfortunately, in reality we don't want to restrict our searches only to <page>, because each document includes metadata that must also be searchable -- title, author name, etc. That requirement of our application would seem to disallow the use of <searchable-expression>. Or is there some way around this -- perhaps through the use of fields? Many thanks, Greg On May 22, 2012, at 3:05 PM, Damon Feldman wrote: > Greg, > > I suspect your algorithms are still searching for and finding snippets in > entire documents/books, rather than pages. Fragmenting on page allows you to > more efficiently load a smaller number of pages from a book, but you have to > explicitly work with pages to see the benefit. If you load a full book > document you'll get all the pages, and no benefit. > > Adding a searchable-expression to the search:search options may help: > > <searchable-expression xmlns:lib="http://digital.library.ptsem.edu/ia";> > /lib:page > </searchable-expression> > > (above code not tested) > This tells search:search to look for *and return* pages. > > Yours, > Damon > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Murray, Gregory > Sent: Tuesday, May 22, 2012 11:58 AM > To: MarkLogic Developer Discussion > Subject: Re: [MarkLogic Dev General] Efficient full-text searching and > snippeting > > Mike, > > Thanks very much for your quick reply. Clearly there are lots of good ideas > here. Our first plan of attack has been to set <page> as a fragment root and > reindex the database. Our 50,000 documents now consist of 18 million > fragments. Strangely, the situation appears to be even worse than before. The > query resolution time seems reasonable, but snippet processing is now several > times slower than it was before. We're using search:search(), and running a > typical query in QC gives these metrics: > > <search:metrics> > <search:query-resolution-time>PT0.093S</search:query-resolution-time> > <search:facet-resolution-time>PT0.109S</search:facet-resolution-time> > <search:snippet-resolution-time>PT6.582S</search:snippet-resolution-time> > <search:total-time>PT6.926S</search:total-time> > </search:metrics> > > which makes me wonder if the snippet process is still bringing in the entire > document (book) rather than an individual fragment (page). > > In case they're relevant, I should clarify a few things: Our document > structure includes a metadata element that stands outside of any <page> > element. Also, a typical page will contain about 1000 UTF-8 characters, which > is smaller than the optimal size range. And there are inevitably some blank > pages and thus some empty fragments. For example: > > <doc xmlns="http://digital.library.ptsem.edu/ia";> > <metadata> > <id>1edhumanimmortalitjameuoft</id> > <name>James, William, 1842-1910.</name> > <title>Human immortality : two supposed objections to the doctrine</title> > <date>1898</date> > ... > </metadata> > <text type="ocr"> > <page id="1edhumanimmortalitjameuoft_0001" absSeq="1"/> > <page id="1edhumanimmortalitjameuoft_0002" absSeq="2"/> > ... > <page id="1edhumanimmortalitjameuoft_0011" absSeq="11" > xmlns="http://digital.library.ptsem.edu/ia";> > HUMAN IMMORTALITY > Tis a matter unfortunately too often seen in history to call for much > remark, > ... > </page> > ... > </text> > </doc> > > Given that additional info -- do you have any thoughts on why snippeting > would be even slower than before fragmenting on <page>? > > Many thanks, > Greg > > > On May 18, 2012, at 5:23 PM, Michael Blakeley wrote: > >> That can be a tricky question to answer. The bottleneck may be disk or could >> also be CPU, since highlighting several MB of XML can use quite a few >> cycles. If you can establish that the bottleneck is CPU, it may be enough to >> extend your highlight code to set $cts:action to 'skip' or 'break' after the >> first few matches. >> >> Otherwise your goal is to create documents or fragments that average >> something like the 1-100 kB. XML is messy so we cannot hit that number >> precisely, so do not take that target too literally. At the same time, make >> sure that the fragments are well-aligned with search and update requirements. >> >> The simplest thing that might possibly work would be to fragment by page. I >> do not much like that answer myself: the sizes are reasonable, but >> paragraphs and sentences may cross page boundaries. Since search terms are >> indexed by fragment, that may cause problems. Imagine searching for "now is >> the time", but one of the potential matches crosses a page boundary. If the >> book is a single document and the page boundary is a phrase-through, there >> is no problem. But if each page is its own fragment root, then neither page >> will match the full query. Still, this option is worth considering and does >> not require much extra work because each book still looks like a single XML >> document. >> >> Another common approach is to treat each book as a directory >> (http://blakeley.com/blogofile/2012/03/19/directory-assistance/), which >> contains one document for every top-level structural element. Usually these >> are chapters or something at a similar level. That limits the highlighting >> problem to a reasonable amount of XML, and searching chapters independently >> makes sense for most applications. Some chapters may be larger than others, >> but still smaller than the entire book. Crucially, they should be small >> enough to highlight comfortably. Reassembling the entire book can be as >> simple as having a toc.xml or manifest.xml in each directory. >> >> Most book-oriented applications use one or the other of those approaches. >> There are also some clever algorithmic tricks involving cts:walk and >> cts:highlight together, and approaches using duplicated content for >> passage-oriented or sentence-oriented search. But these can be challenging >> to implement, and most applications can use one of the simpler designs >> outlined above. >> >> -- Mike >> >> On 18 May 2012, at 13:48 , Murray, Gregory wrote: >> >>> Hello, >>> >>> I'm working on a project based on full-text XML documents. Each document is >>> a digital representation of a book, containing a short metadata header but >>> consisting mainly of <page> elements, each of which contains a single text >>> node containing a blob of OCR'ed text. We now have 50,000 such documents >>> and are hitting performance/scaling problems. After some profiling, we've >>> concluded that searching isn't the problem but that snippeting is a major >>> culprit. At ML World, my colleagues and I asked Mary Holstege about this, >>> and she indicated that most likely I/O is the bottleneck, because each time >>> a document is processed it must be pulled from disk in its entirety, since >>> of course the <page> elements are not amenable to indexing. She suggested >>> breaking the monolithic documents into pieces, perhaps one XML document per >>> page. At that point, performing a search would involve the search itself >>> but also post-processing to assemble the results back into book-based >>> results for display t o > > >> th >>> e user. >>> >>> I'm sure that would work well if I knew how to collate the page-level >>> search results into book-level results. I'm sure there are readers of this >>> list -- perhaps especially in publishing or government -- who work not just >>> with smaller, data-centric documents but also with large, full-text >>> documents. Is there a way to retain the integrity of the documents as >>> complete books and still have good performance? If not, and if we were to >>> split the books into pages, what's a good strategy for collating the >>> page-level search results so they still appear to be book-level results to >>> the user? >>> >>> Many thanks, >>> Greg >>> >>> Gregory Murray >>> Digital Library Application Developer >>> Princeton Theological Seminary >>> >>> _______________________________________________ >>> General mailing list >>> [email protected] >>> http://community.marklogic.com/mailman/listinfo/general >>> >> >> _______________________________________________ >> General mailing list >> [email protected] >> http://community.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > [email protected] > http://community.marklogic.com/mailman/listinfo/general > _______________________________________________ > General mailing list > [email protected] > http://community.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://community.marklogic.com/mailman/listinfo/general
