I think cts:document-fragment-query might help there. http://docs.marklogic.com/5.0doc/docapp.xqy#display.xqy?fname=http://pubs/5.0doc/apidoc/SearchBuiltins.xml&category=SearchBuiltins&function=cts:document-fragment-query
http://docs.marklogic.com/5.0doc/docapp.xqy#display.xqy?fname=http://pubs/5.0doc/xml/search-dev-guide/cts_query.xml%2388078 I'm not sure how to integrate that with the search API, though. I don't think any of the built-in constraint types will produce cts:document-fragment-query terms. But you could create a custom constraint. http://docs.marklogic.com/5.0doc/docapp.xqy#display.xqy?fname=http://pubs/5.0doc/xml/search-dev-guide/search-api.xml%2349750 Or you might call search:parse, then rewrite the query a bit, then call search:resolve with the final query. http://docs.marklogic.com/5.0doc/docapp.xqy#display.xqy?fname=http://pubs/5.0doc/xml/search-dev-guide/cts_query.xml -- Mike On 22 May 2012, at 13:33 , Murray, Gregory wrote: > Damon, > > Adding <searchable-expression> appears to have a big impact. Searching > without it and looking at xdmp:query-meters() output, I get this: > > <qm:link-cache-hits>8327</qm:link-cache-hits> > <qm:link-cache-misses>9921</qm:link-cache-misses> > > But if I add <searchable-expression>, I get this: > > <qm:link-cache-hits>123</qm:link-cache-hits> > <qm:link-cache-misses>6</qm:link-cache-misses> > > Unfortunately, in reality we don't want to restrict our searches only to > <page>, because each document includes metadata that must also be searchable > -- title, author name, etc. That requirement of our application would seem to > disallow the use of <searchable-expression>. Or is there some way around this > -- perhaps through the use of fields? > > Many thanks, > Greg > > > On May 22, 2012, at 3:05 PM, Damon Feldman wrote: > >> Greg, >> >> I suspect your algorithms are still searching for and finding snippets in >> entire documents/books, rather than pages. Fragmenting on page allows you to >> more efficiently load a smaller number of pages from a book, but you have to >> explicitly work with pages to see the benefit. If you load a full book >> document you'll get all the pages, and no benefit. >> >> Adding a searchable-expression to the search:search options may help: >> >> <searchable-expression xmlns:lib="http://digital.library.ptsem.edu/ia";> >> /lib:page >> </searchable-expression> >> >> (above code not tested) >> This tells search:search to look for *and return* pages. >> >> Yours, >> Damon >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of Murray, Gregory >> Sent: Tuesday, May 22, 2012 11:58 AM >> To: MarkLogic Developer Discussion >> Subject: Re: [MarkLogic Dev General] Efficient full-text searching and >> snippeting >> >> Mike, >> >> Thanks very much for your quick reply. Clearly there are lots of good ideas >> here. Our first plan of attack has been to set <page> as a fragment root and >> reindex the database. Our 50,000 documents now consist of 18 million >> fragments. Strangely, the situation appears to be even worse than before. >> The query resolution time seems reasonable, but snippet processing is now >> several times slower than it was before. We're using search:search(), and >> running a typical query in QC gives these metrics: >> >> <search:metrics> >> <search:query-resolution-time>PT0.093S</search:query-resolution-time> >> <search:facet-resolution-time>PT0.109S</search:facet-resolution-time> >> <search:snippet-resolution-time>PT6.582S</search:snippet-resolution-time> >> <search:total-time>PT6.926S</search:total-time> >> </search:metrics> >> >> which makes me wonder if the snippet process is still bringing in the entire >> document (book) rather than an individual fragment (page). >> >> In case they're relevant, I should clarify a few things: Our document >> structure includes a metadata element that stands outside of any <page> >> element. Also, a typical page will contain about 1000 UTF-8 characters, >> which is smaller than the optimal size range. And there are inevitably some >> blank pages and thus some empty fragments. For example: >> >> <doc xmlns="http://digital.library.ptsem.edu/ia";> >> <metadata> >> <id>1edhumanimmortalitjameuoft</id> >> <name>James, William, 1842-1910.</name> >> <title>Human immortality : two supposed objections to the doctrine</title> >> <date>1898</date> >> ... >> </metadata> >> <text type="ocr"> >> <page id="1edhumanimmortalitjameuoft_0001" absSeq="1"/> >> <page id="1edhumanimmortalitjameuoft_0002" absSeq="2"/> >> ... >> <page id="1edhumanimmortalitjameuoft_0011" absSeq="11" >> xmlns="http://digital.library.ptsem.edu/ia";> >> HUMAN IMMORTALITY >> Tis a matter unfortunately too often seen in history to call for much >> remark, >> ... >> </page> >> ... >> </text> >> </doc> >> >> Given that additional info -- do you have any thoughts on why snippeting >> would be even slower than before fragmenting on <page>? >> >> Many thanks, >> Greg >> >> >> On May 18, 2012, at 5:23 PM, Michael Blakeley wrote: >> >>> That can be a tricky question to answer. The bottleneck may be disk or >>> could also be CPU, since highlighting several MB of XML can use quite a few >>> cycles. If you can establish that the bottleneck is CPU, it may be enough >>> to extend your highlight code to set $cts:action to 'skip' or 'break' after >>> the first few matches. >>> >>> Otherwise your goal is to create documents or fragments that average >>> something like the 1-100 kB. XML is messy so we cannot hit that number >>> precisely, so do not take that target too literally. At the same time, make >>> sure that the fragments are well-aligned with search and update >>> requirements. >>> >>> The simplest thing that might possibly work would be to fragment by page. I >>> do not much like that answer myself: the sizes are reasonable, but >>> paragraphs and sentences may cross page boundaries. Since search terms are >>> indexed by fragment, that may cause problems. Imagine searching for "now is >>> the time", but one of the potential matches crosses a page boundary. If the >>> book is a single document and the page boundary is a phrase-through, there >>> is no problem. But if each page is its own fragment root, then neither page >>> will match the full query. Still, this option is worth considering and does >>> not require much extra work because each book still looks like a single XML >>> document. >>> >>> Another common approach is to treat each book as a directory >>> (http://blakeley.com/blogofile/2012/03/19/directory-assistance/), which >>> contains one document for every top-level structural element. Usually these >>> are chapters or something at a similar level. That limits the highlighting >>> problem to a reasonable amount of XML, and searching chapters independently >>> makes sense for most applications. Some chapters may be larger than others, >>> but still smaller than the entire book. Crucially, they should be small >>> enough to highlight comfortably. Reassembling the entire book can be as >>> simple as having a toc.xml or manifest.xml in each directory. >>> >>> Most book-oriented applications use one or the other of those approaches. >>> There are also some clever algorithmic tricks involving cts:walk and >>> cts:highlight together, and approaches using duplicated content for >>> passage-oriented or sentence-oriented search. But these can be challenging >>> to implement, and most applications can use one of the simpler designs >>> outlined above. >>> >>> -- Mike >>> >>> On 18 May 2012, at 13:48 , Murray, Gregory wrote: >>> >>>> Hello, >>>> >>>> I'm working on a project based on full-text XML documents. Each document >>>> is a digital representation of a book, containing a short metadata header >>>> but consisting mainly of <page> elements, each of which contains a single >>>> text node containing a blob of OCR'ed text. We now have 50,000 such >>>> documents and are hitting performance/scaling problems. After some >>>> profiling, we've concluded that searching isn't the problem but that >>>> snippeting is a major culprit. At ML World, my colleagues and I asked Mary >>>> Holstege about this, and she indicated that most likely I/O is the >>>> bottleneck, because each time a document is processed it must be pulled >>>> from disk in its entirety, since of course the <page> elements are not >>>> amenable to indexing. She suggested breaking the monolithic documents into >>>> pieces, perhaps one XML document per page. At that point, performing a >>>> search would involve the search itself but also post-processing to >>>> assemble the results back into book-based results for display t > o >> >> >>> th >>>> e user. >>>> >>>> I'm sure that would work well if I knew how to collate the page-level >>>> search results into book-level results. I'm sure there are readers of this >>>> list -- perhaps especially in publishing or government -- who work not >>>> just with smaller, data-centric documents but also with large, full-text >>>> documents. Is there a way to retain the integrity of the documents as >>>> complete books and still have good performance? If not, and if we were to >>>> split the books into pages, what's a good strategy for collating the >>>> page-level search results so they still appear to be book-level results to >>>> the user? >>>> >>>> Many thanks, >>>> Greg >>>> >>>> Gregory Murray >>>> Digital Library Application Developer >>>> Princeton Theological Seminary >>>> >>>> _______________________________________________ >>>> General mailing list >>>> [email protected] >>>> http://community.marklogic.com/mailman/listinfo/general >>>> >>> >>> _______________________________________________ >>> General mailing list >>> [email protected] >>> http://community.marklogic.com/mailman/listinfo/general >> >> _______________________________________________ >> General mailing list >> [email protected] >> http://community.marklogic.com/mailman/listinfo/general >> _______________________________________________ >> General mailing list >> [email protected] >> http://community.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > [email protected] > http://community.marklogic.com/mailman/listinfo/general > _______________________________________________ General mailing list [email protected] http://community.marklogic.com/mailman/listinfo/general
