Mike's approach looks right to me. search:parse() will return an XML
representation of a query, which can be manipulated to add the document query.
Note that search:parse() returns an XML representation of a cts:query. To
convert from XML to a "real" cts:query use cts:query($my-xml-representation).
To convert from a cts:query() to the XML form embed it in xml:
<x>{$my-cts-query}</x>/node().
Here are two equivalent examples with the first manipulating the query in XML
and the second converting to cts:query items and manipulating those.
import module namespace search = "http://marklogic.com/appservices/search";
at "/MarkLogic/appservices/search/search.xqy";
let $full-query-as-xml :=
<cts:and-query>
{cts:document-fragment-query(cts:element-value-query(xs:QName("author"),
"Hemmingway"))}
{search:parse("foo and bar")}
</cts:and-query>
return search:resolve($full-query-as-xml)
-----
import module namespace search = "http://marklogic.com/appservices/search"; at
"/MarkLogic/appservices/search/search.xqy";
let $query-with-author :=
cts:and-query((
cts:document-fragment-query(cts:element-value-query(xs:QName("author"),
"Hemmingway")),
cts:query(
search:parse("foo and bar")
)
))
let $full-query-as-xml := <x>{$query-with-author}</x>/node()
return search:resolve($full-query-as-xml)
Yours,
Damon
________________________________________
From: [email protected]
[[email protected]] On Behalf Of Michael Blakeley
[[email protected]]
Sent: Tuesday, May 22, 2012 5:12 PM
To: MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] Efficient full-text searching
and snippeting
I think cts:document-fragment-query might help there.
http://docs.marklogic.com/5.0doc/docapp.xqy#display.xqy?fname=http://pubs/5.0doc/apidoc/SearchBuiltins.xml&category=SearchBuiltins&function=cts:document-fragment-query
http://docs.marklogic.com/5.0doc/docapp.xqy#display.xqy?fname=http://pubs/5.0doc/xml/search-dev-guide/cts_query.xml%2388078
I'm not sure how to integrate that with the search API, though. I don't think
any of the built-in constraint types will produce cts:document-fragment-query
terms. But you could create a custom constraint.
http://docs.marklogic.com/5.0doc/docapp.xqy#display.xqy?fname=http://pubs/5.0doc/xml/search-dev-guide/search-api.xml%2349750
Or you might call search:parse, then rewrite the query a bit, then call
search:resolve with the final query.
http://docs.marklogic.com/5.0doc/docapp.xqy#display.xqy?fname=http://pubs/5.0doc/xml/search-dev-guide/cts_query.xml
-- Mike
On 22 May 2012, at 13:33 , Murray, Gregory wrote:
> Damon,
>
> Adding <searchable-expression> appears to have a big impact. Searching
> without it and looking at xdmp:query-meters() output, I get this:
>
> <qm:link-cache-hits>8327</qm:link-cache-hits>
> <qm:link-cache-misses>9921</qm:link-cache-misses>
>
> But if I add <searchable-expression>, I get this:
>
> <qm:link-cache-hits>123</qm:link-cache-hits>
> <qm:link-cache-misses>6</qm:link-cache-misses>
>
> Unfortunately, in reality we don't want to restrict our searches only to
> <page>, because each document includes metadata that must also be searchable
> -- title, author name, etc. That requirement of our application would seem to
> disallow the use of <searchable-expression>. Or is there some way around this
> -- perhaps through the use of fields?
>
> Many thanks,
> Greg
>
>
> On May 22, 2012, at 3:05 PM, Damon Feldman wrote:
>
>> Greg,
>>
>> I suspect your algorithms are still searching for and finding snippets in
>> entire documents/books, rather than pages. Fragmenting on page allows you to
>> more efficiently load a smaller number of pages from a book, but you have to
>> explicitly work with pages to see the benefit. If you load a full book
>> document you'll get all the pages, and no benefit.
>>
>> Adding a searchable-expression to the search:search options may help:
>>
>> <searchable-expression xmlns:lib="http://digital.library.ptsem.edu/ia";>
>> /lib:page
>> </searchable-expression>
>>
>> (above code not tested)
>> This tells search:search to look for *and return* pages.
>>
>> Yours,
>> Damon
>>
>> -----Original Message-----
>> From: [email protected]
>> [mailto:[email protected]] On Behalf Of Murray, Gregory
>> Sent: Tuesday, May 22, 2012 11:58 AM
>> To: MarkLogic Developer Discussion
>> Subject: Re: [MarkLogic Dev General] Efficient full-text searching and
>> snippeting
>>
>> Mike,
>>
>> Thanks very much for your quick reply. Clearly there are lots of good ideas
>> here. Our first plan of attack has been to set <page> as a fragment root and
>> reindex the database. Our 50,000 documents now consist of 18 million
>> fragments. Strangely, the situation appears to be even worse than before.
>> The query resolution time seems reasonable, but snippet processing is now
>> several times slower than it was before. We're using search:search(), and
>> running a typical query in QC gives these metrics:
>>
>> <search:metrics>
>> <search:query-resolution-time>PT0.093S</search:query-resolution-time>
>> <search:facet-resolution-time>PT0.109S</search:facet-resolution-time>
>> <search:snippet-resolution-time>PT6.582S</search:snippet-resolution-time>
>> <search:total-time>PT6.926S</search:total-time>
>> </search:metrics>
>>
>> which makes me wonder if the snippet process is still bringing in the entire
>> document (book) rather than an individual fragment (page).
>>
>> In case they're relevant, I should clarify a few things: Our document
>> structure includes a metadata element that stands outside of any <page>
>> element. Also, a typical page will contain about 1000 UTF-8 characters,
>> which is smaller than the optimal size range. And there are inevitably some
>> blank pages and thus some empty fragments. For example:
>>
>> <doc xmlns="http://digital.library.ptsem.edu/ia";>
>> <metadata>
>> <id>1edhumanimmortalitjameuoft</id>
>> <name>James, William, 1842-1910.</name>
>> <title>Human immortality : two supposed objections to the doctrine</title>
>> <date>1898</date>
>> ...
>> </metadata>
>> <text type="ocr">
>> <page id="1edhumanimmortalitjameuoft_0001" absSeq="1"/>
>> <page id="1edhumanimmortalitjameuoft_0002" absSeq="2"/>
>> ...
>> <page id="1edhumanimmortalitjameuoft_0011" absSeq="11"
>> xmlns="http://digital.library.ptsem.edu/ia";>
>> HUMAN IMMORTALITY
>> Tis a matter unfortunately too often seen in history to call for much
>> remark,
>> ...
>> </page>
>> ...
>> </text>
>> </doc>
>>
>> Given that additional info -- do you have any thoughts on why snippeting
>> would be even slower than before fragmenting on <page>?
>>
>> Many thanks,
>> Greg
>>
>>
>> On May 18, 2012, at 5:23 PM, Michael Blakeley wrote:
>>
>>> That can be a tricky question to answer. The bottleneck may be disk or
>>> could also be CPU, since highlighting several MB of XML can use quite a few
>>> cycles. If you can establish that the bottleneck is CPU, it may be enough
>>> to extend your highlight code to set $cts:action to 'skip' or 'break' after
>>> the first few matches.
>>>
>>> Otherwise your goal is to create documents or fragments that average
>>> something like the 1-100 kB. XML is messy so we cannot hit that number
>>> precisely, so do not take that target too literally. At the same time, make
>>> sure that the fragments are well-aligned with search and update
>>> requirements.
>>>
>>> The simplest thing that might possibly work would be to fragment by page. I
>>> do not much like that answer myself: the sizes are reasonable, but
>>> paragraphs and sentences may cross page boundaries. Since search terms are
>>> indexed by fragment, that may cause problems. Imagine searching for "now is
>>> the time", but one of the potential matches crosses a page boundary. If the
>>> book is a single document and the page boundary is a phrase-through, there
>>> is no problem. But if each page is its own fragment root, then neither page
>>> will match the full query. Still, this option is worth considering and does
>>> not require much extra work because each book still looks like a single XML
>>> document.
>>>
>>> Another common approach is to treat each book as a directory
>>> (http://blakeley.com/blogofile/2012/03/19/directory-assistance/), which
>>> contains one document for every top-level structural element. Usually these
>>> are chapters or something at a similar level. That limits the highlighting
>>> problem to a reasonable amount of XML, and searching chapters independently
>>> makes sense for most applications. Some chapters may be larger than others,
>>> but still smaller than the entire book. Crucially, they should be small
>>> enough to highlight comfortably. Reassembling the entire book can be as
>>> simple as having a toc.xml or manifest.xml in each directory.
>>>
>>> Most book-oriented applications use one or the other of those approaches.
>>> There are also some clever algorithmic tricks involving cts:walk and
>>> cts:highlight together, and approaches using duplicated content for
>>> passage-oriented or sentence-oriented search. But these can be challenging
>>> to implement, and most applications can use one of the simpler designs
>>> outlined above.
>>>
>>> -- Mike
>>>
>>> On 18 May 2012, at 13:48 , Murray, Gregory wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm working on a project based on full-text XML documents. Each document
>>>> is a digital representation of a book, containing a short metadata header
>>>> but consisting mainly of <page> elements, each of which contains a single
>>>> text node containing a blob of OCR'ed text. We now have 50,000 such
>>>> documents and are hitting performance/scaling problems. After some
>>>> profiling, we've concluded that searching isn't the problem but that
>>>> snippeting is a major culprit. At ML World, my colleagues and I asked Mary
>>>> Holstege about this, and she indicated that most likely I/O is the
>>>> bottleneck, because each time a document is processed it must be pulled
>>>> from disk in its entirety, since of course the <page> elements are not
>>>> amenable to indexing. She suggested breaking the monolithic documents into
>>>> pieces, perhaps one XML document per page. At that point, performing a
>>>> search would involve the search itself but also post-processing to
>>>> assemble the results back into book-based results for display
t
> o
>>
>>
>>> th
>>>> e user.
>>>>
>>>> I'm sure that would work well if I knew how to collate the page-level
>>>> search results into book-level results. I'm sure there are readers of this
>>>> list -- perhaps especially in publishing or government -- who work not
>>>> just with smaller, data-centric documents but also with large, full-text
>>>> documents. Is there a way to retain the integrity of the documents as
>>>> complete books and still have good performance? If not, and if we were to
>>>> split the books into pages, what's a good strategy for collating the
>>>> page-level search results so they still appear to be book-level results to
>>>> the user?
>>>>
>>>> Many thanks,
>>>> Greg
>>>>
>>>> Gregory Murray
>>>> Digital Library Application Developer
>>>> Princeton Theological Seminary
>>>>
>>>> _______________________________________________
>>>> General mailing list
>>>> [email protected]
>>>> http://community.marklogic.com/mailman/listinfo/general
>>>>
>>>
>>> _______________________________________________
>>> General mailing list
>>> [email protected]
>>> http://community.marklogic.com/mailman/listinfo/general
>>
>> _______________________________________________
>> General mailing list
>> [email protected]
>> http://community.marklogic.com/mailman/listinfo/general
>> _______________________________________________
>> General mailing list
>> [email protected]
>> http://community.marklogic.com/mailman/listinfo/general
>
> _______________________________________________
> General mailing list
> [email protected]
> http://community.marklogic.com/mailman/listinfo/general
>
_______________________________________________
General mailing list
[email protected]
http://community.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected]
http://community.marklogic.com/mailman/listinfo/general
