Chances are the snippet code is bringing in other fragments more often than is
optimal.
In situations like this I try to design little experiments - or test cases -
that will narrow down the source of the problem. Here is an example:
(: generate content :)
xquery version "1.0-ml";
declare function local:new($i as xs:integer)
{
xdmp:document-insert(
concat('test/', $i),
<doc xmlns="http://digital.library.ptsem.edu/ia";>
<metadata>
<id>test-{ $i }</id>
<name>Test Author {$i}.</name>
<title>Test Title { $i }</title>
<date>{
xs:date('1999-01-01')
+ xs:dayTimeDuration(concat('P', $i, 'D')) }</date>
</metadata>
{
for $pn in 1 to 100
return element page {
for $word in 1 to 50
return xdmp:integer-to-hex(xdmp:random(255)) }
}
</doc>)
};
local:new(1 to 100)
(: search :)
import module namespace search="http://marklogic.com/appservices/search";
at "/MarkLogic/appservices/search/search.xqy";
search:search(
'aa',
<options xmlns="http://marklogic.com/appservices/search";>
<searchable-expression
xmlns:ia="http://digital.library.ptsem.edu/ia";>
//ia:page</searchable-expression>
</options>),
xdmp:query-meters()
I'm sure this test case is much too simple to reproduce your problem, but it
should give you a starting point. The query-meters output, at the end, reveals
some interesting information right away:
...
<qm:link-cache-hits>111</qm:link-cache-hits>
<qm:link-cache-misses>8</qm:link-cache-misses>
...
<qm:fragments>
<qm:fragment>
<qm:root xmlns="http://digital.library.ptsem.edu/ia";>page</qm:root>
<qm:expanded-tree-cache-hits>10</qm:expanded-tree-cache-hits>
<qm:expanded-tree-cache-misses>0</qm:expanded-tree-cache-misses>
</qm:fragment>
<qm:fragment>
<qm:root xmlns="http://digital.library.ptsem.edu/ia";>doc</qm:root>
<qm:expanded-tree-cache-hits>119</qm:expanded-tree-cache-hits>
<qm:expanded-tree-cache-misses>0</qm:expanded-tree-cache-misses>
</qm:fragment>
</qm:fragments>
...
Since I used randomly-generated content, your numbers might be slightly
different. But the link-cache hits and misses tell us that the query is
traversing fragment boundaries many times. With a page-length of 10 and 119
accesses, every snippet causes an average of 12 fragment accesses. We can also
see that there were 119 expanded-tree cache hits on the /ia:doc fragment, which
is the document root. So it seems clear that the search:search highlighting is
starting in the ia:page fragment, but then accesses the root fragment for some
reason.
This turns out to be slower than your original configuration because following
fragment pointers has some cost. So creating more fragments only makes sense
when it helps you to avoid loading most of the fragments. That was exactly the
idea behind fragmenting on 'ia:page', but it looks like the search snippet code
has not been optimized for this use-case.
Writing a custom snippet function might help, as documented at
http://docs.marklogic.com/5.0doc/docapp.xqy#display.xqy?fname=http://pubs/5.0doc/xml/search-dev-guide/search-api.xml%2358295
To test this idea, I added a search option:
<transform-results apply="raw"/>
This returns the raw ia:page as the snippet. The query-meters output still
shows 79 link-cache accesses. Let's try a more extreme option:
<transform-results apply="empty-snippet"/>
With this option I see 49 accesses to the ia:doc fragment. That is better that
119, and may be close to optimal.
Even a simple cts:search on //ia:page shows 39 accesses. I'm not sure why those
are necessary: it might be worth your while to delve into that with MarkLogic
support.
-- Mike
On 22 May 2012, at 08:58 , Murray, Gregory wrote:
> Mike,
>
> Thanks very much for your quick reply. Clearly there are lots of good ideas
> here. Our first plan of attack has been to set <page> as a fragment root and
> reindex the database. Our 50,000 documents now consist of 18 million
> fragments. Strangely, the situation appears to be even worse than before. The
> query resolution time seems reasonable, but snippet processing is now several
> times slower than it was before. We're using search:search(), and running a
> typical query in QC gives these metrics:
>
> <search:metrics>
> <search:query-resolution-time>PT0.093S</search:query-resolution-time>
> <search:facet-resolution-time>PT0.109S</search:facet-resolution-time>
> <search:snippet-resolution-time>PT6.582S</search:snippet-resolution-time>
> <search:total-time>PT6.926S</search:total-time>
> </search:metrics>
>
> which makes me wonder if the snippet process is still bringing in the entire
> document (book) rather than an individual fragment (page).
>
> In case they're relevant, I should clarify a few things: Our document
> structure includes a metadata element that stands outside of any <page>
> element. Also, a typical page will contain about 1000 UTF-8 characters, which
> is smaller than the optimal size range. And there are inevitably some blank
> pages and thus some empty fragments. For example:
>
> <doc xmlns="http://digital.library.ptsem.edu/ia";>
> <metadata>
> <id>1edhumanimmortalitjameuoft</id>
> <name>James, William, 1842-1910.</name>
> <title>Human immortality : two supposed objections to the doctrine</title>
> <date>1898</date>
> ...
> </metadata>
> <text type="ocr">
> <page id="1edhumanimmortalitjameuoft_0001" absSeq="1"/>
> <page id="1edhumanimmortalitjameuoft_0002" absSeq="2"/>
> ...
> <page id="1edhumanimmortalitjameuoft_0011" absSeq="11"
> xmlns="http://digital.library.ptsem.edu/ia";>
> HUMAN IMMORTALITY
> Tis a matter unfortunately too often seen in history to call for much
> remark,
> ...
> </page>
> ...
> </text>
> </doc>
>
> Given that additional info -- do you have any thoughts on why snippeting
> would be even slower than before fragmenting on <page>?
>
> Many thanks,
> Greg
>
>
> On May 18, 2012, at 5:23 PM, Michael Blakeley wrote:
>
>> That can be a tricky question to answer. The bottleneck may be disk or could
>> also be CPU, since highlighting several MB of XML can use quite a few
>> cycles. If you can establish that the bottleneck is CPU, it may be enough to
>> extend your highlight code to set $cts:action to 'skip' or 'break' after the
>> first few matches.
>>
>> Otherwise your goal is to create documents or fragments that average
>> something like the 1-100 kB. XML is messy so we cannot hit that number
>> precisely, so do not take that target too literally. At the same time, make
>> sure that the fragments are well-aligned with search and update requirements.
>>
>> The simplest thing that might possibly work would be to fragment by page. I
>> do not much like that answer myself: the sizes are reasonable, but
>> paragraphs and sentences may cross page boundaries. Since search terms are
>> indexed by fragment, that may cause problems. Imagine searching for "now is
>> the time", but one of the potential matches crosses a page boundary. If the
>> book is a single document and the page boundary is a phrase-through, there
>> is no problem. But if each page is its own fragment root, then neither page
>> will match the full query. Still, this option is worth considering and does
>> not require much extra work because each book still looks like a single XML
>> document.
>>
>> Another common approach is to treat each book as a directory
>> (http://blakeley.com/blogofile/2012/03/19/directory-assistance/), which
>> contains one document for every top-level structural element. Usually these
>> are chapters or something at a similar level. That limits the highlighting
>> problem to a reasonable amount of XML, and searching chapters independently
>> makes sense for most applications. Some chapters may be larger than others,
>> but still smaller than the entire book. Crucially, they should be small
>> enough to highlight comfortably. Reassembling the entire book can be as
>> simple as having a toc.xml or manifest.xml in each directory.
>>
>> Most book-oriented applications use one or the other of those approaches.
>> There are also some clever algorithmic tricks involving cts:walk and
>> cts:highlight together, and approaches using duplicated content for
>> passage-oriented or sentence-oriented search. But these can be challenging
>> to implement, and most applications can use one of the simpler designs
>> outlined above.
>>
>> -- Mike
>>
>> On 18 May 2012, at 13:48 , Murray, Gregory wrote:
>>
>>> Hello,
>>>
>>> I'm working on a project based on full-text XML documents. Each document is
>>> a digital representation of a book, containing a short metadata header but
>>> consisting mainly of <page> elements, each of which contains a single text
>>> node containing a blob of OCR'ed text. We now have 50,000 such documents
>>> and are hitting performance/scaling problems. After some profiling, we've
>>> concluded that searching isn't the problem but that snippeting is a major
>>> culprit. At ML World, my colleagues and I asked Mary Holstege about this,
>>> and she indicated that most likely I/O is the bottleneck, because each time
>>> a document is processed it must be pulled from disk in its entirety, since
>>> of course the <page> elements are not amenable to indexing. She suggested
>>> breaking the monolithic documents into pieces, perhaps one XML document per
>>> page. At that point, performing a search would involve the search itself
>>> but also post-processing to assemble the results back into book-based
>>> results for display t
o
>
>> th
>>> e user.
>>>
>>> I'm sure that would work well if I knew how to collate the page-level
>>> search results into book-level results. I'm sure there are readers of this
>>> list -- perhaps especially in publishing or government -- who work not just
>>> with smaller, data-centric documents but also with large, full-text
>>> documents. Is there a way to retain the integrity of the documents as
>>> complete books and still have good performance? If not, and if we were to
>>> split the books into pages, what's a good strategy for collating the
>>> page-level search results so they still appear to be book-level results to
>>> the user?
>>>
>>> Many thanks,
>>> Greg
>>>
>>> Gregory Murray
>>> Digital Library Application Developer
>>> Princeton Theological Seminary
>>>
>>> _______________________________________________
>>> General mailing list
>>> [email protected]
>>> http://community.marklogic.com/mailman/listinfo/general
>>>
>>
>> _______________________________________________
>> General mailing list
>> [email protected]
>> http://community.marklogic.com/mailman/listinfo/general
>
> _______________________________________________
> General mailing list
> [email protected]
> http://community.marklogic.com/mailman/listinfo/general
>
_______________________________________________
General mailing list
[email protected]
http://community.marklogic.com/mailman/listinfo/general
