Mike,
Thanks very much for these pointers. In your experiment, would your database
configuration include an index on the <page> element? This may be a very
elementary question. I've been assuming that it doesn't make sense to create a
range index -- or any other kind of index or lexicon -- for <page> because the
content is too long and various. Is that assumption valid?
Many thanks,
Greg
On May 22, 2012, at 1:35 PM, Michael Blakeley wrote:
> Chances are the snippet code is bringing in other fragments more often than
> is optimal.
>
> In situations like this I try to design little experiments - or test cases -
> that will narrow down the source of the problem. Here is an example:
>
> (: generate content :)
> xquery version "1.0-ml";
>
> declare function local:new($i as xs:integer)
> {
> xdmp:document-insert(
> concat('test/', $i),
> <doc xmlns="http://digital.library.ptsem.edu/ia";>
> <metadata>
> <id>test-{ $i }</id>
> <name>Test Author {$i}.</name>
> <title>Test Title { $i }</title>
> <date>{
> xs:date('1999-01-01')
> + xs:dayTimeDuration(concat('P', $i, 'D')) }</date>
> </metadata>
> {
> for $pn in 1 to 100
> return element page {
> for $word in 1 to 50
> return xdmp:integer-to-hex(xdmp:random(255)) }
> }
> </doc>)
> };
>
> local:new(1 to 100)
>
> (: search :)
> import module namespace search="http://marklogic.com/appservices/search";
> at "/MarkLogic/appservices/search/search.xqy";
>
> search:search(
> 'aa',
> <options xmlns="http://marklogic.com/appservices/search";>
> <searchable-expression
> xmlns:ia="http://digital.library.ptsem.edu/ia";>
> //ia:page</searchable-expression>
> </options>),
> xdmp:query-meters()
>
> I'm sure this test case is much too simple to reproduce your problem, but it
> should give you a starting point. The query-meters output, at the end,
> reveals some interesting information right away:
>
> ...
> <qm:link-cache-hits>111</qm:link-cache-hits>
> <qm:link-cache-misses>8</qm:link-cache-misses>
> ...
> <qm:fragments>
> <qm:fragment>
> <qm:root xmlns="http://digital.library.ptsem.edu/ia";>page</qm:root>
> <qm:expanded-tree-cache-hits>10</qm:expanded-tree-cache-hits>
> <qm:expanded-tree-cache-misses>0</qm:expanded-tree-cache-misses>
> </qm:fragment>
> <qm:fragment>
> <qm:root xmlns="http://digital.library.ptsem.edu/ia";>doc</qm:root>
> <qm:expanded-tree-cache-hits>119</qm:expanded-tree-cache-hits>
> <qm:expanded-tree-cache-misses>0</qm:expanded-tree-cache-misses>
> </qm:fragment>
> </qm:fragments>
> ...
>
> Since I used randomly-generated content, your numbers might be slightly
> different. But the link-cache hits and misses tell us that the query is
> traversing fragment boundaries many times. With a page-length of 10 and 119
> accesses, every snippet causes an average of 12 fragment accesses. We can
> also see that there were 119 expanded-tree cache hits on the /ia:doc
> fragment, which is the document root. So it seems clear that the
> search:search highlighting is starting in the ia:page fragment, but then
> accesses the root fragment for some reason.
>
> This turns out to be slower than your original configuration because
> following fragment pointers has some cost. So creating more fragments only
> makes sense when it helps you to avoid loading most of the fragments. That
> was exactly the idea behind fragmenting on 'ia:page', but it looks like the
> search snippet code has not been optimized for this use-case.
>
> Writing a custom snippet function might help, as documented at
> http://docs.marklogic.com/5.0doc/docapp.xqy#display.xqy?fname=http://pubs/5.0doc/xml/search-dev-guide/search-api.xml%2358295
>
> To test this idea, I added a search option:
>
> <transform-results apply="raw"/>
>
> This returns the raw ia:page as the snippet. The query-meters output still
> shows 79 link-cache accesses. Let's try a more extreme option:
>
> <transform-results apply="empty-snippet"/>
>
> With this option I see 49 accesses to the ia:doc fragment. That is better
> that 119, and may be close to optimal.
>
> Even a simple cts:search on //ia:page shows 39 accesses. I'm not sure why
> those are necessary: it might be worth your while to delve into that with
> MarkLogic support.
>
> -- Mike
>
> On 22 May 2012, at 08:58 , Murray, Gregory wrote:
>
>> Mike,
>>
>> Thanks very much for your quick reply. Clearly there are lots of good ideas
>> here. Our first plan of attack has been to set <page> as a fragment root and
>> reindex the database. Our 50,000 documents now consist of 18 million
>> fragments. Strangely, the situation appears to be even worse than before.
>> The query resolution time seems reasonable, but snippet processing is now
>> several times slower than it was before. We're using search:search(), and
>> running a typical query in QC gives these metrics:
>>
>> <search:metrics>
>> <search:query-resolution-time>PT0.093S</search:query-resolution-time>
>> <search:facet-resolution-time>PT0.109S</search:facet-resolution-time>
>> <search:snippet-resolution-time>PT6.582S</search:snippet-resolution-time>
>> <search:total-time>PT6.926S</search:total-time>
>> </search:metrics>
>>
>> which makes me wonder if the snippet process is still bringing in the entire
>> document (book) rather than an individual fragment (page).
>>
>> In case they're relevant, I should clarify a few things: Our document
>> structure includes a metadata element that stands outside of any <page>
>> element. Also, a typical page will contain about 1000 UTF-8 characters,
>> which is smaller than the optimal size range. And there are inevitably some
>> blank pages and thus some empty fragments. For example:
>>
>> <doc xmlns="http://digital.library.ptsem.edu/ia";>
>> <metadata>
>> <id>1edhumanimmortalitjameuoft</id>
>> <name>James, William, 1842-1910.</name>
>> <title>Human immortality : two supposed objections to the doctrine</title>
>> <date>1898</date>
>> ...
>> </metadata>
>> <text type="ocr">
>> <page id="1edhumanimmortalitjameuoft_0001" absSeq="1"/>
>> <page id="1edhumanimmortalitjameuoft_0002" absSeq="2"/>
>> ...
>> <page id="1edhumanimmortalitjameuoft_0011" absSeq="11"
>> xmlns="http://digital.library.ptsem.edu/ia";>
>> HUMAN IMMORTALITY
>> Tis a matter unfortunately too often seen in history to call for much
>> remark,
>> ...
>> </page>
>> ...
>> </text>
>> </doc>
>>
>> Given that additional info -- do you have any thoughts on why snippeting
>> would be even slower than before fragmenting on <page>?
>>
>> Many thanks,
>> Greg
>>
>>
>> On May 18, 2012, at 5:23 PM, Michael Blakeley wrote:
>>
>>> That can be a tricky question to answer. The bottleneck may be disk or
>>> could also be CPU, since highlighting several MB of XML can use quite a few
>>> cycles. If you can establish that the bottleneck is CPU, it may be enough
>>> to extend your highlight code to set $cts:action to 'skip' or 'break' after
>>> the first few matches.
>>>
>>> Otherwise your goal is to create documents or fragments that average
>>> something like the 1-100 kB. XML is messy so we cannot hit that number
>>> precisely, so do not take that target too literally. At the same time, make
>>> sure that the fragments are well-aligned with search and update
>>> requirements.
>>>
>>> The simplest thing that might possibly work would be to fragment by page. I
>>> do not much like that answer myself: the sizes are reasonable, but
>>> paragraphs and sentences may cross page boundaries. Since search terms are
>>> indexed by fragment, that may cause problems. Imagine searching for "now is
>>> the time", but one of the potential matches crosses a page boundary. If the
>>> book is a single document and the page boundary is a phrase-through, there
>>> is no problem. But if each page is its own fragment root, then neither page
>>> will match the full query. Still, this option is worth considering and does
>>> not require much extra work because each book still looks like a single XML
>>> document.
>>>
>>> Another common approach is to treat each book as a directory
>>> (http://blakeley.com/blogofile/2012/03/19/directory-assistance/), which
>>> contains one document for every top-level structural element. Usually these
>>> are chapters or something at a similar level. That limits the highlighting
>>> problem to a reasonable amount of XML, and searching chapters independently
>>> makes sense for most applications. Some chapters may be larger than others,
>>> but still smaller than the entire book. Crucially, they should be small
>>> enough to highlight comfortably. Reassembling the entire book can be as
>>> simple as having a toc.xml or manifest.xml in each directory.
>>>
>>> Most book-oriented applications use one or the other of those approaches.
>>> There are also some clever algorithmic tricks involving cts:walk and
>>> cts:highlight together, and approaches using duplicated content for
>>> passage-oriented or sentence-oriented search. But these can be challenging
>>> to implement, and most applications can use one of the simpler designs
>>> outlined above.
>>>
>>> -- Mike
>>>
>>> On 18 May 2012, at 13:48 , Murray, Gregory wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm working on a project based on full-text XML documents. Each document
>>>> is a digital representation of a book, containing a short metadata header
>>>> but consisting mainly of <page> elements, each of which contains a single
>>>> text node containing a blob of OCR'ed text. We now have 50,000 such
>>>> documents and are hitting performance/scaling problems. After some
>>>> profiling, we've concluded that searching isn't the problem but that
>>>> snippeting is a major culprit. At ML World, my colleagues and I asked Mary
>>>> Holstege about this, and she indicated that most likely I/O is the
>>>> bottleneck, because each time a document is processed it must be pulled
>>>> from disk in its entirety, since of course the <page> elements are not
>>>> amenable to indexing. She suggested breaking the monolithic documents into
>>>> pieces, perhaps one XML document per page. At that point, performing a
>>>> search would involve the search itself but also post-processing to
>>>> assemble the results back into book-based results for display
t
> o
>>
>>> th
>>>> e user.
>>>>
>>>> I'm sure that would work well if I knew how to collate the page-level
>>>> search results into book-level results. I'm sure there are readers of this
>>>> list -- perhaps especially in publishing or government -- who work not
>>>> just with smaller, data-centric documents but also with large, full-text
>>>> documents. Is there a way to retain the integrity of the documents as
>>>> complete books and still have good performance? If not, and if we were to
>>>> split the books into pages, what's a good strategy for collating the
>>>> page-level search results so they still appear to be book-level results to
>>>> the user?
>>>>
>>>> Many thanks,
>>>> Greg
>>>>
>>>> Gregory Murray
>>>> Digital Library Application Developer
>>>> Princeton Theological Seminary
>>>>
>>>> _______________________________________________
>>>> General mailing list
>>>> [email protected]
>>>> http://community.marklogic.com/mailman/listinfo/general
>>>>
>>>
>>> _______________________________________________
>>> General mailing list
>>> [email protected]
>>> http://community.marklogic.com/mailman/listinfo/general
>>
>> _______________________________________________
>> General mailing list
>> [email protected]
>> http://community.marklogic.com/mailman/listinfo/general
>>
>
> _______________________________________________
> General mailing list
> [email protected]
> http://community.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected]
http://community.marklogic.com/mailman/listinfo/general
