Yes, that's correct. Range indexes are for simple elements, and page is a
complex element. Range indexes are used for inequality operators, sorting,
facets, and that sort of thing.
-- Mike
On 22 May 2012, at 12:46 , Murray, Gregory wrote:
> Mike,
>
> Thanks very much for these pointers. In your experiment, would your database
> configuration include an index on the <page> element? This may be a very
> elementary question. I've been assuming that it doesn't make sense to create
> a range index -- or any other kind of index or lexicon -- for <page> because
> the content is too long and various. Is that assumption valid?
>
> Many thanks,
> Greg
>
>
> On May 22, 2012, at 1:35 PM, Michael Blakeley wrote:
>
>> Chances are the snippet code is bringing in other fragments more often than
>> is optimal.
>>
>> In situations like this I try to design little experiments - or test cases -
>> that will narrow down the source of the problem. Here is an example:
>>
>> (: generate content :)
>> xquery version "1.0-ml";
>>
>> declare function local:new($i as xs:integer)
>> {
>> xdmp:document-insert(
>> concat('test/', $i),
>> <doc xmlns="http://digital.library.ptsem.edu/ia";>
>> <metadata>
>> <id>test-{ $i }</id>
>> <name>Test Author {$i}.</name>
>> <title>Test Title { $i }</title>
>> <date>{
>> xs:date('1999-01-01')
>> + xs:dayTimeDuration(concat('P', $i, 'D')) }</date>
>> </metadata>
>> {
>> for $pn in 1 to 100
>> return element page {
>> for $word in 1 to 50
>> return xdmp:integer-to-hex(xdmp:random(255)) }
>> }
>> </doc>)
>> };
>>
>> local:new(1 to 100)
>>
>> (: search :)
>> import module namespace search="http://marklogic.com/appservices/search";
>> at "/MarkLogic/appservices/search/search.xqy";
>>
>> search:search(
>> 'aa',
>> <options xmlns="http://marklogic.com/appservices/search";>
>> <searchable-expression
>> xmlns:ia="http://digital.library.ptsem.edu/ia";>
>> //ia:page</searchable-expression>
>> </options>),
>> xdmp:query-meters()
>>
>> I'm sure this test case is much too simple to reproduce your problem, but it
>> should give you a starting point. The query-meters output, at the end,
>> reveals some interesting information right away:
>>
>> ...
>> <qm:link-cache-hits>111</qm:link-cache-hits>
>> <qm:link-cache-misses>8</qm:link-cache-misses>
>> ...
>> <qm:fragments>
>> <qm:fragment>
>> <qm:root xmlns="http://digital.library.ptsem.edu/ia";>page</qm:root>
>> <qm:expanded-tree-cache-hits>10</qm:expanded-tree-cache-hits>
>> <qm:expanded-tree-cache-misses>0</qm:expanded-tree-cache-misses>
>> </qm:fragment>
>> <qm:fragment>
>> <qm:root xmlns="http://digital.library.ptsem.edu/ia";>doc</qm:root>
>> <qm:expanded-tree-cache-hits>119</qm:expanded-tree-cache-hits>
>> <qm:expanded-tree-cache-misses>0</qm:expanded-tree-cache-misses>
>> </qm:fragment>
>> </qm:fragments>
>> ...
>>
>> Since I used randomly-generated content, your numbers might be slightly
>> different. But the link-cache hits and misses tell us that the query is
>> traversing fragment boundaries many times. With a page-length of 10 and 119
>> accesses, every snippet causes an average of 12 fragment accesses. We can
>> also see that there were 119 expanded-tree cache hits on the /ia:doc
>> fragment, which is the document root. So it seems clear that the
>> search:search highlighting is starting in the ia:page fragment, but then
>> accesses the root fragment for some reason.
>>
>> This turns out to be slower than your original configuration because
>> following fragment pointers has some cost. So creating more fragments only
>> makes sense when it helps you to avoid loading most of the fragments. That
>> was exactly the idea behind fragmenting on 'ia:page', but it looks like the
>> search snippet code has not been optimized for this use-case.
>>
>> Writing a custom snippet function might help, as documented at
>> http://docs.marklogic.com/5.0doc/docapp.xqy#display.xqy?fname=http://pubs/5.0doc/xml/search-dev-guide/search-api.xml%2358295
>>
>> To test this idea, I added a search option:
>>
>> <transform-results apply="raw"/>
>>
>> This returns the raw ia:page as the snippet. The query-meters output still
>> shows 79 link-cache accesses. Let's try a more extreme option:
>>
>> <transform-results apply="empty-snippet"/>
>>
>> With this option I see 49 accesses to the ia:doc fragment. That is better
>> that 119, and may be close to optimal.
>>
>> Even a simple cts:search on //ia:page shows 39 accesses. I'm not sure why
>> those are necessary: it might be worth your while to delve into that with
>> MarkLogic support.
>>
>> -- Mike
>>
>> On 22 May 2012, at 08:58 , Murray, Gregory wrote:
>>
>>> Mike,
>>>
>>> Thanks very much for your quick reply. Clearly there are lots of good ideas
>>> here. Our first plan of attack has been to set <page> as a fragment root
>>> and reindex the database. Our 50,000 documents now consist of 18 million
>>> fragments. Strangely, the situation appears to be even worse than before.
>>> The query resolution time seems reasonable, but snippet processing is now
>>> several times slower than it was before. We're using search:search(), and
>>> running a typical query in QC gives these metrics:
>>>
>>> <search:metrics>
>>> <search:query-resolution-time>PT0.093S</search:query-resolution-time>
>>> <search:facet-resolution-time>PT0.109S</search:facet-resolution-time>
>>> <search:snippet-resolution-time>PT6.582S</search:snippet-resolution-time>
>>> <search:total-time>PT6.926S</search:total-time>
>>> </search:metrics>
>>>
>>> which makes me wonder if the snippet process is still bringing in the
>>> entire document (book) rather than an individual fragment (page).
>>>
>>> In case they're relevant, I should clarify a few things: Our document
>>> structure includes a metadata element that stands outside of any <page>
>>> element. Also, a typical page will contain about 1000 UTF-8 characters,
>>> which is smaller than the optimal size range. And there are inevitably some
>>> blank pages and thus some empty fragments. For example:
>>>
>>> <doc xmlns="http://digital.library.ptsem.edu/ia";>
>>> <metadata>
>>> <id>1edhumanimmortalitjameuoft</id>
>>> <name>James, William, 1842-1910.</name>
>>> <title>Human immortality : two supposed objections to the doctrine</title>
>>> <date>1898</date>
>>> ...
>>> </metadata>
>>> <text type="ocr">
>>> <page id="1edhumanimmortalitjameuoft_0001" absSeq="1"/>
>>> <page id="1edhumanimmortalitjameuoft_0002" absSeq="2"/>
>>> ...
>>> <page id="1edhumanimmortalitjameuoft_0011" absSeq="11"
>>> xmlns="http://digital.library.ptsem.edu/ia";>
>>> HUMAN IMMORTALITY
>>> Tis a matter unfortunately too often seen in history to call for much
>>> remark,
>>> ...
>>> </page>
>>> ...
>>> </text>
>>> </doc>
>>>
>>> Given that additional info -- do you have any thoughts on why snippeting
>>> would be even slower than before fragmenting on <page>?
>>>
>>> Many thanks,
>>> Greg
>>>
>>>
>>> On May 18, 2012, at 5:23 PM, Michael Blakeley wrote:
>>>
>>>> That can be a tricky question to answer. The bottleneck may be disk or
>>>> could also be CPU, since highlighting several MB of XML can use quite a
>>>> few cycles. If you can establish that the bottleneck is CPU, it may be
>>>> enough to extend your highlight code to set $cts:action to 'skip' or
>>>> 'break' after the first few matches.
>>>>
>>>> Otherwise your goal is to create documents or fragments that average
>>>> something like the 1-100 kB. XML is messy so we cannot hit that number
>>>> precisely, so do not take that target too literally. At the same time,
>>>> make sure that the fragments are well-aligned with search and update
>>>> requirements.
>>>>
>>>> The simplest thing that might possibly work would be to fragment by page.
>>>> I do not much like that answer myself: the sizes are reasonable, but
>>>> paragraphs and sentences may cross page boundaries. Since search terms are
>>>> indexed by fragment, that may cause problems. Imagine searching for "now
>>>> is the time", but one of the potential matches crosses a page boundary. If
>>>> the book is a single document and the page boundary is a phrase-through,
>>>> there is no problem. But if each page is its own fragment root, then
>>>> neither page will match the full query. Still, this option is worth
>>>> considering and does not require much extra work because each book still
>>>> looks like a single XML document.
>>>>
>>>> Another common approach is to treat each book as a directory
>>>> (http://blakeley.com/blogofile/2012/03/19/directory-assistance/), which
>>>> contains one document for every top-level structural element. Usually
>>>> these are chapters or something at a similar level. That limits the
>>>> highlighting problem to a reasonable amount of XML, and searching chapters
>>>> independently makes sense for most applications. Some chapters may be
>>>> larger than others, but still smaller than the entire book. Crucially,
>>>> they should be small enough to highlight comfortably. Reassembling the
>>>> entire book can be as simple as having a toc.xml or manifest.xml in each
>>>> directory.
>>>>
>>>> Most book-oriented applications use one or the other of those approaches.
>>>> There are also some clever algorithmic tricks involving cts:walk and
>>>> cts:highlight together, and approaches using duplicated content for
>>>> passage-oriented or sentence-oriented search. But these can be challenging
>>>> to implement, and most applications can use one of the simpler designs
>>>> outlined above.
>>>>
>>>> -- Mike
>>>>
>>>> On 18 May 2012, at 13:48 , Murray, Gregory wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm working on a project based on full-text XML documents. Each document
>>>>> is a digital representation of a book, containing a short metadata header
>>>>> but consisting mainly of <page> elements, each of which contains a single
>>>>> text node containing a blob of OCR'ed text. We now have 50,000 such
>>>>> documents and are hitting performance/scaling problems. After some
>>>>> profiling, we've concluded that searching isn't the problem but that
>>>>> snippeting is a major culprit. At ML World, my colleagues and I asked
>>>>> Mary Holstege about this, and she indicated that most likely I/O is the
>>>>> bottleneck, because each time a document is processed it must be pulled
>>>>> from disk in its entirety, since of course the <page> elements are not
>>>>> amenable to indexing. She suggested breaking the monolithic documents
>>>>> into pieces, perhaps one XML document per page. At that point, performing
>>>>> a search would involve the search itself but also post-processing to
>>>>> assemble the results back into book-based results for display
> t
>> o
>>>
>>>> th
>>>>> e user.
>>>>>
>>>>> I'm sure that would work well if I knew how to collate the page-level
>>>>> search results into book-level results. I'm sure there are readers of
>>>>> this list -- perhaps especially in publishing or government -- who work
>>>>> not just with smaller, data-centric documents but also with large,
>>>>> full-text documents. Is there a way to retain the integrity of the
>>>>> documents as complete books and still have good performance? If not, and
>>>>> if we were to split the books into pages, what's a good strategy for
>>>>> collating the page-level search results so they still appear to be
>>>>> book-level results to the user?
>>>>>
>>>>> Many thanks,
>>>>> Greg
>>>>>
>>>>> Gregory Murray
>>>>> Digital Library Application Developer
>>>>> Princeton Theological Seminary
>>>>>
>>>>> _______________________________________________
>>>>> General mailing list
>>>>> [email protected]
>>>>> http://community.marklogic.com/mailman/listinfo/general
>>>>>
>>>>
>>>> _______________________________________________
>>>> General mailing list
>>>> [email protected]
>>>> http://community.marklogic.com/mailman/listinfo/general
>>>
>>> _______________________________________________
>>> General mailing list
>>> [email protected]
>>> http://community.marklogic.com/mailman/listinfo/general
>>>
>>
>> _______________________________________________
>> General mailing list
>> [email protected]
>> http://community.marklogic.com/mailman/listinfo/general
>
> _______________________________________________
> General mailing list
> [email protected]
> http://community.marklogic.com/mailman/listinfo/general
>
_______________________________________________
General mailing list
[email protected]
http://community.marklogic.com/mailman/listinfo/general
