Chris- Thanks for the information. I'm reading through chapter 18 of the Administrator's Guide to get a better understanding of fragments in MarkLogic.
As for the prototype I'm working on, it's basically a replica of the Newswire Analyst demo: a search engine with facets. I've been working with the code libraries available on the Mark Logic Developer site (lib-search, Versi, search-ui, and lib-uitools) and I have a decent grasp of how they work. What will make my prototype different, though, is the number of documents. Rather than have hundreds of small documents, I've got 10 documents that are larger in size (approximately 400KB each) and I would like to search through them and pull out small segments of information. The DTD for the documents is DocBook-like except that it uses nested <section> elements to represent different heading levels and there are a few inline elements that are specific to our domain (legal publishing). Each <section> element has a <title> and a <p> and will eventually contain <indexterm> elements that will be used for faceted search. Obviously, some <section> elements will contain other <section> elements. I would like to search across <section> elements, weighting the <title> higher than <p> and use the <title> in the display of the search hit. I would also like to get the <title> of the parent of the <section> element, whether it be a <chapter>, <subchapter>, or another <section> element and use that "parent title" as a facet. I was planning on getting the title of the parent container by using some XPath on each search result, although I realize that could be pretty ineffecient. Hopefully, that gives you a better understanding of what I'm trying to achieve. If you would like more details, just let me know. I appreciate the help. John Craft -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Christopher Welch Sent: Wednesday, July 02, 2008 4:37 PM To: General Mark Logic Developer Discussion Subject: RE: [MarkLogic DevGeneral]RestrictingSearchHits ToImmediateParentContainers Hi John, Glad that worked for you :) The cts:score order by statement that you reference is actually what MarkLogic does internally by default with the results of a cts:search expression. I suspect that you don't have fragmenting enabled, which means that your relevance scores will be calculated first by the relevance score of the document, then document order of the result nodes. Scores are calculated at the fragment level. When you return results from cts:search, the cts:score will be one of the following: 1) If the resulting node is a fragment root, then the node score will equal the fragment score. This is the ideal case. 2) If the resulting node is not a fragment root, and contains no fragments, then the node score will equal the parent fragment score. 3) If the resulting node contains fragments, then the score is the highest score of all the encompassed fragments (itself and its constituent fragments) Without fragmentation, the search you ran below falls under case #2. But if you were searching at the document level you would experience case #1. If you choose to fragment, say at the section level, then you would experience case #1 for the search below, and case #3 if you were to search at the document level. These are trade-offs that you would need to make as part of your indexing strategy, and which searches it is most critical to have accurate relevance calculations. It *may* be worthwhile to try fragmenting at the section level to see what happens, but keep in mind that it requires a reindex, so if you have too much content, then you may want to backup your database first and let reindexing run overnight. In terms of searching on title, I believe at that point you would find it easier and faster to search at a section level, not a paragraph level. Then perform a filtering step to remove any sections you're not interested in displaying. If you can share what kind of application or prototype you're trying to build that would be useful. Cheers! Chris -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of John Craft Sent: Wednesday, July 02, 2008 1:27 PM To: General Mark Logic Developer Discussion Subject: RE: [MarkLogic Dev General]RestrictingSearchHits ToImmediateParentContainers Chris- That is precisely what I was trying to do. You nailed it. And thank you Danny and Mike; your input was also very helpful. If I add a simple "order by cts:score($search-hit) descending" to the FLOWR, will that sort the results fairly across all books? Or is the relevance of each book factored into the score, making the score not something that can be sorted on consistently across books? Also, if I plan on adding other elements to the search (ex. <title>), would you recommend using a field as Danny suggested? I have done a little reading through the documentation and it seems like a good approach. Thanks. John Craft -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Christopher Welch Sent: Wednesday, July 02, 2008 11:39 AM To: General Mark Logic Developer Discussion Subject: RE: [MarkLogic Dev General] RestrictingSearchHits ToImmediateParentContainers John, To reiterate what I believe what you're asking for is the ability to generate a list of search results that will find matching paragraphs but return the parent section. We do something similar to this in one of our popular demos, Medbook. You might want to try an approach similar to Mike suggested: fn:distinct-nodes( cts:search(fn:doc()//p, cts:word-query("searchTerm"))/ancestor::section[1] ) Bear in mind that if your results are not ordered and you do not have fragmentation enabled, then the order of the sections will be based on the relevance of the book each "p" element was contained in, and then sub-sorted again by the document order of the matching elements. ~ Chris -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike Sokolov Sent: Wednesday, July 02, 2008 9:28 AM To: General Mark Logic Developer Discussion Subject: Re: [MarkLogic Dev General] Restricting SearchHits ToImmediateParentContainers How about cts:element-query(xs:QName("p"), "searchTerm")/parent::section or cts:element-word-query(xs:QName("p"), "searchTerm")/parent::section ? Danny Sokolsky wrote: > Do you want your searches to always return the top-level "section", but > return it if the match is in a p tag child of *any* section element? > Your concern about returning dups implies that. If so, then you can > rename your top-level section in your xml, and then perform a search > something like: > > let $q := cts:word-query("searchterm") > return > cts:search(/path/to/top-level-section, $q)[( cts:contains(./p, $q) or > cts:contains(.//section/p, > $q) )] > > Given that you want to search a more complicated set of elements, > however, another option to consider is creating a field, specifying the > needed included and excluded elements. Then you could use > cts:field-word-query to search. I am not positive the field will work > for your content, but it sounds like it is worth pursuing. To find out > more about field, see the "Fields Database Settings" chapter of the > Administrator's Guide > (http://developer.marklogic.com/pubs/3.2/books/admin.pdf). > > -Danny > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of John Craft > Sent: Tuesday, July 01, 2008 6:30 PM > To: General Mark Logic Developer Discussion > Subject: RE: [MarkLogic Dev General] Restricting Search Hits > ToImmediateParentContainers > > Danny- > > Thanks for the suggestions. One thing I didn't mention, as I was trying > to keep the example simple, is that I would eventually like to search > additional child elements of <section> (like a <title> element and > possibly <indexterm>) in addition to <p>, weighting them appropriately. > That rules out your third suggestion and may rule out your first > suggestion (not quite sure). > > The second approach won't work because there could be a <section> that > contains a <p> that also contains a <section> that contains a <p> that > contains the search terms. Example: > > <section> > <p /> > <section> > <p>search terms</p> > </section> > </section> > > Using the predicate [fn:exists(./p)], the markup above would return two > results when I would like for it to return one. > > If you think there is an approach that uses cts:query() I would be very > interested. Our content is pretty simple and I have included an outline > of the basic structure below. Of course, I could also send you more (or > a file) if that would be more helpful. > > Content structure (nested sections can go eight levels deep): > > <chapter> > <title /> > <subchapter> > <title /> > <section> > <title /> > <p /> > <section> > <title /> > <p /> > <section> > <title /> > <p /> > </section> > </section> > </section> > </subchapter> > </chapter> > > I'm willing to add/edit elements and attributes if necessary. I just > don't know what would make things easiest for MarkLogic. > > Thanks again. > > John Craft > > > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Danny > Sokolsky > Sent: Tuesday, July 01, 2008 4:27 PM > To: General Mark Logic Developer Discussion > Subject: RE: [MarkLogic Dev General] Restricting Search Hits To > ImmediateParentContainers > > Hi John, > > This can be a little tricky, as it sounds like your "section" elements > can mean different things in different places in the document. One > approach can be to change your section element names for the ones that > have p children to something different, and then search over those. It > would be relatively easy to write a transformation in XQuery to do that. > Ultimately, this might prove to make your content the most searchable > for what you want. > > Another approach is to filter out the results that do not have a direct > p child from the search results. This will probably be OK if the number > of results to filter is small relative to the number of results returned > from the search. This might look something like: > > cts:search(//section, "searchterm")[fn:exists(./p)] > > You can also search below the section element (//section/p), but that > would return p elements. Depending on your content, that might work. > > There may be a cts:query solution here, too, but without knowing your > content very well, it is harder for me to see that. > > -Danny > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of John Craft > Sent: Tuesday, July 01, 2008 12:58 PM > To: General Mark Logic Developer Discussion > Subject: [MarkLogic Dev General] Restricting Search Hits To Immediate > ParentContainers > > I am evaluating MarkLogic and have been playing around with the > cts:element-query() and cts:element-word-query() expressions. So far, I > am having difficulty restricting search results to elements that are > direct parents of the elements that contain the search terms. > > Our content is made up of nested <section> elements and most <section> > elements contain <p> elements, which are our containers for paragraph > text. The <section> elements contain <title> elements and other > information as well. When performing a search, I would like to limit > the results to only the <section> elements whose direct <p> children > contain search terms. I began by creating the following cts:search() > string: > > cts:search(fn:doc()//section, cts:element-query(xs:QName("section"), > cts:element-query(xs:QName("p"), "searchTerm") )) > > This approach was flawed because the search results included <section> > elements that were further up the tree and didn't directly contain <p> > elements (or, rather, <p> elements that contained the search terms). > > My next approach was to use cts:element-word-query() and create an > element-word-query-through for the <p> element: > > cts:search(fn:doc()//section, > cts:element-word-query(xs:QName("section"), "searchTerm") ) > > Again, the search results contain <section> elements that aren't direct > parents of <p> elements that contain search terms. The end result is > that I end up with a lot of <section> elements that are false positives. > > I'm beginning to think the path information on the first cts:search() > argument may be the problem, but I'm not sure. And if it is the > problem, how else can I get search results returned as <section> > elements > > I appreciate any help or suggestions you can provide. > > Thanks. > > John Craft > _______________________________________________ > General mailing list > [email protected] > http://xqzone.com/mailman/listinfo/general > _______________________________________________ > General mailing list > [email protected] > http://xqzone.com/mailman/listinfo/general > _______________________________________________ > General mailing list > [email protected] > http://xqzone.com/mailman/listinfo/general > _______________________________________________ > General mailing list > [email protected] > http://xqzone.com/mailman/listinfo/general > _______________________________________________ General mailing list [email protected] http://xqzone.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://xqzone.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://xqzone.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://xqzone.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://xqzone.com/mailman/listinfo/general
