Hi Mary,
I'm not sure that I entirely follow you on this. First of all, the use of whitespace-sensitive versus whitespace-insensitive is making a difference, which leads me to believe that it is affecting the results of an unfiltered search. Secondly, I understand that the lexicon searches are performed against fragments and not documents, so I set my fragment root to the element level at which only one instance of the JournalTitle element is defined. This leads me to believe that the frequency counts obtained against any JournalTitle in the lexicon are exact counts. I would like to get a better idea of what you meant by a non-exact . value query as related to an unfiltered search. I figure there has to be a certain degree of determinism involved in such a case, that the same request will continue to yield the same results, and that some (if not all) degrees of non-exact results can be avoided by doing things such as configuring the fragment root appropriately, and using cts:element-value-query as a subquery as if it were acting as close as possible to a filtered search. My goal in all of this it to create reports that have counts obtained from the lexicon frequency counts (using cts:frequency) and from co-occurrences frequency counts that provide fairly accurate statistics for indexed elements and their relationship with other indexed elements and attributes. It makes for nice real-time reports, but if the numbers aren't exact, I'd like to be able to understand why, to take corrective action in code (if possible), and to be able to inform users when and why certain results are not reliable. Thanks or your response - I hope I haven't come across harshly - I'm just trying to make optimal use of the MarkLogic and want to understand how to improve my MarkLogic database configuration and coding practices. Tim -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Mary Holstege Sent: Thursday, June 25, 2009 12:55 PM To: General Mark Logic Developer Discussion Subject: Re: [MarkLogic Dev General] Whitespace, Punctuation, Collations, and& The issue here isn't the lexicon and its collation, the issue here is the element-value-query. The query used with a a lexicon operation in this way is executed unfiltered, and a non-exact (i.e. whitespace-insensitive) punctuation-sensitive value query does not resolve the difference between "Personality & Individ. Diff" and "Personality Individ. Diff" in the index. What you need to do is make sure that you only use queries that are accurate against the index (i.e. where fn:count=xdmp:estimate). //Mary On Thu, 25 Jun 2009 09:08:09 -0700, Tim Meagher <[email protected]> wrote: > Hi Folks, > > > I have come across an interesting phenomenon and am trying to understand > it. > I have a (range element index) lexicon configured for JournalTitle using > the > root collation which contains the following distinct values: > > > Personality & Individ. Diff. > > > and > > > Personality Individ. Diff. > > > If I perform a lexicon search for "Personality & Individ. Diff." > using > cts:element-values() with a subquery of cts:element-value-query() > specifying > the equivalent of an exact match in the search options as follows: > > > > element results { > > for $result in > > cts:element-values(xs:QName("JournalTitle"), (), > > ("item-frequency", "item-order", "ascending"), > > cts:and-query(( > > cts:element-value-query(xs:QName("JournalTitle"), > > "Personality & Individ. Diff.", > > ("case-sensitive", "diacritic-sensitive", > > "punctuation-sensitive", "whitespace-sensitive", > > "unstemmed", "unwildcarded")) > > )) > > )[1 to 20] > > return element result {$result} > > } > > > then I get the following results (as expected): > > > <http://markprod.apa.org:8002/eval.xqy?iefix.txt##> - <results> > > <result>Personality & Individ. Diff.</result> > > </results> > > > However, if I change the request options to whitespace-insensitive, then > I > get the following results: > > > <http://markprod.apa.org:8002/eval.xqy?iefix.txt##> - <results> > > <result>Personality & Individ. Diff.</result> > > <result>Personality Individ. Diff.</result> > > </results> > > > This implies to me that the ampersand is treated like whitespace. I > would > have expected it to be treated as punctuation, but I'm not sure exactly > what > character set (including escaped characters) are defined by whitespace > and > punctuation. I've looked into the UCA and ISO-8859-1 specs to try to > understand the default MarkLogic root collation, but I haven't found a > simple list that would help me to understand why I'm getting the above > results. Can anyone shed some light on this? > > > Can someone also help clarify the distinction between the default > MarkLogic > root collation (http://marklogic.com/collation) and the codepoint > collation > (http://marklogic.com/collation/codepoint)? I'm trying to find the ideal > collation for my JournalTitle lexicon collation. > > > Thanks for the help! > > > Tim Meagher - AAOM Consulting > > _______________________________________________ General mailing list [email protected] http://xqzone.com/mailman/listinfo/general
_______________________________________________ General mailing list [email protected] http://xqzone.com/mailman/listinfo/general
